GenEx: Generating an Explorable World

Authors: TaiMing Lu, Tianmin Shu, Alan Yuille, Daniel Khashabi, Jieneng Chen
Primary area: generative models
Keywords: Generative Models, Video Generation, Embodied AI

Abstract:
Understanding, navigating, and exploring the 3D physical real world has long been a central challenge in the development of artificial intelligence. In this work, we take a step toward this goal by introducing *GenEx*, a system capable of planning complex embodied world exploration, guided by its generative imagination that forms expectations about the surrounding environments. *GenEx* generates high-quality, continuous 360-degree virtual environments, achieving robust loop consistency and active 3D mapping over extended trajectories. Leveraging generative imagination, GPT-assisted agents can undertake complex embodied tasks, including goal-agnostic exploration and goal-driven navigation. Agents utilize imagined observations to update their beliefs, simulate potential outcomes, and enhance their decision-making. Training on the synthetic urban dataset *GenEx-DB* and evaluation on *GenEx-EQA* demonstrate that our approach significantly improves agents' planning capabilities, providing a transformative platform toward intelligent, imaginative embodied exploration.

Decision: accept (poster)
Reviewer scores: 5, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730463664391

thread_type:
official_review

<thread_object>
creator: reviewer_gznb

<content>
summary:
The authors of this study investigate the problem of  planning with partial observation, which is important in embodied AI. To achieve this, the authors propose a video generation model Generative World Explorer (Genex), that allows an agent to simulate the world through panoramic representation. The authors also propose an imagination-driven POMDP framework, where generated images assist the agent in decision-making through question-answering (QA).

strengths:
1. Importance of the work: While world models with front-view and multi-view videos are actively investigated by the research community, the generation of panoramic videos is seldom explored. This paper introduces a novel training strategy specifically for panoramic video generation, contributing valuable insights for the community.
2. In this work, the authors aim to define an embodied agent with belief revision driven by imagination, which is able to imagine hidden views through imaginative exploration..
3. Two new datasets called Genex-DB and Genex-EQA have been collected to facilitate the proposed pipeline. The scenarios include a diverse range of styles: Realistic, Animated, Low-Texture, and Geometric.
4. On the proposed dataset Genex-DB and Genex-EQA, the proposed method Genex achieves favorable results in panoramic video generation and embodied QA, compared to other baselines.

weaknesses:
1. My main concern is the **actual impact** of the proposed 'imagination' on embodied QA. While the authors show an approach to link the panoramic video generation with embodied QA, the experiments do not explicitly demonstrate the effectiveness of the ''imagination generation''. How about the results of POMDP without imagination？
2. As far as the reviewer knows, most of the generation models (including the SVD used in this paper) are poor in the reasoning ability, because essentially they are just simulating the probability of objects appearing. In most cases, if there are no explicit constraints like specified object category, the generation model wouldn't expect an ambulance to be here in most cases. This is an open question and the reviewer wants to see the point of the authors. Additionally, could the authors provide more examples of the imagination results, particularly challenging cases like those shown in Fig.12?
3. Some **concerns about the Genex-EQA questions**. The questions and answers in the dataset are quite subjective. For example in the second row and second column in Fig.12, the gt choice is "Signal the car to stop for the pedestrian". This action seems impractical for an autonomous driving vehicle. Further clarification on the methodology and rationale behind the question and answer collection process is needed to understand the dataset's reliability.
4. For panoramic video generation task, though the method serves as a baseline, it is beneficial to have **some comparisons with previous single-view world models** (because they can also perform panoramic video generation task by just replacing the data) and demonstrate why these models fail to generate panoramic videos.

questions:
1. Some typos: L96: imaginatively-imaginative; L186: a-an; Fig.15 Imaginatin-Imagination.
2. Fig link: L379: Fig.2? Maybe this should be Fig.6.

The reviewer has identified four major concerns and would like the authors' responses to these points. Please answer each concern in the rebuttal stage. The reviewer will respond according to the authors' rebuttal in the discussion phase.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667318014

thread_type:
official_review

<thread_object>
creator: reviewer_mnvr

<content>
summary:
The paper works on the problem of decision-making in the partial observation setting. To tackle the task, the authors introduce a novel panorama-based video diffusion model which can imagine the observations from different positions. The authors further combine the generative model and the LLM to help the decision making process. To evaluate the decision making performance, they design a benchmark over 200 scenarios in single and multi-agent settings. The results show that their pipeline achieves better performance by augmenting the agent's imagination ability via the generative model.

strengths:
1. Leveraging generative models to complete the partial observations to a full “world” understanding is reasonable to utilize the priors learned from the data.
2. For the panorama representation, they design the spherical-consistent learning during their learning process to improve the consistency of the panorama image. From their results, the panorama truly shows better consistency and leads to better representation of the scene.
3. The authors conduct extensive experiments and create a benchmark for demonstrating the challenging cases under the partial observation constraints.

weaknesses:
1. In this work, the authors actually construct an explicit representation for “the imagination prior” to make decision making. However, in the benchmark setting, most questions seem only related to a specific case. For single-agents, just try to avoid some unseen cars. And for multi-agent, try to make the other two agents avoid collision. The task setting seems not challenging and common enough to demonstrate the usefulness of such imagination ability. Also it’s hard to see the real performance through such discrete choice-making decision accuracy. Potentially, the method can serve as a role to generate the bird-eye map from a single panorama image and can reveal the hidden cars not in the observation. 
2. The paper mentions such imagination can be further updated based on new observations, however, in this work, there is no integration of the imagination and the new observations.

questions:
1. How to determine what’s the trajectory to explore if the world is unlimited? And how to make sure the information is enough to make a decision?
2. Is there better way to evaluate the imagination ability, like the 3D concept error with GT (there is hidden car or not, how much unobserved information is discove
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687264160

thread_type:
official_review

<thread_object>
creator: reviewer_m4hx

<content>
summary:
The paper introduces the challenge of planning with partial observation in embodied AI and highlights how humans can mentally explore unseen parts of the world to update their beliefs and make informed decisions. To replicate this human-like ability, the authors propose the Generative World Explorer (Genex), a video generation model that enables agents to mentally explore large-scale 3D worlds and acquire imagined observations to update their beliefs. They train Genex using a synthetic urban scene dataset, Genex-DB, and demonstrate that it can generate high-quality and consistent observations during long-horizon mental exploration and improve decision-making in an existing model.

strengths:
+The idea of build a Generative World Explorer is interesting, and I think it will be useful to the development of embodied AI research.

+ It's practical to apply the proposed Genex to the embodied decision making process.

weaknesses:
-There is a gap between the training data (synthesized with unity) and test data (captured from google street), the degrees of freedom of in the observation perspectives, google street seems to more limited compared to the unity. But the gap between training and test data may not  be always "bad", because such gap may show more "Generalizability". 

-In the following sentence “An embodied agent is inherently a POMDP agent (Kaelbling et al., 1998): instead of full observation, the agent has only partial observations of the environment.” , “a POMDP agent” seems to lack rigor. POMDP (Partially Observable Markov Decision Process) is a modeling framework that can be applied to describe the behavior of an agent in an environment where full state information is not available. Visual observation is only one channel for information acquisition. Saying that incomplete visual observation necessarily leads to a POMDP is also not very rigorous.

questions:
Overall I think this is a good paper that can contribute to the subsequent development of the field of embodied AI.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730780313353

thread_type:
official_review

<thread_object>
creator: reviewer_gpur

<content>
summary:
Humans have the capacity to imagine the future and revise their beliefs about the world based on these imagined observations. Building on this concept, the authors have introduced a video generation model, GeNex, which enables an agent to mentally explore future imagined observations. Subsequently, a Large Language Model (LLM) is utilized as the policy model to predict future actions.

strengths:
The concept is intriguing and the explanation is clear and straightforward.

weaknesses:
1. There are some errors in the mathematical formulations presented, particularly in equations (3) and (4), which are confusing.
2. Although SCL is highlighted as a contribution, its effectiveness is not demonstrated in the experimental results.
3. The use of latent diffusion with temporal attention is not a novel architecture.
4. The real-world dynamics of vehicles do not allow for pure rotation, which the paper seems to overlook.
5, Table 3 presents an unfair comparison.

questions:
1. Equation (3) is incorrect
2. The derivation of Equation (4) is unclear. Could you explain how it was formulated?
3. Is the LLM policy model fine-tuned or used as is?
4. The space of 'state' & 'belief' is not clearly defined.
5. It is unclear whether the diffusion model has been overfitted to the dataset, potentially making it inadequate for handling complex real-world interactions.
6. The entire framework appears to have little connection with POMDP.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733293194397

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the discussion period

comment:
Dear Reviewers and Chairs,

We sincerely thank all reviewers for their constructive feedback and for recognizing the contributions of our work.

We have carefully addressed all concerns and provided detailed responses, along with additional experiments to support our claims:

- Clarified multiple points about the experimental setup and included model comparisons for `M4Hx`.
- Expanded on the generalizability of the proposed model in response to reviewer `M4Hx`.
- Addressed reviewer `MnVr`'s concerns regarding the subjectiveness of the benchmark and practical applications of Genex, highlighting its future potential.
- Conducted additional experiments with single-view world models to resolve reviewer `GPuR`'s feedback.

We believe these updates adequately address all concerns raised. Although we have not yet heard back from reviewer `GPuR`, we hope our revisions meet their expectations.

Thank you again for your valuable feedback and engagement.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary**

The paper proposes Generative World Explorer (Genex), which uses a video generation model to imagine taking a sequence of actions and their followup observations.  The imagined observations are used to revise agent beliefs of the world, which are then used for decision making.  The framework can be extended to multi-agents so that decisions can be made while taking into account other agent's beliefs. 

To demonstrate the feasibility of the approach, the work uses a diffusion-based video generation model that generates egocentric equirectangular projected panoramas.  For decision making, a large multimodal model (LMM) is used for the policy model and for mapping observation to belief (the LMM is prompted to select actions based a set of input images).  

A dataset (Genex-DB) of rendered panoramic images from four scenes with different styles is used to train video diffusers (one per scene).  The dataset also contains additional indoor and outdoor panoramic images for testing.  Genex-DB is used to evaluate the video generator.  In addition, an embodied QA dataset (Genex-EQA) consisting of 200 scenarios is constructed to evaluate the decision making ability of the proposed model. 

The main contributions of this work are 1) the proposed Genex framework for using imagining outcomes of different actions and integrating the imagined observations with an LMM for decision making  2) the egocentric panoramic diffuser, 3) Genex-DB and Genex-EQA datasets

**Strengths**

Reviewers noted the following strengths of the work:
1. Idea of generative world explorer is interesting [GPuR,M4Hx,GZnb]
2. Generation of panoramic video is underexplored [GZnb] and spherical consistent learning seems effective [MnVr]
3. Contributed dataset and benchmarks [GZnb,MnVr] 

**Weaknesses**

Reviewers have concerns about the following:
1. Limited experiments
   - Limited comparison to previous single-view world models [GZnb]
   - Experiments do not actually show impact of imagination [GZnb] / EQA questions doesn't really seem to require imagination [MnVr]
2. Limited integration of imagination and new observations [MnVr]
3. Concerns about the quality of the Genex-EQA dataset [GZnb]
4. Concerns about mathematical formulations [GPuR] and connection of the proposed framework to POMDP [GPuR,M4Hx]
5. Concerns about generalizability of trained diffusers [GPuR]

**Recommendation**

Given the overall positive rating from reviewer (3 vote for accept, one for reject), the AC believe that the contributions of the work could be interesting for the ICLR audience.  The AC finds the experimental setup somewhat weak.  

As pointed out by several reviewers, it's unclear how much imagination (vs just exploration the environment) is required for the proposed Genex-EQA benchmark.  There is also limited information about the quality of the Genex-EQA benchmark.  

Nevertheless, the AC believes the contributions (proposed framework with panoramic video diffuser) is sufficiently interesting for acceptance at ICLR.  

**Suggested updates**

The AC notes that the submission can be improved with the following:
1. Add discussion to clarify relation to POMDP and clearly explain how the Equation 3 samples trajectories through exploration.  
2. Provide additional detail on how Genex-EQA is created.  Appendix 4 only consists of A.4.1 (Dataset details) which did not have much information about how Genex-EQA was constructed and how the quality of Genex-EQA was ensured.