EgoSim: Egocentric Exploration in Virtual Worlds with Multi-modal Conditioning

Authors: Wei Yu, Songheng Yin, Steve Easterbrook, Animesh Garg
Primary area: generative models
Keywords: Controllable video generation, Egocentric video prediction, World model

Abstract:
Recent advancements in video diffusion models have established a strong foundation for developing world models with practical applications. The next challenge lies in exploring how an agent can leverage these foundation models to understand, interact with, and plan within observed environments. This requires adding more controllability to the model, transforming it into a versatile game engine capable of dynamic manipulation and control. To address this, we investigated three key conditioning factors: camera, context frame, and text, identifying limitations in current model designs. Specifically, the fusion of camera embeddings with video features leads to camera control being influenced by those features. Additionally, while textual information compensates for necessary spatiotemporal structures, it often intrudes into already observed parts of the scene. To tackle these issues, we designed the Spacetime Epipolar Attention Layer, which ensures that egomotion generated by the model strictly aligns with the camera’s movement through rigid constraints. Moreover, we propose the CI2V-adapter, which uses camera information to better determine whether to prioritize textual or visual embeddings, thereby alleviating the issue of textual intrusion into observed areas. Through extensive experiments, we demonstrate that our new model EgoSim achieves excellent results on both the RealEstate and newly repurposed Epic-Field datasets. For more results, please refer to https://egosim.github.io/EgoSim/.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730611783732

thread_type:
official_review

<thread_object>
creator: reviewer_l7sn

<content>
summary:
This work focuses on improving the controllability of video diffusion models for egocentric exploration with camera ego-motion + text + image as conditioning. The proposed framework incorporates epipolar constraints to better focus on the relevant parts in the attention mechanism, referred to as Spacetime Epipolar Attention (SEAL). It is then combined with existing text-to-video and image-to-video diffusion models. Experiments on RealState and EPIC-Fields datasets show the effectiveness of the proposed approach over existing methods.

strengths:
- This work focuses on improving the controllability of video diffusion models with multiple conditioning factors: combinations of camera ego-motion, text, and image, which is important from a practical use perspective.
- The idea of incorporating epipolar constraints into the attention mechanism is simple and intuitive.
- Experiments on RealState (Tab.1, Fig.4) and EPIC-Fields (Tab.1, Fig.5) datasets show the effectiveness of the proposed approach over existing methods.
- Ablations in Tab.2 and visualizations on the website provide a better understanding of the capabilities of the proposed approach.

weaknesses:
- In Tab.1, several values are missing, which makes it difficult to compare different models. There is some justification in the text (L426-429). It'd be helpful to have more details:
    - Why can't MotionCtrl and EgoSim (SVD) be evaluated on EPIC-Fields? Both are I2V methods and EPIC-Fields contains both frames and camera egomotion.
    - For CameraCtrl + SparseCtrl, why can't TransErr and RotErr be computed? Since ground truth is available, the camera trajectory from the video diffusion output needs to be computed. Is it because COLMAP optimization does not converge on the outputs? Is there some other reason?
    - It'd also be useful to have T2V and I2V settings on EPIC-Fields to better understand the trends across different datasets. Since text description is available for EPIC-Fields (Fig.5), is there any reason to not use these settings?
- There are 2 mentions of efficiency benefits in the text. It'd be helpful to verify these benefits quantitatively, in terms of memory usage and train/inference time.
    - L205-206: The use of epipolar attention introduces additional sparsity, enabling us to utilize memory-efficient operations.
    - L236-238: employ pixel unshuffle Shi et al. (2016) to adjust the size while preserving as much fine-grained positional information as possible. This approach is sufficient and also helps to save computational resources. 
- L245-246 mentions: 'a particular patch in a specific frame should be explained either by text or by the context frame, but not both'. It'd be interesting to see if this is indeed the case. One way to do this is to check the values of $\mu$ in Eq.4, which should be close to 0 or 1.
- Some experimental details are missing:
    - L288-290: details on how EPIC-Fields is processed.
    - L323: what does 'more difficult random trajectories' mean? how are they sampled?
    - Are the baselines re-trained in the same setting or used in a zero-shot manner?
- It'd be helpful to clarify these aspects:
    - L19-20: while textual information compensates for necessary spatiotemporal structures, it often intrudes into already observed parts of the scene
    - L25: issue of textual intrusion into observed areas
    - L109-110: override the interference of other features in video generation
    - L112-114: leverages camera movement information to further assist in clearly defining the boundaries between the text and visual elements
    - L189-191: need a control method that operates relatively independently of these features

questions:
There are 3 major concerns (details above):
- Several values are missing in Tab.1 which makes it difficult to understand the trends. It'd be helpful to provide details on these missing values and EPIC-Fields experiment setting.
- The text mentions efficiency benefits in 2 places and disentangling text & image features. It'd be useful to verify if this is indeed the case.
- Several aspects of the text need further clarifications.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730624826707

thread_type:
official_review

<thread_object>
creator: reviewer_swtr

<content>
summary:
This paper tackles video generation with multi-modal condition signals: text, image, and camera pose. It introduces several model designs including employing epipolar attention to the spacetime domain for precise camera motion control and a CI2V adaptor that balances text and vision guidance based on camera information. Further, it repurposes the EPIC Fields dataset as the new dynamic scene dataset with camera annotations. Extensive experiments show the effectiveness of each proposed module.

strengths:
+ The multi-modality control of video generation is an interesting topic. The proposed method fills in the gap in precise camera-conditioned image-to-video generation. 
+ Most of the past camera-conditioned video generations trained on static 3D scene datasets, i.e. Realestate, DL3DV. The proposed method provides an effective practice to repurpose video understanding benchmarks for generation and to some extent shows a way to resolve the data scarcity of dynamic scene data with camera pose annotations.
+ Balancing different control signals is an intuitive challenge in multi-modal guided video generation. The proposed CI2V adapter is a simple and effective strategy to handle it.

weaknesses:
- Given that Realestate is a large-scale dataset with 100 times the number of scenes compared to Epic-Field, how do you prevent overfitting your generations to static scenes?
- It would be interesting to compare the proposed method with Viewcrafter [M1] in terms of the preciseness of camera controls in a static 3D scene. 
- The camera trajectories in the results are quite simple and mostly object-centric, it would be better to infer with longer, more complex trajectories in open scenes. 
- [Minor] The examples in 'Interacting with the World' contain too many noticeable artifacts, e.g., hand disappearance, hand merging into objects, etc. 

[M1] Yu, Wangbo, et al. "Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis." arXiv 2024.

questions:
Please refer to the weaknesses section. Overall I think the task addressed is important and interesting. Most of the simple cases look fine. I suggest the authors add more comparisons with more recent methods, i.e., viewcrafter.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730947344610

thread_type:
official_review

<thread_object>
creator: reviewer_4w3n

<content>
summary:
This paper presents a novel video diffusion architecture capable of handling multiple conditioning inputs, including image, text, and camera poses, in a unified framework. The work makes meaningful technical contributions to controllable video generation, though there are areas where clarity could be improved.

strengths:
The authors integrate multiple conditioning types in a coherent framework, demonstrating a novel adaptation of epipolar attention from 3D generation to video generation. The introduction of the SEAL and CI2V-adapter shows thoughtful consideration of the challenges in multi-modal video generation. The evaluation demonstrated on both static (RealEstate) and dynamic (Epic-Field) datasets, supported by both qualitative and quantitative improvements over existing methods. The extensive ablation studies further strengthen the technical contributions.

weaknesses:
The paper suffers from several clarity issues that should be addressed. The experimental setup and results sections (3.2, 3.3) lack clear organization and the data preparation is not detailed throughout the paper, making it difficult to fully understand the implementation.

questions:
1.1 What is the difference between EgoSim (SVD) and EgoSim in Table 1? 

1.2 For Epic-Field experiments, what are the input conditions (text, image, or both) in Table 1?

1.3 Regarding LoRA usage (Line 437): Why was an additional LoRA necessary for Epic-Field when the model was already fine-tuned? This seems redundant and needs justification. Did the authors use LoRA to fine-tune the pre-trained model?

1.4 How were camera poses obtained/annotated for RealEstate and Epic-Field datasets?

1.5 How is the training and testing dataset split? What are the respective dataset sizes?

1.6 No training detail is provided. Please mention your setup such as earning rates, optimization parameters, batch sizes, number of training iterations, hardware specifications, and training time.

2. How is K* and V* calculated in practice?

3.1 L265 "attachted"" revise my review for this paper.

These clarifications would significantly improve the paper's reproducibility and technical clarity.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a novel video diffusion architecture for controllable video generation conditioned on multiple modalities, including text, images, and camera poses. Specifically, it introduces (i) a spacetime epipolar attention layer that can support better control of camera motion, and (ii) a CI2V adapter to balance text and vision guidance. To obtain camera annotations, the authors further repurpose video understanding dataset (Epic-Field) for the task of video generation. While there are some concerns regarding the clarity of the paper and experimental setups, such as camera trajectories used in experiments are overly simple and object-centric, limiting generalizability to open, complex scenes, or comparisons with more recent methods are missing. The authors have addressed most of them during the rebuttal phase. After careful discussion, the ACs agree the pros outweigh cons. The ACs urge the authors to incorporate the feedbacks from the reviewers into the final version of the paper. The ACs recommend acceptance.