EgoSim: Egocentric Exploration in Virtual Worlds with Multi-modal Conditioning
Authors: Wei Yu
Songheng Yin
Steve Easterbrook
Animesh Garg
Venue: ICLR 2025 Poster (2025)
Keywords: Controllable video generation
Egocentric video prediction
World model
Primary area: generative models

Abstract: Recent advancements in video diffusion models have established a strong foundation for developing world models with practical applications. The next challenge lies in exploring how an agent can leverage these foundation models to understand, interact with, and plan within observed environments. This requires adding more controllability to the model, transforming it into a versatile game engine capable of dynamic manipulation and control. To address this, we investigated three key conditioning factors: camera, context frame, and text, identifying limitations in current model designs. Specifically, the fusion of camera embeddings with video features leads to camera control being influenced by those features. Additionally, while textual information compensates for necessary spatiotemporal structures, it often intrudes into already observed parts of the scene. To tackle these issues, we designed the Spacetime Epipolar Attention Layer, which ensures that egomotion generated by the model strictly aligns with the camera’s movement through rigid constraints. Moreover, we propose the CI2V-adapter, which uses camera information to better determine whether to prioritize textual or visual embeddings, thereby alleviating the issue of textual intrusion into observed areas. Through extensive experiments, we demonstrate that our new model EgoSim achieves excellent results on both the RealEstate and newly repurposed Epic-Field datasets. For more results, please refer to https://egosim.github.io/EgoSim/.

Published as a conference paper at ICLR 2025
EGOSIM: EGOCENTRIC EXPLORATION IN VIRTUAL WORLDS WITH MULTI-MODAL CONDITIONING
Wei Yu Songheng Yin', Steve Easterbrook', Animesh Garg University of Toronto, Vector Institute. 'Columbia University "NVIDIA Georgia Tech
ABSTRACT
Recent advancements in video diffusion models have established a strong founda- tion for developing world models with practical applications. The next challenge lies in exploring how an agent can leverage these foundation models understand, interact with, and plan within observed environments. This requires adding more controllability to the model, transforming it into a versatile game engine capable of dynamic manipulation and control. To address this, investigated three key conditioning factors: camera, context frame, and text, identifying limitations current model designs. Specifically, the fusion of camera embeddings with video features leads to camera control being influenced by those features. Additionally, while textual information compensates for necessary spatiotemporal structures, it often intrudes into already observed parts of the scene. To tackle these issues, we designed the Spacetime Epipolar Attention Layer, which ensures that egomotion generated by the model strictly aligns with the camera's 's movement through rigid constraints. Moreover, we propose the CI2V-adapter, which uses camera informa- tion to better determine whether to prioritize textual or visual embeddings, thereby alleviating the issue of textual intrusion into observed areas. Through extensive experiments, we demonstrate that our new model EgoSim achieves excellent results on both the RealEstate and newly repurposed Epic-Field datasets, For more results, please refer to httpss//egos github. 1 //EgoSim/
INTRODUCTION
The success of diffusion models Ho et al. (2020) has revolutionized the field of generative models, enabling advancements from realistic image generation Rombach et al. (2022) to consistent video generation Blattmann et al..[.222)) (202bb). Recent works on long-term video generative
training(Opena (2024) have further shown that diffusion models effectively capture the complex dynamics of the physical world, laying a strong foundation for developing world models Ha & Schmidhuber (2018) with practical value.
The next challenge is to investigate how an agent can leverage this foundational model for under- standing, interacting with, and planning within observed environments. To this end, need first identify what conditioning factors are necessary for constructing an effective world model. Our ultimate aim is to empower an agent to visualize potential scenarios based on observed environmental data, much like playing game, thereby enhancing its ability to predict and respond to different situations Yang et al. (2023).
Conditioning with Camera, Frame and Text: To create a playable simulation engine, the primary desired feature is enable agents to freely explore the simulated world. This necessitates using the agents' egomotion data, e.g. camera poses, as conditions to guide the generation process of the video diffusion model. Fortunately, as the majority of pixel changes in the video stem from the observer's egomotions rather than dynamic alterations in the surrounding environment, the pre-trained video diffusion models have already successfully internalized prior knowledge about typical patterns and transformations in the 3D world Blattman al. (2023b); Voleti et al. (2024). Therefore, task is to determine how to efficiently extract this prior knowledge and ensure that the videos generated by the model precisely align with the specified camera motion instructions.
Existing methods Wang et al. 2023);; He et al. (224)) typically combine the transformed camera information directly with intermediate features before feeding them into the temporal transformer

---

Published as a conference paper at ICLR 2025
a. Prompt: a kitchen with a stove top oven and a sink.
EGGMOTION
b. Image-to-video generation: No prompt.
OBSERVATION
BoC IMAGINATION
Prompt: bedroom with bed and dresser in it.
Figure 2: To create playable world simulator that supports free exploration and imagination based on observation, we have identified several primary bottlenecks: (a). We attempt to generate video of moving forward in front of a stove, but the model fails to fulfiil it because it has never seen such training data of moving above a stove. (b). Image-to-vided model equipped with camera control, as emphasized inside the blue box, can only depict details already observed in the first frame. (c). Enable text-to-video models with image conditioning. The first row generated video while the second row is the groundtruth. As illustrated inside the red ellipse, introducing textual information can erode the already observed part, leading to a disruptive transition from realistic scenes to complete fantasy.
layer. This approach can indeed roughly learn the camera motion. However, is quite intuitive to notice that the intermediate features, along with the camera poses. jointly determine how the video generated. As shown Figure 2h. influenced by the distribution of training data, the model may determine that even if the input motion is valid in certain situations, the camera cannot move forward because it has never seen an example of forward movement in such scenarios. Therefore, the generated video in this case will severely deviate from the user-specified motion.
Additionally, we aim for the model to seamlessly integrate observed environmental information, usually presented context frames, into the video generation process while simultaneously inferring unobserved structures and predicting future interactions with the environment. Our preliminary experiments indicate that although the pretrained image-to-video model Blattmann al. (2023a) can produce authentic videos with minor movements, increasing the amplitude of the movement results outputs that only depict the details visible the context frame, illustrated in Figure Zp. This suggests that the model needs supplementary information to imagine the unobserved parts.
Moreover, through further analysis on text-to-video models Guo al. (203cc), found that adding textual information can effectively compensate for previous shortcomings by injecting extra spatiotemporal structure, leading to more coherent and realistic video sequences. However, as shown in Figure 2. it became evident that this added spatiotemporal structure intrudes upon already observed parts, causing a disruptive transition from realistic scenes to complete fantasy.
In summary, our detailed examination reveals that while each condition in multi-condition inputs has an effective controlling method, integrating these conditions together often leads to their embeddings interfering with negating each other, significantly undermining the reliability of the generated video. This indicates that using a naively unified embedding control interface may inevitably harm the accuracy of video generation. Consequently, this paper focuses resolving the interactions between different conditioning modalities to construct a coordinated and compositional world model. The contributions of our paper can be categorized follows:
2

---

Published as a conference paper at ICLR 2025
1. We introduce novel plug-in DiT Peebles & Xie (203) module called the Spacetime Epipolar Attention layer (SEAL) to override the interference of other features in video generation, making more accurately aligned with camera motion.
2. We propose CI2V-adapter to arrange text information and visual conditions in mutually exclusive relationship. CI2V-adapter leverages camera movement information to further assist in clearly defining the boundaries between the text and visual elements, ensuring they do not interfere with each other during the generation process.
3. We repurpose the Epic-Field dataset for egocentric video generation and established a new benchmark to evaluate video diffusion models in more interactive and dynamic settings.
We evaluate the proposed method EgoSim on two competitive benchmarks, RealEstate and Epic- Field datasets, in a mulli-condition-inge setting, Extensive experimental results show that our model achieves precise control that previous methods could not accomplish. It not only generates video that accurately follows camera movements but also fills in unobserved new information into existing observations and interacts with the environment.
ORCHESTRATING DIVERSE CONDITIONS
Given reference image, text prompt and sequence of camera poses, our goal is to generate a video sequence which starts from the context frame, faithfully obeys the user-specified motion and interact with environment in accordance with the textual description. As discussed earlier, the current control methods for these conditioning factors often interfere with each other, leading to the generation of videos that are inconsistent and unexpected.
Our initial analysis has pinpointed two main sources of this conflict: the interaction between camera control and intermediate features, and the interplay between text and context frames. This chapter will begin by providing an overview of video generation models and then address these two critical issues separately to resolve the problems.
2.1 PRELIMINARY
Diffusion Models Ho et al. (020)) are a class of generative models designed to produce high-quality samples through multi-step process. Starting with Gaussian noise, these models iteratively refine and denoise the initial random input. In the case of video generation Blattmann et al. (2023b), sequence of N images their latent features) =)) are progressively subjected to noise e, transforming them into a normal distribution over T steps. A neural network c" is then trained to predict the added noise from these noised inputs. During training, the network aims minimize the mean squared error (MSE) between its predictions and the actual noise. The training objective function is defined as follows:
(())= E Es...e.ced [||e (1le-E0(2!" (0 cc. +]
where e, represents conditional embeddings.
2.1.1 CONTROLLABLE VIDEO GENERATION
Controllability plays important role in video generation as it enables users to craft content precisely as they envision. In this work, we aim to integrate three distinct control conditions autonomous camera motion, reference images, and text -into a single pipeline.
First, regarding camera motion, it's important to note that the field of video diffusion models still relatively new. Control over camera motion has only been tentatively explored so far. Recent methods Wang al. (003)) He et al. (2024) primarily achieve control by combining camera embeddings with intermediate features. On the other hand, there is wealth of research on both text-to-vided Guo et al. (2023c) and image-to-vide remains rare. Typically, text-to-video generation incorporates CLIP embeddings Radford al. 2021 of texts through multiple cross- tross-attention layers while image-to- image-to-video models are mainly trained from scratch by concatenating repeated first-frame image features to noised input.
generation Blattmann et al. (2023a), but combining these two approaches
3

---

Published as a conference paper at ICLR 2025
Modules
Atanninn
Creso Atcation
Temporal
SEAL
Triinale
Emboddlings
Thet CLIP ClIPeeddiii
Image embodde mge
Cumsou uniondkings
CLIP
Upipslar Attention
Signold guttig
pen
adapece
Video
addition
Figure 3: EgoSim Overview. Left: A simple illustration of how epipolar attention is calculated. Each patch only attends to patches from other frames which directly intersect or are near the re- re-proeection of the unprojected epipolar line. Right: This diagram shows how various embeddings are injected into the video diffusion model. All modules along with the entire SEAL inside the dashed box with a flammable sign are set trainable. The computation graph of the CI2V-adapter is shown within the purple box. For simplicity, ResNets are omitted and trainable parts are not repeatedly emphasized with flammable sign for other blocks.
2.2
CAMERA CONTROL
In this paper, attempt to develop more robust and reliable method for controlling camera trajectories video generation. Existing approaches, which concatenate or add camera embeddings with intermediate features Wang et al. (2023); He al. (024). often result in camera movements being influenced by these features. Therefore, we need control method that operates relatively independently of these features and enforces rigid constraints derived from camera movement.
2.2.1 SPACETIME EPIPOLAR ATTENTION LAYER
We notice that UNet-based diffusion models have an inherent limitation egocentric video generation. Specifically, current publicly available pre-trained models all utilize factorized space-time attention, meaning a patch in one frame cannot directly attend to a patch in another frame. This design helps save computational resources but leaves frame-to-frame consistency solely controlled by the text prompt. Introducing additional camera information, which governs egomotion, is likely to disrupt the control provided by the textual guidance. Therefore, it becomes necessary introduce mechanism that enables direct interaction between patches across different frames.
Inspired by SORA OpenAT (2024), we addressed this limitation by first rearranging the latent representation into spacetime patches and introducing the DřT Peebles & Xie (2023) structure to enable interaction between patches. Moreover, we propose to leverage geometric constraints derived from epipolar lines to precisely guide camera movements. The use of epipolar attention introduces additional sparsity, enabling us to utilize memory-efficient operations. We call this module Spacetime Epipolar Attention Layer (SEAL).
Next, we will describe detail how to utilize the newly established channel to enforce camera control by leveraging the epipolar geometry. Consider patch coordinate (u, e) in the target frame 4,, where the intrinsic parameters K,, and extrinsic parameters, including rotation R, and translation T relative canonical frame, are known. The epipolar line 1 corresponding to this patch can be calculated using the fundamental matrix:
1, F, 1, KT- ([Ti)x RR) KT 12, 1)7
(1)
4

---

Published as a conference paper at ICLR 2025
As illustrated in Figure [] with the help of epipolar lines, model can identify which regions in other frames to focus on for any given patch as follows:
QK? Vdz
EpipolarAttentionn( (Q,K, V) = softmax
V
where K, and V. are keys and values calculated from patches directly crossed by near the projection of epipolar lines. After constructing such links between all patches, the model can learn to effectively extract knowledge about viewpoint changes in the 3D world from video generative pre-tratee pre-training.
2.2.2 INJECTION OF CAMERA EMBEDDINGS
Since the temporal transformer mainly focuses on learning motion information, integrating the camera embedding Wang et al. (2023;; He et al. (2024) into the temporal transformer can achieve the most direct impact on egomotion generation. More importantly, is compatible with epipolar attention. Therefore, this paper retains and improves the method of injecting camera embedding.
Methods like MotionCtrl Wang et al. (2023) which directly flatten the camera pose, can also capture camera motion but overlook intrinsic parameters. Similar to concurrent work, CameraCtri He ct al. 2024), we concatenate the camera positional encoding Vaswani et al. (2017) with Plücker embedding Sitzzaann et al. (22TT) to get more precise geometric interpretation for each patch. However, unlike CameraCtrl, we found that introducing an additional encoder to generate multi-scale camera embeddings is unnecessary. Instead, employ pixel unshuffle (Shi al. (2016) to adjust the size while preserving as much fine-grained positional information as possible. This approach is sufficient and also helps save computational resources.
2.3 FRAME V.S. TEXT
As previously mentioned, after we equipped the video model with camera movement control, relying solely on the context frame was insufficient for the model to envision the details of unobserved scenes. This underscores the necessity of text input, which supplements the imagined spatiotemporal structure. However, we also realized that textual information could be intrusive and should therefore be mutually exclusive with visual input. In other words, a particular patch in a specific frame should be explained either by text or by the context frame, but not both. To address this, notice that the shift between imagination and observation primarily driven by egomotion. Therefore, incorporating camera motion data is essential to better determine which source of information to prioritize.
2.3.1 CI2V-ADAPTER
To preserve the intricate text-to-video capabilities that the cross-attention layers have learned, we decide to start from text-to- viovideo generation model and then adapt it to support image-to- moage-to-videe -iideooo generation. Specifically, our improvement built based on the 12V-adapter Guo et al.) (2023)). The 12V-adapter pipeline consists of two key components: (a). Additional trainable cross-attentie operations performed each self-attention layer for the first frame where ground truth is provided. (b). IP-adepters Ye et al. (2023) attached after cross-attention layers through adding the output of extra artes-at computations with image embeddings. We upgrade 12V-adapter into C12V-adapter by enabling both components incorporate the influence of camera motion.
For the first component, we notice that it actually involves bles SEAL computation. Hence, we can directly incorporate epipolar attention masks precomputed in SEAL, relative to the first frame, into the cross-attention mechanism. This allows for more precise feature fusion across frames.
interaction that closely resem-
For the IP-adapter component, we want to mitigate mutual interference between textual and visual control through providing additional guidance from camera informatiom. More specifically, inter- mediate features attached with corresponding camera embeddings are used as input for patch-wise sigmoid gating functions μ form a weighted sum of two types of information. This integration is calculated as follow:
14 @WW ReL.U(W, Ca], + bz) + bz)
5

---

Published as a conference paper at ICLR 2025
21 =1 (1-p)) @ (T, + Mr @ X-attn(I1, +
where 2t the intermediate feature of frame 4, T,I are transformed text, image and camera embeddings, á is sigmoid activation, * is the convolution operator, @ is the Hadamard product and X-attn cross-attention operation.
2.4 TRAINING DATA
Data is the most critical component of video generation models. Previous work video generation with camera control has often been limited to datasets featuring static scenes, c.g. RealEstate Zhou al. (2018). which restricts the model's ability generalize more dynamic environments. This is due to the fact that labeling camera poses for dynamic scene data is very challenging and complex. Fortunately, we have identified highly suitable dataset, Spic-Field Damen et al. (2018). which, although not previously used in video generation research, aligns perfectly with our requirements.
However, the Epic-Field dataset is not ready to be used directly. transform into resource suitable for egocentric video generation, we installed two major and laborious upgrades: (a). we re-annotated the entire dataset's text labels using Cog VL.M2 Hong al. (2024), and (b). due to the significant camera motion, applied NAFNET Chu et al. (2022) to perform motion deblurring each frame. We applied the same modifications to the RealEstare dataset as well.
3 EVALUATION
We will now evaluate the proposed method, EgoSim, and demonstrate its applications across various scenarios. In particular, will highlight the following key capabilities of the model: (a). precise control of camera movement, and (b). generating meaningful interactions with the environment based on both observation and imagination.
To demonstrate these two capabilities, the experiments are organized into three distinct scenarios: (1) camera text-to-video (Section 3.2.1), (2) camera image-to-video (Section 3.2.2), and (3) camera image + text-to-video (Sections 3.2.3 and 3.3).
3.1 EXPERIMENTAL. SETUP
We leverage two pretrained video diffusion model as base to implement our method. More specifically, for the image-to-vides case, Stable Video Diffusion (SVD) Blattmann et al. (2023a). For experiments that require text input, we use AnimateDiff Guo et al. (2023c).
Pretrained weights:
Baselines: To the best of our knowledge, there no video model that can simultaneously accept camera information, video frames, and text conditional inputs. Consequently,
only attempt some simplified experimental scenarios, or we can modify existing methods for comparative purposes. CameraCtri He et al. (2024) and MotionCtrl Wang al. (2023) included baselines for video diffusion model equipped with camera motion control, Methods which can modify AnimateDiff for image conditioning involve V2Vaadape Guo et al. (2023a) and SparseCurl Guo
(2023b). We
reproduced the above models following the respective research papers and publicly available code.
Metrics: To estimate the fidelity of egocentric video prediction, SSIM (Wang al.] (2004) and the Fréchet Video Distance (FVD) Unterthiner et al. (2019) are calculated between the predictions and groundtruths. Besides, we recruit Colmap Colmap|Sayab et al. assess the camera poses of generated videos to see if they authentically follow camera movements. TransErr and RotErr He et al. (2024) are computed by comparing the ground truth camera poses [R T] and estimation [R, as follows.
tr( (r(RRRIT RiT)
RotErr
arccos
and TransErr Σ T 12
2
We use a mixture of groundtruth trajectories and more difficult random trajectories to calculate the above errors.

---

Published as a conference paper at ICLR 2025
Text-tovvideo generation. Prompt: A bathroom with toilet. Input camera moving forward, left and then forward are shown on the left. CameraCtrl depicts it wrong by not moving left and by stopping before toilet.
EgoSim
CameraCtrl
b. Image-to-videss generation: No prompt. SVD base. Input camera moving toward lower right shown on the left. MotionCtrl depicts it wrong by moving backward.
EgoSim
MotionCtrl
c. Prompt: An aerial view of suburban neighborhood Camera focuses on the center while moving circular motion.
Baseline depicts it wrong by adding irrelvant details.
EgoSim
CameraCtrl
12V-Adapter
Figure Qualitative Comparison on RealEstate Datasets.
7

---

Published as a conference paper at ICLR 2025
3.2 REALESTATE
RealEstate Zhou al. (0118) consists large number open house video tours. In the case of AnimateDiff, we resize the video resolution to 256 x 384. We train and test the model with a length of 16 frames. For SVD, we resize to 256 512 and use T
3.2.1 TEXT-TO- TEXD-OO-VIDEO GENERATION WITH CAMERA CONTROL
In this section, will first evaluate the effectiveness of SEAL for text-to-video model equipped with camera control on the RealEstate dataset. As presented in Table I the proposed method outperforms all previous methods by wide margin on all metries.
Qualitative analysis in Fig 4a further demonstrates the limitations of previous methods and the superiority of our approach. We can see that the baseline model is unable to generate out-of- distribution movements because its control is influenced by intermediate features. In contrast, our method perfectly adheres the specified camera movements. addition, SEAL also improves the consistency and realism of the videos thanks to its spacetime structure.
Method
Transirr, RotEar]
FVDJ
Method
Transtir, RotEm]]
FVD.
SSIMT
MotionCud
9.88
1.23
793.5
MotionCtri
7.74
0.88
330.1
0.678
CameraCtal
9.67
1.23
782.4
CameraCul
6.98
0,81
318.3
0.691
EgoSim
722.0
EgoSinn (SVD)
0.831
Table I: RealEstate- T2V results.
Table RealEstate-12V results.
RealEstate-T-ID3
Epic-Field-T-I2VV
Mothod
TransEm: RotEmr)
FVD1
SSIMt
Method
TransEr: RotEr.) FVD
SSIM?
MotionCull
14.10
488.3
MotionCtrl
16.31
1.70
CameraCull +SparseCtrl
1360.8
0.353
CameraCul +SparsoCiil
1566.2
0.331
CameraCurl + 12V
13.95
1.68
472.9
0.645
CameraCUll + 12V
15.95
1.77
1172.1
0.625
EguSim
6.75
0.77
293.7
0.808
EgoSim
12.41
1.27
663.2
0.755
Table 3: RealEstate-T+12V results.
Table Epic-FFild-TT112 results.
3.2.2 IMAGE-TO- TO-G-OOOVIDDO GENERATION WITH CAMERA CONTROL
Next, we move to the pure image-to-video setting and compare our method with MotionCtrl + SVD and CameraCtrl + SVD. it should be noted that estimating SSIM becomes practically meaningful in the case of frame-conditioned generation because SSIM can more effectively tell us if the model strictly follows the given camera motion for given groundtruth-generaiion pair. Besides, as SVD uses more parameters, calculations and data compared to AnimateDiff, the overall generated video quality is better.
Results: As shown in Table 2 EgaSim significantly improves on all metrics compared to the baseline methods. In Figure 4p. we can see that because the context frame we provide is a long corridor, MotionCtrl would assume that movement the lower right corner should not occur even if it's valid movement. On the contrary, our model can faithfully generate the corresponding movement to the lower right corner expected.
3.2.3 CAMERA, FRAME AND TEXT
As we mentioned earlier, due to the lack of existing baselines in this setting, we can only create baseline models for comparison by combining existing methods. More specifically, we tried the following three combinations: MotionCtrl + 12V-Adapter, CameraCtrl + 12V-Adapter and CameraCtrl + SparseCtrrl Guo et al. (2023b). It is worth noting that CameraCtrl + 12V-Adapter in general performed well, but were unable to successfully reproduce the results with CameraCtrl + SparseCtrl and cannot calculate TransErr and RotErr even though we directly used the publicly available code and the pre-trained weights provided by the authors.
8

---

Published as a conference paper at ICLR 2025
Results:
Quantitative results summarized in Table and EgoSim achieves the best performance. In Figure4:: we can see that CameraCtrl + 12V-Adapter suffers from the intrusion issue while the proposed method can add reasonable imagination top of observed details.
3.3 EPIC-FIELD
Finally, we move the most challenging setting, Epic-Field Damen et al. (2018). Compared with RealEstate, Epic-Field involves not only camera movements but also a significant amount of interaction with the environment. Thus, model needs to learn additionally how to generate videos of these actions. We trained an additional LORA for Epic-Field, and as in the previous section, we conducted experiments using MotionCtrl + 12V-Adapter CameraCtrl + 12V-Adatter and CameraCurl + SparseCtrl as baselines. We resized the video frames 256 x 448 and use T = 14.
The quantitative comparisons are provided in Table and EgoSim achieves the best scores on all metries. The qualitative analysis in Fig further reveals the advantage of our method. We encourage readers to view more impressive visual results the project page. EgoSim not only learned generate precise autonomous movements but also can perform environmental interactions such as washing dishes and opening drawers. This significantly broadens the range of potential applications for EgoSim. Past inverse dynamics approaches Du et al. (2024) were limited in their ability control autonomous movements, restricting test scenarios to fixed camera positions. However, with the advent of EgoSim, can now attempt learn more advanced inverse dynamics models. This enables dynamic camera positioning and broader testing capabilities, paving the way for more robust and versatile autonomous systems.
Results:
3.4 ABLATION STUDY
In this section, we evaluate the contribution of each module to the model's improvement through extensive ablation experiments. As shown in Table [ each modification contributes either the improvement of the video quality or better accuracy of camera control.
RealEstate-12
Epic-Field
Method
TransErr, RotEr
FVDJ
SSIM
TransEm,
RotEr,)
FVD4
SSIM
MiimattDiff-I2-=Mootio
14.10
1.71
488.3
0.637
16.31
1.70
1223.5
0.612
Plucker embedding
13.96
1.68
466.5
0.644
16.08
1.77
1213.0
0.611
SEAL
8.12
366.7
0.739
13.81
1.41
786.2
0.698
C12VVadapter
6.75
0.77
293.7
0.808
12.41
1.27
663.2
0.755
Table 5: Ablation study. Note that for TransErr, RotErr and FVD, lower number indicates better performance while higher SSIM means better.
4 RELATED WORK
4.1 VIDEO GENERATION
Since the advent of the deep learning era, video generation has been an active area of research. Initially, ConvLSTM model Shi al. (2015): Wang al.] (2017); Yu (2020) was popular, followed by GAN models Skorokhodov et al. 2022]; Yu et al. (2022), and more recently, video diffusion models Blattmann al. (2023b) have become the mainstream. Video diffusion model Blattmann et al.) 2023a) usually extends a 2D image diffusion architecture to handle video data, enabling joint training on both images and videos from the ground up. To leverage powerful pre- trained image generators like Stable Diffusion Rombach et al. 2022). subsequent approaches expand the 2D architecture by integrating temporal layers between the pre- pre-trained 2D layers. This new model then fine-tuned on large video dataset. The latest advancement is SORA OpenAl (2024). which abandons the affrrementioned strategy and directly uses diffusion transformers Peebles & Xie (2023) train on large-scale videos of different sizes. Its generation quality is exceptionally impressive.

---

Published as a conference paper at ICLR 2025
Prompt: a person is preparing a pizza in front of stove
Baseline depicts it wrong by adding distorted hallucination.
GT
EgoSim
CameraCtrl
12V-Adapter
Figure 5: Qualitative Comparison on Epic-Field Dataset.
4.2 VIDEO GENERATION WITH CAMERA CONTROL
Research on video generation with camera control Yu et al. (024)) is still in its early stages. The first two works, MotionCtrl Wang et al. (2023) and CameraCtri et al. (2024), relied on directly concatenating adding camera embeddings to control camera movement. Researchers soon recog- nized the limitations of such methods. Epipolar attention Kant et al. (2024); Du al. (2023) has been commonly studied in the context of 3D generation, but its potential for video generation remains largely unexplored. It is worth noting that two other concurrent works, CamCo Xu et al. (2024) and CVD Kuann et al. (2024), have also acknowledged the importance of incorporating epipolar attention. However, unlike their approaches, our work not only leverages epipolar attention but also, independently of their efforts, identifies and addresses the limitations present in their methods. CamCo focused only on image-to-video (I2V), which our early analysis showed I2V fails to depict sigmificant camera movements, covering only part of our research. CVD, meanwhile, requires gener- ating multiple videos and cannot directly produce single onc. While they recognized the importance of cross-framd frame interaction, they took an indirect approach by establishing interaction between two different video trajectories.
5 CONCLUSION AND FUTURE WORK
This paper introduces EgoSim, a compositional world simulator that can egocentrically explore and interact with the observed environment. To achieve this, we identify two major obstacles that prevent the creation of meaningful videos that accurately adhere to user-specified instructions and tackle them with spacetime epipolar attention and C12V-adapter. Our designed model has demonstrated unprecedented controllability and the potential uses of such an egocentric world simulator are also diverse and impactful. Here, we summarize two interesting directions for future work:
DIT video diffusion: more powerful DiT diffusion models often employ 3D VAE to compress the temporal dimension. This compression complicates the process of the 3D projection used in epipolar attention directly correspond to specific spacetime patches, making it more difficult to maintain precise attention.
Although our method is plug-and hodd-paayy key challenge arises from the fact that
Inverse dynamics: The most suitable application scenario for this method is inverse dynamics. We hope to test EgoSim soon for training robots, giving embodied agents the ability to plan and simulate the future through imagination.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et Stable video diffusion: Scaling latent video diffusion models to large datasets, arXiv preprint arXiv:2311.1512) 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023b.
Xiaojie Chu, Liangyu Chen, and Wenqing Yu. Nafssr: Stereo image super-resolution using nafnet. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
1239-1248, 2022.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV), pp. 720-736, 2018.
Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning render novel views from wide-baseline stereo pairs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023.
Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances Neural Information Processing Systems, 36, 2024.
Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel. Scenescape: Text-driven consistent
scene generation. Advances in Neural Information Processing Systems, 36, 2024.
Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Chongyang Ma, Weiming Hu, Zhengjun Zha, Haibin Huang, Pengfei Wan, al. 12v-adapter: A general image-to-video adapter for video diffusion models. arXiv preprint arXiv:2312. 2023a.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. arXiv preprins arXiv:2311. 2023b.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang. Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint wrXiv:2307.04725. 2023c.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint 1803. 2018.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cam- eractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint urXiv:2408. 16500, 2024.
Yash Kant, Ziyi Wu, Michael Vasilkovsky, Guocheng Qian, Jian Ren, Riza Alp Guler, Bernard Ghanem, Sergey Tulyakov, Igor Gilitschenski, and Aliaksandr Siarohin. Spad: Spatially aware multiview diffusers. arXiv preprint arXiv:2402.05235 2024.
Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hongsheng Li, Leonidas Guibas, and Gordon Wetzstein. Collaborative video diffusion: Consistent multi-video generation with camera control. arXiv preprint arXiv:2405. /7414, 2024.
11

---

Published as a conference paper at ICLR 2025
Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from a single image. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 14458-14467, 2021.
OpenAL. Sora. https://www.opent wtw.com//oooooo sora 2024. URL https://www openai Sora is an Al model that can create realistic and imaginative scenes from text
Com/ sora
instructions.
William Peebles and Saining Xic. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748 8163.8763..66 PMLR, 2021.
Xuanchi Ren and Xiaolong Wang. Look outside the room: Synthesizing a consistent long-term 3d scene video from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3563-3573, 2022.
Robin Rombach, Patrick Esser, and Björ Ommer. Geometry- -reee view synthesis: Transformers and no 3d priors. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 14356-14366, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
M Sayab, M Paananen, D Aerden, P Saarela, and Dragon Mining. 'structure- view stereo' strone-based photogrammetry: An efficient virtual toolkit for measuring structural elements 3-d on inaccessible outerops. In 3rd Finnish National Colloquiun of Geosciences Espoo, 15-16 March 2017, pp. 97.
and 'multi-
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1874-1883, 2016.
Xingjian Shi, Zhourong Chen, Hao Wang. Din
Yan Yeung, Wai-Kin Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. In Advances in neural information processing systems, pp. 802-810, 2015.
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 34: 19313-19-933 2021.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3626-3623333 3636, 2022.
Hung-Yu Tseng, Qinbo Li, Changil Kim, Suhib Alsisan, Jia-Bin Huang, and Johannes Kopf. Con- sistent view synthesis with pose-guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (6773-16783, 2023.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017
12

---

Published as a conference paper at ICLR 2025
Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from single image using latent video diffusion. arXiv preprint arXiv:2403.12008 2024.
Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and S Yu Philip. Predrnn: Recur- rent neural networks for predictive learning using spatiotemporal Istms. In Advances in Neural Information Processing Systems, pp. 879-888, 2017.
Zhou Wang, Alan Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4):600-612, 2004.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionetrl: A unified and flexible motion controller for video generation. arXiv preprint arXiv:2312.03641, 2023.
Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Camco: Camera-controlllblle 3d-consistent image-to-vides generation. arXiv preprint (xXiv:2406.02509 2024.
Mengjiao Yang. Yilun Du. Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114, 2023.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.00672,, 2023.
Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang. Xiangjun Gao, Tien- Tsin Wong, Ying Shan, and Yonghong Tian. Viewerafter: Taming video diffusion models for high-fidelity novel view synthesis. arXiv preprint arXiv:2409. 02048, 2024.
Wei Yu, Yichao Lu, Steve Easterbrook, and Sanja Fidler. Efficient and informatioon future frame prediction and beyond. 2020.
Wei Yu, Wenxin Chen, Songheng Yin, Steve Easterbrook, and Animesh Garg. Modular action concept grounding in semantic video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3605-3614, 2022.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: 2018.
Learning view synthesis using multiplane images. arXiv preprint
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 TECHNICAL DETAILS
A.1.1 DATA
For RealEstate dataset, we use 67,477 scenes for training and 7,289 scenes for testing. For Epic-Field we use 611 scenes for training and 88 scenes for testing. should be noted that the duration of each scene of Epic-Field is much longer than that in RealEstate. That's why the number of scenes in Epic-Field is much smaller.
We found that large number of videos in Epic-Field suffer from significant motion blur due to rapid movement, which is detrimental model learning. Therefore, we applied NAFNet Chu et al. (2022) to perform motion deblurring on all frames.
During the training, we use sample stride of for RealEstate and for Epic-Field. We also used several additional techniques for data augmentation. In RealEstate, since most scenes are static, we can reverse the videos to generate additional motion trajectories. In both datasets, we can randomly increase or decrease the sample stride by one step obtain video clips with different speeds, As result, each training sample is consist of 14-frame video clip, a text prompt and camera poses for all frames.
A.1.2 TRAINING AND INFERENCE
In the of SVD and AnimateDifff we keep the original parameters of our base models fixed and only optimize the newly introduced layers and their subsequent layers using the AdamW optimizer with learning rate 2 x 10-4, All models are trained on 8 NVIDIA A A100 GPUs for 300k iterations using an effective batch size 32. We use BF16 precision for training SVD.
We use DDIM scheduler with 1000 steps during training and 25 steps during inference.
A.1.3 EVALUATION
For caleulation of TransErr and RotErr, we basically follow the protocol from CameraCtrl. The main difference is that we adopted different camera poses. We sorted the translation and rotation of the real camera poses from two datasets, and then randomly selected poses from the top 20% to 40% and 40% to 60% ranges classify them into hard and medium groups, respectively. Additionally, we designed simple linear motions and rotations to create the simple group. We combined the hard, medium, and simple groups in 1:1:1 ratio adequately test the camera poses. Therefore, our selection process, which filters out poses with larger motion ranges, results in more challenging evaluation. For FVD) and SSIM, we follow the common practice.
A.2 RELATED WORK
The exploration of egocentric 3D generation through direct camera control has undergone several paradigm shifts. InfiniteNature [Lu et al. (202T) stands out as an early influential work in this field. This project primarily utilizes traditional computer vision techniques. takes paired RGB image and disparity map to construct a textured mesh and then renders from new perspectives by warping the textures adjust disparities. To complete the process, a refinement network addresses and corrects any gaps or missing parts in the final output. This method does not fully leverage the learning capabilities of neural networks.
After the rise of generating images using transformerbbseee autoregressive models, GeoGPT Rombach al. (202)) posits that all camera motion control conditions can be transformed into tokens or directly summed embeddings. They experimented with a wide variety of ways for incorporating control conditions to assess their impact on the generated results. However, their fusion methods only included concatenation and addition. Therefore, GeoGPT encounters the same issues CameraCtrl and MotionCtrl, specifically limited ranges generated motion. Therefore, subsequent studies primarily focused on how introduce more explicit geometrical control. "Look Outside the Room" RRnn & Wang (2022) proposed using Gamera-Awand Bias by computing a bias with a Multi- Multi-Layer Perceptron (MLP), which takes the relative camera position input during the calculation of the
14

---

Published as a conference paper at ICLR 2025
attention score. The paper Du et al. (2023) utilized epipolar attention to aggregate information from patches under different viewpoints to generate the final render. These efforts have achieved significant progress.
Subsequently, the era of diffusion models arrived. Researchers Tseng et al. (202)) introduced epipolar attention into the of diffusion models for 3D generation. SceneScape Fridman al. (2024) returned to the framework of InfinitNature, but with the aid more powerful diffusion model. integrated control capabilities and achieved better results.
15


Reviews:
[{"review_id": "SgT25yv21R", "reviewer": "Reviewer_4w3N", "summary": "This paper presents a novel video diffusion architecture capable of handling multiple conditioning inputs, including image, text, and camera poses, in a unified framework. The work makes meaningful technical contributions to controllable video generation, though there are areas where clarity could be improved.", "questions": "1.1 What is the difference between EgoSim (SVD) and EgoSim in Table 1? \n\n1.2 For Epic-Field experiments, what are the input conditions (text, image, or both) in Table 1?\n\n1.3 Regarding LoRA usage (Line 437): Why was an additional LoRA necessary for Epic-Field when the model was already fine-tuned? This seems redundant and needs justification. Did the authors use LoRA to fine-tune the pre-trained model?\n\n1.4 How were camera poses obtained/annotated for RealEstate and Epic-Field datasets?\n\n1.5 How is the training and testing dataset split? What are the respective dataset sizes?\n\n1.6 No training detail is provided. Please mention your setup such as earning rates, optimization parameters, batch sizes, number of training iterations, hardware specifications, and training time.\n\n2. How is K* and V* calculated in practice?\n\n3.1 L265 \"attachted\"\" revise my review for this paper.\n\nThese clarifications would significantly improve the paper's reproducibility and technical clarity.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The authors integrate multiple conditioning types in a coherent framework, demonstrating a novel adaptation of epipolar attention from 3D generation to video generation. The introduction of the SEAL and CI2V-adapter shows thoughtful consideration of the challenges in multi-modal video generation. The evaluation demonstrated on both static (RealEstate) and dynamic (Epic-Field) datasets, supported by both qualitative and quantitative improvements over existing methods. The extensive ablation studies further strengthen the technical contributions.", "weaknesses": "The paper suffers from several clarity issues that should be addressed. The experimental setup and results sections (3.2, 3.3) lack clear organization and the data preparation is not detailed throughout the paper, making it difficult to fully understand the implementation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7LORy0DWyR", "reviewer": "Reviewer_SwtR", "summary": "This paper tackles video generation with multi-modal condition signals: text, image, and camera pose. It introduces several model designs including employing epipolar attention to the spacetime domain for precise camera motion control and a CI2V adaptor that balances text and vision guidance based on camera information. Further, it repurposes the EPIC Fields dataset as the new dynamic scene dataset with camera annotations. Extensive experiments show the effectiveness of each proposed module.", "questions": "Please refer to the weaknesses section. Overall I think the task addressed is important and interesting. Most of the simple cases look fine. I suggest the authors add more comparisons with more recent methods, i.e., viewcrafter.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ The multi-modality control of video generation is an interesting topic. The proposed method fills in the gap in precise camera-conditioned image-to-video generation. \n+ Most of the past camera-conditioned video generations trained on static 3D scene datasets, i.e. Realestate, DL3DV. The proposed method provides an effective practice to repurpose video understanding benchmarks for generation and to some extent shows a way to resolve the data scarcity of dynamic scene data with camera pose annotations.\n+ Balancing different control signals is an intuitive challenge in multi-modal guided video generation. The proposed CI2V adapter is a simple and effective strategy to handle it.", "weaknesses": "- Given that Realestate is a large-scale dataset with 100 times the number of scenes compared to Epic-Field, how do you prevent overfitting your generations to static scenes?\n- It would be interesting to compare the proposed method with Viewcrafter [M1] in terms of the preciseness of camera controls in a static 3D scene. \n- The camera trajectories in the results are quite simple and mostly object-centric, it would be better to infer with longer, more complex trajectories in open scenes. \n- [Minor] The examples in 'Interacting with the World' contain too many noticeable artifacts, e.g., hand disappearance, hand merging into objects, etc. \n\n[M1] Yu, Wangbo, et al. \"Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis.\" arXiv 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qWh6bvEQ57", "reviewer": "Reviewer_L7sN", "summary": "This work focuses on improving the controllability of video diffusion models for egocentric exploration with camera ego-motion + text + image as conditioning. The proposed framework incorporates epipolar constraints to better focus on the relevant parts in the attention mechanism, referred to as Spacetime Epipolar Attention (SEAL). It is then combined with existing text-to-video and image-to-video diffusion models. Experiments on RealState and EPIC-Fields datasets show the effectiveness of the proposed approach over existing methods.", "questions": "There are 3 major concerns (details above):\n- Several values are missing in Tab.1 which makes it difficult to understand the trends. It'd be helpful to provide details on these missing values and EPIC-Fields experiment setting.\n- The text mentions efficiency benefits in 2 places and disentangling text & image features. It'd be useful to verify if this is indeed the case.\n- Several aspects of the text need further clarifications.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This work focuses on improving the controllability of video diffusion models with multiple conditioning factors: combinations of camera ego-motion, text, and image, which is important from a practical use perspective.\n- The idea of incorporating epipolar constraints into the attention mechanism is simple and intuitive.\n- Experiments on RealState (Tab.1, Fig.4) and EPIC-Fields (Tab.1, Fig.5) datasets show the effectiveness of the proposed approach over existing methods.\n- Ablations in Tab.2 and visualizations on the website provide a better understanding of the capabilities of the proposed approach.", "weaknesses": "- In Tab.1, several values are missing, which makes it difficult to compare different models. There is some justification in the text (L426-429). It'd be helpful to have more details:\n    - Why can't MotionCtrl and EgoSim (SVD) be evaluated on EPIC-Fields? Both are I2V methods and EPIC-Fields contains both frames and camera egomotion.\n    - For CameraCtrl + SparseCtrl, why can't TransErr and RotErr be computed? Since ground truth is available, the camera trajectory from the video diffusion output needs to be computed. Is it because COLMAP optimization does not converge on the outputs? Is there some other reason?\n    - It'd also be useful to have T2V and I2V settings on EPIC-Fields to better understand the trends across different datasets. Since text description is available for EPIC-Fields (Fig.5), is there any reason to not use these settings?\n- There are 2 mentions of efficiency benefits in the text. It'd be helpful to verify these benefits quantitatively, in terms of memory usage and train/inference time.\n    - L205-206: The use of epipolar attention introduces additional sparsity, enabling us to utilize memory-efficient operations.\n    - L236-238: employ pixel unshuffle Shi et al. (2016) to adjust the size while preserving as much fine-grained positional information as possible. This approach is sufficient and also helps to save computational resources. \n- L245-246 mentions: 'a particular patch in a specific frame should be explained either by text or by the context frame, but not both'. It'd be interesting to see if this is indeed the case. One way to do this is to check the values of $\\mu$ in Eq.4, which should be close to 0 or 1.\n- Some experimental details are missing:\n    - L288-290: details on how EPIC-Fields is processed.\n    - L323: what does 'more difficult random trajectories' mean? how are they sampled?\n    - Are the baselines re-trained in the same setting or used in a zero-shot manner?\n- It'd be helpful to clarify these aspects:\n    - L19-20: while textual information compensates for necessary spatiotemporal structures, it often intrudes into already observed parts of the scene\n    - L25: issue of textual intrusion into observed areas\n    - L109-110: override the interference of other features in video generation\n    - L112-114: leverages camera movement information to further assist in clearly defining the boundaries between the text and visual elements\n    - L189-191: need a control method that operates relatively independently of these features", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)