VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
Authors: Sherwin Bahmani
Ivan Skorokhodov
Aliaksandr Siarohin
Willi Menapace
Guocheng Qian
Michael Vasilkovsky
Hsin-Ying Lee
Chaoyang Wang
Jiaxu Zou
Andrea Tagliasacchi
David B. Lindell
Sergey Tulyakov
Venue: ICLR 2025 Poster (2025)
Keywords: video generation
3d
diffusion
Primary area: generative models
TL;DR: 3D camera control for video diffusion transformers

Abstract: Modern text-to-video synthesis models demonstrate coherent, photorealistic generation of complex videos from a text description. However, most existing models lack fine-grained control over camera movement, which is critical for downstream applications related to content creation, visual effects, and 3D vision. Recently, new methods demonstrate the ability to generate videos with controllable camera poses---these techniques leverage pre-trained U-Net-based diffusion models that explicitly disentangle spatial and temporal generation. Still, no existing approach enables camera control for new, transformer-based video diffusion models that process spatial and temporal information jointly. Here, we propose to tame video transformers for 3D camera control using a ControlNet-like conditioning mechanism that incorporates spatiotemporal camera embeddings based on Plucker coordinates. The approach demonstrates state-of-the-art performance for controllable video generation after fine-tuning on the RealEstate10K dataset. To the best of our knowledge, our work is the first to enable camera control for transformer-based video diffusion models.

Published as a conference paper at ICLR 2025
VD3D: TAMING LARGE VIDEO DIFFUSION TRANSFORMERS FOR 3D CAMERA CONTROL
Sherwin Bahmani
1.2.3 1,2.3 Ivan Skorokhodoy
Aliaksandr Siarohin
Willi Menapace"
Guocheng Qian
Michael Vasilkovsky
Hsin- Hinn-Ying Lee
Chaoyang Wang
Jiaxu Zou
Andrea Tagliasacchi David Lindell
Sergey Tulyakov
University of Toronto
Vector Institute
Snap Inc. *SFU
httoss//anap-rrseecchhh.ci
1o/ Lo/vd3d
"Otters with chef hats, skillfully preparing a miniature sushi feast on lily pad"
Camera
"A dog wearing vr goggles on a boat"
Figure I: 3D camera control for text-to-vides generation. We introduce method that can control camera poses for text-to-video generation using video diffusion transformers. (left) The method takes imput set of camera poses used to generate each frame of rendered video. (center, right) Applying multiple camera trajectories with the same text prompt enables synthesis of complex scenes from varied set of viewpoints. Video results: https: /ssap- resea github. oovvd3d
ABSTRACT
Modern text-to-video synthesis models demonstrate coherent, photorealistic gener- ation of complex videos from a text description. However, most existing models lack fine-gu reneinrdeedd control over camera movement, which is critical for downstream applications related to content creation, visual effects, and 3D vision. Recently, new methods demonstrate the ability to generate videos with controllable cam- era poses- these techniques leverage pre-trained U-Net-based diffusion models that explicitly disentangle spatial and temporal generation. Still, existing ap- proach enables camera control for new, transformer- video diffusion models that process spatial and temporal information jointly. Here, we propose tame video transformers for 3D camera control using a ControlNet-like conditioning mechanism that incorporates spatiotemporal camera embeddings based Plücker coordinates. The approach demonstrates state-of-the-art performance for control- lable video generation after fine-tuning on the RealEsstate 10K dataset. To the best of our knowledge, our work is the first to enable camera control for transformer-base video diffusion models.
INTRODUCTION
Text-to-video foundation models achieve unprecedented visual quality (Brooks et al., 2024; Sharma et al. 2024). They are trained on massive collections of images and videas and learn to synthesize remarkably consistent and physically plausible visualizations of the world. Yet, they lack built-in mechanisms for explicit 3D control during the synthesis process, requiring users manipulate outputs through prompt engineering and trial and error a slow, laborious, and computationally expensive process. For example, Fig. shows, ofate-of-the-a video models struggle to follow even simple "zoom-in" or "zoom-out" camera trajectories using text prompt instructions (see supplemental webpage). This lack of controllability limits interactivity and makes existing video generation

---

Published as a conference paper at ICLR 2025
"A space shuttle launching"
4D-fy
space shuttle launching, camera zoom
SnapVideo Video
space shuttle launching"
VD3D
Figure 2: Comparing text-to-video, text-to-4D, and camera-conditioned text-to-video generation. We show time progressing across columns for generated videos from different approaches. Text- to-4D approaches, such as 4D-fy (Bahmani et al.) 2024b) (top). have complete control over the camera through a representation but lack photorealism. We visualize time progressing while simultaneously changing the viewpoint. (middle) Methods for text-to-video generation (Menapace ct al., 2024) create realistic videos but do not provide explicit control over the viewpoint. In contrast, camera-condiin conditionddd text-to-video generation (bottom) bridges the gap between the paradigms by extending text-to-video generators with 3D camera control without using an explicit 3D representation. Specifically, condition VD3D with zoom-in camera trajectory to control the generation.
techniques challenging use for artists or other end users. We augment 2D video generation models with control over position and orientation of the camera, providing finer-grained control compared to text prompting, and facilitating of video generation models for downstream applications.
Several contemporary works (Wang et al. 2023c He et al. 2024a Yang et al.) 2024b) propose methods for camera control of state-of-the-art, open-source video diffusion models. The key technical insight proposed by these methods is to add camera control by fine-tuning the temporal conditioning layers of U-Net-based based video generation model dataset with high-quality camera annotations. While these techniques achieve promising results, they are not applicable to more recent, high-quality transformer-based architectures (Vaswani al. 2017; Peebles & Xie 2023), such as Sora (Brooks) et al. 2024), Video [Menapace et al. 2024). and Lumina- T2X (Gao et al. 2022a),, as these latest works simply do have standalone temporal layers amenable to camera conditioning.
Large video transformers represent a video as (possibly compressed) sequence of tokens, applying self-attention layers to all the tokens jointly (Brooks al. 2024 Menapace al., 2024). Consequently, as diffusion transformers do not have standalone temporal layers, they are incompatible with current camera conditioning approaches. As the community shifts towards large video transformers to jointly model spatiotemporal dependencies in the data, it is critical to develop methods that provide similar capabilities for camera control. Our work designs a camera conditioning method tailored to the joint spatiotemporal computation used in large video transformers and takes step towards taming them for controllable video synthesis.
We develop our work on top of our implementation of SnapVideo (Menapace al. 2024 state-of- the-art video diffusion model, that employs Far-reaching Interleaved Transformers (FIT) blocks (Chen & Li 2023) for efficient video modeling in the compressed latent space. We investigate various camera conditioning mechanisms in the fine-tuning scenario and explore trade-offs terms of visual quality preservation and controllability. Our findings reveal that simply adapting existing approaches
2

---

Published as a conference paper at ICLR 2025
to video transformers does not yield satisfactory results: they either enable some limited amount of control while reducing the visual quality of the output video, or they entirely fail to control camera motion. Our key technical insight is to enable the control through spatiotemporal camera embeddings, which we derive by combining Plücker coordinates with the network input through a separately trained cross-attention layer. To the best of knowledge, our work is the first to explore a ControlNet-like (Zhang et al. 2023) conditioning mechanism for spatiotemporal transformers.
We evaluate the method on a collection of manually crafted text prompts and unseen camera trajecto- ries and compare to baseline approaches that incorporate previous camera control methods into a video transformer. Our approach achieves state-of of-the-art results in terms of camera controllability and video quality, and also enables downstream applications such as multi-view, text-to-video genera- tion, as depicted in Figure [] In contrast to existing image-to-3ll methods (e.g., (Liu et al. 2024a] Qian et al. 2024b; Voleti et al. 2024a]), which are limited to object-centric scenes, our approach synthesizes novel views for real input images with complex environments.
Overall, our work makes the following contributions.
We propose a new method to tame large video transformers for 3D camera control. Our approach uses a ControlNet-like conditioning mechanism that incorporates spatiotemporal camera embeddings based on Plücker coordinates.
We thoroughly evaluate this approach, including comparisons to previous camera control methods, which we adapt to the video transformer architecture.
We show state-of-the-ar results in camera-controllable video synthesis by applying the proposed conditioning method and fine-tuning scheme to the Snap Video-based model (Mena pace et al.) 2024).
2 RELATED WORK
Our method is connected to techniques related to text-to-video, text-to-3D, and text-to-4D generation. As this is a popular and fast-moving field, this section provides only partial overview with focus on the most relevant techniques; we refer readers to Po et al. (2023) and Yunus et al. (2024) for
more thorough review of related techniques.
Text-to-video generation. Our work builds on recent developments in 2D video generation models. One such class of these techniques works by augmenting exx-tto-mmag models with layers that operate on the temporal dimension to facilitate video generation (Blattmann et al., 2223b;; [Singer jtt al., 2023a; Wu al.) 2023) Guo et al.) |024| [Blatmann et al.) (023a). Video generation models can be trained in a hybrid fashion on both images and videos to improve the generation quality (Bain
cet al. 2021 Wang et al. 2023b Xue et al. 2022] Ho et al. 2022a Guo et al. 2024 He et al. 2022
Wang al. 2023c Zhou al. 2022). While they are primarily based on convolutional, U-Net-style architectures, a recent shift towards transformer- basedd architectures enables synthesis much longer videos with significantly higher quality (Brooks et al. 2024 Ma et al. 2024b Menapace et al. 2024 Ma et al. 2024a). Still, these methods do not enable synthesis with controllable camera motion.
4D generation. Previous methods also tackle the problem of 4D generation, i.e., generating videos of dynamic 3D scenes from controllable viewpoints, usually from an input text prompt or image. Since the initial work on this topic using large-scale generative models (Singer et al. 2023b), significant improvements in the visual quality and motion quality of generated scenes have been achieved (Ren)
al.] [2023] Ling et al. [2024a; [Bahmani et al. 2024b] Zheng et al. [0244a] Bahmani et al. 2024a).
While these methods generate scenes based on text conditioning other approaches convert an input
image or video a dynamic 3D scene (Ren et al. 2023 Zhao et al. 2023 Yin et al. 2023 Pan et al.
2024 Zheng et al. 2024a] Ling et al. 2024a Gao et al. [2024b] Zeng et al. 2024 Chu et al. 2024). Another line of work (Bahmani et al. 2023a Xu et al. (2023) extends 3D GANs into 4D by training on 2D videos, however the quality is limited and models are trained on single category datasets. All of these methods are focused on object-centric generation, typically based on 3D volumetric representations. As such, they typically do not incorporate background elements, and overall. they do not approach the level of photorealism demonstrated by the state-of-t ofte-----thhrr art video generation models used in our technique (see Fig.
3

---

Published as a conference paper at ICLR 2025
Controllable generation with diffusion models. Methods for controllable generation using diffusion models have had significant impact, both in the context of image and video generation. For example, existing techniques allow controllable image generation conditioned on text, depth maps, edges, pose, or other signals (Zhang et al. 2023 Ye et al. 2023). While Chen et al. (2024) developed a ControlNet--aseed mechanism for transformer- measeeeee-- diffusion, limited to spatial conditioning with spatial signals, we explore conditioning mechanisms for camera poses (a spatio-temporal signal with an intricate temporal component) in a spatio-temporal transformer. Furthermore, there line of work for 3D generation that conditions diffusion models on camera poses for view-consistent multi- view generation (Watson et
2023 T seng et al., 2023] Chan et al. 2023] Yu al., 2023a Kumari et al. 2024 Müller al. 2024 Gao et al. 2024c). Our approach is most similar to related techniques
in video generation that seek to control the camera position. For example, MotionCtrl (Wang et al.
2023e) designs camera and object control mechanisms for the VideoCrafterl (Chen et al. (023b) and
SVD Blattmaan et al., 2023a) models. However, MotionCtrl is designed for U-Net-based approaches and does not directly apply to video diffusion transformers.
Concurrent 3D camera control methods. Concurrent approaches enable camera control by con- ditioning the temporal layers of the network with camera pose information, e.g., using Plücker coordinates (He et al.
2024a Guo et al. 2024] Xu et al.] 2024b] Kuang et al., (024) or other em- beddings YYang et al.
Interestingly. it also possible incorporate camera control into video generation models without additional training through manipulation and masking of attention layers. though this requires additional tracking, segmentation, or depth for each input video (Hu et al.) (024)XXia et al. 2024 Hou et al., 2024). Another recent work (Ling et al. 2024b) transfers motion, including camera motion, other generated videos.
2024b).
Although these approaches demonstrate promising results for U-Net-based video diffusion models, the techniques are not applicable to modern video transformers that model spatio-temporal dynamies jointly. While another concurrent work ( (Watson et al. (0244) uses a transformer-base based architecture for space and time, it does not tackle text-based generation for dynamic scenes but focuses on novel view synthesis from an input image. In our work, we design an efficient mechanism that enables camera control in video diffusion transformers using a ControlNet inspired mechanism without sacrificing visual quality.
METHOD
3.1 LARGE TEXT-TO-VIDEO TRANSFORMERS
Text-to-video generation. Diffusion models have emerged as the dominant paradigm for large- scale video generation (Ho et al. 2022b a Brooks et al. 2024). The standard setup considers the conditional distribution p y of videos æ E R'"xx x resolution) given their text descriptions y E y", consisting of (possibly padded) tokens from the alphabet V: Following (Karras et al. 2022), our video diffusion framework assumes denoising model Do : (æ: y, ++ á that predicts a clean video à from the corresponding noised input à x + de, where (0, I) is standard Gaussian noise and sampled from log-normal distribution with location P Pmean and scale Pss. The model is parametrized by neural network Fo (æ; 0, Dø y, o) Coul (a) (o (a)(())) ) y. + Cakip()) j where (a) (aur(ar and Caip (a) are input, output and residual scaling factors from (Krras et al. 2022).
(consisting of F frames of H x W
Iog mcann P2) is the noise strength,
The minimization objective is defined as
(())=
E
[| oE:3.0) x
p(..
We refer the reader to (Menapace et al.) 2024 and KKrrra et al. 2022 for further details on the diffusion setup, which we adopted without modifications.
Spatiotemporal transformers. Following Video (Menapace et al. 2024), our video generator consists two models: the base 4B-parameters generator, operating on fr-fes 36 x 64 resolution videos, and 288 x 512 upsampler. The latter, diffusion model itself, is fine-tuned from the base model and conditioned on the generated low-resolution videos. Each model uses FIT transformer blocks (Chen & Li| 2023 Jabri et al.) 2022) for efficient self-attention operations (see Fig. 33. An FIT model consists of B blocks (we have B = 6 in all the experiments) and first partitions each frame in input video into patches (Dosovitskiy al. 2021) of resolution hp Wy (we
4

---

Published as a conference paper at ICLR 2025
"running horse"
text enc.
noise std.
compression
framerate
joint s.t. block
resolution
docompression
time
XAttn + FF
FIT Block (repcated Bx)
latent tok.
SAtm + FF
latent tok
SA + FF
latent tok:
SAIH + FF
latent tok:
patches pixels
video pixels to patches
Plücker conditioning
( K f)
ConVies
FF
FF
Plucker coords.
Plücker conditioning
XAttn FF
FFttn+FF
XAttn.cm
p 6
FFa
Plücker tokens
v
video tokens
Plicker coords to patches
Figure 3: Overview of architecture. We adapt FIT-based architecture (Chen & Li 2023 to incorporate camera control. We take as input the noisy input video ă; camera extrinsics Cj, and camera intrinsics Ky y for each video frame f. We compute the Plücker coordinates for each pixel within the video frames using the camera parameters. Both the input video and Plücker coordinate frames are converted to patch tokens. and we condition the video patch tokens using a mechanism similar to ControlNet (Zhang et al. 2023) ("Plücker conditioning" block). Then, the model estimates FIT blocks (Chen & Li. 2023). Each block reads
the denoised video à by recurrent application information from the patch tokens into small set of latent tokens on which computation is performed. The results are written to the patch tokens in an iterative denoising diffusion process.
hp = u, = 4 in all the experiments). These video patches are then independently projected via feedforward (FF) layer obtain a sequence of video tokens Ve (#1, .... U1) E Lxd of length I, = F (H/hp) x (W/wp) and dimensionality d. Next, each FIT block "reads" the information from this video sequence into a much shorter sequence of M latent tokens = & (21,..., ZMM) through a "read" cross-attention layer, followed by feedforward layer. The processing with self-attention layers is performed in this latent space, and then the result written back the video tokens through corresponding "write" cross-attention layer (also followed by FF layer). The latent tokens in each subsequent FIT block are initialized from the previous one, which helps to propagate the computational results throughout the network. In this way, the entire computation occurs jointly in both spatial and temporal axes, which yields superior scalability (Menapace et al.) 2024). However, abandons the decomposed spatial/temporal computation nature modern video diffusion U-Nets, which modern camera conditioning techniques (Wang et al. 2023c Yang et al. 2024b) rely on enforce control without compromising visual quality.
3.2 CAMERA CONTROL FOR SPATIOTEMPORAN TRANSFORMERS
Spatiotemporal camera representation. The standard way of representing camera parameters for a video (in the pinhole model) via a trajectory of extrinsics and intrinsics camera parameters (C₁, K () for each f-th frame. The matrix C = RR t| 3 describes the camera rotation R E 33x3 and translation (and also horizontal/vertica skew coefficient, but it is always 0 in our setup). To control camera motion, existing methods condition the temporal attention layers of U-Net-based video generators on embeddings computed from these camera parameters (Wang et al.) 2023e Yang al. 2024b Hu et al. 2024). Such pipeline provides good conditioning signal for convolutional video generators with decomposed spatial/temporal computation, but experiments demonstrate that it works poorly for spatiotemporal transformers: they either fail to pick up any controllability (when being added transformed residuals to the latent tokens), or degrade the visual quality of the output (when
TR3 and K₃ E R contains the focal length and principal point
5

---

Published as a conference paper at ICLR 2025
the original network parameters are being fine-tuned). This motivates to design a better camera conditioning scheme, tailored for modern large-scale spatiotemporal transformers.
First, we propose to normalize the camera parameters w.r.t the first frame. For this, recompute the rotations and translations for each f-th frame as R' = Ri R2 and t'J t, t,. This procedure results in normalized camera extrinsics as C ; t'yl and establishes a consistent coordinate system across different samples in the dataset. After that, found essential enrich the conditioning information by switching from temporal frame-level camera parameters to pixel-wise spatiotemporal ones. This is achieved by computing the Plücker coordinates for each pixel, providing a fine-grained positional representation.
Plücker coordinates provide a convenient parametrization of lines in the 3D space, and we use them to compute fine-gre tme-grained positional representations of each pixel in each frame of video. Given the extrinsic and intrinsic camera parameters R'y. K of the f-th frame, we parametrize each (h, w--th pixel as a Plücker embedding ff,h. E R from the camera position to the pixel's center as
Pyh,w = (tz' x d d.s,,w),
dr,b... = jdd.k.ulll'
dz = R2 K (w, h 1] + tt'
This approach mirrors the technique used in recent 3D works (Sitzmann et al. 2021 Chen et al.
2023a Kant al. 2024) as well as CameraCtrl (He et al., 2024a), concurrent study focusing camera control in U-Net-based video diffusion models. The motivation for using Plücker coordinates is that geometric manipulations in the Plücker space can be performed through simple arithmetics on the coordinates, which makes it casier for the network to use the positional information stored in such a disentangled representation.
Computing Plücker coordinates for each pixel results in P R6xFxxxx spatiotemporal camera representation for video. To input it into the model, we first perform the equivalent ViT-like (Doso vitskiy et al. 2021 hp x Wip patchification procedure. is followed by learnable 2-layered MLP MLP; # with GELU (Hendrycks & Gimpel 2016) non-linearity to obtain the Plücker camera tokens sequence Č+LL E Rl.dd of the same length L = F x (H/hp) X (W/up) and dimensionality d as the video tokens sequence (/)) This spatiotemporal representation carries fine-grained positional information about each pixel in a video, making it easier for the generator to accurately follow the desired camera motion.
Camera conditioning. To input the Plücker embeddings into our video generator, we design an efficient ControlNet like (Zhang et al.) 2023) mechanism tailored for large transformer models (see Fig. 3. This mechanism is guided by two main objectives: 1) the model should be amenable to rapid fine-tuning from small dataset with estimated camera positions; and 2) the visual quality shouldn's 't be compromised during the fine-tuning stage. We found that meeting these objectives is more challenging for spatiotemporal transformers compared to U-Net- based models with decomposed spatial/temporal computation, since even minor interventions into their design quickly lead degraded video outputs. We hypothesize that the core reason for it is the entangled spatial/temporal computation of video transformers: any attempt alter the temporal dynamics (such as camera motion) influences spatial communication between the tokens, leading to unnecessary signal propagation and overfitting during the fine-tuning stage. To mitigate this, we input the camera information gradually through read cross-attention layers, zero-initialized from the original network parameters of the corresponding layers.
Specifically, in each b-th FIT block of our video generator, replace its standard read cross- attention operation (see aabri et al. 2022 Menapace et al. 2024) for details):
(b) M:M
FF FF(0)(XAtn (b) (x
(b) meAs: (b) D...),
(3)
where FF(-) and XAttn(,,) -) denote feed-forward and cross-attention layers respectively. Our revised formulation given as:
m:M (F(1) [XAttn (2) (= (.
(2(0))
Cony
(b) res
(F(() (XAttn (b)
cam (2 čeLL + o((L))
(4)
cam
where (P()) (b)
and XAttn cam are learnable layers, Conv
(b) is a 1-dimensional convolution that processes res z' z(b)
camera-augmented latents. The produced latents
m: are then passed to the sequence of four

---

Published as a conference paper at ICLR 2025
sheep enjoying spughetti together"
"Melling cream dripping down the cone"
MotionCtrl
CameraCtrl
Ours
Figure 4: Camera-conditioned text-to-video generation. Comparison the proposed approach to MotionCtrl and CameraCtrl for the same camera trajectory input. We adapted MotionCtrl and CameraCtrl to the same transformer- based model for fair comparisons. MotionCtrl exhibits worse quality due fine-tuning existing layers, and CameraCtrl See the supplementary webpage for video results.
sensitive to camera conditioning.
self-attention layers, which are the core computational component of FIT. It crucial to instantiate the weights of the output convolutions Conv! from zeros preserve the model initialization. Besides,
we initialize the weights of FF and XAttn (b)
cam from the corresponding parameters of the original network. This approach helps to preserve visual quality at initialization and facilitates rapid fine- tuning small dataset. As a result, we obtain the method for fine-grained 3D camera control in
large video diffusion transformers. We name it VD3D and visualize its architecture Figure]
3.3 TRAINING DETAILS
To ensure comparability with prior work such as Waan et al. 2023e), we train our video generator on the same RealFEstate 10K dataset (Zhou al. 2018). We optimize only the newly added parameters ) > and keep the rest of the network frozen. We found that training only the base 36 x 64 model was sufficient, as the 288 x 512 upsampler already accurately follows the camera motion of low-resolution video.
FF and (FF()) XAttn
3.4 DATASET
We fine-tune a text-to-video model, pre-trained on 2D video data, on RealEstatc (Zhou et al. 2018). The training split for fine-tuning consists of roughly 65K video clips, and is the same as is used in concurrent work (MotionCtrl (Wang et al. 2023e) and CameraCtrl (He et al. 2024a)).
3.5 METRICS
We conduct a user study to evaluate our approach for camera-controlled text-to-video generation. The study participants are presented with 20 side-by-side comparisons between the proposed approach and the basclines as well as a reference video from RealEstate 10K with the same trajectory to better judge the camera alignment. We ask 20 participants for each generated video sequence to indicate which generated video they prefer based on multiple submetrics, namely, camera alignment, motion quality, text alignment, visual quality, and overall preference. The user study involved negligible risk to the participants and was conducted with appropriate institutional review board and legal approval. We evaluate our method using 20 camera trajectories sampled from the RealEstate 10K test split that not seen during training for the user study. We use the full test split combined with unseen text prompts for the automated camera evaluations, ie., 6928 unseen camera trajectories combined with out-of-distribution text prompts.
7

---

Published as a conference paper at ICLR 2025
3.6 BASELINES
We compare our work to MotionCtri (Wang et al., (203e)) and the concurrent work CameraCtrl (He) al., 2024a) by adapting their publicly released code the same pre-trained video model as ours. Note that both baselines were originally designed for space-time disentangled U-Net video diffusion models. Hence, their approaches are not directly applicable to the spatio-temporal transformers, and so we adapt them to this setting as follows. For MotionCtrl, we omit their object motion control module and use their proposed camera motion control module to encode the camera parameters into a context vector used with the SnapVideo model. We fine-tune both the camera motion control module and the cross attention between the latent vectors and the patches. For Camera we fine-tune the original camera encoder module and use this produce the latent vectors in the SnapVideo During fine-tuning the model weights are kept frozen-i.e., the same as in our proposed approach. Furthermore, we include comparisons to the original MotionCtrl and CameraCtrl works built on the U-Net-based AnimateDiff (Guo et al., (024) architecture. For fair comparison, we evaluate all metrics for MotionCtrl (U-Net) and CameraCtrl (U-Net) on the same test sets as our method using the publicly provided code and checkpoints. Moreover, we provide metrics for the base model used across all our experiments, i.e., a model without camera injection. We provide more baseline variants of MotionCtrl and CameraCtrl in the appendix in Sec. B.3
Video model.
EXPERIMENTS
4.1 ASSESSMENT
We provide a qualitative and quantitative assessment of our approach compared to the baselines in Fig. aand in Tab. [ Following CameraCtrl (He et al.] 2024a), we also evaluate the camera pose accuracy using ParticleSfM (Zhao et al. 2022) on generated videos in Tab. 2 We use generations for text prompts from RealEstate 10K (Zhou et al. 2018) and MSR- VTT Xu et al. 2016), testing both in- and -iitt--distritttion prompts. Note that we adjust the CameraCtrl (He et al. 2024a) evaluation pipeline by normalizing all cameras into unified scale as COLMAP provides different scales across different scenes. This prevents scenes with large scale to have a higher impact on the errors. Please also refer the supplementary webpage for additional video results.
In Fig. 4 we observe that adapting the camera conditioning method from the MotionCtrl degrades visual quality and text alignment, likely because this approach adjusts the weights of the base video model. In the space-time U-Net for which this approach was proposed, the temporal layers can be fine-tuned without sacrificing visual fidelity. Since spatio-temporal transformers do not decompose temporal and spatial attributes in the same way, the model overfits to the small dataset used to fine-tune the cross- attention layer. While we observe some agreement with the camera poses used to condition the model, the text alignment is generally low in our experiments (see supplemental webpage). In contrast, CameraCtrl keeps the pre-trained video model weights frozen and only trains camera encoder. This leads to strong visual quality, but the generated videos show little agreement with the input camera poses, For fair comparison, we trained all models for the same number of iterations (described in Sec. 3.3
The results of the user study in Tab. [show that most participants prefer the generated videos using the proposed camera conditioning mechanism across all evaluated sub-metrics. We also observe a pronounced preference for the camera alignment of the proposed method compared to the other baselines. That is, 82% and 78% of participants prefer the camera alignment of the proposed method compared to our respective adaptations of MotionCtrl and CameraCtrl to the video transformer model. All results are significant at the p < 0.001 level as evaluated using a x² test. We further present image-based metrics for multi-view generation and video generation quality metrics in the appendix in Sec. and and Tab. B.2 respectively. The results of our camera pose accuracy evaluation in Tab. further demonstrate that our approach clearly outperforms previous works. Note that the previous works MotionCtrl and CameraCtrl especially struggle with the rotation accuracy.
4.2 ABLATIONS
Plücker embedding. We motivate our Plücker embedding conditioning mechanism by training a variant using the raw camera matrices. Concretely, flatten and concatenate extrinsics and intrinsies
8

---

Published as a conference paper at ICLR 2025
Table Quantitative results. We compare our method MotionCtrl and CameraCtri implemented on the same base video models as ours. The methods are evaluated in user study in which participants indicate their preference based on camera aligntment (CA), motion quality (MQ), text alignment (TA), visual quality (VQ), and overall preference (Overall). The percentages indicate preference for VD3D vs. the alternative method (in each row). All results are statistically significant with p < 0.001 as evaluated using x2 test.
Human Preference
Method
CA
MQ
TA
VQ
Overall
VD3D vs. MotionCtrl 82%
%
86%
81 %
84%
VD3D vs. CameraCtrl
78%
64%
63%
65%
66%
Table 2: Camera pose evaluation. We evaluate all models using reference camera trajectories from the RealEstate 10K test We compute translation and rotation errors based estimated camera poses from generations using ParticleSfM (Zhao et al. 2022)
RealEstate 10K
MSR-VTT
Method
TransError ())
RotError (4)
TransError (4)
RotError (L)
Base Model
0.616
0.207
0.717
0.216
MotionCtrl (U-Net)
0.477
0.094
0.593
0.137
CameraCtrl (U-Net)
0.465
0.089
0.587
0.132
MotionCtrl
0.518
0.161
0.627
0.148
CameraCtrl
0.532
0.165
0.578
0.220
Ours
0.409
0.043
0.504
0.050
w/o Plucker
0.517
0.161
0.676
0.156
w/o ControlNet
0.573
0.182
0.787
0.179
w/o weight copy
0.424
0.044
0.513
0,063
w/o add context
0.602
0.212
0.702
0.128
w/o Plucker context
0.487
0.088
0.627
0.091
matrices in the channel dimension and repeat the values in the spatial patch dimensions. We observe that Plücker embeddings provide an essential spatial conditioning mechanism, as shown in Tab. 2
ControlNet conditioning, Our ControlNettinspiree conditioning mechanism ensures fast and precise learning of the conditioning signal distribution. Instead using a ControlNet block, we simply add zero-initialized Plücker embedding features the patches and observe close no camera control. We observe training cross-attention layers in the Control block is key to learning camera control while preserving the original model weights. This confirmed by camera evaluation in Tab.
ControlNet weight copy. While the ControlNet block is essential, copying the pre-trained weights into the new copy has rather minor impact, shown in Tab. 2 To verify this, train model where we randomly initialize the cross-attentin block between patches and latents instead of copying the weights. We observe similar results, showing that rather the architecture and zero-iiitialization are the key component of this design.
Add context vector. We train a model where we use camera matrices. linearly encode them, and add them to the context vector as simple conditioning mechanism for transformer-basee models. The rendered videos show little to no correlation with the input camera matrices, resulting poor camera pose accuracy, shown in Tab. 2 This highlights the importance of carefully incorporating cameras into the spatio-temporal patches with our Plücker embedding block.
Plücker in context vector. We train model where we integrate Plücker embeddings into the context vector instead the patches. We similarly compute Plücker embeddings, but instead of patchifying them, we flatten and linearly map the embeddings into vector that has the same shape the context vector. We fuse the Plücker features and context vector using the same Plücker conditioning mechanism used for the patches. We observe higher camera pose errors in Tab. highlighting that

---

Published as a conference paper at ICLR 2025
Camera
"A bedroom with a bed, lamps and a window"
"A house sitting the middle of grassy field"
Figure 5: Conditional multi-view generation real image. We can generate arbitrary camera trajectories from given real image for multi-view synthesis, paving the way to single-image scene reconstruction using camera-controlled video models.
crucial incorporate the spatio-temporal Plücker embeddings into the spatio-temporal patches instead of using the context vector.
4.3 APPLICATIONS
Image-to-video generation. We synthesize camera-controlled videos based on different camera poses as shown Fig. [ For this task we use a version of our pre-trained text-to-video model that we fine-tune on video sequences where a random subset of the input frames are masked. At inference time, we can provide image guidance for any the generated frames, providing an additional dimension of controllability when paired with proposed method for camera control.
To demonstrate image-to-vides generation in Fig. | we condition the model using camera poses along with image guidance from the first frame of generated video sequence. Note that method provides no control over motion within the scene itself; hence, scene motion can differ depending on the random seed or the provided camera poses.
Image-to-multiview generation. We also explore multi-view generation for static scenes as shown in Fig. 5 Given a real input image complex scene unseen during training, our camera-conditioned model generates view-consistent renderings of that scene from arbitrary viewpoints. These multi-view renderings could be directly relevant downstream 3D reconstructions pipelines, e.g., based on NeRF (Mildenhall et al.) (020)) or 3D Gaussian Splatting (Kerbl et al.) (003).. We show the potential of camera-conditione image-to-me mouultivieew generation for complex 3D scene generation, but we leave more extensive exploration of this topic for future work.
5 CONCLUSION
Large-scale video transformer models show immense promise to solve many long-standing chal- lenges in computer vision, including novel-view synthesis, single-image 3D reconstruction, and text-conditioned scene synthesis. Our work brings additional controllability these models, enabling a user to specify the camera poses from which video frames are rendered.
Limitations and future work. There several limitations our work, which highlight important future research directions. For example, while rendering static scenes from different camera view- points produces results that appear consistent, dynamic scenes rendered from different camera viewpoints can have inconsistent motion (see supplemental videos). We envision that future video generation models will have fine-grained control over both scene motion and camera motion to address this issue. Further, our approach applies camera conditioning only to the low-resolution Snap model and we keep their upsampler model frozen (i.c., without camera conditioning)-it may be possible further improve camera control through joint training, though this brings additional architectural engineering and computational challenges. Finally, our approach currently limited to generation of relatively short videos (16 frames), based the design and training scheme of the Snap Video model. Future work to address these limitations will enable new capabilities for applications in computer vision, visual effects, augmented and virtual reality, and beyond.
10

---

Published as a conference paper at ICLR 2025
6 ETHICS STATEMENT
Broader Impact. Recent vidco generation models demonstrate coherent, photorcalistic synthesis of complex scenes--caaailittes that are highly sought after for numerous applications across computer vision, graphics, and beyond. Our key technical contributions relate to camera control of these models, which be applied to a wide range of methods. As with all generative models and technologies, underlying technologies can be misused by bad actors in unintended ways. While these methods continue to improve, researchers and developers should continue consider safeguards, such as output filtering, watermarking, access control, and others.
Data. To develop the camera control methods proposed in this paper, we used RealEstate 100K (Zhou et al. 2018). RealEstate 1OK is released and open-sourced by Google LLC under a Creative Commons Attribution 4.0 International License, and sourced from content using a CC-BY license. The dataset can be found under the following URL: https:://///. /ttt:::://oog// google qi t huub wttp///eeeesttteeeeeoe The RealEstate dataset was published as part of research paper by Zhou et al. (2018): "Stereo Magnification: Learning View Synthesis using Multiplane Images" at SIGGRAPH 2018. The RcalEstate dataset is a commonly used dataset for 3D reconstruction (Charatan et al. 2024).
3D generation (Gao et al. (024c)), and camera-controlled video generation (Wang et al. 2023e).
Similarly, closely related baselines such as MotionCtrl (Wang et al. (023e) and Camera Ctrl (He al. 2024a) the identical dataset train and evaluate their camera-contre controled video diffusion models. Hence, we follow an established training and evaluation pipeline without any modifications the data.
7 REPRODUCIBILITY STATEMENT
We have structured our paper ensure comprehensive reproducibility of our camera control method. Section[]pnee provides a detailed description of our method, including theoretical foundations and core algorithmic components. Section 4 presents thorough experimental details, covering evaluation protocols, metrics, and comparisons with baselines. All implementation specifics, including train- ing procedures, architectural choices, and hyperparameters, are documented Appendix C. To further facilitate reproducibility, we include the source code of our camera-controlled FIT block as supplementary material. The provided implementation contains the core components necessary to replicate approach, accompanied by documentation and usage examples. We welcome requests for additional technical details or clarifications to support the research community in reproducing and building upon work.
ACKNOWLEDGEMENTS
DBL acknowledges support from NSERC under the RGPIN program, the Canada Foundation for Innovation, and the Ontario Research Fund.
REFERENCES
Sherwin Bahmani, Jeong Joon Park, Despoina Paschalidou, Hao Tang, Gordon Wetzstein, Leonidas Guibas, Lue Van Gool, and Radu Timofte. 3d-aware video generation. In TMLR, 2023a.
Sherwin Bahmani, Jeong Joon Park, Despoina Paschalidou, Xingguang Yan, Gordon Wetzstein, Leonidas Guibas, and Andrea Tagliasacchi. CC3D: Layout-conditioned generation of composi- tional 3D scenes. In Proc. ICCV, 2023b.
Sherwin Bahmani, Xian Liu, Wang Yifan, Ivan Skorokhodov, Victor Rong, Ziwei Liu, Xihui Liu, Jeong Joon Park, Sergey Tulyakov, Gordon Wetzstein, Andrea Tagliasacchi, and David B. Lindell. Tc4d: rrajectorycconditioned text-to-4d generation. In Proc. ECCV, 2024a.
Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B. Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling. In Proc. CVPR, 2024b.
11

---

Published as a conference paper at ICLR 2025
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end- and-to-end retrieval. In Proc. ICCV, 2021.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprini urXiv:2311.15122 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proc. CVPR, 2023b.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. OpenAl technical reports, 2024. URL https: /tpe:////ppa..... com/ asr-ssssssss-sdodoo wor www.adssimmmtt..r...
Researchh/viee--ae oeee:rcdd.....o......o.ttaooarrr
Zenghao Chai, Chen Tang. Yongkang Wong, and Mohan Kankanhalli. Star: Skeleton-awar text- based 4d avatar generation with in-network motion retargeting. arXiv preprint arXiv:2406. 04629, 2024.
Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, al. Efficient geometry-aware 3D generative adversarial networks. Proc. CVPR, 2022.
Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3d-aware diffusion models. In Proc. ICCV, 2023.
David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proc. CVPR, 2024.
Eric Ming Chen, Sidhanth Holalkere, Ruyu Yan, Kai Zhang, and Abe Davis. Ray conditioning: Trading photo-consistency for photo-realism in multi-view image generation. In Proc. ICCV, 2023a.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang. Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, and Ying Shan. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.1955 2023b.
Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. Pixart-( delta]: Fast and controllable image generation with latent consistency models, arXiv preprint arXiv:2401.05252. 2024.
Kevin Chen, Christopher B Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese. Text2Shape: Generating shapes from natural language by learning joint embeddings. Proc. ACCV, 2018.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3D: Disentangling geometry and appearance for high-quality text-to-3D content creation. arXiv preprint 2rXiv:2303.1387. 2023c.
Ting Chen and Lala Li. Fit: Far-reaching interleaved transformers. arXiv preprint arXiv:2305. /2689, 2023.
Wen- Chu, Lei Ke, and Katerina Fragkiadaki. Dreamscene4d: Dynamic multi-object scene generation from monocular videos. arXiv preprint arXiv:2405.02280 2024.
Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W Taylor, and Joshua M Susskind. Unconstrained scene generation with locally conditioned radiance fields. Proc. ICCV, 2021.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. Proc. ICLR, 2021.
12

---

Published as a conference paper at ICLR 2025
Qijun Feng, Zhen Xing, Zuxuan Wu, and Yu-Gang Jiang. FDGaussian: Fast Gaussian splatting from single image via geometric-awane diffusion model. arXiv preprint arXiv:2403.10242, 2024a.
Yutao Feng, Yintong Shang. Xiang Feng, Lei Lan, Shandian Zhe, Tianjia Shao, Hongzhi Wu, Kun Zhou, Hao Su, Chenfanfu Jiang, et al. Elastogen: 4d generative elastodynamics. arXiv preprint 200::2405.15056, 2024b.
Peng Gao, Le Zhuo, Ziyi Lin, Dongyang Liu, Ruoyi Du, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Transforming text into any modality, resolution, and duration via flow- baw-based large diffusion transformers. arXiv preprint arXiv:2455.05945 2024a.
Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma, Le Chen, Danhang Tang, and UIrich Neumann. Gaussianflow: Splatting Gaussian dynamics for 4D content creation. arXiv preprint arXiv:2403. 12365, 2024b.
Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualle Pratul Srinivasan, Jonathan T Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view diffusion models. arXiv preprint arXivv2405. 10314, 2024c.
William Gao, Noam Aigerman, Thibault Groueix, Vova Kim, and Rana Hanocka. TextDeformer: Geometry manipulation using text guidance. Proc. SIGGRAPH, 2023.
Jiatao Gu, Alex Trevithick, Kai-En Lin, Joshua M Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi. NerIDiff: Single-image view synthesis with NeRF-guided distillation from 3D-aware diffusion. Proc. ICML, 2023.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. Proc. ICLR, 2024.
Junlin Han, Filippos Kokkinos, and Philip Torr. VFusion3D: Learning scalable 3D generative models from video diffusion models. arXiv preprint arXiv:2403. 12034, 2024.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cam- eractrl: Enabling camera control for tett-to-video generation. arXiv preprint 144002240..02101, 2024a.
Xianglong He, Junyi Chen, Sida Peng, Di Huang, Yangguang Li, Xiaoshui Huang. Chun Yuan, Wanli Ouyang, and Tong He. GYGEN: Text-to-3D generation with volumetric representation. arXiv preprint arXiv:2403.1 12957, 2024b.
Yingqing He, Tianyu Yang. Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.1322. 2022.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv: 1oXiv:6006.8800 08415, 2016.
arXiv preprint
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Berhaard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Proc. NewrIPS, 2017.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint
2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Proc. NeurIPS, 2022b.
Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. In Proc. ICCV, 2023.
Lukas Höllein, Aljaž Božič, Norman Müller, David Novotny, Hung- Yu Tseng, Christian Richardt, Michael Zollhöfer, and Matthias Nießner. ViewDiff: 3D-consistent image generation with text-to- image models. Proc. CVPR, 2024.
13

---

Published as a conference paper at ICLR 2025
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: Large reconstruction model for single image to 3D. Proc. ICLR, 2024.
Chen Hou, Guoqiang Wei, Yan Zeng, and Zhibo Chen. Training-fre free camera control for video generation. arXiv preprint arXiv:2406. 10126, 2024.
Teng Hu, Jiangning Zhang, Ran Yi, Yating Wang, Hongrui Huang, Jieyu Weng, Yabiao Wang, and Lizhuang Ma. Motionmaster: Training-free camera motion transfer for video generation. arXiv preprint Arxi::2404. 10 15789, 2024.
Tranyu Huang, Yihan Zeng, Hui Li, Wangmeng Zuo, and Rynson WH Lau. Dreamphysics: Learn- ing physical properties of dynamic 3d gaussians with video diffusion priors. arXiv preprini arXiv:2406.014 2024.
Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. arXiv preprini arXiv:2212.11972 2022.
Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text- ext--guided object generation with dream fields. Proc. CVPR, 2022.
Nikolay Jetchev. ClipMatrix: Text-controlled creation of 3D textured meshes. arXiv preprint arXiv:2/09. /2922, 2021.
Lutao Jiang and Lin Wang. Brightdreamer: Generic 3D Gaussian generative framework for fast text-to-3D synthesis. arXiv preprint 1rXiv:24033112733 2024.
Yanqin Jiang, Li Zhang, Jin Gao, Weimin Hu, and Yao Yao. Consistent4d: Consistent 360 {'deg] dynamic object generation from monocular video. Proc. ICLR, 2024.
Yash Kant, Aliaksandr Siarohin, Ziyi Wu, Michael Vasilkovsky, Guocheng Qian, Jian Ren, Riza Alp Guler, Bernard Ghanem, Sergey Tulyakov, and Igor Gilitschenski. Spad: Spatially aware multi- view diffusers. In Proc. CVPR, 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Proc. NeurlPS, 2022.
Oren Katzir, Patashnik, Daniel Cohen-Or, and Dani Lischinski. Noise-free score distillation. Proc. ICLR, 2024.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM TOG, 2023.
Seung Wook Kim, Bradley Brown, Kangxue Yin, Karsten Kreis, Katja Schwarz, Daiqing Li, Robin Rombach, Antonio Torralba, and Sanja Fidler. NeuralField-LDM: Scene generation with hierarchi- cal latent diffusion models. Proc. CVPR, 2023.
Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hongsheng Li, Leonidas Guibas, and Gordon Wetzstein. Collaborative video diffusion: Consistent mulli-vides generation with camera control. arXiv preprint 11t..i::2005114144 2024.
Nupur Kumari, Grace Su, Richard Zhang, Taesung Park, Eli Shechtman, and Jun-Yan Zhu, Cus- tomizing text-to-image diffusion with camera viewpoint control. arXiv preprint 2XXv:2444.11233,, 2024.
Kyungmin Lee, Kihyuk Sohn, and Jinwoo Shin. DreamFlow: High-quality text-to-3D generation by approximating probability flow. Proc. ICLR, 2024a.
Yao-Chih Lee, Yi-Ting Chen, Andrew Wang, Ting Hsuan Liao, Brandon Y Feng, and Jia-Bin Huang. Vividdream: Generating 3d scene with ambient dynamics. arXiv preprint 24X1:24455.2334 2024b.
Bing Li, Cheng Zheng, Wenxuan Zhu, Jinjie Mai. Biao Zhang, Peter Wonka, and Bernard Ghanem. Vivid-zoo: Multi-view video generation with diffusion model. arXiv preprint arXiv:2406.0s8 08659, 2024a.
14

---

Published as a conference paper at ICLR 2025
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3D: Fast text-to-3D with sparse-view genera- tion and large reconstruction model. Proc. ICLR, 2024b.
Renjie Li. Panwang Pan, Bangbang Yang. Dejia Xu, Shijie Zhou, Xuanyang Zhang, Zeming Li, Achuta Kadambi, Zhangyang Wang, and Zhiwen Fan. 4k4dgen: Panoramic 4d generation at 4k resolution. arXiv preprint arXiv:2406. 13527, 2024c.
Zhiqi Li, Yiming Chen, Lingzhe Zhao, and Peidong Liu. Controllable text-to-3D generation via surface-alignee Gaussian splatting, arXiv preprint (rXiv:2403009981,1 2024d.
Hanwen Liang, Yuyang Yin, Dejia Xu, Hanxue Liang, Zhangyang Wang, Konstantinos N Plataniotis, Yao Zhao, and Yunchao Wei. Diffusion4d: Fast spatial-temport consistent 4d generation via video diffusion models. arXiv preprint arXiv:2405. 16645, 2024.
Yixun Liang Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Lucid- dreamer: Towards high-fidelity text-to-3D generation via interval score matching. arXiv preprint arXiv:2311.11284 2023.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Y Yi Lin. Magic3D: High-resolution text-to-3D content creation. Proc. CVPR, 2023a.
Yukang Lin, Haonan Han, Chaoqun Gong, Zunnan Xu, Yachao Zhang, and Xiu Li. Consistent123: One image to highly consistent 3D asset using case-aware diffusion priors. arXiv preprint 1.Xiv:23099172611 2023b.
Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis. Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. In Proc. CVPR, 2024a.
Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. Motionclone: Training-free motion cloning for controllable video generation. arXiv preprint arXiv:2406.05338 2024b.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Proc. NeurlPS, 2024a.
Pengkun Liu, Yikai Wang, Fuchun Sun, Jiafang Li, Hang Xiao, Hongxiang Xue, and Xinzhou Wang. Isotropic3D: Image-to-3D generation based on single clip embedding. arXiv preprint arXiv:2403.10395. 2024b.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3; Zero-shot one image to 3D object. Proc. ICCV, 2023.
Xian Liu, Xiaohang Zhan, Jiaxiang Tang, Ying Shan, Gang Zeng, Dahua Lin, Xihui Liu, and Ziwei Liu. HumanGaussian: Text-driven 3D human generation with Gaussian splatting. Proc. CVPR, 2024c.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. SyneDreamer: Generating multiview-consisse images from single-view image. Proc. ICLR, 2024d.
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma. Song- Haig-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3D: Single image to 3D using cross-domain diffusion. Proc. CVPR, 2024.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint XXX::4001.0000 2024a.
15

---

Published as a conference paper at ICLR 2025
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024b.
Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, al. Snap video: Scaled spatiotemporal transformers for text-to-vides synthesis. Proc. CVPR, 2024.
Qiaowei Miao, Yawei Luo, and Yi Yang. Pla4d: Pixel-level alignments for text-to-4d gaussian splatting, arXiv preprint arXiv:2405. 2ttV:2205.19977, 2024.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proc. ECCV, 2020.
Norman Müller, Katja Schwarz, Barbara Rössle, Lorenzo Porzi, Samuel Rota Bulò, Matthias Nießner, and Peter Kontschieder. Multidiff: Consistent novel view synthesis from single image. In Proc. CVPR, 2024.
Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher- Shlizerman. StyleSDF: High-resolution 3D-consistent image and geometry generation. Proc. CVPR, 2022.
Zijie Pan, Zeyu Yang. Xiatian Zhu, and Li Zhang. Fast dynamic 3d object generation from single-view video. arXiv preprint arXiv:2401.08742 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. Proc. ICCV, 2023.
Ryan Po, Wang Yifan, Vladislav Golyaník, Kfir Aberman, Jonathan T Barron, Amit H Bermano, Eric Ryan Chan, Tali Dekel, Aleksander Holynski, Angjoo Kanazawa, et al. State of the diffusion models for visual computing. arviv preprint arXiv:2310.07224, 2023.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching- Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720, 2024.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D diffusion. In Proc. ICLR, 2023.
Guocheng Qian, Junli Cao, Aliaksandr Siarohin, Yash Kant, Chaoyang Wang, Michael Vasilkovsky, Hsin-Ying Ying Lee, Yuwei Fang. Ivan Skorokhodov, Peiye Zhuang, al. Atom: Amortized text-to- mesh using 2d diffusion. arXiv preprint wrXiv:2402.00867, 2024a.
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Yin Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et Magic One image to high-quality 3d object generation using both 2d and 3d diffusion priors. Proc. ICLR, 2024b.
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Yor Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al. Magic123; One image high-quality 3D object generation using both 2D and 3D diffusion priors. Proc. ICLR, 2024c.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. Proc. ICML, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Proc. JMLR, 2020.
Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. DreamGaus- sian4D: Generative 4D Gaussian splatting. arXiv preprint arXiv:23/2.17142 2023.
Jiawei Ren, Kevin Xie, Ashkan Mirzaei, Hanxue Liang, Xiaohui Zeng, Karsten Kreis, Ziwei Liu, Antonio Torralba, Sanja Fidler, Seung Wook Kim, et al. L4gm: Large 4d gaussian reconstruction model, arXiv preprint 4406 100244 2024.
16

---

Published as a conference paper at ICLR 2025
Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin- Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan. Corg-Forge: Towards zero-shot text-to-shape generation. Proc. CVPR, 2022.
Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger. VoxGRAF: Fast 3D-aware image synthesis with sparse voxel grids. Proc. NeurIPS, 2022.
Abhishek Sharma, Adams Yu, Ali Razavi, Andeep Toor, Andrew Pierson, Ankush Gupta, Austin Waters, Daniel Tanis, Dumitru Erhan, Eric Lau, Eleni Shaw, Gabe Barth-Maron, Greg Shaw, Han Zhang, Henna Nandwani, Hernan Moraldo, Hyunjik Kim, Irina Blok, Jakob Bauer, Jeff Donahue, Junyoung Chung, Kory Mathewson, Kurtis David, Lasse Espeholt, Marc van Zee, Matt McGill, Medhini Narasimhan, Miaosen Wang, Mikotaj Binkowski, Mohammad Babaeizadeh, Mohammad Taghi Saffar, Nick Pezzotti, Picter-Jan Kindermans, Poorva Rane, Rachel Hornung, Robert Riachi, Ruben Villegas, Rui Qian, Sander Dieleman, Serena Zhang, Serkan Cabi, Shixin Luo, Shlomi Fruchter, Signe Norly, Srivatsan Srinivasan, Tobias Pfaff, Tom Hume, Vikas Verma, Weizhe Hua, William Zhu, Xinchen Yan, Xinyu Wang, Yelin Kim, Yuging Du, and Yutian Chen. Veo, 2024. URL bttps:// http///ppw.....ooleee////oooooee /veo/
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3D generation. Proc. ICLR, 2024.
Jaidev Shriram, Alex Trevithick, Lingjie Liu, and Ravi Ramamoorthi. Realmdreamer: Text-driven 3d scene generation with inpainting and depth diffusion. arXiv preprint arXiv:2404.07199, 2024.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. Proc. ICLR, 2023a.
Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Iurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Tex dynamic scene generation. In Proc. ICML, 2023b.
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Proc. NeurIPS, 2021.
Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. DreamCraft3D: Hierarchical 3D generation with bootstrapped diffusion prior. Proc. ICLR, 2024a.
Qi Sun, Zhiyang Guo, Ziyu Wan, Jing Nathan Yan, Shengming Yin, Wengang Zhou, Jing Liao, and Houqiang Li. Eg4d: Explicit generation of 4d object without score distillation. arXiv preprint wwX11:2405.18132. 2024b.
Stanislaw Szymanowicz. Christian Rupprecht, and Andrea Vedaldi. Viewset diffusion:(O-) image- conditioned 3d generative models from 2d data. In Proc. ICCV, 2023.
Stanislaw Szymanowicz. Eldar Insafutdinov, Chuanxia Zheng, Dylan Campbell, João F Henriques, Christian Rupprecht, and Andrea Vedaldi. Flash3d: Feed-forward generalisable 3d scene recon- struction from single image. arXiv preprini MxXiv:2406.04343 2024a.
Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3D reconstruction. Proc. CVPR, 2024b.
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. LGM: Large multi-view gaussian model for high-resolution 3d content creation. Proc. ECCV, 2024.
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3D: High-fidelity 3D creation from a single image with diffusion prior. arXiv preprint arXiv:2303. 14184, 2023.
Ayush Tewari, Tianwei Yin, George Cazenavette, Semon Rezchikov, Josh Tenenbaum, Frédo Durand, Bill Freeman, and Vincent Sitzmann. Diffusion with forward models: Solving stochastic inverse problems without direct supervision. Proc. NearIPS, 2023.
17

---

Published as a conference paper at ICLR 2025
Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3D object reconstruction from single image. arXiv preprint 24X1:2403.0215// 2024.
Hung- Yu Tseng, Qinbo Li, Changil Kim, Suhib Alsisan, Jia- Bin Huang, and Johannes Kopf. Consis- tent view synthesis with pose-guided diffusion models. In Proc. CVPR, 2023.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A metric & challenges. arXiv preprint arXiv:/8/2.01777, 2018
Lukas Uzolas, Elmar Eisemann, and Petr Kellnhofer. Motiondreamer: Zero-shot 3d mesh animation from video diffusion models. arXiv preprint arXiv:2405.20155 2024.
Basile Van Hoorick, Rundi Wu, Ege Ozguroglu, Kyle Sargent, Ruoshi Liu, Pavel Tokmakov, Achal Dave, Changxi Zheng, and Carl Vondrick. Generative camera dolly: Extreme monocular dynamic novel view synthesis. arXiv preprint arXiv:2405. 14868, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention all you need. Proc. NeurIPS, 2017.
Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-viww synthesis and 3d generation from single image using latent video diffusion. arXiv preprint arXiv:2403. 12008, 2024a.
Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Chris- tian Laforte, Robin Rombach, and Varun Jampani. SV3D: Novel multi-view synthesis and 3D generation from single image using latent video diffusion. arXiv preprint arXiv:2403.12008, 2024b.
Ziyu Wan, Despoina Paschalidou, Ian Huang, Hongyu Liu, Bokui Shen, Xiaoyu Xiang, Jing Liao, and Leonidas Guibas. CAD: Photorealistic 3D generation via adversarial distillation. Proc. CVPR, 2024.
Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-NeRF: Text-and-image driven manipulation of neural radiance fields. Proc. CVPR, 2022.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score Jacobian chaining: Lifting pretrained 2d diffusion models for 3D generation. Proc. CVPR, 2023a.
Wenjing Wang. Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation. arXiv preprint arXiv:2305. 2023b.
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. arXiv preprint 20XXX:23006.0008.. 2023c.
Yikai Wang, Xinzhou Wang, Zilong Chen, Zhengyi Wang, Fuchun Sun, and Jun Zhu. Vidu4d: Single generated video to high-fidelity 4d reconstruction with dynamic gaussian surfels. arXiv preprint arXiv:2405.16822 2024.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- Dreamer: High-fidelity and diverse text-to-3D generation with variational score distillation. Proc. NeurIPS, 2023d.
Zhouxia Wang. Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia. Ping Luo, and Yin Shan. Motionetrl: unified and flexible motion controller for video generation. In arXiv preprint 1oXiv:2312.036441 2023e.
Daniel Watson, William Chan, Ricardo Martin Bruuala, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. Proc. ICLR, 2023.
18

---

Published as a conference paper at ICLR 2025
Daniel Watson, Saurabh Saxena, Lala Li, Andrea Tagliasacchi, and David J Fleet. Controlling space and time with diffusion models. arXiv preprint arXiv:2407.00 000600, 2024.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104. 14806, 2021.
Ruiqi Wu, Liangyu Chen, Tong Yang, Chunle Guo, Chongyi Li, and Xiangyu Zhang. Lamp: Learn motion pattern for few-shot-base baset video generation. arXiv preprint arXiv:2310. 2023.
Zijie Wu, Chaohui Yu, Yanqin Jiang, Chenjie Cao, Fan Wang, and Xiang Bai. Sc4d: Sparse-controlled video-to-4d generation and motion transfer. arXiv preprint vrXiv22404003736, 2024.
Zeqi Xiao, Yifan Zhou, Shuai Yang, and Xingang Pan. Video diffusion models are training-free motion interpreter and controller. arXiv preprint arXiv:2405. 14864, 2024.
Kevin Xie, Jonathan Lorraine, Tianshi Cao, Jun Gao, James Lucas, Antonio Torralba, Sanja Fidler, and Xiaohui Zeng. LATTE3D: Large-scale amortized text-to-enhanced3D synthesis. Proc. ECCV, 2024.
Dejia Xu, Hanwen Liang, Neel P Bhatt, Hezhen Hu, Hanxue Liang, Konstantinos N Plataniotis, and Zhangyang Wang. Comp4d: Llm- Aln-giided compositional 4d scene generation. arXiv preprint arXiv:2403.1699. 16993, 2024a.
Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Camco:
Camera-controllable 3d-consistent image-to-vin generation. arXiv preprint arXiv:2406.022500 2024b.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proc. CVPR, 2016.
Yinghao Xu, Zifan Shi. Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. GRM: Large Gaussian reconstruction model for efficient 3D reconstruction and generation. Proc. ECCV, 2024c.
Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. DMV3D: Denoising multi-viww diffusion using 3D large reconstruction model. Proc. ICLR, 2024d.
Zhongeong Xu, Jianfeng Zhang, Jun Hao Liew, Wenqing Zhang, Song Bai, Jiashi Feng, and Mike Zheng Shou. Pv3d: A 3d generative model for portrait video generation. Proc. ICLR, 2023.
Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high- reghuresolution video- lideo-language representation with large-scale video transcriptions. Proc. CVPR, 2022.
Qitong Yang, Mingtao Feng, Zijie Wu, Shijie Sun, Weisheng Dong, Yaonan Wang, and Ajmal Mian. Beyond skeletons: Integrative latent mapping for coherent 4d sequence generation. arXiv preprint arXiv:2403.13238 2024a.
Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. arXiv preprint arXiv:2402.03 162, 2024b.
Zeyu Yang, Zijie Pan, Chun Gu, and Li Zhang. Diffuusion: Dynamic 3d content generation via score composition of orthogonal diffusion models. arXiv preprint 2404.02143 2024c
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06077 2024d.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint 2wXiv:2308.0672/ 2023.
19

---

Published as a conference paper at ICLR 2025
Junliang Ye, Fangfu Liu, Qixiu Li, Zhengyi Wang, Yikai Wang. Xinzhou Wang, Yueqi Duan, and Jun Zhu. DreamReward: Text-to-3D generation with human preference. arXiv preprint arXiv:2403.146133 2024.
Yuyang Yin, Dejia Xu, Zhangyang Wang. Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content generation with spatial-temporal consistency. arXiv preprint arXiv:2312.17225, 2023,
Paul Yoo, Jiaxian Guo, Yutaka Matsuo, and Shixiang Shane Gu. DreamSparse: Escaping from Plato's cave with 2D diffusion model given sparse views. arXiv preprint arXivv2306.033414: 2023.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv: /90.49994..0.... 2019.
Heng Yu, Chaoyang Wang, Peiye Zhuang, Willi Menapace, Aliaksandr Siarohin, Junli Cao, Laszlo A
Jeni, Sergey Tulyakov, and Hsin- Ying Lee. 4real: Towards photorealistic 4d scene generation via video diffusion models. arXiv preprint arXiv:2406.07444 077477 2024.
Jason Yu, Fereshteh Forghani, Konstantinos Derpanis, and Mareus Brubaker. Long-term photometric consistent novel view synthesis with diffusion models. In Proc. ICCV, 2023a.
Xin Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Song- Hai Zhang, and Xiaojuan Qi. Text-to-3D with classifier score distillation, arXiv preprint arXiv:2310.1940 19415, 2023b.
Raza Yunus, Jan Eric Lenssen, Michael Niemeyer, Yiyi Liao, Christian Rupprecht, Christian Theobalt, Gerard Pons-Moll, Jia-Bin Huang, Vladislav Golyanik, and Eddy lIg. Recent trends in 3d reconstruction of general non-rigid scenes. In Computer Graphics Forum, 2024.
Yifei Zeng, Yangin Jiang, Siyu Zhu, Yuanxun Lu, Youtian Lin, Hao Zhu, Weiming Hu, Xun Cao. and Yao Yao. Stag4d: Spatial-temporal anchored generative 4d gaussians. arXiv preprint arXiv:2403. 14939, 2024.
Bowen Zhang, Tianyu Yang, Yu Li, Lei Zhang, and Xi Zhao. Compress3D: a compressed latent space for 3D generation from a single image. arXiv preprint arXiv:2403. 13524, 2024a.
Haiyu Zhang, Xinyuan Chen, Yaohui Wang, Xihui Liu, Yunhong Wang, and Yu Qiao. 4diffusion: Multi-view video diffusion model for 4d generation. arXiv preprint 20X5::240..20674, 2024b.
Hao Zhang, Di Chang, Fang Li, Mohammad Soleymani, and Narendra Ahuja. Magicpose4d:: Crafting articulated models with appearance and motion control. arXiv preprint arXiv:2405. 14017, 2024c.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. Proc. ICCV, 2023.
Tianyuan Zhang, Hong-Xing Yu, Rundi Wu, Brandon Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William T' Freeman. Physdreamer: Physics-based interaction with 3d objects via video generation. arXiv preprint 2rXiv22404133026 2024d.
Wang Zhao, Shaohui Liu, Hengkai Guo, Wenping Wang, and Yong-Jin Jin Liu. Particlesfm: Exploiting dense point trajectories for localizing moving cameras in the wild. In Proc. ECCV, 2022.
Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee. Animate1244 Animating one image 4d dynamic scene. arXiv preprint arXiv:2311.1444 2023.
Yufeng Zheng. Xueting Li. Koki Nagano, Sifei Liu, Otmar Hilliges, and Shalini De Mello. A unified approach for text-and image-guided 4d scene generation. Proc. CVPR, 2024a.
Zangwei Zheng, Xiangyu Peng Tianji Yang. Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024b. URL httptttss///tthhatet htt..//hww....o//aaaaecccc//c//e
Sora
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11011 2022.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ACM TOG, 2018
20

---

Published as a conference paper at ICLR 2025
A RELATED WORK
Due to space constraints, we summarize related 3D and an extended list of 4D works in the appendix.
3D generation. Limited to single category trainings, early work on 3D generation extends GANs
into 3D using a neural renderer as an inductive bias (DeVsies et al. 2021: Chan et al.) 2022: Or
El et al. generation, CLIP-based supervision [Radford et al.] 2021) enabled based generation and editing
2022. Schwwrz al. 2022 Bahhani al. 2023b), Towards more diverse and flexible
3D assets (Chen al. 2018 Jain et al. [022] Sanghi et al. 2022 [Jecche| 2021 Gao et al.) 2023
Wang et al.
2022). With recent advances in diffusion models, Score Distillation Sampling (SDS)
replaces CLIP supervision with diffusion model supervision (Poole et 2023) Wang et al. 2023d
Lin et al.
2023a Chen et 2023cc [iang et al. [223]] WWnn et al. [023a Li et al. 2024d He et al.
2024b: Ye et al.]
2024] et | [222cc] Yu et al.] 2023b; Katzir al. [0244] Lee et al. 2024aa] |unn
et al. 2024a) for higher quality generation. In order to improve the 3D structure of scenes, another
2023 [Shi et al. line work generates multiple views of a scene (Lin et al.) 2023b Liu al.
2024]
Feng ct al. 2024a]] Liu et al. 2024b] Kim et al. 2023; Voleti et al., [024b
Höllein et al. 2024).
Alternatively, other methods use iterative inpainting for 3D scene generation (Höllein et al. 2023
Shriram al. 2024). Recent methods lift input images into 3D (Chan et al. 2023 lang et al. 2023
Gu et al. [2023] Liu et al.] 2024d] Yoo et al.] 22233] Tewari et al. 2023] Qian et al.] 2024c
Long
al. 2024] Wan al. [022|] |Szymanowicz al. (023) using NeRF (Mildenhall al.) 2020). 3D
Gaussian Splatting (Kerbl et al. (2023), or Meshes in combination with diffusion models. Other
recent methods (Hong al. [2224]] Li al. 2024b; Xu al. 2024d c Zhang et al. 2024a Han et al.
2024; Jiang& & Wang 2024, Xie et al.) 20244] Tang et al. 20244) Tochilkin et al., 2024; Qian et al.
2024a: Szymanowicz et al. 22244 a) tackle fast foed-forward 3D generation, directly predicting a 3D
generation from input images or text. Different from our approach, these methods can only synthesize
static scenes.
4D generation. Previous methods also tackle the problem of 4D generation, i.e., generating videos of dynamic 3D scenes from controllable viewpoints, usually from an input text prompt or image. Since
the initial work on this topic using large-scale generative models (Singer et al.] 2023b), significant
improvements in the visual quality and motion quality of generated scenes have been achieved (Ren
ctt al.
2023 Ling et al. 2024at Bahmmani et al., 2024b] Zheng al. 2202aa Gao ct al.
2024b Yang
ctt al.
2024a Jiang et al., 2024, Bahmani et al., 2024a| Zhang et al. 2024d| Xu ct al.
2024a Miao
al.
2024| Li et al.) 2024a). While these methods generate scenes based on text conditioning other
approaches convert an input image or video to dynamic 3D scene (Ren et al. [023 /hho et al.)
[023
Yin et al. 2023 Pan et al. [2024] Zheng et al. 2024aa] Ling et al. 2024a] Gao et al.] 2024b) Zeng et al.
2024| Chu et al. 2024 Wu et al.
2024 Yang et al. [024c] Wang et al.
2024
Feng et al. 2024b Sun
al.
2024b] Zhang et al. 2024b; Yu al. 2024 Ren et al. |024:] Lee al.
2024b
Li al. 2024c]
Van Hoorick et al. 2024] Uzolas et al. [224|] Huang et al. 2024 Chai et al.
2024 Liang et al. 2024
Zhang et al. 2024c). Another line of work (Bahmani et al.) 2023a Xu et al.
2023) extends 3D GANs
into 4D by training on 2D videos, however the quality is limited and models are trained on single
category datasets. Still, all of these methods are focused on object-centric generation, typically based
on 3D volumetric representations. As such, they typically do not incorporate background elements, and overall, they do not approach the level of photorealism demonstrated by the state-of-the-art video generation models used in our technique.
B QUANTITATIVE EVALUATION
We further evaluate all models for the task of single provide results for established 2D video generation metries.
generation. Furthermore, we
B. MULTI-VIEW GENERATION
We evaluate our model for image- to- multiview generation. Due to ground truth correspondences for the RealEstate (Zhou et al.) 2018) dataset, we can condition model on given camera trajectory and assess image based metrics, i.e., PSNR, SSIM, and LPIPS. We provide results for the low-resolution base model and the upsampled high-resolution results in Tab.
21

---

Published as a conference paper at ICLR 2025
B.2 QUALITY METRICS
Moreover, we evaluate all models on established image and video generation metrics, namely, FID (Heusel et al. 2017). FVD (Unterthiner et al., 2018). and CLIPSIM (Wu al. 202T We provide results for RealEstate106 (Zhou et al. 2018) and MSS-VVTT- (Xu et al. 2016) in Tab. 4
is important to highlight that FID and FVD are not direct measurements of visual quality metrics measure similarities between dataset distributions. So while fine-tuning on dataset will degrade these scores relative to the original model, this degradation is expected because the model is fitting different data distribution than the one used in the original training run. The same degradation in FID after fine-tuning is observed in the original ControlNet paper (Zhang et al. 2023), where the authors report an increase in FID when comparing the StableDiffusion base model (6.09 FID) to their fine-tuned ControlNet model (15.27 FID; see their Table 3). We believe that joint training with 2D video data may help to alleviate this degradation, and we will explore joint training strategies part of future work.
B.3 BASELINE VARIANTS
We provide further variants of MotionCtrl and CameraCtrl adopted to the SnapVideo base model. These variants performed worse than the ones presented in the main paper, hence we mainly show them here for completeness to show the different implementations we explored.
On top of that, we train variant of MotionCtrl where instead of integrating the camera matrices into the context vector, we integrate them into the output of the cross-attention layer between the latent tokens and patches (as in our approach). We observe degraded camera control compared to our approach. Moreover, we train variant of CameraCtrl where we incorporate the Plucker embeddings of the CameraCtrl camera encoder into the patches instead of the latents. We observe worse camera control compared to our model. Furthermore, we train a variant of MotionCtrl where we freeze all base model layers similar CameraCtrl and our approach. We use the same batch size, same number of iterations, and same parameter size as our proposed method for fair comparisons. Note that MotionCtrl unfreezes the attention layer after injecting camera features into the base model. This experiment highlights that we outperform MotionCtrl independent of training freezing these attention layers. We show results for camera pose accuracy, multi-view generation, and video quality in Tab. 5 Tab. T and Tab. /respectivly.
B.4 GENERALIZATION OF CAMERA TRAJECTORIES
We conduct additional experiments with horizontal and vertical panning, where the camera trajectory is defined by rotation-only camera matrices without any translation. Specifically, we manually construct each trajectory by randomly selecting either the x-, y-, z-axis and randomly sampling the angular extent of the trajectory (from to 120 degrees). We set the camera translation to the zero vector. We show results in Tab. g and observe higher accuracy for our methods. Furthermore, we include results for non-random, user-defined camera trajectories that involve camera movements with significant directional changes including both rotations and translations in Tab, g These include following trajectories: rotation around clockwise; rotation around anticlockwise; rotation clockwise without translation; rotation anticlockwise without translation; zoom out, then up translation; trans- lation right, then rotation anticlockwise; translation left, then rotation clockwise; translation left; translation right; translation up: translation down. We observe that our method generalizes to input camera trajectories with variable rotations and translations.
B.5 EXPERIMENTS WITH VANILLA DIT
Instead of building upon the FIT CCen & Li 2023) architecture, we implemented our VD3D method on top of pre-trained text-to-video DiT model (Peebles & Xie, 2023) in the latent space of the Cog VideoX (Yang et al. 2024d) autoencoder. We include these results in Tab. 10 and Tab. [T Instead of building on top of read attention in FIT, we incorporate the NontrolNet conditioning
top of the vanilla attention mechanism of the actual tokens. We observe that the vanilla DiT version further improves quality and camera accuracy diistiiiiitttnoionnnn prompts (MSR-VTT). We believe that
22

---

Published as a conference paper at ICLR 2025
our proposed method of -patio-temporal Plucker tokens and aligning them with video patch tokens through a ControlNet-type of conditioning mechanism is agnostic to the transformer architecture and will serve as a starting point for follow-up works.
Base video DiT architecture details. The video DiT architecture follows the design of other contemporary video DiT models (e.g., Sora (Brooks al.
2024). MovieGen (Polyak al. 2024). OpenSora (Zheng et al. 2024b), LuminaT2X (Gao et al. 2024a), and Cog CogVideoX (Yang et al. 2024dh). As the backbone, it incorporates a transtormer-baaee
architecture with 32 DiT blocks. Each DiT block includes cross-attention layer for processing text embeddings (produced by the T5-11B model), a self-attention layer, and fully connected network with x4 dimensionality expansion. Attention layers consist of 32 heads with RMSNorm for query and key normalization. Positional information is encoded using 3D RoPE attention, where the temporal, vertical, and horizontal axes are allocated fixed dimensionality within each attention head (using a 2:1:1 ratio). LayerNorm is applied to normalize activations within each DiT block. A pre-trained Cog VideoX autoencoder is utilized for video dimensionality reduction, employing causal 3D convolution with a 4x8x8 compression rate and 16 channels per latent token. The model features a hidden dimensionality of 4,096 and comprises 11.5B parameters. It leverages block modulations condition the video backbone on rectified flow timestep information, SiLU activations, and 2x2 ViT-like patchification of input latents to reduce sequence size.
Base video DiT training details. The base DiT model is optimized using AdamW, with learning rate of 0.0001 and weight decay of 0.01. It is trained for 750,000 iterations with a cosine learning rate scheduler in bfloat16. Image animation support is incorporated by encoding the first frame with the Cog VideoX encoder, adding random Gaussian noise (sampled independently from the video noise levels), projecting via a separate learnable ViT-like patchification layer, repeating sequence-wist to match video length, and summing with the video tokens. Training incorporates loss normalization and is conducted jointly on images and videos with variable resolutions (256, 512, and 1024), aspect ratios (16:9 and 9:16 for videos; 16:9, 9;16, and 1:1 for images), and video lengths (ranging from 17 to 385 frames). Videos are generated at 24 frames per second, and variable-FPP training avoided due to observed performance decreases for target framerates without fine-tuning.
Base video DiT inference details. Inference uses standard rectified flow without stochasticity. We find forty steps to balance quality and sampling speed effectively. For higher resolutions and longer video generation, time-shifting strategy similar to Lumina-T2 m2X is used, with a time shift of v/2 for 1024-resolution videos.
C TRAINING DETAILS
Compute details. A single training run for the smaller 700M parameter generator takes approximately 1 day on node equipped with 8x NVIDIA A100 40GB GPUs, connected via NVIDIA NVLink, along with 960 GB of RAM and 92 Intel Xeon CPUs. The larger 4B parameter model was trained on 8 such nodes for 1,5 days, totaling 64 NVIDIA A100 40GB GPUs. In total, we conducted =150 training runs for the smaller model during the development stage of 4B generator. Consequently. the project's total compute utilization amounted approximately 2700 NVIDIA A100 40GB GPU-days.
Optimization details. We experiment with two model variants: a smaller generator with approxi- mately 700 million parameters for ablations and initial explorations, and larger 4 billion parameter model, which we use for the main results in this paper. Both models were trained with a batch size of 256 over 50,000 optimization steps with the LAMB optimizer (You et al., 2019). The learning rate was warmed up for the first 10,000 iterations from 0 to 0.005 and then linearly decreased to 0.0015 over subsequent iterations. For the large model, VD3D contains 230M trainable parameters in total which corresponds to around 5% of the total amount. Since the original video diffusion model trained the soay--rrr-coodtiooniin
pipeline (Menapace et al. 2024), we can produce variable camera trajectories from the same starting frame. For text conditioning, we use the T5-1 1B (Raffel ct al. 2020) language model to encode text 1024-dimensional embeddings into 128-length sequences. For training efficiency, they were precomputed for the entire dataset. The rest of the training details have been adopted from (Menapace et al. 2024).
23

---

Published as a conference paper at ICLR 2025
Table 3: Multi-view generation. We evaluate all models using reference camera trajectories and single-view input images RealEstate I0K test set. We compute reconstruction metrics based on the subsequent frames for the low-resolution and upsampled high-resolution generations.
Low-resolution
High-resolution
Method
PSNR (1)
SSIM (t)
LPIPS (4)
PSNR (1)
SSIM (1)
LPIPS (4)
Base Model
14.74
0.320
0.334
13.23
0.459
0.572
MotionCtrl (U-Net)
15.35
0.387
0.294
13.64
0.473
0.548
CameraCtrl (U-Net)
15,86
0.412
0.266
13.81
0.479
0.540
MotionCtrl
15.07
0.348
0.308
13.42
0.467
0.560
CameraCtrl
14.81
0.327
0.330
13.21
0.456
0.571
Ours
17.23
0.534
0.211
14.90
0.499
0.499
w/o Plucker
14.89
0.346
0.308
13.05
0.455
0,573
w/o ControlNet
14.66
0.313
0.340
13.10
0.450
0.573
w/o weight copy
16.96
0.509
0.220
14.75
0.495
0,504
w/o add context
14.45
0.303
0.368
13.06
0.446
0,588
w/o Plucker context
14.76
0.322
0.318
13.28
0.463
0.579
Table 4: Quality metrics evaluation. We evaluate all models using text prompts from the
RealEstate 10K and MSR-VTT test sets respectively.
RealEstate 10K
MSR-VTT
Method
FID (L)
FVD (L)
CLIPSIM (t)
FID (L)
FVD (4)
CLIPSIM (t)
Base Model
8.22
160.37
0.2677
3.50
141.26
0.2774
MotionCtrl (U-Net)
2.99
61.70
0.2646
16.85
283.12
0.2411
CameraCtrl (U-Net)
2.48
55.64
0.2681
12.33
201.33
0,2505
MotionCtrl
1.50
52.30
0.2708
9.97
183.57
0.2677
CameraCtrl
2.28
66.31
0.2730
8.47
181.90
0,2690
Ours
1.40
42.43
0.2807
7.80
165.18
0.2689
w/o Plucker
1.17
43.65
0.2715
9.84
152.91
0.2660
w/o ControlNet
3.66
137.06
0.2766
8.34
185.79
0.2674
w/o weight copy
1.38
42.00
0,2710
10.09
218.43
0.2647
w/o add context
1,45
44.74
0.2735
9.56
173.43
0.2657
w/o Plucker context
1.54
43.88
0.2724
8.83
168.95
0,2615
Table Camera pose evaluation with additional baselines. We evaluate additional variants the baselines using reference camera trajectories from the RealEstate 10K test set. We compute translation and rotation errors based on estimated camera poses from generations using ParticleSIM (Zhao et al.
2022
RealEstate
MSR-VTT
Method
TransError (4)
RotError (L)
TransError (4)
RotError (L)
MotionCtrl (latents)
0.549
0.183
0.650
0.145
CameraCtrl (patches)
0.587
0.197
0.648
0.233
MotionCtrl (frozen)
0.607
0.205
0.678
0.122
Ours
0.409
0.043
0.504
0.050
24

---

Published as a conference paper at ICLR 2025
Table 6: Multi-view generation. We evaluate additional variants the baselines using refer- camera trajectories and single-view imput images of the RealEstate 10K test set. We compute reconstruction metrics based on the subsequent frames for the low-resolution and upsampled high- resolution generations.
Low-resolution
High-resolution
Method
PSNR (1)
SSIM (1)
LPIPS (4)
PSNR (1)
SSIM (t)
LPIPS (4)
MotionCtrl (latents)
14.67
0.323
0.331
13.23
0.467
0.562
CameraCtrl (patches)
14.42
0.304
0.366
13.04
0,450
0.577
MotionCull (frozen)
14.59
0.308
0.340
13.11
0.455
0.573
Ours
17.23
0.534
0.211
14.90
0.499
0.499
Table 7: Quality metrics evaluation with additional baselines. We evaluate additional variants of the baselines using text prompts from the RealEstate and MSR-V VTT test sets respectively.
RealEstate
MSR-VTT
Method
FID (4)
FVD (L)
CLIPSIM (f)
FID (L)
FVD (L)
CLIPSIM (1)
MotionCtrl (latents)
1.83
77.39
0.2788
10.21
187.42
0.2636
CameraCtrl (patches)
2.57
71.04
0.2703
9.84
184.22
0.2612
MotionCtrl (frozen)
3.53
142.15
0.2772
8.19
165.48
0.2679
Ours
1.40
42.43
0.2807
7.80
165.18
0.2689
Table 8: Camera pose evaluation for random rotation trajectories. We evaluate all models using trajectories with randomly selected y-, or z-axis and randomly sampled angular extent of the trajectory. We compute translation and rotation errors based on estimated camera poses from generations using ParticleSfM (Zhao et al. 2022)
RealEstate10K
MSR-VTT
Method
TransError (L)
RotError (L)
TransError (4)
RotError (L)
MotionCtrl
0.396
0.087
0.417
0.120
CameraCtrl
0.381
0.092
0.433
0.138
Ours
0.202
0.037
0.265
0.044
Table Camera pose evaluation for random trajectories with translations and rotations. We evaluate all models using 11 trajectories for 1000 prompts with trajectories that involve significant directional changes including both rotations and translations. We compute translation and rotation errors based estimated camera poses from generations using ParticleSIM (Zhao al. 2022
RealEstate100 10K
MSR-VTT
Method
TransError (L)
RotError (4)
TransError (4)
RotError (L)
MotionCtrl
0.451
0.095
0.456
0.146
CameraCtrl
0.369
0.088
0.479
0.135
Ours
0.236
0.041
0,258
0.050
25

---

Published as a conference paper at ICLR 2025
Table 10: Camera pose evaluation with vanilla video DiT backbone. We incorporate our approach into a pre-trained vanilla DiT model in the latent space of VideoX (Yagg et al.) 2024d). We
compute translation and rotation errors based on estimated camera poses from generations using ParticleSIM (Zhao et al. 2022
RealEstate
MSR-VTT VTT
Method
TransError (L)
RotError (4)
TransError (4)
RotError (L)
MotionCtrl
0.501
0.145
0.602
0.152
CameraCtrl
0.513
0.138
0.559
195
Ours
0.421
0.056
0.486
0.047
Table 11: Quality metrics evaluation with vanilla video DiT backbone. We incorporate our approach into pre-trained vanilla DiT model in the latent space of CogVid (Yang al. 2024d). We evaluate additional variants of the baselines using text prompts from the RealEstate 1 OK and MSR-VT ETR test sets respectively.
RealEstate
MSR-VTT
Method
FID (L)
FVD (4)
CLIPSIM (t)
FID (L)
FVD (L)
CLIPSIM (t)
MotionCtrl
1.37
44.62
0.2752
9.68
157.90
0.2684
CameraCtrl
2.13
53.72
0.2748
8.32
152.88
0.2723
Ours
1.21
38.57
0.2834
6.88
137.62
0.2790
26


Reviews:
[{"review_id": "BLNCUZyzqR", "reviewer": "Reviewer_B4pY", "summary": "This paper presents a camera control method for transformer-based video generation models that enhances control while ensuring visual quality. The proposed approach aligns the video perspective with predefined camera trajectories, improving controllability. The authors claim this is the first study to employ ControlNet-like guidance for global spatiotemporal transformer video generation models, in contrast to the more commonly used U-Net architecture. Moreover, the evaluation demonstrates that both the video quality and adherence to the input camera trajectories are state-of-the-art.", "questions": "- Have there been additional relevant experiments regarding camera trajectories, such as comparisons of control quality for generated trajectories of varying complexity?\n\nAt this time, I have no further questions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Camera control during the video generation process is a significant issue. As more foundational models adopt transformer architectures, exploring control mechanisms for these models becomes crucial. This paper is the first to investigate how to better utilize camera trajectory parameters for transformer-based video generation models, using SnapVideo as the foundational model. The design is well thought out, and the evaluation is rigorous. The strengths of the paper are as follows:\n\n- Unlike the spatiotemporal decoupling generation of U-Net structures, transformer-based video generation considers spatiotemporal video tokens globally, which means it cannot directly leverage the advantages of spatiotemporal decoupling. This paper overcomes this limitation by being the first to explore control specifically for spatiotemporal transformers. This shift in foundational model structure is critical and provides a solid engineering foundation for future work.\n  \n- The authors use Plücker embeddings to convert camera intrinsic and extrinsic parameters into pixel-level controls, which match the shape of video tokens. This information is then introduced through read cross-attention layers. While this approach is a straightforward combination of existing methods, it has been validated as effective for transformer-based video generation models, providing valuable experimental insights.\n  \n- The paper includes comprehensive evaluations and ablation studies, conducting both qualitative and quantitative experiments regarding video content quality and camera control, with well-defined criteria. The evaluation of baseline models is fair, making the transition to the new foundational model structure more convincing.", "weaknesses": "- While camera control is the central problem addressed in this paper, the camera trajectories used primarily come from the RealEstate10K dataset, which, as observed in the visual results, mostly follow smooth, straight lines. There is a lack of consideration and experimentation with trajectories of varying difficulty, such as those involving significant directional changes. This raises some questions regarding the trajectory settings.\n  \n- There have been several prior works in the 3D multi-view generation field that focus on similar camera control issues, such as the referenced *Cat3D*, which also employed Plücker embeddings and attention calculations. The distinction between spatiotemporal decoupling and other network characteristics is a design feature intrinsic to the architecture. In exploring DiT-based generation, there have also been multiple studies investigating spatiotemporal decoupling, such as *Latte: Latent Diffusion Transformer for Video Generation*. Therefore, the novelty of this work lies more in applying existing designs to spatiotemporal transformers rather than presenting a technological innovation. However, the state-of-the-art results under the new configuration indeed serve as an important engineering reference for future directions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CMqlCLFTYm", "reviewer": "Reviewer_uN7u", "summary": "The method proposes a controlnet-like architecture for a private video diffusion model by including plucker coordinates as camera control.", "questions": "Can the authors please comment on the above-mentioned weaknesses?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed controlnet design outperforms other model variants designed by the authors. The evaluations are thoroughly conducted for the design choices. Detailed ablations are provided.", "weaknesses": "- The proposed framework overfits on the trajectories that are seen during training. Though the authors provide quantitative comparisons in Tab. 8, no visual comparisons are provided. \n- Though the performance is impressive, the technical contribution is limited in the proposed framework. Training a ControlNet for diffusion transformer is not new, as shown in [1]. Using Plucker coordinates for camera control is not new, as shown in CameraCtrl (He et al., 2024a).\n\n[1] Chen J, Wu Y, Luo S, et al. Pixart-{\\delta}: Fast and controllable image generation with latent consistency models[J]. arXiv preprint arXiv:2401.05252, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9BcHupT7Zt", "reviewer": "Reviewer_qX6r", "summary": "This paper provides a method for adding the control of camera movements to video diffusion transformers. The core idea is to represent camera conditions as pixel-level ray condition with Plucker embeddings. A ControlNet inspired module is used to process the camera condition. The model is finetuned on RealEstate10k and is compared to two similar methods both qualitatively and quantitatively.", "questions": "Overall I think this paper provides an effective solution to amend video diffusion transformers with camera control. See weaknesses for questions and discussion.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- paper is easy to follow\n- the proposed design including the Plücker embedding is reasonable and effective\n- comprehensive experiments are conducted and presented in the main manuscript and appendix\n- supplemental materials contain video samples to demonstrate the effectiveness", "weaknesses": "- The proposed method has been evaluated only on one video diffusion transformer, which raises some concerns on whether its performance can generalize to other pretrained video diffusion transformers.\n- I'm curious about the distribution of camera movements evaluated in the experiments, in terms of its diversity and similarity to natural camera movements.\n- The novelty is slightly limited, as the task is not new, and ControlNet-like module as well as Plücker embedding have been explored and used before.\n\nminor:\n- CameraCtrl was appeared on arXiv in April 2024, which should not be considered as a concurrent work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DkQ83qxzWe", "reviewer": "Reviewer_DiqV", "summary": "The paper proposes a first method to condition spatio-temporal transformer-based video diffusion models on camera poses (previous methods focused on pre-trained U-Net-based diffusion models). To this end, they propose a ControlNet-like block that conditions the model on camera embeddings that are based on Plucker coordinates. The paper evaluates the choices made and demonstrates good results, both qualitatively and quantitatively.", "questions": "Nothing crucial.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a nice idea on to enable spatio-temporal transformer-based video diffusion models with camera control. It is the first to do so by using a ControlNet-like block in combination with Plucker coordinates, and presents a nice contribution. I value the ablation studies in the paper and the reasoning as to how they arrived at this particular architecture. The paper is quite well-written overall. It's easy enough to follow and appreciate the contribution.", "weaknesses": "While I'm positive overall, there are a few weaknesses. \n\nA criticism that could be made is that this paper is a bit of an A+B paper, where A is the ControlNet block and B is the Plucker coordinates. I don't think that's a useful criticism, because at the end of the day it's a sensible thing to do and the authors demonstrate that you need to do a few things to make it work (Table 2).\n\nThe paper reads well overall, however, I'm not a fan of Figure 3. It's hard to interpret what goes where, and how this relates to the formulas. I'd suggest to clearly delineate what are the video patch tokens vs. the plucker patch tokens, maybe add the variables from the formulas to the appropriate boxes, and overall structure the figure so that the separate blocks are clearly separate.\n\nFinally, in terms of results, I would have liked to see more examples of the same scenes with different camera control (there are only three examples). Furthermore, most examples use input camera trajectories from scenes that are completely unrelated to the target scene. It be nice to see some that are related -- makes it easier to judge if the generated camera path is good or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LROnH4Nhnt", "reviewer": "Reviewer_ZZhZ", "summary": "This paper presents a novel approach to incorporating camera movement control into Transformer-based video diffusion models. While camera control has been extensively studied in UNet-based video diffusion models, this area remains largely uncharted for Transformer architectures. The author introduces an additional ControlNet-like network to inject tokenized Plücker embeddings, demonstrating that this design enhances both visual quality and controllability.", "questions": "1. Line 272, R_1, t_1 can be interpreted as the original extrinsic or the extrinsic after normalization. It would be better to use R'_1, and t'_1 to represent the extrinsic after normalization;\n\n2. I am not quite clear about the baseline implementation details. From my understanding, for MotionCtrl, one additional learned token is concatenated to the video patch tokens, while for CameraCtrl, the camera encoder produces additional latent tokens. Is that correct?\n\n3. While previous works sacrifice motion dynamics when training on RealEstate10K (mostly static scenes), the video in the supplementary material exhibits better and larger motions. What would be the possible reasons for the difference?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors made an important step forward in camera control for the Transformer-based video diffusion model, which is unexplored by previous research;\n2. The visual quality and motion dynamics of the generated videos are excellent;\n3. The methodology is clearly explained and easy to understand.", "weaknesses": "1. The main components of the method have been validated by previous works, for instance, the Plucker embedding as camera representation (CameraCtrl), and training on RealEstate10K(MotionCtrl, CameraCtrl);\n\n2. As mentioned in 1., I would say the specified designs for Transformer architecture are the major contribution of the paper. However, SnapVideo's architecture is not a typical DiT (it has the \"read\" operation, and the attention is not performed on the actual tokens). It's not clear how to extend the proposed method for a standard video DiT and how the performance would be.\n\n3. For the MotionCtrl baseline, the visual quality degrades when the base model is fine-tuned. Would it be better to freeze the backbone?\n\n4. Would be better to also provide the trainable parameter scale of the two baselines and the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)