VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control

## Abstract

Abstract Modern text-to-video synthesis models demonstrate coherent, photorealistic generation of complex videos from a text description.
However, most existing models lack fine-grained control over camera movement, which is critical for downstream applications related to content creation, visual effects, and 3D vision.
Recently, new methods demonstrate the ability to generate videos with controllable camera poses—these techniques leverage pre-trained U-Net-based diffusion models that explicitly disentangle spatial and temporal generation.
Still, no existing approach enables camera control for new, transformer-based video diffusion models that process spatial and temporal information jointly.
Here, we propose to tame video transformers for 3D camera control using a ControlNet-like conditioning mechanism that incorporates spatiotemporal camera embeddings based on Plucker coordinates.
The approach demonstrates state-of-the-art performance for controllable video generation after fine-tuning on the RealEstate10K dataset.
To the best of our knowledge, our work is the first to enable camera control for transformer-based video diffusion models.

## 1 Introduction

Figure: Figure 2: Comparing text-to-video, text-to-4D, and camera-conditioned text-to-video generation. Text-to-4D approaches, such as 4D-fy [] (top) have full control over camera through use of a 3D representation, but they lack photorealism. (middle) Methods for text-to-video generation [] create realistic videos, but do not provide explicit control over the viewpoint. In contrast, camera-conditioned text-to-video generation (bottom) bridges the gap between the two paradigms by extending text-to-video generators with 3D camera control without using an explicit 3D representation. Please see the supplementary webpage for the corresponding video results.
Refer to caption: x2.png

Text-to-video foundation models achieve unprecedented visual quality [, ].
They are trained on massive collections of images and videos and learn to synthesize remarkably consistent and physically plausible visualizations of the world.
Yet, they lack built-in mechanisms for explicit 3D control during the synthesis process, requiring users to manipulate outputs through prompt engineering and trial and error—a slow, laborious, and computationally expensive process.
For example, as Fig. 2 shows, state-of-the-art video models struggle to follow even simple “zoom-in” or “zoom-out” camera trajectories using text prompt instructions (see supplemental webpage).
This lack of controllability limits interactivity and makes existing video generation techniques challenging to use for artists or other end users.
We augment 2D video generation models with control over the position and orientation of the camera, providing finer-grained control compared to text prompting, and facilitating use of video generation models for downstream applications.

Several contemporary works [, , ] propose methods for camera control of state-of-the-art, open-source video diffusion models.
The key technical insight proposed by these methods is to add camera control by fine-tuning the temporal conditioning layers of a U-Net-based video generation model on a dataset with high-quality camera annotations.
While these techniques achieve promising results, they are not applicable to more recent, high-quality transformer-based architectures [, ], such as Sora [], SnapVideo [], and Lumina-T2X [], as these latest works simply do not have standalone temporal layers amenable to camera conditioning.

Large video transformers represent a video as a (possibly compressed) sequence of tokens, applying self-attention layers to all the tokens jointly [, ].
Consequently, they do not have standalone temporal layers, which are essential for current camera conditioning approaches to integrate viewpoint information, making them inapplicable to such models.
As the community shifts towards large video transformers to jointly model spatiotemporal dependencies in the data, it is critical to develop methods that provide similar capabilities for camera control.
Our work designs a camera conditioning method tailored to the joint spatiotemporal computation used in large video transformers and takes a step towards taming them for controllable video synthesis.

We develop our work on top of our implementation of SnapVideo [], a state-of-the-art video diffusion model, that employs FIT-blocks [] for efficient video modeling in the compressed latent space.
We investigate various camera conditioning mechanisms in the fine-tuning scenario and explore trade-offs in terms of visual quality preservation and controllability.
Our findings reveal that simply adapting existing approaches to video transformers does not yield satisfactory results: they either enable some limited amount of control while reducing the visual quality of the output video, or they entirely fail to control camera motion.
Our key technical insight is to enable the control through spatiotemporal camera embeddings, which we derive by combining Plucker coordinates with the network input through a separately trained cross-attention layer.
To the best of our knowledge, our work is the first to explore a ControlNet-like [] conditioning mechanism for spatiotemporal transformers.

We evaluate the method on a collection of manually crafted text prompts and unseen camera trajectories and compare to baseline approaches that incorporate previous camera control methods into a video transformer.
Our approach achieves state-of-the-art results in terms of camera controllability and video quality, and also enables downstream applications such as multi-view, text-to-video generation, as depicted in Fig. 1.
In contrast to existing image-to-3D methods (e.g., [, , ]), which are limited to object-centric scenes, our approach synthesizes novel views for real input images with complex environments.

Overall, our work makes the following contributions.

- •
We propose a new method to tame large video transformers for 3D camera control. Our approach uses a ControlNet-like conditioning mechanism that incorporates spatiotemporal camera embeddings based on Plucker coordinates.
- •
We thoroughly evaluate this approach, including comparisons to previous camera control methods, which we adapt to the video transformer architecture.
- •
We demonstrate state-of-the-art results in controllable video generation by applying the proposed conditioning method and fine-tuning scheme to the SnapVideo model [].

## 2 Related work

Our method is connected to techniques related to text-to-video, text-to-3D, and text-to-4D generation.
As this is a popular and fast-moving field, this section provides only a partial overview with a focus on the most relevant techniques; we refer readers to Po et al. [] and Yunus et al. [] for a more thorough review of related techniques.

Text-to-video generation.
Our work builds on recent developments in 2D video generation models.
One such class of these techniques works by augmenting text-to-image models with layers that operate on the temporal dimension to facilitate video generation [, , , , ].
Video generation methods can be trained in a hybrid fashion on both images and video to improve the generation quality [, , , , , , , ].
While these techniques are primarily based on convolutional, U-Net-style architectures, a recent shift towards transformer-based models enables synthesis of much longer videos with significantly higher quality [, , ].
Still, these recent techniques do not enable synthesis with controllable camera motion.

4D generation.
Previous methods also tackle the problem of 4D generation, i.e., generating videos of dynamic 3D scenes from controllable viewpoints, usually from an input text prompt or image.
Since the initial work on this topic using large-scale generative models [], significant improvements in the visual quality and motion quality of generated scenes have been achieved [, , , , , , , , , , , ].
While these methods generate scenes based on text conditioning, other approaches convert an input image or video to a dynamic 3D scene [, , , , , , , , , , , , , , , , , , , , , , , , ].
Another line of work [, ] extends 3D GANs into 4D by training on 2D videos, however the quality is limited and models are trained on single category datasets.
Still, all of these methods are focused on object-centric generation, typically based on 3D volumetric representations.
As such, they typically do not incorporate background elements, and overall, they do not approach the level of photorealism demonstrated by the state-of-the-art video generation models used in our technique.

Controllable generation with diffusion models.
Methods for controllable generation using diffusion models have had significant impact, both in the context of image and video generation.
For example, existing techniques allow controllable image generation conditioned on text, depth maps, edges, pose, or other signals [, ].
Furthermore, there is a line of work for 3D generation which conditions diffusion models on camera poses for view-consistent multi-view generation [, , , , , , ].
Our approach is most similar to related techniques in video generation that seek to control the camera position.
For example, MotionCtrl [] designs camera and object control mechanisms for the VideoCrafter1 [] and SVD [] models.

Concurrent 3D camera control methods.
Concurrent approaches enable camera control by conditioning the temporal layers of the network with camera pose information, e.g., using Plucker coordinates [, , , ] or other embeddings [].
Interestingly, it is also possible to incorporate camera control into video generation models without additional training through manipulation and masking of attention layers, though this requires additional tracking, segmentation, or depth for each input video [, , ]. Another recent work [] transfers motion, including camera motion, to other generated videos.

Although these approaches demonstrate promising results for U-Net-based video diffusion models, the techniques are not applicable to modern video transformers that model spatio-temporal dynamics jointly. While another concurrent work [] uses a transformer-based architecture for space and time, it does not tackle text-based generation for dynamic scenes but mainly novel view synthesis for 3D scenes.
In our work, we design an efficient mechanism that enables camera control in video diffusion transformers using a ControlNet inspired mechanism without sacrificing visual quality.

## 3 Method

Figure: Figure 3: Overview of architecture. We adapt the Snap Video FIT architecture [] to incorporate camera control. We take as input the noisy input video $\tilde{\bm{x}}$, camera extrinsics $\bm{C}_{f}$, and camera intrinsics $\bm{K}_{f}$ for each video frame $f$. Using the camera parameters, we compute the Plücker coordinates for each pixel within the video frames. Both the input video and Plücker coordinate frames are converted to patch tokens, and we condition the video patch tokens using a mechanism similar to ControlNet [] (“Plücker conditioning” block). Then, the model estimates the denoised video $\hat{\bm{x}}$ by recurrent application of FIT blocks []. Each block reads information from the patch tokens into a small set of latent tokens on which computation is performed. The results are written to the patch tokens in an iterative denoising diffusion process.
Refer to caption: x3.png

### 3.1 Large text-to-video transformers

Text-to-video generation.
Diffusion models have emerged as the dominant paradigm for large-scale video generation [, , ].
The standard setup considers the conditional distribution $p(\bm{x}|\bm{y})$ of videos $\bm{x}\in\mathbb{R}^{F\times H\times W}$ (consisting of $F$ frames of $H\times W$ resolution) given their textual descriptions $\bm{y}\in\mathcal{Y}^{L}$, consisting of $L$ (possibly padded) tokens from the alphabet $\mathcal{Y}$.
Following [], our video diffusion framework assumes a denoising model $D_{\bm{\theta}}:(\tilde{\bm{x}};\bm{y},\sigma)\mapsto\hat{\bm{x}}$ that predicts a clean video $\hat{\bm{x}}$ from the corresponding noised input $\tilde{\bm{x}}=\bm{x}+\sigma\bm{\varepsilon}$, $\sigma\in\mathbb{R}_{+},\bm{\varepsilon}\sim\mathcal{N}(\bm{0},\bm{I})$.
The model is parametrized by a neural network $F_{\bm{\theta}}(\tilde{\bm{x}};\bm{y},\sigma)$ as $D_{\bm{\theta}}(\tilde{\bm{x}};\bm{y},\sigma)=c_{\text{out}}(\sigma)F_{\bm{
\theta}}(c_{\text{in}}(\sigma)\tilde{\bm{x}};\bm{y},\sigma)+c_{\text{skip}}(
\sigma)\tilde{\bm{x}}$.
The optimization objective is defined as follows:

$$ $\displaystyle\mathcal{L}(\bm{\theta})\triangleq\underset{(\bm{x},\bm{y}), \sigma,\bm{\varepsilon}}{\mathbb{E}}\left[\|D_{\bm{\theta}}(\bm{x}+\sigma\bm{ \varepsilon};\bm{y},\sigma)-\bm{x}\|_{2}^{2}\right]\longrightarrow\min_{\bm{ \theta}}$ (1) $$

We refer a reader to [] and [] for further details on the diffusion setup, which we adopted without modifications.

Spatiotemporal transformers.
Following SnapVideo [], our video generator consists of two models: the base 4B-parameters generator, operating on 16-frames $36\times 64$ resolution videos, and a $288\times 512$ upsampler.
The latter, a diffusion model itself, is fine-tuned from the base model and conditioned on the low-resolution videos.
Each model uses FIT transformer blocks [, ] for efficient self-attention operations (see Fig. 3).
An FIT model consists of $B$ blocks (we have $B=6$ in all the experiments) and first partitions each frame in an input video into patches [] of resolution $h_{p}\times w_{p}$ (we use $h_{p}=w_{p}=4$ in all the experiments).
These video patches are then independently projected via a feedforward (FF) layer to obtain a sequence of video tokens $[\bm{v}_{\ell}]_{\ell=1}^{L}\triangleq(\bm{v}_{1},...,\bm{v}_{L})\in\mathbb{R}
^{L\times d}$ of length $L=F\times(H/h_{p})\times(W/w_{p})$ and dimensionality $d$.
Next, each FIT block “reads” the information from this video sequence into a much shorter sequence of *latent* tokens $[\bm{z}_{m}]_{m=1}^{M}\triangleq(\bm{z}_{1},...,\bm{z}_{L})$ through a “read” cross-attention layer, followed by a feedforward layer.
The core processing with self-attention layers is performed in this latent space, and then the result is written back to the video tokens through a corresponding “write” cross-attention layer (also followed by an FF layer).
The latent tokens in each next FIT block are initialized from the previous one, which helps to propagate the computational results throughout the network.
In this way, the entire computation occurs jointly in both spatial and temporal axes, which yields superior scalability properties [].
However, it abandons the decomposed spatial/temporal computation nature of modern video diffusion U-Nets, which is vital for modern camera conditioning techniques [, ] to enforce control without compromising visual quality.

### 3.2 Camera control for spatiotemporal transformers

Spatiotemporal camera representation.
The standard way of representing camera parameters for a video (in the pinhole model) is via a trajectory of extrinsics and intrinsics camera parameters $(\bm{C}_{f},\bm{K}_{f})_{f=1}^{F}$ for each $f$-th frame, where $\bm{C}_{f}=[\bm{R};\bm{t}]\in\mathbb{R}^{3\times 4}$, describes the camera rotation $\bm{R}\in\mathbb{R}^{3\times 3}$ and translation $\bm{t}\times\mathbb{R}^{3}$, and $\bm{K}_{f}\in\mathbb{R}^{3\times 3}$ contains the focal length and principal point (and also horizontal/vertical skew coefficient, but it is always 0 in our setup).
To control camera motion, existing methods such as [, , ] condition the temporal attention layers of U-Net-based video generators on embeddings computed from these camera parameters.
Such a pipeline provides a good conditioning signal for convolutional video generators with decomposed spatial/temporal computation, but our experiments demonstrate that it works poorly for spatiotemporal transformers: they either fail to pick up any controllability (when being added as transformed residuals to the latent tokens), or degrade the visual quality of the output (when the original network parameters are being fine-tuned).
This motivates us to design a better camera conditioning scheme, tailored for modern large-scale spatiotemporal transformers.

First, we propose to normalize the camera parameters to the very first frame, forcing the first frame’s rotation be an identity matrix $\bm{R}_{1}=\bm{I}$ and translation be zero: $\bm{t}_{1}=\bm{0}$.
We then recompute the extrinsics for each $f$-th frame as $\bm{C}_{f}=[\bm{R}_{1}^{-1}\bm{R}_{f};\bm{t}_{f}-\bm{t}_{1}]$, establishing a consistent coordinate system across videos.
After that, we found it essential to enrich the conditioning information by switching from temporal frame-level camera parameters to pixel-wise *spatiotemporal* camera parameters.
This is achieved by computing the Plücker coordinates for each pixel, providing fine-grained positional representation.

Plücker coordinates provide a convenient parametrization of lines in the 3D space, and we use them to compute fine-grained positional representations of each pixel in each frame of a video.
Given the extrinsic and intrinsic camera parameters $\bm{R},\bm{t}$, $\bm{K}_{f}$ of the $f$-th frame, we parametrize each $(h,w)$-th pixel as a Plücker embedding $\ddot{\bm{p}}_{f,h,w}\in\mathbb{R}^{6}$ from the camera position to the pixel’s center as

$$ $\displaystyle\ddot{\bm{p}}_{f,h,w}=(\bm{t}_{f}\times\hat{\bm{d}}_{f,h,w},\hat{ \bm{d}}_{f,h,w}),\quad\hat{\bm{d}}_{f,h,w}=\frac{\bm{d}}{\|\bm{d}_{f,h,w}\|}, \quad\bm{d}_{f,h,w}=\bm{R}_{f}\bm{K}_{f}[w,h,1]^{\top}+\bm{t}_{f}.$ (2) $$

This approach mirrors the technique used in recent 3D works [, , ] as well as CameraCtrl [], a concurrent study focusing on camera control in U-Net-based video diffusion models.
The motivation for using Plücker coordinates is that geometric manipulations in the Plücker space can be performed through simple arithmetics on the coordinates, which makes it easier for the network to use the positional information stored in such a disentangled representation.

Computing Plücker coordinates for each pixel results in a $\ddot{\bm{P}}\in\mathbb{R}^{6\times F\times H\times W}$ spatiotemporal camera representation for a video.
To input it into the model, we first perform the equivalent ViT-like [] $h_{p}\times w_{p}$ patchification procedure.
It is followed by a learnable feedforward layer $\text{FF}_{\ddot{p}}$ to obtain the Plücker camera tokens sequence $[\ddot{\bm{c}}_{\ell}]_{\ell=1}^{L}\in\mathbb{R}^{L\times d}$ of the same length $L=F\times(H/h_{p})\times(W/w_{p})$ and dimensionality $d$ as the video tokens sequence $[\bm{v}_{\ell}]_{\ell=1}^{L}$.
This spatiotemporal representation carries fine-grained positional information about each pixel in a video, making it easier for the generator to accurately follow the desired camera motion.

Camera conditioning.
To input the rich spatiotemporal camera information in the form of Plücker embeddings into our video generator, we design an efficient ControlNet like [] mechanism tailored for large transformer models (see Fig. 3).
This mechanism is guided by two main objectives: 1) the model should be amenable to rapid fine-tuning from a small dataset with estimated camera positions; and 2) the visual quality shouldn’t be compromised during the fine-tuning stage.
We found that meeting these objectives is more challenging for spatiotemporal transformers compared to U-Net-based models with decomposed spatial/temporal computation, since even minor interventions into their design quickly lead to degraded video outputs.
We hypothesize that the core reason for it is the entangled spatial/temporal computation of video transformers: any attempt to alter the temporal dynamics (such as camera motion) influences spatial communication between the tokens, leading to unnecessary signal propagation and overfitting during the fine-tuning stage.
To mitigate this, we input the camera information gradually through read cross-attention layers, zero-initialized from the original network parameters of the corresponding layers.

Specifically, in each $b$-th FIT block of our video generator, we replace its standard read cross-attention operation (see [, ] for details):

$$ $\displaystyle[\bm{z}^{(b)}_{m}]_{m=1}^{M}=\text{FF}^{(b)}(\text{XAttn}^{(b)}([ \bm{z}^{(b)}_{m}]_{m=1}^{M},[\bm{v}^{(b)}_{\ell}]_{\ell=1}^{L}]),$ (3) $$

where FF($\cdot$) and XAttn($\cdot,\cdot$) denote feed-forward and cross-attention layers respectively, with:

$$ $\displaystyle[\bm{z}^{(b)}_{m}]_{m=1}^{M}$ $\displaystyle=\text{FF}^{(b)}(\text{XAttn}^{(b)}([\bm{z}^{(b)}_{m}]_{m=1}^{M}, [\bm{v}^{(b)}_{\ell}]_{\ell=1}^{L}])$ (4) $\displaystyle+\text{Conv}_{\text{res}}^{(b)}(\text{FF}_{\text{cam}}^{(b)}( \text{XAttn}_{\text{cam}}^{(b)}([\bm{z}^{(b)}_{m}]_{m=1}^{M},\text{Conv}_{ \text{pl\"{u}ck}}^{(b)}([\ddot{\bm{c}}_{\ell}]_{\ell=1}^{L}]))),$ (5) $$

where $\text{FF}_{\text{cam}}^{(b)}$, and $\text{XAttn}_{\text{cam}}^{(b)}$ are learnable layers, and $\text{Conv}_{\text{pl\"{u}ck}}^{(b)},\text{Conv}_{\text{res}}^{(b)}$ are 1-dimensional convolutions that process plucker camera tokens and their final representations inside, respectively.
It is crucial to instantiate the weights of these convolutions from zeros to preserve the model initialization.
Besides, we initialize the weights of $\text{FF}_{\text{cam}}^{(b)}$ and $\text{XAttn}_{\text{cam}}^{(b)}$ from the corresponding parameters of the original network.
This approach helps to preserve visual quality at initialization and facilitates rapid fine-tuning on a small dataset.
As a result, we obtain the method for fine-grained 3D camera control in large video diffusion transformers.
We name it VD3D and visualize its architecture in Fig. 3.

### 3.3 Training details

Optimization details.
To ensure comparability with prior work such as [], we train our video generator on the same RealEstate10k dataset [].
We optimize only the newly added parameters $\text{FF}_{\ddot{p}}$ and $(\alpha^{(b)},\text{FF}^{(b)},\text{XAttn}^{(b)})_{b=1}^{B}$, and keep the rest of the network frozen.
We found that training only the base $36\times 64$ model was sufficient, as the $288\times 512$ upsampler already accurately follows the camera motion of a low-resolution video.
We experiment with two model variants: a smaller generator with approximately 700 million parameters for ablations and initial explorations, and a larger 4 billion parameter model, which we use for the main results in this paper.
Both models were trained with a batch size of 256 over 50,000 optimization steps with the LAMB optimizer [].
The learning rate was warmed up for the first 10,000 iterations from 0 to 0.005 and then linearly decreased to 0.0015 over subsequent iterations.
Since the original video diffusion model is trained in the any-frame-conditioning pipeline [], we can produce variable camera trajectories from the same starting frame.
For text conditioning, we use the T5-11B [] language model to encode text $1024$-dimensional embeddings into 128-length sequences.
For training efficiency, they were precomputed for the entire dataset.
The rest of the training details have been adopted from [].

Compute details.
A single training run for the smaller 700M parameter generator takes approximately 1 day on a node equipped with $8\times$ NVIDIA A100 40GB GPUs, connected via NVIDIA NVLink, along with 960 GB of RAM and 92 Intel Xeon CPUs.
The larger 4B parameter model was trained on $8$ such nodes for 1,5 days, totaling $64\times$ NVIDIA A100 40GB GPUs.
In total, we conducted ${\approx}$150 training runs for the smaller model during the development stage of 4B generator.
Consequently, the project’s total compute utilization amounted to approximately 2700 NVIDIA A100 40GB GPU-days.

### 3.4 Dataset

We fine-tune the pre-trained text-to-video model (SnapVideo []), on RealEstate10K [].
The training split consists of roughly 65K video clips, and is the same as is used in concurrent work (MotionCtrl [] and CameraCtrl []).
We evaluate our method using 20 camera trajectories sampled from the RealEstate10K test split that were not seen during training for the user study. For the automated evaluations, we use the full test split.

### 3.5 Metrics

We conduct a user study to evaluate our approach for camera-controlled text-to-video generation.
The study participants are presented with 20 side-by-side comparisons between the proposed approach and the baselines as well as a reference video from RealEstate10K with the same trajectory to better judge the camera alignment.
We ask 20 participants for each generated video sequence to indicate which generated video they prefer based on multiple submetrics, namely, camera alignment, motion quality, text alignment, visual quality, and overall preference.
The user study involved negligible risk to the participants and was conducted with appropriate institutional review board and legal approval.
Participants were compensated 0.30 USD per question.

### 3.6 Baselines

We compare our work to the two concurrent works MotionCtrl [] and CameraCtrl [] by adapting their publicly released code to the SnapVideo model [].
Note that both baselines were originally designed for space-time disentangled U-Net video diffusion models.
Hence, their approaches are not directly applicable to the spatio-temporal transformers, and so we adapt them to this setting as follows.
For MotionCtrl, we omit their object motion control module and use their proposed camera motion control module to encode the camera parameters into a context vector used with the SnapVideo model.
We fine-tune both the camera motion control module and the cross attention between the latent vectors and the patches. We also train one version of MotionCtrl where we freeze the original base model to prevent quality degradation.
For CameraCtrl, we fine-tune the original camera encoder module and use this to produce the latent vectors in the SnapVideo model.
During fine-tuning the model weights are kept frozen—i.e., the same as in our proposed approach.

## 4 Experiments

### 4.1 Assessment

We provide a qualitative and quantitative assessment of our approach compared to the baselines in Fig. 4 and in Tab. 1. Following CameraCtrl [], we also evaluate the camera pose accuracy using ParticleSfM [] on generated videos in Tab. 2. We using generations for text prompts from RealEstate10K [] and MSR-VTT [], testing both in- and out-of-distribution prompts. Note that we adjust the CameraCtrl [] evaluation pipeline by normalizing all cameras into a unified scale as COLMAP provides different scales across different scenes. This prevents scenes with large scale to have a higher impact on the errors.
Please also refer to the supplementary webpage for additional video results.

In Fig. 4 we observe that adapting the camera conditioning method from the MotionCtrl degrades visual quality and text alignment, likely because this approach adjusts the weights of the base video model.
In the space-time U-Net for which this approach was proposed, the temporal layers can be fine-tuned without sacrificing visual fidelity.
Since spatio-temporal transformers do not decompose temporal and spatial attributes in the same way, the model overfits to the small dataset used to fine-tune the cross-attention layer.
While we observe some agreement with the camera poses used to condition the model, the text alignment is generally low in our experiments (see supplemental webpage).
In contrast, CameraCtrl keeps the pre-trained video model weights frozen and only trains a camera encoder.
This leads to strong visual quality, but the generated videos show little agreement with the input camera poses.
We hypothesize that this is due random initialization of the new temporal attention layers and convolutions.
For fair comparison, we trained all models for the same number of iterations (described in Sec. 3.3).
We believe that CameraCtrl requires substantially longer training time to converge in comparison to our ControlNet-inspired conditioning mechanism (see the Plucker conditioning module in Fig. 3), which is initialized using a copy of the cross-attention and feed-forward layer and fine-tuned.

The results of the user study in Tab. 1 show that most participants prefer the generated videos using the proposed camera conditioning mechanism across all evaluated sub-metrics.
We also observe a pronounced preference for the camera alignment of the proposed method compared to the other baselines. That is, 82% and 78% of participants prefer the camera alignment of the proposed method compared to our respective adaptations of MotionCtrl and CameraCtrl to the video transformer model.
All results are significant at the $p<0.001$ level as evaluated using a $\chi^{2}$ test.

Figure: Figure 4: Camera-conditioned text-to-video generation. Comparison of the proposed approach to MotionCtrl and CameraCtrl for the same camera trajectory input. MotionCtrl exhibits worse quality due to fine-tuning of existing layers and CameraCtrl is not sensitive to camera conditioning. See the supplementary webpage for video results.
Refer to caption: x4.png

**Table 1: Quantitative results. We compare our method to MotionCtrl and CameraCtrl implemented on the same base video models as ours. The methods are evaluated in a user study in which participants indicate their preference based on camera aligntment (CA), motion quality (MQ), text alignment (TA), visual quality (VQ), and overall preference (Overall). The percentages indicate preference for VD3D vs. the alternative method (in each row). All results are statistically significant with $p<0.001$ as evaluated using a $\chi^{2}$ test.**
|  | Human Preference |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Method | CA | MQ | TA | VQ | Overall |
| VD3D vs. MotionCtrl | 82% | 81% | 86% | 81% | 84% |
| VD3D vs. CameraCtrl | 78% | 64% | 63% | 65% | 66% |

**Table 2: Camera pose evaluation. We evaluate all models using reference camera trajectories from the RealEstate10K test set. We compute translation and rotation errors based on estimated camera poses from generations using ParticleSfM [].**
| Method | RealEstate10K | MSR-VTT |  |  |
| --- | --- | --- | --- | --- |
| TransError ($\downarrow$) | RotError ($\downarrow$) | TransError ($\downarrow$) | RotError ($\downarrow$) |  |
| Base Model | 0.616 | 0.207 | 0.717 | 0.216 |
| MotionCtrl | 0.518 | 0.161 | 0.627 | 0.148 |
| MotionCtrl (frozen) | 0.607 | 0.205 | 0.678 | 0.122 |
| CameraCtrl | 0.532 | 0.165 | 0.578 | 0.220 |
| Ours | 0.409 | 0.043 | 0.504 | 0.050 |
| w/o Plucker | 0.517 | 0.161 | 0.676 | 0.156 |
| w/o ControlNet | 0.573 | 0.182 | 0.787 | 0.179 |
| w/o weight copy | 0.424 | 0.044 | 0.513 | 0.063 |

### 4.2 Ablations

Plucker embedding. We motivate our Plucker embedding conditioning mechanism by training a variant using the raw camera matrices. Concretely, we flatten and concatenate extrinsics and intrinsics matrices in the channel dimension and repeat the values in the spatial patch dimensions. We observe that Plucker embeddings provide an essential spatial conditioning mechanism, as shown in Tab. 2.

ControlNet conditioning. Our ControlNet inspired conditioning mechanism ensures fast and precise learning of the conditioning signal distribution. Instead of using a ControlNet block, we simply add zero-initialized Plucker embedding features to the patches and observe close to no camera control. We observe training cross-attention layers in the ControlNet block is key to learning camera control while preserving the original model weights. This is confirmed by our camera evaluation in Tab. 2.

ControlNet weight copy. While the ControlNet block is essential, copying the pre-trained weights into the new copy has rather minor impact, as shown in Tab. 2. To verify this, we train a model where we randomly initialize the cross-attention block between patches and latents instead of copying the weights. We observe similar results, showing that rather the architecture and zero-initialization are the key component of this design.

### 4.3 Applications

Image-to-video generation. We synthesize camera-controlled videos based on different camera poses as shown in Fig. 1.
For this task we use a version of the SnapVideo model that we fine-tune on video sequences where a random subset of the input frames are masked.
At inference time, we can provide image guidance for any of the generated frames, providing an additional dimension of controllability when paired with our proposed method for camera control.
To demonstrate image-to-video generation in Fig. 1, we condition the model using camera poses along with image guidance from the first frame of a generated video sequence.
Note that our method provides no control over motion within the scene itself; hence, scene motion can differ depending on the random seed or the provided camera poses.

Image-to-multiview generation. We also explore multi-view generation for static scenes as shown in Fig. 5.
Given a real input image of a complex scene unseen during training, our camera-conditioned model generates view-consistent renderings of that scene from arbitrary viewpoints.
These multi-view renderings could be directly relevant to downstream 3D reconstructions pipelines, e.g., based on NeRF [] or 3D Gaussian Splatting [].
We show the potential of camera-conditioned image-to-multiview generation for complex 3D scene generation, but we leave more extensive exploration of this topic for future work.

Figure: Figure 5: Conditional multi-view generation on a real image. We can generate arbitrary camera trajectories from a given real image for multi-view synthesis, paving the way to single-image scene reconstruction using camera-controlled video models.
Refer to caption: x5.png

## 5 Conclusion

Large-scale video transformer models show immense promise to solve many long-standing challenges in computer vision, including novel-view synthesis, single-image 3D reconstruction, and text-conditioned scene synthesis.
Our work brings additional controllability to these models, enabling a user to specify the camera poses from which video frames are rendered.

Limitations and future work. There are several limitations to our work, which highlight important future research directions.
For example, while rendering static scenes from different camera viewpoints produces results that appear 3D consistent, dynamic scenes rendered from different camera viewpoints can have inconsistent motion (see supplemental videos).
We envision that future video generation models will have fine-grained control over both scene motion and camera motion to address this issue.
Further, our approach applies camera conditioning only to the low-resolution SnapVideo model and we keep their upsampler model frozen (i.e., without camera conditioning)—it may be possible to further improve camera control through joint training, though this brings additional architectural engineering and computational challenges.
Finally, our approach is currently limited to generation of relatively short videos (16 frames), based on the design and training scheme of the SnapVideo model.
Future work to address these limitations will enable new capabilities for applications in computer vision, visual effects, augmented and virtual reality, and beyond.

#### Broader impacts.

Recent video generation models demonstrate coherent, photorealistic synthesis of complex scenes—capabilities that are highly sought after for numerous applications across computer vision, graphics, and beyond.
Our key technical contributions relate to camera control of these models, which can be applied to a wide range of methods.
As with all generative models and technologies, underlying technologies can be misused by bad actors in unintended ways.
While these methods continue to improve, researchers and developers should continue to consider safeguards, such as output filtering, watermarking, access control, and others.

## Appendix A Related Work

#### 3D generation.

Limited to single category trainings, early work on 3D generation extends GANs into 3D using a neural renderer as an inductive bias [, , , , ],
Towards more diverse and flexible generation, CLIP-based supervision [] enabled text-based generation and editing of 3D assets [, , , , , ]. With recent advances in diffusion models, Score Distillation Sampling (SDS) replaces CLIP supervision with diffusion model supervision  [, , , , , , , , , , , , , ] for higher quality generation.
In order to improve the 3D structure of scenes, another line of work generates multiple views of a scene [, , , , , , , ]. Alternatively, other methods use iterative inpainting for 3D scene generation [, ].
Recent methods lift input images into 3D  [, , , , , , , , , ] using NeRF [], 3D Gaussian Splatting [], or Meshes in combination with diffusion models. Other recent methods [, , , , , , , , , , , , ] tackle fast feed-forward 3D generation, directly predicting a 3D generation from input images or text.
Different from our approach, these methods can only synthesize static scenes.

## Appendix B Quantitative evaluation

We further evaluate all models for the task of single image-to-multiview generation. Furthermore, we provide results for established 2D video generation metrics.

### B.1 Multi-view generation

We evaluate our model for image-to-multiview generation. Due to ground truth correspondences for the RealEstate10K [] dataset, we can condition our model on a given camera trajectory and assess image based metrics, i.e., PSNR, SSIM, and LPIPS. We provide results for the low-resolution base model and the upsampled high-resolution results in Tab. 3.

### B.2 Quality metrics

Moreover, we evaluate all models on established image and video generation metrics, namely, FID [], FVD [], and CLIPSIM []. We provide results for RealEstate10K [] and MSR-VTT [] in Tab. 4.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 8
presentation: 3
questions: - Have there been additional relevant experiments regarding camera trajectories, such as comparisons of control quality for generated trajectories of varying complexity?

At this time, I have no further questions.
soundness: 3
strengths: Camera control during the video generation process is a significant issue. As more foundational models adopt transformer architectures, exploring control mechanisms for these models becomes crucial. This paper is the first to investigate how to better utilize camera trajectory parameters for transformer-based video generation models, using SnapVideo as the foundational model. The design is well thought out, and the evaluation is rigorous. The strengths of the paper are as follows:

- Unlike the spatiotemporal decoupling generation of U-Net structures, transformer-based video generation considers spatiotemporal video tokens globally, which means it cannot directly leverage the advantages of spatiotemporal decoupling. This paper overcomes this limitation by being the first to explore control specifically for spatiotemporal transformers. This shift in foundational model structure is critical and provides a solid engineering foundation for future work.
  
- The authors use Plücker embeddings to convert camera intrinsic and extrinsic parameters into pixel-level controls, which match the shape of video tokens. This information is then introduced through read cross-attention layers. While this approach is a straightforward combination of existing methods, it has been validated as effective for transformer-based video generation models, providing valuable experimental insights.
  
- The paper includes comprehensive evaluations and ablation studies, conducting both qualitative and quantitative experiments regarding video content quality and camera control, with well-defined criteria. The evaluation of baseline models is fair, making the transition to the new foundational model structure more convincing.
summary: This paper presents a camera control method for transformer-based video generation models that enhances control while ensuring visual quality. The proposed approach aligns the video perspective with predefined camera trajectories, improving controllability. The authors claim this is the first study to employ ControlNet-like guidance for global spatiotemporal transformer video generation models, in contrast to the more commonly used U-Net architecture. Moreover, the evaluation demonstrates that both the video quality and adherence to the input camera trajectories are state-of-the-art.
weaknesses: - While camera control is the central problem addressed in this paper, the camera trajectories used primarily come from the RealEstate10K dataset, which, as observed in the visual results, mostly follow smooth, straight lines. There is a lack of consideration and experimentation with trajectories of varying difficulty, such as those involving significant directional changes. This raises some questions regarding the trajectory settings.
  
- There have been several prior works in the 3D multi-view generation field that focus on similar camera control issues, such as the referenced *Cat3D*, which also employed Plücker embeddings and attention calculations. The distinction between spatiotemporal decoupling and other network characteristics is a design feature intrinsic to the architecture. In exploring DiT-based generation, there have also been multiple studies investigating spatiotemporal decoupling, such as *Latte: Latent Diffusion Transformer for Video Generation*. Therefore, the novelty of this work lies more in applying existing designs to spatiotemporal transformers rather than presenting a technological innovation. However, the state-of-the-art results under the new configuration indeed serve as an important engineering reference for future directions.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Can the authors please comment on the above-mentioned weaknesses?
soundness: 2
strengths: - The proposed controlnet design outperforms other model variants designed by the authors. The evaluations are thoroughly conducted for the design choices. Detailed ablations are provided.
summary: The method proposes a controlnet-like architecture for a private video diffusion model by including plucker coordinates as camera control.
weaknesses: - The proposed framework overfits on the trajectories that are seen during training. Though the authors provide quantitative comparisons in Tab. 8, no visual comparisons are provided. 
- Though the performance is impressive, the technical contribution is limited in the proposed framework. Training a ControlNet for diffusion transformer is not new, as shown in [1]. Using Plucker coordinates for camera control is not new, as shown in CameraCtrl (He et al., 2024a).

[1] Chen J, Wu Y, Luo S, et al. Pixart-{\delta}: Fast and controllable image generation with latent consistency models[J]. arXiv preprint arXiv:2401.05252, 2024.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: Overall I think this paper provides an effective solution to amend video diffusion transformers with camera control. See weaknesses for questions and discussion.
soundness: 3
strengths: - paper is easy to follow
- the proposed design including the Plücker embedding is reasonable and effective
- comprehensive experiments are conducted and presented in the main manuscript and appendix
- supplemental materials contain video samples to demonstrate the effectiveness
summary: This paper provides a method for adding the control of camera movements to video diffusion transformers. The core idea is to represent camera conditions as pixel-level ray condition with Plucker embeddings. A ControlNet inspired module is used to process the camera condition. The model is finetuned on RealEstate10k and is compared to two similar methods both qualitatively and quantitatively.
weaknesses: - The proposed method has been evaluated only on one video diffusion transformer, which raises some concerns on whether its performance can generalize to other pretrained video diffusion transformers.
- I'm curious about the distribution of camera movements evaluated in the experiments, in terms of its diversity and similarity to natural camera movements.
- The novelty is slightly limited, as the task is not new, and ControlNet-like module as well as Plücker embedding have been explored and used before.

minor:
- CameraCtrl was appeared on arXiv in April 2024, which should not be considered as a concurrent work.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Nothing crucial.
soundness: 3
strengths: The paper presents a nice idea on to enable spatio-temporal transformer-based video diffusion models with camera control. It is the first to do so by using a ControlNet-like block in combination with Plucker coordinates, and presents a nice contribution. I value the ablation studies in the paper and the reasoning as to how they arrived at this particular architecture. The paper is quite well-written overall. It's easy enough to follow and appreciate the contribution.
summary: The paper proposes a first method to condition spatio-temporal transformer-based video diffusion models on camera poses (previous methods focused on pre-trained U-Net-based diffusion models). To this end, they propose a ControlNet-like block that conditions the model on camera embeddings that are based on Plucker coordinates. The paper evaluates the choices made and demonstrates good results, both qualitatively and quantitatively.
weaknesses: While I'm positive overall, there are a few weaknesses. 

A criticism that could be made is that this paper is a bit of an A+B paper, where A is the ControlNet block and B is the Plucker coordinates. I don't think that's a useful criticism, because at the end of the day it's a sensible thing to do and the authors demonstrate that you need to do a few things to make it work (Table 2).

The paper reads well overall, however, I'm not a fan of Figure 3. It's hard to interpret what goes where, and how this relates to the formulas. I'd suggest to clearly delineate what are the video patch tokens vs. the plucker patch tokens, maybe add the variables from the formulas to the appropriate boxes, and overall structure the figure so that the separate blocks are clearly separate.

Finally, in terms of results, I would have liked to see more examples of the same scenes with different camera control (there are only three examples). Furthermore, most examples use input camera trajectories from scenes that are completely unrelated to the target scene. It be nice to see some that are related -- makes it easier to judge if the generated camera path is good or not.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Line 272, R_1, t_1 can be interpreted as the original extrinsic or the extrinsic after normalization. It would be better to use R'_1, and t'_1 to represent the extrinsic after normalization;

2. I am not quite clear about the baseline implementation details. From my understanding, for MotionCtrl, one additional learned token is concatenated to the video patch tokens, while for CameraCtrl, the camera encoder produces additional latent tokens. Is that correct?

3. While previous works sacrifice motion dynamics when training on RealEstate10K (mostly static scenes), the video in the supplementary material exhibits better and larger motions. What would be the possible reasons for the difference?
soundness: 3
strengths: 1. The authors made an important step forward in camera control for the Transformer-based video diffusion model, which is unexplored by previous research;
2. The visual quality and motion dynamics of the generated videos are excellent;
3. The methodology is clearly explained and easy to understand.
summary: This paper presents a novel approach to incorporating camera movement control into Transformer-based video diffusion models. While camera control has been extensively studied in UNet-based video diffusion models, this area remains largely uncharted for Transformer architectures. The author introduces an additional ControlNet-like network to inject tokenized Plücker embeddings, demonstrating that this design enhances both visual quality and controllability.
weaknesses: 1. The main components of the method have been validated by previous works, for instance, the Plucker embedding as camera representation (CameraCtrl), and training on RealEstate10K(MotionCtrl, CameraCtrl);

2. As mentioned in 1., I would say the specified designs for Transformer architecture are the major contribution of the paper. However, SnapVideo's architecture is not a typical DiT (it has the "read" operation, and the attention is not performed on the actual tokens). It's not clear how to extend the proposed method for a standard video DiT and how the performance would be.

3. For the MotionCtrl baseline, the visual quality degrades when the base model is fine-tuned. Would it be better to freeze the backbone?

4. Would be better to also provide the trainable parameter scale of the two baselines and the proposed method.

Meta-review: This paper proposes a camera-control video diffusion model. The central claimed contribution is its transformer-based architecture. The method builds on SnapVideo and introduces a new attn-based conditional module for injecting the camera information. STOA performances are obtained compared to the existing approaches.

Reviewers highlighted the key advantages of the method, particularly the effectiveness of its transformer-based architecture and Plücker embedding conditioning, as evidenced by its strong experimental results and qualitative performance.

A common concern among reviewers was the limited novelty, noting that the main contribution involved replacing the linear layer in the condition block with attention layers, alongside some necessary design adjustments for Plücker embedding injection—an approach that also has been previously explored in camera-controllable video generation.

The AC acknowledged the solid contribution of the work, as validated by its experimental results, and recognized its value in advancing the field.

Decision: Accept (Poster)