CameraCtrl: Enabling Camera Control for Text-to-Video Generation

## Abstract

Abstract Controllability plays a crucial role in video generation since it allows users to create desired content.
However, existing models largely overlooked the precise control of camera pose that serves as a cinematic language to express deeper narrative nuances.
To alleviate this issue, we introduce CameraCtrl , enabling accurate camera pose control for text-to-video (T2V) models.
After precisely parameterizing the camera trajectory, a plug-and-play camera module is then trained on a T2V model, leaving others untouched.
Additionally, a comprehensive study on the effect of various datasets is also conducted, suggesting that videos with diverse camera distribution and similar appearances indeed enhance controllability and generalization.
Experimental results demonstrate the effectiveness of CameraCtrl in achieving precise and domain-adaptive camera control, marking a step forward in the pursuit of dynamic and customized video storytelling from textual and camera pose inputs. Our project website is at: https://hehao13.github.io/projects-CameraCtrl/ .

###### Keywords:

## 1 Introduction

Figure: Figure 1: Illustration of CameraCtrl. As a plug-and-play module, it can control the camera trajectory for both general text-to-video generation and personalized one, shown in the first three rows. Additionally, CameraCtrl can collaborate with visual controllers, such as the RGB encoder from SparseCtrl [] to generate videos given the first images while simultaneously managing camera movements, The condition image is in the bottom right corner of the first image in the last row.

Recently, diffusion models have significantly advanced video generation [, , , , ], revolutionizing digital content design workflows.
Controllability plays a crucial role in practical video generation applications, allowing for better customization according to user needs.
This enhances the quality, realism, and usability of the generated videos.
While text and image inputs are commonly used for achieving controllability, they may lack precise control over content and motion.
To address this, some approaches have been proposed, leveraging control signals such as optical flow [, ], pose skeleton [, ], and other multi-modal signals [, ], enabling more accurate control for guiding video generation.

However, existing models lack precise control over adjusting or simulating camera viewpoints in video generation.
The ability to control the camera is crucial not only for enhancing the realism of generated videos but also for increasing user engagement by allowing customized viewpoints.
This is particularly important in virtual reality, augmented reality, and game development.
Moreover, skillful management of camera movements enables creators to emphasize emotions, highlight character relationships, and guide the audience’s focus, which holds significant value in the film and advertising industries.
Recent efforts have been made to introduce camera control in video generation. For example, AnimateDiff [] incorporates a MotionLoRA module on top of its motion module, enabling specific types of camera movement.
Nevertheless, it struggles to generalize to camera trajectories customized by users.
MotionCtrl [] offers more flexible camera control by conditioning its text-to-video (T2V) model on a sequence of camera poses, but relying solely on numerical values without geometric cues of camera parameters may not ensure precise camera control.
Additionally, MotionCtrl [] lacks the capability to generalize camera control across other personalized video generation models.

We thus introduce CameraCtrl, learning a precise plug-and-play camera module that could control the camera viewpoints in video generation.
Considering that seamlessly integrating a customized camera into existing T2V models is challenging, we investigate how to represent and inject the camera effectively.
Concretely, we adopt plücker embeddings [] as the primary form of camera parameters.
This choice is attributed to their encoding of geometric interpretations for each pixel in a video frame, offering a comprehensive description of camera pose information.
To ensure the applicability and the generalizability of our CameraCtrl after training, we introduce a camera control module that only takes the plücker embedding as input, thus agnostic to the appearance of the training dataset.
To effectively train the camera control model, a comprehensive study is also conducted to investigate how various training data affect from the realistic data to synthetic data.
Experimental results suggest that data (*e.g.,* RealEstate10K []) with similar appearance to the original base model and diverse camera pose distribution achieves the best trade-off between generalizability and controllability.

We implement our CameraCtrl on top of AnimateDiff [], enabling precise camera control in video generation across various personalized ones, some of which are shown in the first three rows of Fig. 1. demonstrating its versatility and utility in a wide range of video creation contexts.
In addition, as shown in the last row of Fig. 1, it is also compatible with other plug-and-play module *e.g.,* SparseCtrl [] to control viewpoints under the image-to-video settings.

In summary, our main contributions are as follows:

- •
We introduce CameraCtrl, empowering video diffusion models with flexible and precise controllability over camera viewpoints.
- •
The plug-and-play camera control module can be adapted to various personalized video generation models, producing smooth and visually appealing camera control.
- •
We provide a comprehensive analysis of datasets to train the camera control module. We hope this will be helpful for future research in this direction.

## 2 Related Work

Text-to-video generation.
Recent attempts at text-to-video (T2V) generation [, , , , , ] mainly leverage diffusion models [, , ] for their stability in training and well-established open-sourced communities.
As a pioneer in this field, Video Diffusion Model [] expands a 2D image diffusion architecture to accommodate video data and jointly train the model on image and video from scratch.
To utilize the powerful pre-trained image generator such as Stable Diffusion [], later works inflate the 2D architecture by interleaving temporal layers between the pre-trained 2D layers and finetune the new model on large video dataset [].
Among them, Align-Your-Latents [] efficiently turns T2I into video generators by aligning independently sampled noise maps, while AnimateDiff [] utilizes a pluggable motion module to enable high-quality animation creation on personalized image backbones [].
To enhance temporal coherency, Lumiere [] replaces the commonly used temporal super-resolution module and directly generates full-frame-rate videos.
Other significant attempts include leveraging scalable transformer backbone [], operating in space-temporal compressed latent space, *e.g.*, W.A.L.T. [] and Sora [], and using discrete token with language model for video generation [].

Controllable video generation.
The ambiguity of solely used text descriptions often leads to weak control for text-to-video models.
To provide enhanced guidance, some work with precise signals, *e.g.*, depth/skeleton sequence, to precisely control the scene/human motion in the synthesized videos [, , , , , ].
Other methods [, ] adopt the images as the control signal to the video generator, contributing to high video quality or accurate temporal relationship modeling.

This paper focuses on camera control during the video generation process. AnimateDiff [] adopts efficient LoRA [] finetuning to obtain model weights specified for different shot types.
Direct-a-Video [] proposes a camera embedder to control the camera pose during the video generation process, but it only conditions on three camera parameters, which limits its camera control ability to the most basic types, like pan left.
MotionCtrl [] designs a motion controller that takes more camera parameters as input and produces the corresponding videos with more complex camera poses. However, the necessity to fine-tune part of the video diffusion model’s parameter can hamper its generalization ability.
In this study, we aim to control the camera poses during the video generation process precisely, and expect the corresponding camera control model can be used in various personalized video generation models.

## 3 CameraCtrl

Introducing precise control of the camera into existing video generation methods is challenging, but holds significant value in terms of achieving desired results.
To accomplish this, we address the problem by considering three key questions:
(1) How can we effectively represent the camera condition to reflect the geometric movement in 3D space?
(2) How can we seamlessly inject the camera condition into existing video generators without compromising frame quality and temporal consistency?
(3) What type of training data should be utilized to ensure proper model training?

This section is thus organized as follows: Sec. 3.1 presents the preliminary of video generation models; Sec. 3.2 introduces the camera representation; Sec. 3.3 presents the camera model $\mathrm{\Phi}_{c}$ for injecting camera representation into text-to-video (T2V) models. The data selection method is discussed in Sec. 3.4.

### 3.1 Preliminary of Video Generation

Text-to-video diffusion models. Text-to-video (T2V) diffusion models have seen significant advancements in recent years.
Some approaches [, ] train video generators from scratch, while others [, ] utilize powerful text-to-image (T2I) diffusion models.
These models often adhere to the original formulation used for image generation.
Concretely, a sequence of $N$ images (or their latent features) $z_{0}^{1:N}$ are first added noises $\epsilon$ gradually to normal distribution at $T$ steps. Given the noised input $z_{t}^{1:N}$, a neural network $\hat{\epsilon}_{\theta}$ is thus trained to predict the added noises.
During the training, the network tries to minimize the mean squared error (MSE) between its prediction and the ground truth noise scale; the training objective function is formulated as follows:

$$ ${\mathcal{L}}(\theta)=\mathbb{E}_{z_{0}^{1:N},\epsilon,c_{t},t}[\|\epsilon- \hat{\epsilon}_{\theta}(z_{t}^{1:N},c_{t},t)],$ (1) $$

where $c_{t}$ represents embeddings of the corresponding text prompts.

Controllable text-to-video generation.
In the realm of text-to-video generation, there have been further advancements in enhancing controllability. By incorporating additional structural control signals $s_{t}$ (*e.g.*, depth maps and canny maps) into the process, controllability for both image and video generation can be enhanced. Typically, these control signals are first fed into an additional encoder $\mathrm{\Phi}_{s}$ and then injected into the generator through various operations [, , ]. Consequently, the objective of training this encoder can be defined as follows:

$$ ${\mathcal{L}}(\theta)=\mathbb{E}_{z_{0}^{1:N},\epsilon,c_{t},s_{t},t}[\| \epsilon-\hat{\epsilon}_{\theta}(z_{t}^{1:N},c_{t},\mathrm{\Phi}_{s}(s_{t}),t)].$ (2) $$

In this work, we strictly follow this objective to train our camera encoder $\mathrm{\Phi}_{c}$.

### 3.2 Representing Camera Condition Effectively

Before diving into the training of the camera encoder, we first investigate which kind of camera representation could precisely reflect the movement in 3D space.

Camera representation. Typically, the camera pose refers to the intrinsic and extrinsic parameters, denoted as $\mathbf{K}\in\mathbb{R}^{3\times 3}$ and $\mathbf{E}=[\mathbf{R};\mathbf{t}]$, respectively, where $\mathbf{R}\in\mathbb{R}^{3\times 3}$ representations the rotation part of the extrinsic parameters, and $\mathbf{t}\in\mathbb{R}^{3\times 1}$ is the translation part.

To let a video generator condition on camera pose, one straightforward choice is to feed raw values regarding the camera parameters into the generators. However, such a choice may not contribute to accurate camera control for several reasons: (1) While the rotation matrix $\mathbf{R}$ is constrained by orthogonality, the translation vector $\mathbf{t}$ is typically unconstrained in magnitude, leading to a mismatch in the learning process that can affect the consistency of control. (2) Direct use of raw camera parameters makes it difficult for the model to correlate these values with image pixels, limiting precise control over visual details.
We thus choose plücker embeddings [] as the camera pose representation.
Specifically, for each pixel $(u,v)$ in the image coordinate space, its plücker embedding is $\mathbf{p}_{u,v}=(\mathbf{o}\times\mathbf{d}_{u,v},\mathbf{d}_{u,v})\in\mathbb
{R}^{6}$. Where $\mathbf{o}\in\mathbb{R}^{3}$ is the camera center in world coordinate space, which equals to $\mathbf{t}$, and $\mathbf{d}_{u,v}\in\mathbb{R}^{3}$ is a direction vector in world coordinate space from the camera center to the pixel $(u,v)$, it is calculated as,

$$ $\mathbf{d}_{u,v}=\mathbf{RK^{-1}}[u,v,1]^{T}+\mathbf{t}.$ (3) $$

Then, it is normalized to ensure it has a unit length. For the $i$-th frame in a video sequence, its plücker embedding can be expressed as $\mathbf{P}_{i}\in\mathbb{R}^{6\times h\times w}$, where $h$ and $w$ are the height and width for the frame.

Note that Eq. 3 represents the inverse process of camera projection, which maps point from the 3D world coordinate space into the pixel coordinate system through the use of matrices $\mathbf{E}$ and $\mathbf{K}$. Thus, the plücker embedding has some geometric interpretation for each pixel of a video frame and can provide a more elaborate description of camera pose information. Besides, the value ranges of each item in the plücker embedding are more uniform, which is beneficial for the learning process of the model. After obtaining the plücker embedding $\mathbf{P}_{i}$ for the camera pose of the $i$-th frame, we represent the entire camera trajectory of a video as a plücker embedding sequence $\mathbf{P}\in\mathbb{R}^{n\times 6\times h\times w}$, where $n$ denotes the total number of frames in the video.

Figure: Figure 2: Framework of CameraCtrl. (a) Given a pre-trained T2V model (*e.g.* AnimateDiff []), CameraCtrl trains a camera encoder on it. The camera encoder takes the plücker embedding as input and outputs multi-scale camera representations. These features are then integrated into the temporal attention layers of U-Net at their respective scales to control the video generation process. (b) Details of the camera injection process. The camera features $c_{t}$ and the latent features $z_{t}$ are first combined through the element-wise addition. A learnable linear layer is adopted to further fuse two representations which are then fed into the first temporal attention layer of each temporal block.

### 3.3 Introducing Camera Controllability into Video Generators

As camera trajectory is parameterized into the plücker embedding sequence *i.e.,* spatial maps, we could follow the prior literature [, ] by first using an encoder model to extract the camera features and then fusing the camera features into video generators.

Camera encoder.
Similar to T2I-Adaptor [], we introduce a camera encoder $\mathrm{\Phi}_{c}$ specifically designed for videos. This camera encoder includes a temporal attention module after each convolutional block, allowing it to capture the temporal relationships of camera poses throughout the video clip.
As is shown in Fig. 2(a), $\mathrm{\Phi}_{c}$ only takes the plücker embeddings as input and delivers multi-scale features.
Through empirical analysis, we observed that using camera conditions along with latent codes, like ControlNet [], may result in information leakage from the training data, thereby limiting its generalization across various domains.

Camera fusion.
After obtaining the multi-scale camera features, we aim to integrate these features seamlessly into the U-Net architecture of the T2V model. Thus, we further investigate the fusion of layers in the original video generator, aiming to determine which layers should be used to effectively incorporate the camera information.

Recall that current video generators usually adopt a U-Net-like architecture that contains both spatial and temporal attention.
We inject the camera representations into the temporal attention blocks. This decision stems from the capability of the temporal attention layer to capture temporal relationships, aligning with the inherent sequential and causal nature of a camera trajectory, while the spatial attention layers always picture the individual frames.
This camera fusion process is shown in  Fig. 2(b). The image latent features $z_{t}$ and the camera pose features $c_{t}$ are directly combined through pixel-wise addition. Then, this integrated feature is passed through a linear layer, whose output is fed directly into the first temporal attention layer of each motion module.

### 3.4 Learning Camera Distribution in Data-Driven Manner

Training the aforementioned camera encoder on a video generator usually requires a lot of videos with text and camera annotations.
One can obtain the camera trajectory through structure-from-motion (SfM) *e.g.,* COLMAP [] for realistic videos while others could collect videos with ground-truth camera from rendering engine, such as Blender.
We thus investigate the effect of various training data on the camera-controlled generator.

Dataset selection.
We aim to select a dataset with appearances that closely match the training data of the base T2V model and have as wide a camera pose distribution as possible. We choose three datasets as the candidates, they are Objaverse [], MVImageNet [], and RealEstate10K []. Samples from the three datasets can be found in the supplementary material.

Indeed, datasets generated from virtual engines such as Objaverse [] exhibit diverse camera distribution since we can control the camera parameters during the rendering process, while it often suffers from a distribution gap in appearance when compared to real-world datasets, such as WebVid-10M [] which is used to train our base T2V model.
When dealing with real-world datasets, such as MVImageNet and RealEstate10K, the distribution of camera parameters is often not very broad. In this case, a balance needs to be found between the complexity of individual camera trajectories and the diversity among multiple camera trajectories. The former ensures that the model learns to control complex trajectories during each training process, while the latter guarantees that the model does not overfit to certain fixed patterns. In reality, while the complexity of camera trajectories in MVImageNet may slightly exceed that of RealEstate10K for individual trajectories, the camera trajectories of MVImageNet are typically limited to horizontal rotations. In contrast, RealEstate10K showcases a wide variety of camera trajectories. Considering our goal to apply the model to a wide range of custom trajectories, we ultimately selected RealEstate10K as our training dataset. Besides, there are some other datasets with characteristics similar to RealEstate10K, such as ACID [] and MannequinChallenge [], but their data volume is much smaller than that of RealEstate10K.

Measuring the camera controllability.
To monitor the training process of our camera encoder, we propose the camera alignment metric to measure the camera control quality by quantifying the error between the input camera conditions and the camera trajectory of generated videos.
Concretely, we utilize the COLMAP [] to extract the camera pose sequence of generated videos,
which consists of the rotation matrixes $\mathbf{R}_{gen}\in\mathbb{R}^{n\times 3\times 3}$ and translation vectors $\mathbf{T}_{gen}\in\mathbb{R}^{n\times 3\times 1}$ of camera. Furthermore, since the rotation angles and the translation scales are two different mathematical quantities, we measure the angle and error translation errors separately and term them as RotErr and TransErr. Motivated from  [], the RotErr of a generated camera pose sequence is computed by comparing the ground truth rotation matrix $\mathbf{R}_{gt}$ and $\mathbf{R}_{gen}$, formulated as,

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mbox{{RotErr}}}=\sum_{j =1}^{n}\arccos{\frac{tr(\mathbf{R}_{gen}^{j}\mathbf{R}_{gt}^{j\mathrm{T}}))-1} {2}},$ (4) $$

where $\mathbf{R}_{gt}^{j}$ and $\mathbf{R}_{gen}^{j}$ represent the ground truth rotation matrix and generated rotation matrix for the $i$-th frame, respectively. And tr is the trace of a matrix.
To quantify the translation error, we use the $L2$ distances between the ground truth translation vector $\mathbf{T}_{gt}$ and $\mathbf{T}_{gen}$, that is,

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mbox{{TransErr}}}=\sum_ {j=1}^{n}\|\mathbf{T}_{gt}^{j}-\mathbf{T}_{gen}^{j}\|_{2},$ (5) $$

where $\mathbf{T}_{gt}^{j}$ and $\mathbf{T}_{gen}^{j}$ is the translation vector for the ground truth camera pose and generated camera pose in the $j$-th frame.

## 4 Experiments

In this session, we evaluate CameraCtrl with other methods and show its applications in different settings. Sec. 4.1 presents the implementation details. Sec. 4.2 compares CameraCtrl with other baseline methods AnimateDiff [] and MotionCtrl []. Sec. 4.3 shows the comprehensive ablation studies of CameraCtrl. Sec. 4.4 express the various applications of CameraCtrl.

### 4.1 Implementation details

T2V base model.
We choose the AnimateDiff V3 [] as the base text-to-video (T2V) model and implement our CameraCtrl on it.
Benefiting from the unique training strategy of AnimateDiff, its motion module can be integrated with various text-to-image (T2I) LoRAs, or T2I base models to accommodate the video generation across different domains and genres.
This feature helps us in evaluating the generalization ability of our CameraCtrl.

Training.
We use the Adam optimizer to train our model with a constant learning rate of 1$e^{-4}$.
As stated in Sec. 3.4, we choose RealEstate10K as the dataset, which has around 65$K$ video clips for training.
For the alignment with AnimateDiff V3, we train our camera control model at the resolution of 256 $\times$ 384.
We take the random horizontal flip as one of the argumentation methods to expand the camera poses. We train the camera encoder and the linear layers for camera feature injection together. We use 16 NVIDIA A100 GPUs to train them with a batch size of 2 per GPU for 50$K$ steps, taking about 1 day.

Evaluation metrics.
To ensure that our camera model does not negatively impact the video generation capabilities of the original T2V model, we utilize the Fréchet Inception Distance (FID) to assess the video appearance quality, comparing the generated video quality before and after incorporating the camera model.
For our reference videos, we have chosen 1,000 videos from the RealEstate10K test set.
Additionally, the quality of camera control is evaluated using the metrics RotErr and TransErr introduced in the Sec. 3.4.

**Table 1: Quantitative comparisons. Considering AnimateDiff [] only supports several basic trajectories (*e.g.,* zoom-in, pan left), we compare against prior work on basic trajectory and random trajectory respectively. TransErr and RotErr are reported as the metrics.**
| Method | Basic trajectory | Random trajectory |  |  |
| --- | --- | --- | --- | --- |
|  | TransErr ↓ | RotErr ↓ | TransErr ↓ | RotErr ↓ |
| AnimateDiff [] | 9.81 | 1.03 | - | - |
| MotionCtrl [] | 9.02 | 0.87 | 13.59 | 1.41 |
| CameraCtrl (Ours) | 8.83 | 0.95 | 12.91 | 1.35 |

### 4.2 Comparisons with other methods

To proof the effectiveness and the generalization ability of CameraCtrl, we compare it with another two camera control methods: AnimateDiff [], and MotionCtrl []. Despite Direct-a-Video [] can also be used to control camera pose during the video generation, its model and code are not open source.
Given that AnimateDiff supports only eight basic camera trajectories, the comparison between our method and AnimateDiff is limited to these basic trajectories.
In contrast, for comparisons with MotionCtrl, besides base camera trajectories, we select 1,000 random camera trajectories from the RealEstate10K test set, generate videos using these trajectories along with their corresponding captions, and subsequently evaluate them using RotErr and TransErr. Notability, CameraCtrl and the MotionCtrl utilize different T2V base models, evaluating the quality of appearance using metrics like FID may not yield accurate comparisons. Therefore, we do not compare these metrics.

The quantitative results are shown in Tab. 1. Compared to MotionCtrl and the MotionLoRA AnimateDiff, it is evident that our approach outperforms Animatediff in basic trajectory and MotionCtrl in complex trajectory.

### 4.3 Ablation study

We break down the camera control problem into three challenges, regarding the selection of camera representation in Sec. 3.2, the architecture of camera control model in Sec. 3.3, and the learning process of camera control model in Sec. 3.4. In this session, we comprehensively ablate the design choices to each of them. The rotation error RotErr and translation error TransErr are evaluated using the 1,000 complex camera trajectories.
Besides, to assess the impact of our camera control model to video appearance quality, we measure FID using the original AnimateDiff V3 model, yielding a baseline score 102.3. This score provides a reference for video quality comparison after introducing camera control.

**Table 2: Ablation study on camera representation, condition injection and effect of various datasets.**
| Representation type | FID↓ | TransErr ↓ | RotErr ↓ |  |
| --- | --- | --- | --- | --- |
| Raw Values | 109.8 | 14.01 | 1.41 |  |
| Raw Values + Plücker embedding | 104.3 | 16.98 | 1.45 |  |
| Plücker embedding | 100.4 | 12.91 | 1.25 |  |

Plücker embedding represents camera precisely.
Beside plücker embedding to represent the camera parameters, we could directly use the numerical values of camera parameters or a combination of both as a hybrid representations. For the direct use approach, we repeat the 16 camera parameters of each video frame along the spatial dimension. In the hybrid approach, we first repeat the 16 parameters with the aforementioned method then concatenate it with the plücker embedding along the channel dimension. The experimental results are illustrated in LABEL:table:ablation_cam_rep, using the plücker embedding as the camera representation yields the best camera control results, the FID metric is very similar with the reference value 102.2, yielding no loss of appearance. These quantitative results further confirm our intuition in Sec. 3.2 that plücker embedding enables more precise camera control. This accuracy stems from plücker embedding’s ability to provide a geometric interpretation for every pixel. Incorporating original numerical values directly with plücker embedding could compromise this geometric interpretation. Moreover, relying solely on numerical values might lead to numerical mismatches, adversely affecting the camera model’s learning efficiency.

Noised latents as input limit the generalization.
In ablating the architecture of camera encoders, we differentiate between ControlNet [], whose input is the summation of image features and plücker embedding, and the T2I-Adaptor, solely using plücker embedding as input. This distinction is crucial as the use of noised latent, mentioned in SparseCtrl [], has been associated with appearance leakage, effectively limiting the generalization capability of the model. Besides, to enhance inter-frame camera coherence, we also consider adding a temporal attention block to each encoder. Thus, Our experiment covers four configurations: ControlNet, T2I-Adaptor, and their temporal attention-enhanced variants.

Figure: Figure 3: Effect of camera encoder architectures. Row(a) to row(d) represent the results generated with the camera encoder as ControlNet, ControlNet with temporal attention, T2I-Adaptor, T2I-Adaptor with temporal attention, respectively. All the videos are generated using the same text prompt and camera trajectory (zoom-in).

In this ablation study, we use a personalized T2I backbone ToonYou[] to generate cartoon characters.
According to the Fig. 3, with ControlNet as the camera encoder, the appearance quality is suboptimal. Considering that the ControlNet takes the rgb images as additional input, it may learn some bias towards controlling T2V model to generate contents similar to the training data. This phenomenon is contrary to our objective of creating a camera control model versatile enough to be applicable across various video themes.
For the models utilizing the T2I-Adaptor, it is observable that the model with additional temporal attention module exhibits more obvious camera movement, aligning more closely with the intended camera trajectory. Therefore, we chose the T2I-Adaptor with temporal attention module as our camera encoder.

Injecting camera condition into temporal attention.
We then investigate where the derived camera features should be inserted within the pre-trained U-Net architecture. We conduct three experiments to insert the features into the spatial self attention, spatial cross attention, and temporal attention layers of the U-Net, respectively. The results are presented in the LABEL:tab:abla_camera_feat_inj, indicate that inserting camera features into the temporal attention layers yields better outcomes. This improvement could be attributed to the fact that camera motion typically induces global view changes across frames. Integrating camera poses with the temporal blocks of the LVDM (Latent Video Diffusion Model) resonates with this dynamic nature, thereby enhancing control over camera movements throughout video generation.

Videos with similar appearance distribution and diverse camera help controllability.
To test our argument on dataset selection as discussed in  Sec. 3.4, we selected three representative datasets for our experiments. The Objaverse [] dataset, with camera poses derived from a rendered engine, has the widest distribution of camera poses but has a significantly different appearance from WebVid-10M.
For the real-world datasets, compared to MVImageNet, RealEstate10K possesses a more diverse range of camera trajectories.

The results, as shown in the table LABEL:tab:abla_diff_data, displayed that compared to RealEstate10K, both FID scores and camera errors are significantly higher with MVImageNet. For Objaverse, COLMAP struggles to extract a sufficient number of camera poses to yield meaningful camera error metrics. One possible explanation of this result is that the difference in the dataset appearance may prevent the model from effectively distinguishing between camera pose and appearance, leading to lower video quality (as indicated by poorer FID scores) and inaccurate COLMAP results. Consequently, we ultimately choose for RealEstate10K.

Figure: Figure 4: Introducing CameraCtrl into text-to-video generation. The first two rows represent the videos generated by the general T2V model. The following three rows showcase the results of a personalized generator RealisticVision []. The video of the last row is produced by another personalized model ToonYou [].

### 4.4 Application of CameraCtrl

Apply CameraCtrl to different domains of video generation.
As detailed in Sec. 3.3, our camera control model exclusively uses plücker embedding as input, making it independent of the appearance of the training dataset Besides, as mentioned in Sec. 3.4, we select a dataset with an appearance closely resembling that of the training data of the base T2V model.
Benefiting from these design choices, our camera control model can focus solely on learning camera control-related information. This enables its application across various video domains. We demonstrate this versatility with three distinct video domains, results are shown in the Fig. 4. The first and second rows depict natural scenes, illustrating how our method can generate videos using different camera trajectories based on the narrative intent of the same scene. Rows three to five showcase the scenes with styles divergent from the typical reality, such as the buildings of a cyberpunk city, the nature landscape in a black and white video, and coastlines with uncommon color schemes during the stormy weather. The last row represents a video of a cartoon character. Across these varied video generation types, CameraCtrl consistently demonstrates effective control over the camera trajectories, showcasing its broad applicability and effectiveness in enhancing video narratives through dynamic camera trajectory control. These results further prove the correctness of our design choices.

Integrate CameraCtrl with other video control methods.
Thanks to the plug-and-play nature of our method, not only can it be used during the generation processes of different personalized videos, but it can also be integrated with other video generation control techniques together to produce videos.
In this session, we utilized SparseCtrl [], a recent approach that controls the overall video generation by manipulating a few sparse frames. This control can be based on RGB images, sketch maps, or depths. Here, we adopt the RGB encoder and sketch encoder of SparseCtrl, results are shown in Fig. 5. The result of RGB encoder is presented in the first row. It has a high level of consistency between the object rabbit in the generated video and that in the reference frame, and can be conditioned on a given camera trajectory. The results in the second and third rows utilize the sketch encoder of SparseCtrl. It is evident that, based on the provided sketch maps and camera trajectories, the model is capable of generating videos with content that is both reasonable and aligns with the specified camera movements. The successful integration with SparseCtrl further demonstrates the generalization capabilities of CameraCtrl and enhances its application prospects.

Figure: Figure 5: Intergating CameraCtrl into controllable text-to-video generation. The first row displaces video generated through the combination use of SparseCtrl’s RGB encoder and our method. The last two rows showcase videos produced with the sketch encoder of SparseCtrl alongside our method. Condition images of SparseCtrl are shown in the bottom right corners of the first images for each row. All control signals from SparseCtrl are assigned to the first image.

## 5 Conclusion and Discussion

In this work, we present CameraCtrl, a method that addresses the limitations of existing models in precise camera control for video generation.
By learning a plug-and-play camera module, CameraCtrl enables accurate control over camera viewpoints.
Plücker embeddings are adopted as the primary representation of camera parameters, providing a comprehensive description of camera pose information by encoding geometric interpretations.
Through a comprehensive study on training data, it is found that using data with similar appearance to the base model and diverse camera pose distributions, such as RealEstate10K, achieves the best trade-off between generalizability and controllability.
Experimental results demonstrate its effectiveness in enhancing the realism of generated videos and enabling customized experiences.
Combined with other existing tools, we believe that CameraCtrl will inspire further research in the field and contribute to the ongoing evolution of content generation workflows.

Limitations and future work.
Although CameraCtrl has already achieved appealing control of camera, there remains several potential directions to explore. For instance, the generalization of camera control heavily replies on the diversity of training videos. Current choice could meet the demand to some extent while collecting more videos with more comlicated camera movement could further improve the controllability. In addition, AnimateDiff [] provides more than one generator, *i.e.,* a family of text-to-video generators where we evaluate CameraCtrl. However, these different generators including others (*e.g.,* VideoCrafter [], SVD []) all adopt U-Net like architecture that combines convolution and attention operations.
Regarding transformer generator (*e.g.,* Sora []), we believe CameraCtrl, as a plug-in module, could be also compatible, which we leave in future to explore.
CameraCtrl: Enabling Camera Control for Text-to-Video Generation
Supplementary Material

Hao He Yinghao Xu Yuwei Guo
Gordon Wetzstein Bo Dai Hongsheng Li Ceyuan Yang

This supplementary material provides more discussions on data selection, implementation details, additional ablation experiment, and more visual results.

In the visual results, the first image in each row represents the camera trajectory of a video. Each small tetrahedron on this image represents the position and orientation of the camera for one video frame. Its vertex stands for the camera location, while the base represents the imaging plane of the camera. The red arrows indicate the movement of camera position but do not depict the camera rotation. The camera rotation can be observed through the orientation of the tetrahedrons. For a clearer understanding of the camera control effects, we highly recommend that readers watch the videos provided in our supplementary file.

The organization of this supplementary material is as follows: Sec. 6 presents more discussions on the dataset selection process. Sec. 7 gives more implementation details. Sec. 8 depicts one extra ablation study on model architecture. Finally, more visualization results are showcased in Sec. 9.

## 6 More Discussions on Dataset Selection

When selecting the dataset for training our camera control model,
we choose three datasets as candidates, Objaverse [], MVImageNet [], and RealEstate10K [].

For the Objaverse dataset, its images are rendered with software like Blender, enabling highly complex camera poses. However, as seen in row one to row three of Fig. 6, its content mainly focuses on objects against white backgrounds. In contrast, the training data for many text-to-video (T2V) models, such as WebVid-10M [], encompasses both objects and scenes against more intricate backgrounds. This notable difference in appearance can detract from the model’s ability to concentrate solely on learning camera control.
For MVImageNet data, it has some backgrounds and complex individual camera trajectories. Nevertheless, as demonstrated in the row four to row six of Fig. 6, most of the camera trajectories in the MVImageNet are horizontal rotations. Thus, its camera trajectories lack diversity, which could lead the model to learn a fixed pattern.
Regarding RealEstate10K data, as shown in the row seven to row nine of Fig. 6, it features both indoor and outdoor scenes and objects. Besides, each camera trajectory in RealEstate10K is complex and there exists a considerable variety among different camera trajectories. Therefore, we choose the RealEstate10K dataset to train our camera control model.

Figure: Figure 6: Samples of different datasets. Row one to row three are samples from Objaverse dataset, which has random camera poses for each rendered image. Row four to row six show the samples from the MVImageNet dataset. Samples of the RealEstate10K dataset are presented from row seven to row nine.

## 7 More Implementation Details

Training. We use the LAVIS [] to generate the text prompts for each video clip of the RealEstate10K dataset. For each training sample, we sample 16 images from one video clip with the sample stride equal to 8, then resize their resolution to 256 $\times$ 384. After that, we use the random horizontal flip for both images and poses with a 50 percent probability. We use a linear beta schedule, where $\beta_{start}$ = 0.00085, $\beta_{end}$ = 0.012, and $T$ = 1000. To let the camera control model better focus on learning camera poses, similar to AnimateDiff [], we first train an image LoRA on the images of RealEstate10K dataset. Then, based on the T2V model enhanced with LoRA, we train the camera control model. Note that, after the camera control model is trained, the image LoRA can be removed.
Inference. By utilizing structure-from-motion methods such as COLMAP [], along with existing videos, we can extract the camera trajectory within a video. This extracted camera trajectory can then be fed into our camera control model to generate videos with similar camera movements. Additionally, we can also design custom camera trajectories to produce videos with desired camera movement. During the inference, we use different guidance scales for different domains’ videos and adopt a constant denoise step 25 for all the videos.

## 8 Extra Ablation Study

Injecting camera features into both encoder and decoder of U-Net. In the vanilla T2I-Adaptor [], the extracted control features are only fed into the encoder of U-Net. In this part, we explore whether injecting the camera features to both U-Net encoder and decoder could result in performance improvements. The experiment results are shown in Tab. 3.
The improvements of TransErr and RotErr indicate that, compared to only sending camera features to U-Net encoder, injecting the camera features to both the encoder and decoder enhances camera control accuracy.
This result could be attributed to the fact that similar to text embedding, the plücker embedding inherently lacks structural information. Such that, this integrating choice allows the U-Net model to leverage camera features more effectively.
Therefore, we ultimately choose to feed the camera features to both the encoder and decoder of the U-Net.

**Table 3: Ablation study of the camera feature injection place.**
| Injection Place | FID↓ | TransErr ↓ | RotErr ↓ |
| --- | --- | --- | --- |
| U-Net Encoder | 99.0 | 13.82 | 1.45 |
| U-Net Encoder + Decoder | 100.4 | 12.91 | 1.25 |

## 9 More Visualization results

This section provides additional visualization results of CameraCtrl. All the results are generated using the same camera control model.

### 9.1 Visualization results of various domain videos

Figure: Figure 7: RealEstate10K visual results. The video generation results of CameraCtrl. The control camera trajectories and captions are both from RealEstate10K test set.

Visual results of RealEstate10K domain. First, with the aforementioned image LoRA model trained on RealEstate10K dataset, and using captions and camera trajectories from RealEstate10K, CameraCtrl is capable of generating videos within the RealEstate10K domain. Results are shown in Fig. 7, the camera movement in generated videos closely follows the control camera poses, and the generated contents are also aligned with the text prompts.

Figure: Figure 8: Using CameraCtrl on the same caption and different camera trajectories. The camera control results of CameraCtrl. Camera trajectories are from RealEstate10K test set, all videos utilize the same text prompts.

Figure: Figure 9: Visual results of natural objects and scenes. The natural video generation results of CameraCtrl. CameraCtrl can be used to control the camera poses during the video generation process of natural objects and scenes.

Visual results of original T2V model domain. We choose the AnimateDiff V3 [] as our video generation base model, which is trained on the WebVid-10M dataset. Without the RealEstate10K image LoRA, CameraCtrl can be used to control the camera poses during the video generation of natural objects and scenes. As shown in Fig. 8, with the same text prompts, taking different camera trajectories as input, CameraCtrl can generate almost the same scene, and closely follows the camera trajectories. Besides, Fig. 9 shows more visual results of natural objects and scenes.

Figure: Figure 10: Visual results of stylized objects and scenes. With the personalized generator RealisticVision [], CameraCtrl can be used in the video generation process of stylized videos.

Figure: Figure 11: Visual results of cartoon characters. With the personalized generator ToonYou [], CameraCtrl can be used in the video generation process of cartoon character videos.

Visual results of some personalized video domain. By replacing the image generator backbone of T2V model with some personalized generator, CameraCtrl can be used to control the camera poses in the personalized videos. With the personalized generator RealisticVision [], Fig. 10 showcases the results of some stylized objects and scenes, like some uncommon color schemes in the landscape and coastline. Besides, with another personalized generator ToonYou [], CameraCtrl can be used in the cartoon character video generation process. Some results are shown in Fig. 11. Note that, in both domains, the camera trajectories in the generated videos closely follow the control camera poses.

Figure: Figure 12: Integrating CameraCtrl with other video generation control methods. Row one to row three express the results by integrating the CameraCtrl with RGB encoder of SparseCtrl [], and row four to row six, shows videos produced with the sketch encoder of SparseCtrl. The condition RGB images and sketch maps are shown in the bottom right corners of the second images for each row. Note that, the camera trajectory of the last row is zoom-in.

### 9.2 Integrating CameraCtrl with other video control method

Fig. 12 gives some generated results by integrating the CameraCtrl with another video control method SparseCtrl []. The content of the generated videos follows the input RGB image or sketch map closely, while the camera trajectories of the videos also effectively align with the conditioned camera trajectories.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Please see the weakness part.
soundness: 2
strengths: The motivation is clear.

The results appear promising and solid.

The experiments are thorough.

The writing is easy to follow.
summary: CameraCtrl enhances video generation by enabling precise camera pose control in video diffusion models, preserving narrative nuances through cinematic language. This approach integrates a plug-and-play camera pose control module and demonstrates improved controllability and generalization, with experimental results showcasing its effectiveness across different models.
weaknesses: Applying a controlnet-like network to incorporate control signals into video diffusion isn’t particularly novel, though the use of Plücker embeddings adds a degree of originality.

Since the RealEstate10K dataset mainly features static videos with few moving objects, could this lack of motion introduce a bias that undermines the base model’s natural grasp of object motion? The foreground objects in the examples provided show limited movement.

There is a need for comparisons and discussion with recent camera control methods, such as VD3D.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: * Why are static scene video datasets mainly used without general video datasets? (For example, one could use general video datasets for training along with estimated camera trajectories from various models like particleSFM.)
* This didn't affect my rating, but I'm curious about the authors' opinions on whether this method can be generally applied to DiT-based methods that do not separate temporal and spatial aggregation.
soundness: 3
strengths: * This is one of the early works that enables adding camera control to existing video models. I believe it will have a positive impact on various downstream tasks.
* While the way of achieving camera control itself isn't technically very novel, the paper presents an intuitive design and provides extensive experimental comparisons on various conceivable design choices (e.g., where to inject the control and different camera representations).
* It is interesting that camera control can be achieved with a lightweight additional adapter without further tuning the video model.
summary: This paper proposes a plug-and-play module for control of camera pose. This module can be integrated to various video diffusion models. This module leverages Plücker embeddings to encode camera movements, offering detailed spatial representation compared to traditional numerical values (e.g., extrinsic matrix).
weaknesses: * Since it is fine-tuned on static scene data, it tends to show somewhat reduced ability in generating dynamic motions compared to the original video diffusion baseline (as shown in ODD). But, it shows better performance compared to other methods.
* In object-centric video generation, it appears that the movement of the object is minimal compared to the background. Ideally, models fine-tuned on Objaverse or MVImgNet should be able to generate camera trajectories covering 360 degrees of the object. I'm curious if this model can effectively generate videos when provided with such camera trajectories as input (e.g., generating a video from a frame showing the front of a person to one showing the back).

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - The choice of the Plucker representation appears effective when compared to Euler angles, quaternions and raw values, though some properties of the Plucker representation appear suboptimal for the purpose of achieving camera control. In particular, the set of points along a certain camera ray will have a constant embedding. Sitzmann (2021) introduces the use of Plucker coordinates in the context of LFNs, for which this property is well justified and beneficial. In the context of the current paper, though, Plucker embeddings appear to obscure the camera origin, a fundamental information for the model. With each Plucker embedding defining a set of points along a ray, the only way for the model to recover the camera origin is to solve a system of equations involving multiple rays. This is a necessary, but likely difficult task for the model to perform. Why is this property desirable for this work? Did the authors experiment with alternative representations not suffering from this drawback such as (o, d) or (normalized_o, d) (for some choice of normalization for o) which would make the camera origin readily available to the model?
soundness: 3
strengths: - The problem treated in the paper is of high significance to video generation, and 3D/4D generation.
- The paper is very clearly written. Every section of the paper was easy to follow and it appears the information in the paper would be sufficient to replicate the presented method and results. In addition, code is available.
- I appreciate the simplicity of the method. The authors identified the most crucial components to enable camera control: use of Plucker embeddings, camera encoder architecture, how to couple the encoder to the retrained backbone, and training data recipe. Without adding bells and whistles, the authors show good camera control performance.
- Evaluation is solid and informative. Every of the most critical design dimensions (use of Plucker embeddings, camera encoder architecture, how to couple the encoder to the retrained backbone, and training data recipe) is validated in a respective ablation, providing useful insights. The model is demonstrated on diverse video generation backbones
- Qualitative evaluation is rich. Limitations are shown and highlighted
summary: The authors explore the problem of conditioning a pretrained video generation model on a desired camera motion. The problem is of high relevance not only to the video generation community, but also in 3D and 4D generation. The authors propose the use of a frozen video generation backbone, coupled with a camera encoder block, injecting camera information into the backbone. The method is simple, sound, clearly written and accompanied by a comprehensive evaluation validating the importance of each of the main components: camera embeddings, encoder architecture, feature injection, and training datasets.
weaknesses: - Discussion of the camera representation could be strengthened by an accompanying figure.
- While Plucker embeddings were found more effective than a set of straightforward alternative representations, their properties do not appear to be completely aligned to the nature of the proposed approach. See questions.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: * How is scale handled by your model? For example, what is the meaning/effect of translating the camera by, say, one unit to the right? What about two units?

* How do you make sense of the camera motion if you don't know what is going to be generated in the video?

* Does the choice of focal length (camera intrinsics) affect the generated video?

* Can you comment on the potential issues on the evaluation discussed above?
soundness: 2
strengths: * The problem considered by the paper is practically important: camera control is an important feature in many potential applications of video generation

* The approach is technically simple. The architecture design is reasonably well motivated. The performance is shown to be better than close competitors.

* Several applications beyond text-to-video generation with camera control are demonstrated, showing that the method successfully combines with other control types.
summary: The paper introduces a general-purpose encode for a moving camera that can be used to add camera conditioning to different video generator architectures. The paper considers three technical problems: representing the camera, injecting the representation in a video generator, and developing training data sufficient for learning the model. The camera encoding is based on Plucker rays, it is injected in a video generator via cross-attention layers added to the temporal self-attention of the video generator, and the training data combines synthetic and real-life datasets of static contest where the camera is known.

Empirically, the new module is integrated AnimateDiff and in and Stable Video Diffusion (SVD), two different video generators prompted using text. Compared to alternative like MotionCtrl, the approach has better metrics in terms of camera reconstruction and image quality.
weaknesses: I remain unconvinced about the formulation of the problem (which is a limitation shared by prior works).

The paper aims at controlling camera motion accurately, but camera motion can only be interpreted in *relation* to the content generated in the video, and this relationship is not well defined in this work. This has several manifestations, discussed next.

First, there is the issue of scale: how a particular value for the camera translation should be interpreted depends on the scale of the generated scene, which is unknown and/or arbitrary. For example, a translation of one meter to our right is entirely negligible if we are looking at the sun, but if we are looking at the chair in front of us it is sufficient to take the object out of view entirely. There is no indication on whether the model is learning a "metric" (or canonical) interpretation of scale; this seems in fact unlikely given the nature of the training datasets which do not posses a well defined scale. The appendix briefly mentions scale as an issue of COLMAP, which requires scale normalisation for evaluation. But this is not a minor issue compared to the  much more serious concern that we cannot assume that the scale of the training data is know. Even if it has "some" value, the latter is often arbitrary and inconsistent (many real-life datasets like MVImageNet use COLMAP for camera annotation; synthetic ones like Objaverse have arbitrary scale).

Even if the training data was "metric", so that one could at least attempt to learn a meaningful canonical notion of scale, and thus interpret camera translation accordingly, this would still be difficult to do for a model. This is a reflection o the fact that very similar content can exist at different physical scales in the world (e.g., model of a city vs city), so learning  a canonical reference scale, insofar as one can be defined, is difficult.

Second, along with scale there is the issue of framing and aligning content and camera motion. Suppose you wanted to move the camera around an object. You could specify a camera trajectory that pivots around a given arbitrary physical point in front of you, in a turn-table-like fashion. Whether the network decides to place an object in correspondence of the pivot point, thus obtaining a turn-table-like effect, is not clear. Camera motion is only useful when its relation to the content is meaningful, so that the subject of the video is framed in the intended way.

Thirdly, there is the issue of the intrinsic parameters of the camera. It is not clear how these are handled in the paper. Specifically, the Plucker ray should be able to encode any camera model/focal length, but it is not obvious whether the authors train the model with different sets of cameras, and if the model is accurate enough to capture focal length correctly. Note that focal length would affect the apparent size of an object in an image, so this also has a strong effect in attempting to learn a canonical/metric interpretation of scale, which seems necessary in this formulation of the problem (see above).

Note that the issue of scale trickles down to evaluation. Even if the content of the video is not well matched to the camera motion (i.e., the framing and/or scale is not what one would expect), the model can still generate a video with the prescribed camera motion. The latter can then recovered using, e.g., COLMAP and compared to the input camera up to scale (for the translation component). The latter is described in appendix, but evaluation of camera reconstruction has been considered extensively int he literature; why not using one of the many standard scale-invariant metrics instead of proposing a new one (line 990)?

More worryingly, the authors state (line 983) that they manually discard badly generated videos where COLMAP fails in their evaluation (line 983). This can very well introduce a selection bias, where their method is favoured by simply discarding difficult videos more liberally than for others. While it is difficult to compare video generators directly due to the aleatoric nature of their task, nevertheless the authors should consider a metric that would at least remove this problem. For example, they could measure success rate, as in the percentage of generated videos where the camera is reconstructed at different levels of accuracy, generating corresponding curves. By prompting different models with the same textual prompts and the same camera motions, such curves would at least be more comparable. 

The novelty of the paper is reasonable, but not outstanding. A very similar problem is considered in MotionCtrl and the paper is mostly an iteration that innovates on the three technical components discussed above (better camera encoding, better camera injection, and better data). There is nothing wrong with it, and the results are a bit better, but the conceptual innovation does not run very deep.

# Minor issues

* Line 195: the sentence structure is broken

* Line 460: Do you mean that COLMAP cannot reconstruct the generated videos? Is this an issue with COLAMP or with the quality of generation?

Meta-review: (a) Claims and Findings

This paper targets the challenge of controlling camera pose in text-to-video (t2v) generation models. It introduces the use of Plücker embeddings for camera trajectory representation and proposes an adaptor-like module that can encode camera information independent of appearance. Experimental evaluations on two t2v models demonstrate the effectiveness of this approach.

(b) Strengths

The reviewers find that the paper addresses an important problem in text-to-video generation. The paper is well written, and the proposed method is both simple and effective. The extensive experiments and ablation studies provide strong empirical evidence supporting the method’s effectiveness.

(c) Weakness
* The technical novelty is limited, where the main distinction with prior works come from adopting Plücker embedding as the camera representation.
* The properties of Plücker embedding may not perfectly align with the intended camera control task
* The datasets used in the experiments are biased toward static scene and limits the dynamics in resulting videos
* A fundamental issue in the problem formulation, as pointed out by reviewer UQ23, remains unresolved

(d) Reasons for decision

On the positive side, the paper addresses an important problem, is well presented, and offers a method that is straightforward and supported by substantial empirical evidence. On the negative side, however, the paper lacks technical novelty, and the fundamental concerns about the problem formulation remain unaddressed. If one accepts the problem as currently framed (following prior works), the paper is well written and provide solid improvement for camera control in text-to-video generation model and meets the acceptance criteria. However, the AC agrees with reviewer UQ23 that a more thoughtful and principled problem formulation is strongly desired.

Decision: Accept (Poster)