Training-free Camera Control for Video Generation

## Abstract

Abstract We propose a training-free and robust solution to offer camera movement control for off-the-shelf video diffusion models. Unlike previous work, our method does not require any supervised finetuning
on camera-annotated datasets or self-supervised training via data augmentation. Instead, it can be plugged and played with most pretrained video diffusion models and generate camera controllable videos with a single image or text prompt as input. The inspiration of our work comes from the layout prior that intermediate latents hold towards generated results, thus rearranging noisy pixels in them will make output content reallocated as well. As camera move could also be seen as a kind of pixel rearrangement caused by perspective change, videos could be reorganized following specific camera motion if their noisy latents change accordingly.
Established on this,
we propose our method CamTrol , which enables robust camera control for video diffusion models. It is achieved by a two-stage process. First, we model image layout rearrangement through explicit camera movement in 3D point cloud space. Second, we generate videos with camera motion using layout prior of noisy latents formed by a series of rearranged images. Extensive experiments have demonstrated the robustness our method holds in controlling camera motion of generated videos.
Furthermore, we show that our method can produce impressive results in generating 3D rotation videos with dynamic content.
Project page at https://lifedecoder.github.io/CamTrol/ .

## 1 Introduction

Figure: Figure 1: Training-free video camera control by CamTrol. CamTrol could handle basic camera motions, hybrid motions and complicated trajectories with precise camera coordinates. Besides, CamTrol produces impressive results on generating 3D rotation videos in various styles. Note that all human faces presented in this paper are synthesized using text-to-image models.
Refer to caption: x1.png

As a more appealing and content-richer modality, videos differ from images by including an extra temporal dimension. This temporal aspect provides increased versatility for depicting diverse and dynamic movements, which can be decomposed into object motion, background transitions and perspective changes.
Recent years have witnessed the rapid development and splendid breakthrough of video generation with text prompt or images as input instructions ; ; ; ; ; ; ; ; , and
demonstrated the inestimable potential of diffusion models to synthesis realistic videos.
While these video generation models have made progress in generating videos with highly dynamic objects and backgrounds ; ; , most of them fail to provide camera control for the generated videos.

The difficulty of controlling camera trajectory in videos primarily arises from two aspects. The initial challenge lies in the inadequacy of annotated data. Most video annotations lack of descriptions, especially precise descriptions of video’s camera movements. As a result, video generation models trained on these data often fail to interpret text prompts related to camera motions and generate correct outputs. One solution to mitigate the data insufficiency problem is to mimic videos with camera movements through simple data augmentation . However, these methods could only handle simple camera motions like zoom or truck, and have trouble in dealing with more complicated ones.
The second challenge lies in the effort of additional finetuning required for controlling camera movements. As camera trajectories could be sophisticated, they sometimes cannot be accurately elaborated using naïve text prompts alone.
Common solutions ; proposed to embed camera parameters into diffusion models through learnable encoders and perform extensive finetuning on large-scale datasets with detailed camera trajectories. However, such datasets like RealEstate10k and MVImageNet are intensively limited in scale and diversity due to the difficulty associated with data collection, in this way, these finetuning methods demand substantial resources but exhibit limited generalizability to other types of data.
Lack of annotations and heavy finetuning effort make camera control a challenging task in video generations.

In this work, we attempt to address these issues through a training-free solution to offer camera control for off-the-shelf video diffusion models. We begin by introducing two core observations underpin that video diffusion models can achieve camera movement control in a training-free manner.
First, we find that base video models could produce results with rough camera moves by integrating specific camera-related text into input prompts, such as camera zooms in or camera pans right. This simple implementation, though not very accurate and always leads to static or wrong motions, shows the natural prior knowledge learnt by pretrained models about following different camera trajectories.
The other observation is the effectiveness video models have exhibited in adapting to 3D generation tasks. Recent works ; ; find that leveraging pretrained video models as initialization helps drastically improve the performance of multi-view generations, demonstrating their strong ability of handling perspective change.
The two crucial observations reveals the hidden power of video models for camera motion control, thus, we seek to find a way to evoke this innate ability, as it already exists in the model itself.

We propose CamTrol, which offers camera control for off-the-shelf video diffusion models in a training-free but robust manner.
CamTrol is inspired by the layout prior that noisy latents hold towards generation results: As pixels in noisy latents change their positions, corresponding rearrangement will also occur to the output and leads to layout modification. Considering camera moves could also be seen as a kind of layout rearrangement, this prior can serve as an efficacious hint providing video model with information of specific camera motions. Specifically, CamTrol consists of a two-stage procedure. In stage I, explicit camera movements are modeled in 3D point cloud representations and produce a series of rendered images indicating specific camera movements. In stage II, layout prior of noisy latents are utilized to guide video generations with camera movements.
We conduct extensive experiments to validate the effectiveness of our proposed CamTrol. Both quantitative and qualitative results demonstrate the robustness of CamTrol as a useful tool of controlling camera motion for video diffusion models. Furthermore, we show that CamTrol  produces impressive results of dynamic 3D rotation videos in various styles.

## 2 Related Work

#### Camera Control for Video Generation

While methods aim for controlling video foundation models constantly emerge ; ; , there are few works explore how to manipulate camera motion of generated videos. Earlier work controls motion trajectory via warping image through densified sparse flow and pixel fusion, similar ideas also appear later in ; . Besides utilizing optical flow, two main techniques for implementing video camera control are via self-supervised augmentation or additional finetuning.
disentangles object motion with camera movement and incorporates extra layers to embed camera motions, where model is trained in a self-supervised manner by augmenting input videos to stimulate simple camera movements. and train an additional camera encoder and integrate the output into temporal attention layers of U-Net. learns new motion pattern via LoRA and finetuning with multiple reference videos.

#### Noise Prior of Latents in Diffusion Model

One of the most natural advantages of diffusion model comes from its pixel-wise noisy latents formed during denoising process.
These latents hold strong causuality towards output and directly determine what the result looks like, meanwhile have robust error-resilience as they are perturbed by Gaussian noises across different scales.
Numerous work have exploited the convenience of this noise prior to attain controllable generation, such as image-to-image translation , pixel-level manipulation ; , image inpainting and semantic editing ; ; . Recent research has shown that even sampled from Gaussian distribution, the initial noise of diffusion process still have significantly influence to the layout of generated contents .
In other work, noise prior is used to guarantee temporal consistency among video frames , or to trade-off between fidelity and diversity of image editing .

#### Video Model for 3D Generation

Similar to how most video generation models using the groundwork laid by image foundation models ; ; ; , training of 3D generation model also relies heavily on pretrained 2D video models ; ; ; ; . These methods either finetunes with rendered videos directly ; ; ; , or adds camera embedding for each view as extra condition ; . Video foundation models have shown to be particularly beneficial in generating consistent multi-view rendering of 3D objects, demonstrates their inherent abundant prior knowledge for handling camera pose change.

## 3 Training-free Camera Control for Video Generation

CamTrol takes two stages to evoke the innate camera control ability hidden in base video models. In Sec. 3.1, we will describe how to model explicit camera motion in point cloud space. In Sec. 3.2, we will elaborate on the camera-controllable video generation with the guidance of noise layout prior.

Figure: Figure 2: Pipeline of CamTrol. In stage I, camera movements are modeled through explicit 3D point cloud . In stage II, layout prior of noisy latents are utilized to guide video generation.
Refer to caption: x2.png

### 3.1 Camera Motion Modeling

To evoke pretrained video diffusion model’s ability of dealing with camera perspective changes, hints of camera motion should be injected to diffusion model in a proper way.
While simply concatenating camera trajectories with text prompt is incomprehensible for original model, previous works ; introduce additional embedder to encode camera parameters and finetune with limited annotated data ; , which are data-hungry yet lack of generalization ability. Other methods construct camera motions by self-supervised augmentations, but could only handle a few easy camera controls. Thus, we seek a more efficient and robust way to guide the model towards camera controllable.

Considering perspective change of video is originally caused by camera movements in 3D space, we resort to 3D representation for providing explicit motion hints to video diffusion models. Specifically, we choose point cloud as the intermediate representation, in which space we can expediently manipulate camera poses and positions for simulating diverse camera movements. Besides explicit camera modeling, introducing point cloud brings extra benefits: The first is its data-efficiency. By utilizing inpainting techniques, only one single input image is required for the whole point cloud reconstruction, this sidesteps the effort of large-scale finetuning. Second, consistency between multi-view renderings can be easily ensured, as the known points will remain unchanged once the reconstruction finished.

#### Point Cloud Initialization

We start by lifting the pixels in input image plane to 3D point cloud representations. In practice, the input image can be either user-defined or created by image generators like Stable Diffusion . Given an input image $\mathbf{I}_{0}\in\mathbb{R}^{3\times H\times W}$, we first estimate its depth map $\mathbf{D}_{0}$ using off-the-shelf monocular depth estimator ZeoDepth . By combining image and its depth map, point cloud $\mathcal{P}_{0}$ can be initialized as:

$$ $\mathcal{P}_{0}=\phi([\mathbf{I}_{0},\mathbf{D}_{0}],\mathbf{K},\mathbf{P}_{0}),$ (1) $$

where $\phi$ denotes the mapping function from RGBD to 3D point cloud, $\mathbf{K}$ and $\mathbf{P}_{0}$ represent camera’s intrinsic and extrinsic matrices set by convention as they’re usually intractable.

#### Camera Trajectories

To get consistent images from multiple viewpoints, we set camera motion as a pre-defined trajectory of extrinsic matrix $\left\{\mathbf{P}_{1},...,\mathbf{P}_{N-1}\right\}$, each of which includes a rotation matrix and translation matrix representing camera’s pose and position.
At each step $i$, we project the point cloud back to camera plane using function $\psi$ and get a rendered image with perspective change: $\mathbf{I}_{i}=\psi(\mathcal{P}_{i},\mathbf{K},\mathbf{P}_{i})$. By calculating extrinsic matrices of corresponding movement, we obtain a series of camera motions including zoom, tilt, pan, pedestal, truck, roll and rotate, enabling flexible camera movements. Detailed definitions of these movements are elaborated in Appendix A.1. By combining basic trajectories, hybrid camera movements can be attained and produce videos with cinematic charm. What’s more, benefit from explicit camera motion modeling, our method could support trajectories with precise coordinates, which means it can generate videos with any complicated camera motion(Fig. 1).

#### Multi-view Rendering

When perspective changes, there can be vacancies appear as some areas are unoccupied within the point cloud. To get more reasonable results, we employ image inpainting model to fill up the holes for new renderings, with a mask distinguishing the known points from nonexistent ones. After inpainted in 2D space, image is lifted again onto 3D space and gradually complete the whole point cloud representation. During this process, misalignment between adjacent views may occur since depth estimator only estimates relative depth, further leads to inconsistency of rendered images. We adopt depth coefficient optimization to avoid this situation, which can be formed as:

$$ $d_{i}=\operatorname*{argmin}_{d}\left(\sum_{M}\left\|\phi([\tilde{\mathbf{I}}_ {i},d\tilde{\mathbf{D}}_{i}],\mathbf{K},\mathbf{P}_{i})-\mathcal{P}_{i-1} \right\|\right),$ (2) $$

where $\tilde{\mathbf{I}}_{i}$ and $\tilde{\mathbf{D}}_{i}$ refer to the inpainted image and its depth map respectively, $d_{i}$ denotes depth coefficient to be optimized, and $M$ refers to the overlapping region between $\mathcal{P}_{i}$ and $\mathcal{P}_{i-1}$, as other areas are not shared for calculating $\ell_{1}$ loss.

Thus, we get a set of images refer to the input and indicate specific camera movement:

$$ $\left\{\mathbf{I}_{0},...,\mathbf{I}_{N-1}\right\}=\left\{\psi(\mathcal{P}_{i} ,\mathbf{K},\mathbf{P}_{i})|i\in\left[0,N-1\right]\right\}.$ (3) $$

### 3.2 Layout Prior of Noise

With camera motion modeling, we obtain a sequence $\mathbf{V}_{0}=\left[\mathbf{I}_{0},...,\mathbf{I}_{N-1}\right]\in\mathbb{R}^{
N\times 3\times H\times W}$ of rendered images adhering to a specific camera trajectory. Note that quality of rendered images are not perfect as single input image only leads to sparse point cloud reconstruction, besides, these renderings are static, thus they could not use directly as video frames. To form an ideal video, we need to find a way that satisfies the following three requirements: 1) camera motions should be maintained; 2) video should be encouraged with more dynamics; and 3) quality imperfection should be compensated.

#### Camera Motion Inversion

Recent work on diffusion models have demonstrated the strong controllability of its noisy latents ; , the causality and error-resilience they hold towards final output make them a convenient yet powerful tool for controllable generation of diffusion models.
Particularly for initial noise, even sampled from Gaussian distribution, it still have significant influence on the layout of generated image , so that rearranging the noise pixels will make content in output relocate as well. For instance, if all pixels in initial noise shift to right by a certain distance, it is likely that generated output reflect a similar shift. This reminds us that the impact of camera movement on images could also be regarded as a kind of layout rearrangement, where pixels change their positions caused by viewpoint change. In a similar way, videos can be reorganized following camera motion if their noisy latents change accordingly.

Inspired by this,
we first construct a series of noisy latents indicating corresponding camera movements.
It can be intuitively done by employing diffusion model’s inversion process on rendered image sequence $\mathbf{V}_{0}$. Latent at timestep $t_{0}$ can be calculated as:

$$ $\mathbf{V}_{t_{0}}=\sqrt{\bar{\alpha}_{t_{0}}}\mathbf{V}_{0}+\sqrt{1-\bar{ \alpha}_{t_{0}}}\epsilon,\quad\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I}),$ (4) $$

where $\bar{\alpha}_{t}$ are variances used in DDIM scheduler. Because the rendered images $\mathbf{V}_{0}$ share common pixels in certain regions, their latents also have relevance to each other in a way indicating pixels’ move. Moreover, while being perturbed with random noise, blank spaces and flawed regions in $\mathbf{V}_{0}$ can be further filled with randomness,
providing video model with more possibilities to generate and correct them.

#### Video Generation

After camera motion inversion, noisy latents presenting camera movements are then passed through the backward process of video diffusion model, utilizing their layout controllability to guide video generation. Leveraging prior knowledge of base video model, the generation process also bestows video with rational dynamic information. In this way, explicit camera movements are injected into video diffusion model in an appropriate and training-free fashion.
Starting from noisy motion latents at timestep $t_{0}$, the generation step can be represented as:

$$ $\hat{\mathbf{V}}_{t-1}=\sqrt{\alpha_{t-1}}\left(\frac{\mathbf{V}_{t}-\sqrt{1- \alpha_{t}}\epsilon^{(t)}_{\theta}(\mathbf{V}_{t})}{\sqrt{\alpha_{t}}}\right)+ \sqrt{1-\alpha_{t-1}-\sigma^{2}_{t}}\epsilon^{(t)}_{\theta}(\mathbf{V}_{t})+ \sigma_{t}\epsilon,\quad t\in\left[1,t_{0}\right].$ (5) $$

Here we use DDIM as sampling scheduler and $\epsilon_{\theta}$ denotes the video model for noise prediction. $\sigma_{t}$ determines whether the denoising process is deterministic or probabilistic, we set $\sigma=1$ to encourage diversity of generation results.

#### Trade-off Between Fidelity and Diversity

Leveraging noise prior guidance in diffusion model could lead to trade-off problem between generation’s fidelity and diversity ; ; , where results that hold more faithfulness towards guidance tend to decline in generation quality or diversity. In this task, similar circumstance also occurs as model is required to receive guidance from a series of imperfect renderings while generate a reasonable and dynamic video. The key factor to balance the trade-off problem lies in the choice of $t_{0}$. When larger $t_{0}$ applied, generation bears more resemblance to original guidance $\mathbf{V}_{0}$, yet lacks of rationality and dynamics to be an appealing video. Instead, smaller $t_{0}$ leads to well-generated video, but is less aligned with desired camera motion. In our experiments, we find larger $t_{0}$ works better for motions with moderate intensity, and for those with relatively drastic move, smaller $t_{0}$ shows preferable performance.

## 4 Experiments

### 4.1 Experimental Settings

#### Implementation Details

The major results presented in this paper are grounded on . To ensure a fair comparison with other state-of-the-art methods, we employ as base model for quantitative evaluation.
Inversion and generation steps among all methods are set to 25. Our method doesn’t require any additional training utilizing camera trajectories.

#### Evaluation Details

We compare CamTrol with state-of-the-art works: AnimateDiff , MotionCtrl and CameraCtrl , all three methods are finetune on SVD using specific data. In quantitative evaluation, FVD , FID and IS are used to assess video generation quality, while CLIPSIM quantifies the similarity between generated video and input prompt. With regard to the accuracy of camera motion, we adopt ParticleSFM and produce estimated camera trajectories from generated videos, with the use of Absolute Trajectory Error(ATE) measuring their differences compared to ground truth. Relative Pose Error(RPE) is calculated to assess between consecutive frames how well the relative motions match expected ones including their transition(RPE-T) and rotation part(RPE-R). Settings of evaluation dataset follow those established in MotionCtrl , but include more complex trajectories. Specifically, we extract 51 distinct trajectories from RealEstate10k, each paired with 10 prompts mentioned in MotionCtrl , resulting in 510 samples in total for assessment. As AnimateDiff lacks the ability to handle complicated trajectories, it is only included in qualitative analysis part evaluating with 8 basic camera motions. Quantitative comparisons contain AnimateDiff can be found in Appendix A.3.

### 4.2 Comparisons with State-of-the-art Methods

#### Quantitative Evaluation

Quantitative evaluations are shown in Table  1. Building upon the same base model as MotionCtrl and CameraCtrl , our method demonstrates superior performance across all quantitative metrics concerning both video quality and camera motion accuracy. Despite being training-free, CamTrol outperforms those that rely heavily on extensive finetuning and large-scale annotated data, attaining the lowest score in ATE, RPE-T and RPE-R. Benefit from explicit camera movements modeling and motion inversion, our method is capable to produce videos align best with those tricky trajectories while maintaining their visual qualities.

**Table 1: Quantitative comparisons.**
| Method | Video Quality | Motion Accuracy |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| FVD<br>$\downarrow$ | FID<br>$\downarrow$ | IS<br>$\uparrow$ | CLIP-SIM<br>$\uparrow$ | ATE<br>$\downarrow$ | RPE-T<br>$\downarrow$ | RPE-R<br>$\downarrow$ |  |
| MotionCtrl | 3576.40 | 239.10 | 7.58 | 0.2933 | 4.006 | 1.086 | 0.106 |
| CameraCtrl | 2922.99 | 243.98 | 8.07 | 0.2915 | 4.200 | 1.487 | 0.080 |
| SVD+CamTrol | 2832.59 | 227.36 | 8.07 | 0.3100 | 3.917 | 0.947 | 0.017 |

Figure: Figure 3: Qualitative comparisons with finetuned methods. CamTrol generates videos that adhere to desired camera movements, without compromising their dynamics and rationality.
Refer to caption: x3.png

#### Qualitative Analysis

Qualitative comparisons of different methods are illustrated in Fig. 3. All methods generate videos align well with desired camera motion, however, the constraints imposed by camera trajectories are sometimes too strong, that output videos have to compromise their quality and rationality to satisfy these movements. Some examples that demonstrate this are results of Zoom Out and Roll Anticlockwise, which looks more like flat images being zoomed or rolled on 2D plane instead of videos recorded in 3D spaces. The reason might lies on the limitations of proprietary dataset used for finetuning ; , where most scenes are static, making pretrained model forget about knowledge of dynamic contents.
In contrast, CamTrol makes full use of the powerful dynamics in base model and generate plausible videos with camera perspective change utilizing layout prior guidance, handles these circumstances gracefully.
More comparisons at Appendix A.3.

### 4.3 Ablation Study

Figure: Figure 4: Comparison with base model. Controlling camera motion via prompt engineering doesn’t work at most times. Instead, CamTrol offers robust control towards video’s camera movement in a training-free manner.
Refer to caption: x4.png

Figure: Figure 5: Effectiveness of layout prior. Layout prior guidance compensates for the vacancies(Left) and static contents(Right) caused by point cloud rendering.
Refer to caption: x5.png

#### Comparison to Base Model

To demonstrate that changes of camera motion are attributed to our method rather than the innate capability of video model, we conduct ablation study to assess its effectiveness. We add prompts describing certain camera moves(e.g. zooms out), letting video model understand by itself. The results are shown in Fig. 4. It could be observed that even provided with prompts indicating how camera should move, base model fail to produce correct results. Instead, CamTrol  is able to implement designated motion control without any instructions from text prompts.

#### Effectiveness of Layout Prior

We employ ablation study to validate the effectiveness of layout prior guidance, illustrating its necessity from two aspects: the completeness of vacancies and dynamic of generated video. In Fig. 5, we showcase frames before and after noise prior guidance. With camera pose changes, there appears regions unfilled in point cloud and causes blank spaces in rendered images(left part); Besides, due to the static nature of point cloud, rendered images remain stationary(right part). Noise layout prior could compensate for these flaws, finally produce videos with inpainted vacancies and rationalized dynamics.

#### Effect of Timestep t 0 subscript 𝑡 0 t_{0} italic_t start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT

$t_{0}$ is a crucial factor that influences the trade-off between generated video’s diversity and its faithfulness to camera motions requirements. To investigate its effect on the output, we conduct experiments with various $t_{0}$ values, relevant results are shown in Fig. 7. As illustrated, videos generated with larger $t_{0}$ tend to conform better to camera motion requirements, but suffer from decrease in dynamics; On the contrary, smaller $t_{0}$ leads to more plausible generations but fails to meet camera’s requirements, as latents at these timesteps carry more randomness.

#### Generalization to Diverse Base Models

Our proposed CamTrol can be seamlessly plugged and played with most pretrained video diffusion models to achieve training-free camera control. We present visual results showcasing its applications based on different video base models, including SVD and VideoFusion , in Fig. 8. Our approach remains effective applied to alternative video base models, demonstrating its strong robustness and generalization ability. More results with other base models can be found in Appendix A.5.

### 4.4 Further Applications

Figure: Figure 6: Effect of $t_{0}$. Smaller $t_{0}$ encourages dynamics while larger $t_{0}$ preserves camera movements (Pedestal Down).
Refer to caption: x6.png

Figure: Figure 8: Applied onto SVD and VideoFusion . CamTrol could be plugged and played with most video diffusion models to offer camera movement control.
Refer to caption: x8.png

#### 3D Rotation Videos

One of the most advantages of our method is it can generate videos rotating around some objects and produce outputs similar to 3D generation models ; . While these 3D models need large-scale training on 3D dataset and could only handle inputs in specific styles, our approach is able to deal with any type of images and achieve this in a completely zero-shot manner. Some results are shown in 1. We offer more results on multi-view synthesis in Appendix A.2.

#### Hybrid and Complex Camera Movements

By combining different basic camera trajectories, CamTrol can support hybrid camera movements and endow generated video with cinematic charm. Besides this, explicit motion modeling also equips CamTrol with the abilities to support trajectories containing precise coordinates, which means it can generate videos presenting any complicated camera movement. Results about hybrid and complex motions are shown in Fig. 1 and Appendix A.2.

#### Camera Motion at Different Scales

CamTrol supports camera movements at controllable scales. We provide some results in Fig. 7. By specifying different magnitudes of camera’s extrinsic matrix within point cloud spaces, rendered images will exhibit varying degrees of motion, leading to videos with distinct scales of camera movements. This further demonstrate the powerful controllability of CamTrol, and provides a new pathway for video’s customized camera control.

## 5 Conclusion

In this paper, we propose a training-free and robust method CamTrol to offer camera control for off-the-shelf video diffusion models. It consists of two-stage procedure including explicit camera motion modeling in 3D point cloud space and video generation utilizing layout prior of noisy latents.
Compared to previours work, CamTrol  does not require any additional finetuning on camera-annotated datasets, or self-supervised training via data augmentation, instead, it could be plugged and played with most video diffusion models to generate camera controllable videos with single image or text prompt as input.
Comprehensive experimental results demonstrate the effectiveness and robustness of CamTrol for controlling camera motion for videos. Besides, we show that CamTrol produces impressive results in generating videos with 3D rotations, complicated trajectories, and different moving scales.

## Appendix A Appendix

### A.1 Definitions of Basic Camera Motions

We refer to the terminology in cinematography to describe different camera motions, definitions of each type are detailed in Table 2.

**Table 2: Definitions of basic camera motions.**
| Camera Motion | Directions | Definition |
| --- | --- | --- |
| Zoom | In | Camera moves towards or away from a subject. |
| Out |  |  |
| Tilt | Up | Rotating the camera vertically from a fixed position. |
| Down |  |  |
| Pan | Left | Rotating the camera horizontally from a fixed position. |
| Right |  |  |
| Pedestal | Up | Moving a camera vertically in its entirety. |
| Down |  |  |
| Truck | Left | Moving a camera horizontally in its entirety. |
| Right |  |  |
| Roll | Clockwise | Rotating a camera in its entirety in a horizontal manner. |
| Anticlockwise |  |  |
| Rotate | Clockwise | Moving a camera around a subject. |
| Anticlockwise |  |  |
| Hybrid | Arbitrary | Combination of other motions. |

### A.2 More Results on Camera Control

In this section, we showcase additional qualitative results of CamTrol on various camera movements. Fig. 9 shows video frames with camera move Rotate Anticlockwise and Rotate Clockwise. These outputs share sort of similarity with outputs of 3D generation models, as they all exhibit in a turning-table like way, which camera rotates around some objects. The difference here is that 3D model, as only trained on specific datasets, could only generate outputs in certain styles, e.g., single static object with no background. Instead, our model could handle arbitrary image as input, and generate a rotating-around video with proper dynamics. From this aspect, our method could be seen as a infinite source of attaining 3D data. And by utilizing our method with stronger backbones, video foundation models could truly become the largest source of 3D data as it should be.

Besides, we provide additional results on hybrid motions and basic motions, which is illustrated in Fig. 10, Fig. 11 and Fig. 13, respectively. We also present pre-loaded complex camera motions in Fig. 12.

### A.3 More Comparisons to State-of-the-Art Methods

More comparison results with state-of-the-art methods are shown in Fig. 14 and Fig. 15. Compared to other works, our method matches well with input texts and camera requirements, meanwhile has wide-ranging diversities on generated contents.

We also present quantitative comparisons that include AnimateDiff as part of the evaluation. As AnimateDiff lacks the ability to handle complicated trajectories, experiments are conducted focusing on 8 basic trajectories described in A.1. To expand the diversity of the models evaluated, we employ CamTrol on for assessment. With regard to basic trajectories, we apply UCF-101 as the reference of calculating FID and FVD, from where we randomly sample 125 prompts per camera motion type and form 1000 samples in total for assessment. Relevant results are exhibited in Table 3.

**Table 3: Quantitative comparisons on basic trajectories.**
| Method | Video Quality |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| FVD<br>$\downarrow$ | FID<br>$\downarrow$ | IS<br>$\uparrow$ | CLIP-SIM<br>$\uparrow$ |  |  |
| AnimateDiff | 1582.93 | 73.79 | 13.39 | 0.3196 |  |
| MotionCtrl | 1768.25 | 78.26 | 14.60 | 0.3143 |  |
| CameraCtrl | 1383.68 | 76.18 | 14.04 | 0.3123 |  |
| CamTrol+ | 1134.71 | 69.70 | 15.87 | 0.3253 |  |

### A.4 Ablation on Depth Optimization

Ablation results relevant to the effectiveness of depth optimization are presented in Table 4. While it affects little on CLIPSIM, depth optimization exhibits benefits on other video quality assessments(FVD, FID, IS) and camera trajectory criteria(ATE, RPE). Experiments are conducted on complicated 10 trajectories extracted from RealEstate10k , each with 10 prompts mentioned in MotionCtrl . We assume that depth optimization offers better alignment between adjacent views rendered from point cloud, which benefits both perceptual quality and camera motion accuracy of generated videos.

**Table 4: Ablation on depth coefficient optimization.**
| CamTrol+SVD | FVD$\downarrow$ | FID$\downarrow$ | IS$\uparrow$ | CLIP-SIM$\uparrow$ | ATE$\downarrow$ | RPE-T$\downarrow$ | RPE-R$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| w/o optimization | 5033.44 | 219.01 | 5.67 | 0.2949 | 4.093 | 1.131 | 0.046 |
| w/ optimization | 4977.88 | 218.36 | 5.72 | 0.2947 | 3.906 | 0.978 | 0.049 |

### A.5 More Results on other Base Video Models

Our methods could be plugged and played with arbitrary video diffusion models to achieve camera control. We showcase its performance based on two alternative video generation backbones including SVD and VideoFusion , relevant results are exhibited in Fig. 16.

Figure: Figure 9: Results with camera move Rotate Anticlockwise and Rotate Clockwise. Our method could generated 3D rotation-like videos with dynamics.
Refer to caption: x9.png

Figure: Figure 10: Results with camera move Hybrid I. Here we combine Zoom Out, Truck Left, Pedestal Up, Pan Right and Tilt Down together, form a cinematic like effect on generated videos.
Refer to caption: x10.png

Figure: Figure 11: Results with camera move Hybrid II. Here the hybrid motion is defined as Zoom In first, then Pedestal Up.
Refer to caption: x11.png

Figure: Figure 12: Results with complex camera motions.
Refer to caption: x12.png

Figure: Figure 13: Results with basic camera motions.
Refer to caption: x13.png

Figure: Figure 14: Comparison to state-of-the-art methods on Truck Left, Truck Right, Pedestal Up, Pedestal Down.
Refer to caption: x14.png

Figure: Figure 15: Comparison to state-of-the-art methods on Zoom In, Zoom Out, Roll Anticlockwise, Roll Clockwise.
Refer to caption: x15.png

Figure: Figure 16: Results with other video generation base models including SVD and VideoFusion . Our method works well with these backbones.
Refer to caption: x16.png

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. What is the evaluation dataset for the Table.1 ? If the references are come from the RealEstate10K dataset (Line 322), it is strange that the Video quality metrics of MotionCtrl and CameraCtrl are worse than the CamTrol, since both MotionCtrl and CameraCtrl are fine-tuned on the RealEstate10K dataset.
2. What is the base video generation models of MotionCtrl and CameraCtrl in Table.1 ? If the base video generators are not SVD, this comparison in Table.1 is unfair. Besides, can you provide the Video Quality of the vanilla SVD model for a reference?
3. Since the ParticleSFM may fail sometimes, a natural question is: how accurate is the camera computed ParticleSFM on the ground truth videos? It would be good to have some reference number on the ground truth dataset to show the lower bounds of Motion Accuracy metrics in Table.1 
4. In some cases of the provides webpage,  with the same camera trajectory (especially on complex trajectories), different videos do not have the same camera movement. For example, the complex trajectory II, the first video shows downward at the end of the video, the third video shows upward, the last video ignores the last camera movement. However, MotionCtrl and CameraCtrl do not have this kind of inconsistency. Can the author provide some quantitative results on complex camera trajectories?
soundness: 2
strengths: 1. CamTrol is a training-free solution, it does not need additional training, making it computationally efficient and easy to integrate with existing video diffusion models.
2. It investigate the feasibility to adopt the “noise prior of latent” technique to control the camera viewpoint without direct supervision.
3. CamTrol outperforms competing methods in both perceptual quality and motion alignment, particularly in complex camera movements, as demonstrated in both quantitative and qualitative experiments.
summary: The paper introduces a method named CamTrol to enable controllable camera movements for video generation using off-the-shelf video diffusion models. The innovation lies in its training-free approach—no need for additional supervised or self-supervised training. The method employs a two-stage process:
1. 3D Camera Motion Modeling: It use a single view image as input to regressive a 3D point cloud, then gradually rendering, refining the 3D point cloud. The rendering process adopt the camera poses to simulate camera motion. The generated image sequence is later used to guide the perspective and layout in generated video sequences.
2. Noise Prior Guidance: It utilizes the “layout prior” of noisy latents, first adding noise of the image sequence, injecting the camera motion prior to the noisy latents. Then, providing the noisy latents into a pretrained video generation model to generation a reasonable and camera viewpoint alignment video.
weaknesses: 1. Since the whole pipeline is complex (depth estimation -> point cloud lifting -> rendering -> inpainting -> depth coefficient optimization), CamTrol’s quality is vulnerable. A problem at each of the current steps will have a bad effect on the next step. For example, if the depth estimation model cannot give the precise results, the structure of the entire scene may be strange when viewed from the different perspectives. If the inpainting model is not good enough, the generated videos may have much inconsistency.
2. For different videos (and different camera trajectories), the best timestep t_0 may be inconsistent. It is hard to design a rule to choose the best t_0.
3. The inpainting model is designed for single images, which may lead to inconsistent content and difficulty in generating reasonable new content. Additionally, the noisy latent not only provides layout priors but also carries some semantic information, which is unfavorable for the video generation model when creating new content. This makes CamTrol potentially unsuitable for generating long videos.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Please check the weaknesses
soundness: 2
strengths: The writing is pretty clear, starting from the observation that camera movement could be regarded as one latent layout rearrangement. 
The two-stage framework is well-presented and easy to understand. 
The ablation study is comprehensive for multiple designs of proposed methods.
summary: This work proposes a training-free solution for camera control in video generation, requiring no supervised fine-tuning. A given image is first projected into 3D point cloud space and a sequence of views are then rendered given the camera trajectory. The latter serves as a structure condition for generation through inversion.
weaknesses: There are multiple major concerns:
- The novelty of this pipeline is limited which combines point cloud reconstruction and inversion. For example, Infinite Nature [a] also uses a RGBD image together with rendering to generate novel views and then refine. 
- The claim for training-free could be further clarified since depth coefficient optimization is also adopted (as mentioned at L190). Although the base model is not tuned, this optimization could be empirical and time-comsuming, preventing it from the practical applications. 
- As stated at L210, video should be encouraged with more dynamics, which is quite important. Yet, we do not find any design/modification for this.
- From one side, inversion provides the trade-off between fidelity and diversity (L261). From the other, it is challenging to choose which timestep should be used for the generation with given camera conditions. 
- How the proposed method tackle text-to-video generation tasks? Fig.1 suggests that an input image is always given.


[a] Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Besides the first point in the weakness, I have the following questions.
1. In Table 1, what are the base video generator for MotionCtrl and CameraCtrl? Can you use the same base video generator for Motion, CameraCtrl, and CamTrol, like SVD to quantitatively evaluate them?
2. In the camera motion modeling step, from the same first image, it seems that the CamTrol can generate different image sequences, according to the camera trajectories. If seeding these image sequence into the second step (layout prior of noise). Can we get some videos with similar appearance but different camera trajectories? Can you have some comparisons with the CVD (Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control) method?
3. How to find the best trade-off of t_0 for each videos?
soundness: 4
strengths: 1. The framework is totally training free, combining some pre-trained models.
2. The written is easy to follow.
3. Visualization results demonstrate the effectiveness of each part in this pipeline.
summary: This paper proposes a framework to control the camera viewpoints of the video generation process. This frame work is training free, utilizing some existing pre-trained models, like depth estimation model, and the depth coefficient optimization model. It can be used on many different base video generators. Given a real or T2I-generated images, this frame first generate a series of images according to the camera trajectories, with the point clouds as the middle 3d representation. Then, these images are noised to a certain degree. After that, these noised latents are denoised with the video generator, getting the videos with desired camera trajectory.
weaknesses: 1. In Table.1 , some quantitative comparison between the SVD and CamTrol+SVD is lacked, using the FVD, FID, IS, and CLIP-SIM, metrics. This comparison can reflect the impact of using proposed pipeline on the pretrained video generators.
2. In the demo video, there are some obvious consistency or unreasonable on objects in generated videos. One possible reason is the in painting model cannot handle some situation well, for example, when the camera motion is large, leading much holes in the images, the inpainting model cannot get enough context to generate reasonable contents for the holes.
3. Continue with the second point, this pipeline may have some difficulty in properly use the pre-trained video generator to generate consistent and reasonable videos.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In the quantitative evaluation, could you clarify the differences in the experimental settings for MotionCtrl, CameraCtrl, and other related papers? I noticed that there does not appear to be a uniform experimental setup in the field of camera control video generation.
2. The paper states that it compares CamTrol with state-of-the-art works, including AnimateDiff, MotionCtrl, and CameraCtrl. While it mentions that "AnimateDiff is excluded from quantitative analysis due to its incapacity to handle complicated trajectories," I still have not seen a qualitative comparison with AnimateDiff. Additionally, could you consider comparing your work with other relevant studies, such as CamCo[1] and Vd3d[2]?
3. The current results are excellent in terms of trajectory control, but they do not satisfy me in terms of video quality. Given the availability of a superior fundamental video model (with larger resolution and longer duration), and considering the combination with CogVideoX presented in this paper, I would appreciate it if you could provide some generated samples or comparisons for a more thorough evaluation. I will adjust the final score based on these results.

[1] Xu, Dejia, et al. "CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation." arXiv preprint arXiv:2406.02509 (2024).
[2] Bahmani, Sherwin, et al. "Vd3d: Taming large video diffusion transformers for 3d camera control." arXiv preprint arXiv:2407.12781 (2024).
soundness: 2
strengths: The presentation of this paper is clear. The proposed method is reasonable, effectively generating videos with camera trajectories from a higher-dimensional space (3D point cloud). Notably, the proposed method is training-free, allowing for easy adaptation to new fundamental video generation models and related research.
summary: This paper propose a training-free camera movement control for off-the-shelf video diffusion models. Firstly, the image is converted to 3D point cloud space using depth estimator, and the video frame is re-rendered using the camera motion trajectory. In this process, inpaint is used to repair the vacant part of the point cloud in the rendering result, and finally, generate videos using layout prior of noisy latents(inversion from the rendering result). The experiments and demo show good results.
weaknesses: 1. While the overall pipeline is effective, it is relatively complex, including depth estimation, rerendering, 2D image inpainting, video fundamental model redrawing, and other processes. Additionally, the integration of these various components is not particularly seamless. This complexity can lead to reduced generation efficiency and may result in inconsistencies between the different stages, which could ultimately compromise the quality of the output.
2. The presentation of the experimental part was poor, and the comparison with the relevant work and the combination with the latest video base model were not seen in the demo web page, although they mentioned it in the paper.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: I currently rate this paper below the acceptance threshold. I have some concerns about the result quality and moreover it is not clear how large the computational overhead at inference time is compared to just having some simple embedding module as in MotionCtrl and CameraCtrl.
I would like authors to address following questions:
- Why does the scene often change with camera movement?
- Why is the camera often not that accurate even for simple movements?
- What is the computational overhead at inference time?

I am open to adjusting my rating based on the rebuttal.
soundness: 3
strengths: - Control over motion strength: The method can synthesize videos with different camera motion strengths. Previous methods struggle due to having no knowledge of scale. Because the method uses a metric depth estimator, this problem should be fixed.
summary: This paper proposes CamTrol, a training-free method for camera control in pre-trained video diffusion models. While previous works mainly train a camera control module using a fine-tuning dataset with camera annotations, this work does not require any external dataset to control the camera. The pipeline uses 3D point clouds and inversion to guide the generation process, enabling flexible camera control. CamTrol outperforms previous works which rely on external data, while enabling metric scale control based on the metric depth estimator.
weaknesses: - Scene inconsistency: The scene changes when the camera moves. When looking at the supplementary website, “Tilt Down” changes the top part of the scene or “Pedestal Down” changes the whole arrangement of the scene. But the point of camera control is that the scene remains intact, so in the best case the scene could be reconstructed from the video.
- Inaccurate camera: When looking at the supplementary website, the cameras are not that accurate. E.g., “Pan Left” also moves forward in the scene and doesn’t look like panning but rotating on the spot.
- Missing computational costs analysis: The approach seems very expensive. The authors motivate that previous works fine-tune some camera control modules on a camera annotated dataset which requires additional training. However, those approaches can do inference with basically zero computational overhead after training. This optimization-based approach requires computational overhead for each inference run and it is not clear how significant that is. E.g., if a user has to wait tens of minutes to hours to generate one result, there is a trade-off reached after a few prompts where it is more practical to train a camera control module once and then use it without overhead. It’s important to analyze both optimization time and memory usage.
- Missing quantitative ablations: The ablations mentioned in Sec. 4.3 are missing quantitative comparisons. The depth optimization seems to change nearly nothing in Tab. 2. When looking at relative differences between metrics, the metrics differences are basically 0.
- Writing quality: While the paper starts ok in terms of grammar, it gets worse throughout the paper and it makes the paper hard to read, especially the method section. I recommend putting the paper through some grammar correction tool.
- Unclear motivation of generalizability: The paper mentions in the introduction that methods fine-tuned on RealEstate10K suffer from generalizability issues. But there are no experiments or visuals which demonstrate this. Does the method really work better when cameras totally different from the training distribution are used?

Meta-review: The paper uses pre-trained video diffusion models to introduce a training-free method for controllable camera movements in video generation. By leveraging 3D point cloud modeling, camera motion simulation, and layout priors in noisy latent, it achieves flexible camera control without requiring additional datasets or fine-tuning, outperforming prior methods.

Agreed by most reviewers, CamTrol's strengths include (1) its training-free approach, which is flexible and adaptable for use with different foundation models; (2) the innovative use of noise priors in latent space to control camera motion without supervision; and (3) strong performance in visual quality and motion alignment, particularly for complex camera movements, as validated through experiments and ablation studies.

As pointed out by several reviewers, the limitations include (1) its complex multi-stage pipeline; (2) inconsistencies in scene and camera motion, with unintended changes or inaccuracies in trajectories; (3) the inpainting model’s inability to handle large camera motions, leading to unreasonable content generation; (4) significant computational overhead from depth coefficient optimization, and (5) missing quantitative comparisons in key experiments. The authors addressed the concerns about optimization overhead and quantitative comparisons, and partially addressed issues related to scene consistency and camera motion. Pipeline complexity and inpainting model limitations are not adequately discussed, but most reviewers were happy with the responses.

Decision: Accept (Poster)