Training-free Camera Control for Video Generation
Authors: Chen Hou
Zhibo Chen
Venue: ICLR 2025 Poster (2025)
Keywords: video generation
camera control
training-free
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: We propose a training-free and robust solution to offer camera movement control for off-the-shelf video diffusion models. Unlike previous work, our method does not require any supervised finetuning on camera-annotated datasets or self-supervised training via data augmentation. Instead, it is plug-and-play with most pretrained video diffusion models and can generate camera-controllable videos with a single image or text prompt as input. The inspiration for our work comes from the layout prior that intermediate latents encode for the generated results, thus rearranging noisy pixels in them will cause the output content to relocate as well. As camera moving could also be seen as a type of pixel rearrangement caused by perspective change, videos can be reorganized following specific camera motion if their noisy latents change accordingly. Building on this, we propose **CamTrol**, which enables robust camera control for video diffusion models. It is achieved by a two-stage process. First, we model image layout rearrangement through explicit camera movement in 3D point cloud space. Second, we generate videos with camera motion by leveraging the layout prior of noisy latents formed by a series of rearranged images. Extensive experiments have demonstrated its superior performance in both video generation and camera motion alignment compared with other finetuned methods. Furthermore, we show the capability of CamTrol to generalize to various base models, as well as its impressive applications in scalable motion control, dealing with complicated trajectories and unsupervised 3D video generation. Videos available at https://lifedecoder.github.io/CamTrol/.

Published as a conference paper at ICLR 2025
TRAINING-FREE CAMERA CONTROL FOR VIDEO GEN- ERATION
Chen Hou, Zhibo Chen
University of Science and Technology of China (houchen@ma!] chenzh bo@ justc. edu.cn Justc.edu.ccm
ABSTRACT
We propose training-free and robust solution to offer camera movement con- trol for off-the-shelf video diffusion models. Unlike previous work, our method does not require any supervised finetuning on camera-annotated datasets or self- supervised training via data augmentation. Instead, it is plug-and-play with most pretrained video diffusion models and can generate camera-controllable videos with a single image or text prompt as input. The inspiration for our work comes from the layout prior that intermediate latents encode for the generated results, thus rearranging noisy pixels in them will cause the output content to relocate well. As camera moving could also be seen ment caused by perspective change, videos can be reorganized following spe- cific camera motion if their noisy latents change accordingly. Building this, we propose
type of pixel rearrange-
which enables robust camera control for video diffusion models. is achieved by a two-stage process. First, we model image layout rearrangement through explicit camera movement in 3D point cloud space. Sec- ond, we generate videos with camera motion by leveraging the layout prior of noisy latents formed by series of rearranged images. Extensive experiments have demonstrated its superior performance in both video generation and cam- era motion alignment compared with other finetuned methods. Furthermore, we show the capability of CamTrol to generalize to various base models, as well its impressive applications in scalable motion control, dealing with compli- cated trajectories and unsupervised 3D video generation. Videos available at http:///feddeeodeeeggihbbbllaaaa
INTRODUCTION
As more appealing and content-richer modality, videos differ from images by including extra temporal dimension. This temporal aspect provides increased versatility for depicting diverse and dynamic movements, which can be decomposed into object motion, background transitions, and per- spective changes. Recent years have witnessed the rapid development and splendid breakthroughs of video generation with text prompts or images input instructions (Li et al.) 2023 Hong al.,
2022 Ho et al.) 2022 Luo et al., 2023 Zeng et al., 2024]] Blattmann et al. 2023a] | Brooks et al,,
Ge al. 2023 Fei al. 2023), demonstrating the immense potential of diffusion models to synthesize realistic videos. While these video generation models have made progress in generating videos with highly dynamic objects and backgrounds (Zeng al. 2024 Blattmann al.
2024
2023a
et al. 2023), most of them fail to provide camera control for the generated videos.
The difficulty of controlling camera motion in videos arises primarily from two aspects. The initial challenge lies in the inadequacy of annotated data, as most video annotations lack detailed descrip- tions, particularly about the camera movements. these data often fail to interpret text prompts related to camera motions and generate correct outputs. One solution to mitigate the data insufficiency problem to mimic videos with camera movements through simple data augmentation (Yang et al. 2024a). However, these methods can only handle simple camera motions like zoom or truck, and have trouble dealing with more complicated ones. The second challenge is the extra finetuning required for camera control and its inherent limitations. As camera trajectories could be sophisticated, they sometimes cannot be accurately elaborated us- ing naive text prompts alone. Common solutions (Wang al. 2023
a result, video generation models trained on
He al.] 2024) proposed

---

Published as a conference paper at ICLR 2025
to embed camera parameters into diffusion models through learnable encoders and perform exten- sive finetuning on large-scale datasets with detailed camera trajectories. However, such datasets as RealEstate 10k (Zhou et al. 2018) and MVImageNet (Yu et al. 2023) are highly limited in scale and diversity due to the difficulty associated with data collection; in this way, these finetuning meth- ods demand substantial resources but exhibit limited generalizability to other types of scenes. Lack of annotations and the constraints of finetuning make camera control challenging task in video generations.
In this work, we attempt to address these issues through a training-free solution to offer camera con- trol for aff-the-shelf video diffusion models. We begin by introducing two core observations under- pinning the idea that video diffusion models can achieve camera movement control in a training- free manner. First, we find that base video models could produce results with rough camera moves by integrating specific camera-related text into input prompts, such as camera zooms in or camera pans right. This simple implementation, though not very accurate and always leading to static or wrong motions, shows the natural prior knowledge learned by pretrained models about following different camera trajectories. The other observation is the effectiveness of video models in adapting to 3D generation tasks. Recent works (Voleti et al. 2024 Melas- Kyriazi al. 2024 Shi al. (023) find that leveraging pretrained video models as initializztion helps drastically ly improve the performance of multi-view generations, demonstrating their strong ability to handle perspective change. The two crucial observations reveal the hidden power of video models for camera motion control. Therefore, we seek to find way to activate this innate ability, as it already exists in the model itself.
We propose CamTrol, which offers camera control for off-the-shelf video diffusion models in a training-free but robust manner. CamTrol is inspired by the layout prior that noisy latents possess for the generation results: As pixels in noisy latents change their positions, corresponding rearrange- ment will also occur in the output, leading to layout modification. Considering camera moves can also be seen as a type of layout rearrangement, this prior can serve as an effective hint, providing the video model with information about specific camera motions. Specifically, CamTTrol consists of a two-stage procedure. In stage I, explicit camera movements are modeled in 3D point cloud represen- tation and produce a series of rendered images indicating specific camera movements. In stage II, the layout prior of noisy latents is utilized to guide video generations with camera movements. Com- pared with previous works, Cam TaoTrol requires no additional finetuning on camera-annotated datasets, nor does it need self-super training based data augmentation. Extensive experiments have demonstrated its superior performance in both video generation quality and camera motion align- ment against other finetuned methods. Furthermore, we show CamTrol's capability to generalize to various base models, as well as its impressive applications in scalable motion control, dealing with complicated trajectories and unsupervised 3D video generation.
2 RELATED WORK
Camera Control for Video Generation
While methods aiming to control video foundation mod-
els continue emerge (Ma et al.] 2024 Liu et al. 2023: Feng al. 2023), there are few works exploring how to manipulate the camera motion of generated videos. Earlier works (Hao et al., 2018) control motion trajectory via warping images through densified sparse flow and pixel fusion. Similar ideas also appear later in Chen et al. (2023) and Yin et al. (2023). Apart from utilizing optical flow, two main techniques for implementing video camera control are self-supervised augmentation or additional finetuning. Yang et al. (2024a) disentangles object motion with camera movement and incorporates extra layers to embed camera motions, where the model trained in self-supervised manner by augmenting input videos to simulate simple camera movements. He et al. (2024) and Wang et al. (2023) train additional camera encoders and integrate the outputs into the temporal at- tention layers of U-Net. Guo et al. (2023) learns new motion patterns via LoRA (Hu et al., 2021 and finetuning with multiple reference videos.
Noise Prior of Latents in Diffusion Model One of the most natural advantages of diffusion mod- els comes from its pixel-wise noisy latents formed during the denoising process. These latents have a strong causal effect on the output and directly determine what the result looks like, meanwhile hav- ing robust error resilience they perturbed by Gaussian noise across different scales. Numerous works have exploited the convenience of this noise prior to attain controllable generation, such as image-to-image translation (Meng et al. 2021) pixel-level manipulation (Nichol et al. (022). im-
2

---

Published as a conference paper at ICLR 2025
inpus image le and tex prompt
inversion
onipue video
3D Point Cloud Reconsrrction
corgii playing uhulele....
86
fest prongt p
rendering
Camera Trajectory [X]
Video Foundation Model
depth estimation
stage camera motion modeling
stage II: noise prior guidance
Figure I: Pipeline of Cam'rool In stage I, camera movements are modeled through explicit 3D point cloud. In stage II, the layout prior noisy latents utilized guide video generation.
age inpainting (ugmmyrr et al. 2022) and semantic editing (Choi et al. 2021 Hou et al. 2024). Recent research has shown that, although sampled from a Gaussian distribution, the initial noise of the diffusion process still has significant influence on the layout of generated content (Mao et al.) 2023). In other works, noise prior is used to guarantee temporal consistency between video frames Luo al. 2023), or to trade off between fidelity and diversity of image editing (Kim al. 2022)
Video Model for 3D Generation Similar how most video generation models use the ground- work laid by image foundation models (Blattmann et al.] 2003bb [Esser et al.) 2023] Singer et al.
2022 Wu al. 2023), the training of 3D generation models also relies heavily
pretrained 2D
video models Voleti et al. 2024) Melas- et al. 2024; Shi et al. 2023 Chen et al. 2024 Han
al. 2024). These methods either finetune with rendered videos directly (Blattmann et al. 2023a
Chen et al. 2024| Melas- -yrrizi et al. 2024 Han et al., 2024). or add camera embedding as an condition for each view (Voleti al. 2024] Shi et al. [2023]. Video foundation models have been shown to be particularly beneficial in generating consistent multi-view rendering of 3D objects, demonstrating their inherent and abundant prior knowledge for handling camera pose change.
3 TRAINING-FREE CAMERA CONTROL FOR VIDEO GENERATION
CamTrrol takes two stages to activate the innate camera control ability hidden in foundation models. In Sec. 3.1 we will describe how to model explicit camera movement for video generation. In
Sec. 3.2 will elaborate on video motion control with the guidance of noise layout prior.
3.1
CAMERA MOTION MODELING
To activate pretrained video diffusion models' ability deal with camera perspective changes, hints of camera motion should be injected into the diffusion model in proper way. While simply con- catenating camera trajectories with text prompts is incomprehensible the original model, previous works (Wang et al., 2023 He et al., 2024) introduce additional embedders to encode camera pa- rameters and finetune with limited annotated data (Zhou et al.) 2018:) Yu al. 2023). which are data-hungry yet lack generalization ability. Other methods YYang et al.) 2024a) construct camera motions by self-supervised augmentations, but can only handle few easy camera controls. Thus, seek more efficient and robust way to guide the model towards being camera-controllable
Considering that video perspective changes are originally caused by camera movements in 3D space, we leverage 3D representation to provide generation models with explicit motion hints. Specifically, we choose point cloud as the intermediate representation, in which we can easily manipulate cam- poses to simulate diverse movements. An additional benefit that point cloud brings is its data efficiency: By utilizing inpainting techniques, only one single input image is required for the entire point cloud reconstruction. This eliminates the need of large-scale finetuning.
3

---

Published as a conference paper at ICLR 2025
Point Cloud Initialization We start by lifting pixels in the input image plane into 3D point cloud space. In practice, the imput image can be either user-defined or created by image generators like Stable Diffusion (Rombach et al. 2022). Given an input image In E R depth map Do using the off-the-shelf monocular depth estimator Dephh (Bhat et al. 2023). By combining the image and its depth map, we initialize the point cloud Po as:
first estimate its
Po ( Dol, Po).
(1)
where o denotes the mapping function from RGBD to 3D point cloud, K and Po represent camera's intrinsic and extrinsic matrices set by convention (Chung et al. 2023) as they' 're usually intractable.
Camera Trajectories To get consistent images from multiple views, we model the camera motion as a trajectory of extrinsic matrices (P IP,,... PN each including rotation matrix and a transla- tion matrix representing camera pose and position. At each step we project the point cloud back to the camera plane using & and get a rendered image with perspective change: 1, = (P,, K, P.). By computing the extrinsic matrices of different movements, cluding zoom, tilt, pan, pedestal, truck, roll, and rotation, enabling flexible camera movements. De- tailed definitions of these movements
obtain a series of camera motions in-
provided in Appendix B Basic trajectories are combined to produce hybrid camera movements, adding cinematic appeal to the generated videos. Further- more, benefiting from explicit camera motion modeling, our method can support trajectories with precise extrinsics, which means it can generate videos with arbitrarily complex camera motions.
Multi-view Consistency When perspective changes, holes can appear as some areas are unoc- cupied within the point cloud. To obtain more reasonable results, we employ image inpainting (Rombach et al. 2022) to fill the gaps in new renderings. with mask distinguishing the known points from the nonexistent ones. This operation ensures coherence between the known and novel views in 2D space. After inpainting, the image is lifted back into 3D space and gradually completes the whole point cloud. During this process, points between adjacent views may become misaligned since the depth estimator only estimates relative depth, leading to inconsistencies in 3D point cloud and rendered images. To avoid this situation, we adopt depth coefficient optimization (Chung al. 2023) at each step of the camera movement, formed as:
Σ El dD K,P.) K.P.)).PPPP.. -P-1))
d; argmin
where i, and D; refer to the inpainted image and the corresponding depth map, respectively, d; denotes the depth coefficient to be optimized, and M refers to the overlapping region between P, and Pi-1, other areas are not shared for calculating E loss.
Thus, we get set of images that correspond to the input and indicate specific camera movements:
(Io.... Iv-1l = (w K.P. [0, 1]} -1]]
3.2 LAYOUT PRIOR OF NOISE
With camera motion modeling, obtain sequence Vo [To, IN-1} Nx3xHx of ren- dered images adhering to a specific camera trajectory. Note that the quality of rendered images is not perfect since single input image only leads sparse point cloud reconstruction. Moreover, these renderings are static, thus they cannot be used directly as video frames. To form an ideal video, we need find a way that satisfies three requirements: 1) camera motions should be maintained; 2) the video should be enriched with more dynamics; and 3) quality imperfection should be compensated.
Camera Motion Inversion Recent work on diffusion models has demonstrated the strong control- lability of their noisy latents (Meng et al. 2021 Mao et al. 2023), the causality and error-resilience they possess with respect to the final output make them a convenient yet powerful tool for con- trollable generation in diffusion models. Particularly for initial noise, even when sampled from Gaussian distribution, it still has significant influence on the layout of the generated image (Mao) ett al. 2023). For instance, if all pixels in initial noise shift to the right by certain distance, is
4

---

Published as a conference paper at ICLR 2025
Gaussian (aussian(nearly)
Camera Motion Inversion
Image Distribution
Video
specific layout change
Distribution
Video Generation
motion
to = T
More diversity Less fidelity
More fidelity Less diversity
to
latents of arranged images
gaussian latents with layout change
latents video frames
Figure 2: Transition of samples between two distinct distributions. As the layout-arranged im- ages are inverted by adding random noise, the distribution of their noisy latents will gradually con- verge to that of their video counterpart (green area), eventually forming nearly Gaussian latent with specific layout change. This information is then inherited during the generation process. The inversion step determines the trade-off between video diversity and motion fidelity.
likely that the generated output reflects similar shift. This reminds that the impact of camera movement on images can also be interpreted as type of layout rearrangement, where pixels change their positions due to viewpoint change. In a similar way, videos can be reorganized following camera motion their noisy latents change accordingly.
Inspired by this, we first construct series of noisy latents indicating specific camera movements. It can be done intuitively by employing diffusion's 's inversion process on the rendered image sequence Vo. Latent at timestep can be calculated as follows, where ã₂ is the variance of the scheduler:
(4) Because the rendered images Vo share common pixels in certain regions, their latents are also correlated to each other in way reflecting pixel movement. Moreover, by introducing random noise, blank spaces and flawed regions in Vo can be adaptively completed, providing the video model with more possibilities generate and correct them.
Voa / Vo + VI- E e~ (0,11)
Video Generation After camera motion inversion, noisy latents representing camera movements are then passed through the backward process of the video diffusion model, leveraging their layout controllability to guide video generation. Using prior knowledge of the base model, the generation process also infuses the video with rational dynamic information. In this way, explicit camera move- ments are injected into the video diffusion model in appropriate, training- ffree fashion. Starting from noisy motion latents at timestep to, the generation step can be represented as:
Vi V
0446 (V+)
(1)
V = VoL-1
a--1 - o7 + 1
(Vi) + toic, [1, (5)
Vou
Here 6g denotes the video model for noise prediction and o, determines whether the denoising process is deterministic or probabilistic. We set a to encourage diversity in generation process.
Trade-off Between Fidelity and Diversity Leveraging noise prior guidance in diffusion models could lead trade-off between generation fidelity and diversity (Meng et al. 2021, Hou et al., 2024). where results that are more faithful to the guidance tend to degrade in generation quality. In this task, similar circumstances also occur, as the model required to be guided by some imperfect renderings while generating reasonable video. The key factor in balancing the trade-off problem lies in the choice of to. When a smaller to is applied, the generated output bears more resemblance to original guidance Vo. but lacks rationality and dynamics to be an appealing video. Instead, larger ty leads to better-generated videos but makes them less aligned with the desired camera motion. Empirically. find that smaller to works better for motions with moderate intensity, and for those with relatively drastic movements, larger to shows preferable performance. The process of stage II is illustrated in Fig.
5

---

Published as a conference paper at ICLR 2025
Table 1:
Quantitative comparisons. Our method attains comparable performance with finetuned methods in both video generation quality and camera motion alignment.
Video Quality
Motion Accuracy
Method
FVD 4
FID |
IS
CLIP-SIM
ATE
RPE-T +
RPE-R
SVD MotDonCtI4SVD
1107.93
68.51
7.21
0.3095
4.23
1.79
0.021
810.59
69.03
7.17
0.3076
4.19
1.07
0.012
CaDeraCtr++SD
951.80
67.59
6.82
0.3138
4.22
1.17
0.013
Cam T 778.46
68.06
7.05
0.3110
4.17
1.07
0.010
Reference
3.60
0.89
0.008
A bedroom with large bed and a television Reference
SVD
MotionCtrl
CameraCtel
CamTrol
Figure 3: Qualitative comparisons with finetuned methods. CamTrol's outputs align well with complex trajectories from the reference video, while others fail capture subile changes in camera pose. We provide videos in the supplementary materials for clearer comparison.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Implementation Details We compare our method with state-of-the-art works including MotionC- trl (Wang et al. 2023) and CameraCtrl (He et al. 2024). To ensure fair comparison, we employ SVD (Blatmmann et al. 2023a) as base model for all methods. SVD is originally trained at a res- olution of 576 1024, but SVD-based CameraCtrl only supports 320 576. Since changing the original resolution results in suboptimal generation quality, we use 576 x 1024 for MotionCtrl and CamTrol, then resize the outputs 320 x 576 calculate metrics. For all methods, the number of frames and the decoding size of SVD are set to 14. We use 25 steps for both the inversion and generation processes.
In the quantitative evaluation, FVD (Unterthiner et al. 2018). FID (Heusel al. 2017) and IS (Saito al., 2020) used assess video generation quality, and CLIPSIM (Wu et al. 202T) quantifies the similarity between the generated video and the input prompt. For camera motion accuracy, we adopt ParticleSFM
Evaluation Details
et 2022 estimate camera trajectories from generated videos, with the use of Absolute Trajectory Error (ATE) to measure their differences from ground truth. Relative Pose Error (RPE) is calculated to assess how well the relative mo- tions between consecutive frames match the expected ones, including their translation (RPE-T) and rotation component (RPE-R). Specifically, we randomly sample 500 prompt-trajectory pairs from RealEEstate 10k (Zhou et al.) 2018), and use them references for calculating FVD and FID. Since SVD is an image-to-ven viveeo model, we generate the first frames using Stable Diffusion (Rombach al. 2022) based on text prompts. We also provide the results produced by vanilla SVD as a ref- erence. For camera motion accuracy, we provide the evaluations on ground truth videos as lower bound of these metrics.

---

Published as a conference paper at ICLR 2025
Table 2: Computational analysis of inference process, evaluated under unified settings.
SVD
MotionCtrl
CameraCtrl
CamTrol (ln 15)
Max GPU memory(MB)
11542
31702
26208
11542
pre-process inference
56
Time (s)
11
32
42
8
A backyard with patio and an umbrella.
Reference
Zoom Out
A person, black shirt, typing on koyboard...
MotienCrrl
CameraCtrl
CamTrol
(a) general scenes
(b) dynamics
Figure 4: Generalization comparisons. CamTrol can avoid domain collapse that arises from over- fitting on certain datasets and adapt to more general scenes (Left), meanwhile preserving video's dynamics while adhering to desired camera movements (Right).
4.2 COMPARISONS WITH STATE-OF-THE-ART METHODS
Quantitative Evaluation Quantitative evaluations are shown in Table In the table, the best re- sults are in bold, and the second best are underlined. The performance of vanilla SVD (without mo- tion control) is denoted as SVD, while the lower bound for motion metrics, provided by ground truth videos, is denoted Reference. In terms of video quality, CamTrool attains performance comparable to methods finetuned on the RealEstate 10k dataset. For motion accuracy, CamTrol also achieves the lowest score in ATE and RPE-T/R. The quantitative evaluations demonstrate CamTrol's ' ability to generate videos with both accurate camera motion and high visual quality.
Qualitative Analysis Qualitative comparisons are illustrated in Fig. 3 The reference trajectory in- cludes zoom, pan, and roll. While MotionCtrl and CameraCtrl fail to capture subtle camera motions, resulting in simple pan movements, CamTrol is able to follow the complex trajectory and generate videos with correct motion. We also evaluate the generalizability of different methods in generating more general scenes and dynamic content. The results are shown in Fig. 4 Since both MotionCirl and CameraCtrl are finetuned on limited scenes (i.e., real estate videos) with static content, they have difficulty generalizing to other scenes, such videos in non-realistic styles videos that include people. As illustrated Fig. their motion controls both situations are misaligned with the in- tended movements. Furthermore, finetuning such datasets leads to loss of dynamics, which is crucial element in video generation. comparison, CamTrol preserves most of the prior knowledge from video base models, enabling it to handle general scenes as well as generate dynamic content. Relevant videos are available in the supplementary materials.
Computational Analysis mum GPU memory required and the processing time for all methods during inference. Evaluations are conducted under unified settings. We test at a resolution 576 320 with 25 generation steps. The number of frames and the decoding size of SVD set 14. As training-free method,
We provide the computational analysis in Table 2 including the maxi-

---

Published as a conference paper at ICLR 2025
camera rotates around it.'
camera zooms ouf.
+CamTrol
+ CamTrol
Young wizard swings wand,
ceramic cat,
Figure 5: Comparison with base model. Controlling camera motion via prompt engineering rarely succeeds. Instead, Crol offers robust control to video's 's camera movements.
3d model bulky purple mecha with missiles....
A majestic mounluin range under the clouk of night
Before
After
Figure 6: Effectiveness of layout prior. Layout prior guidance compensates for the voids (Left)
and static elements (Right) in point cloud.
CamTrol requires no extra GPU memory during the inference process compared to the base model. This saves 10-20GB of GPU memory compared other methods under the same circumstances, allowing it to run on a single RTX 3090. The major consumption of CamTrol comes from rendering multi-view images. Since this process is temporally sequential, i.e., to + l - 12, more par- allelized approach may improve time efficiency. The results for 576 1024 resolution and more detailed settings can be found in the Appendix.
4.3 ABLATION STUDY
Comparison Base Model To demonstrate that the changes in camera motion are attributed to our method rather than the innate capability of the video model, we conduct an ablation study to evaluate effectiveness. We add prompts describing specific camera movements (e.g. zooms out) and let the video model interpret them on its own. The results are presented in Fig. 5 It can be observed that, even when provided with prompts indicating how the camera should move, the base model fails to produce correct results. In contrast, CamTrol is able to implement the designated motion control without any instructions from text prompts. In Table I the comparisons with vanilla SVD also demonstrate CamTrol's 's effectiveness.
We conduct ablation study to validate the effectiveness of layout prior guidance, demonstrating its necessity two aspects: the completeness of missing areas and the dynamics of the generated video. In Fig 6 we showcase frames before and after applying noise prior guidance. As camera pose changes, there are regions left unfilled in the point cloud, leading to blank spaces in rendered images (left part); Additionally, due the static nature of point cloud, the rendered images remain stationary (right part). Layout prior guidance compensates for these flaws, ultimately producing videos with inpainted gaps and natural dynamics.
Effectiveness of Layout Prior
8

---

Published as a conference paper at ICLR 2025
Table 3: Quantitative effeet of to.
Video Quality
Motion Accuracy
to
FVD +
FID
IS
CLIP-SIM f
ATE +
RPE-T
RPE-R
Lo
20
1079.88
68.52
7.14
0.3100
4.17
1.09
0.012
15
778.46
68.06
7.05
0.3110
4.17
1.07
0,010
to =
10
754.14
67.98
7.00
0.3107
4.13
1.02
0.008
Child and wolking in the woods...
Figurine Superiman his
25
scale
scall II
sccale III
Figure 7: Effect tg- Larger to encourages dynamics while smaller to preserves camera movements (Pedestal Down).
Figure 3D rotation videos different mo- tion scales. CamTrol supports camera move- ments across multiple scales.
Effeet of Timestep to to a crucial factor that influences the trade-off between generated video's diversity and its faithfulness to camera motion requirements. To investigate its effect the output, we conduct experiments with various to values, the relevant results are shown in Fig. 7 As illus- trated, videos generated with smaller to tend to conform better camera motion requirements, but suffer from decreased dynamics; On the other hand, larger lg leads to more plausible generations but fails meet camera's requirements, since the latents these timesteps carry more random- ness. We also provide quantitative evaluations for different to values in Table 3 As to decreases, CamTrol produces videos resembling static, camera-moving scenes (which have lower FVD use RealEstate 10k as the reference videos) with higher accuracy in motion control.
Generalization to Diverse Situations Our proposed CamTrol can be seamlessly integrated into various scenarios, accommodating different base models, resolutions, and generation lengths, all in a training-free free style. We present visual results of its applications under different settings, including Cog VideoX Yang et al. 2024b) (diffusion transformer model, 720 480 resolution, 49 frames) and VideoFusion (Luo et al. 2023) (decomposed diffusion process, 128 x 128 resolution, 16 frames), in
Fig. 10 Our approach remains effective when applied to alternative video base models, resolutions, and generation lengths, demonstrating its strong robustness and generalizability.
4.4 FURTHER APPLICATIONS
3D Rotation Videos One of the main advantages of our method is that it can generate videos with rotating movements and produce outputs similar to those of 3D generation models (Voleti al. 2024; Melas MallssKKyiazzi et al., 2024). While these 3D models require large-scale training on 3D datasets and can only handle inputs in specific styles, our approach is capable of dealing with any type of image and achieving this a completely zero-shot manner. An example shown Fig.
Hybrid and Complex Camera Movements By combining basic camera trajectories, CamTrol can support hybrid camera movements and produce videos with cinematic appeal. In addition, explicit motion modeling equips CamTrol with the ability to handle trajectories of precise extrinsics, allowing it to generate videos with arbitrarily complicated camera movements. Relevant results can be found at httpss//iiieeecoeegggttbbooooooo CatettuuiioC/CCooollooool/

---

Published as a conference paper at ICLR 2025
saow-capped mountain peak towers above tranquil alpine lake, mirrored perfectly in glassy surface.
Zoom In
Pan Left
Tilt Up
Input
Zoom Out
Pan Right
Tilt Down
Figure 9: Multi-trajectory video generation. CamTrol is able to generate videos with different camera motions for the same scene.
detailed wooden toy ship carpet sea...
Véporwave syrthwwave style Los Angeles street...
Rotate CW
Tift Up
An elderty man sits water's 's adge, engrossed in his artwork...
Aconstruction worker looking down at city...
Out
Hybrid
(a) CogVideoX-2b
(b) VideoFusion
Figure 10: Applied to VogVi Yang et al., 2024b) and VideoFusion (Luo et al. 2023). CamTrol can plug-and-play under various situations, accommodating different base models, dif- ferent resolutions, different generation lengths, all in training- free manner.
Multi-trajectory Video Generation One natural application of our method is generate multi- trajectory videos for a single scene. Since the point cloud remains fixed once the reconstruction has finished, the spatial consistency between different trajectories is guaranteed, making easy to produce multiple camera-moving videos of the same scene. We demonstrate this application in Fig. More results on multi trajectorjeeyyy video generation can be found in the supplementary materials.
Camera Motion at Different Scales CamTrol supports camera movements at controllable scales. By specifying different magnitudes of the camera's 's extrinsic matrix, the rendered images will ex- hibit varying degrees of motion, resulting in videos with distinct scales of camera movements. This further demonstrates the powerful controllability of Cam and provides new pathway for cus- tomized camera control of video generation. We provide relevant visualization Fig.
CONCLUSION
In this paper, we propose training-free and robust method CamTrrll offer camera control for off-the-shelf video diffusion models. It consists of two-stage procedure, including explicit camera motion modeling in 3D point cloud space and video generation utilizing the layout prior of noisy latents. Compared to previous works, CamTrol does not require additional finetuning camera- annotated datasets or self-supervised training via data augmentation. Comprehensive experiments demonstrate its superior performance in both generation quality and motion alignment against other state-of-the-aa methods. Furthermore, we show the ability of CamTrol to generalize to various scenarios. well its impressive applications including unsupervised generation of 3D video, scalable motion control, and dealing with complicated camera trajectories.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero- shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302. 2023.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zäon English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:231 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis, Align your latents: High-resolution video synthesis with latent diffusion mod- cls. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2256--2257, 2023b.
T Brooks, B Peehles, C Homes, W DePue, Y Guo, L Jing, D) Schnurr, J Taylor, T Luhman, E Luh- man, et Video generation models world simulators, 2024.
Tsai-Shien Chen, Chieh Hubert Lin, Hung- Tseng, Tsung-Yi Yi Lin, and Ming-Hsuan Yang. Motion- conditioned diffusion model for controllable video synthesis. arXiv preprint arXiv:2304. 14404, 2023.
Zilong Chen, Yikai Wang. Feng Wang. Zhengyi Wang, and Huaping Liu. V3d: Video diffusion models are effective 3d generators. arXiv preprint FxXiv22433.06388 2024.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong. Youngjune Gwon, and Sungroh Yoon. Ilvr: Con- ditioning method for denoising diffusion probabilistic models. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 14367-14376, 2021.
Jaeyoung Chung, Suyoung Lee, Hyeongjin Nam, Jaerin Lee, and Kyoung Mu Lee. Luciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:2311.13388 2023.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germani- dis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 7346-7356, 2023.
Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, and Tat-Seng Chua. Empowering dynamics- aware text-to-video diffusion with large language models. arXiv preprint arXiv:2308. 13812, 2023.
Ruoyu Feng, Wenming Weng, Yanhui Wang, Yuhui Yuan, Jianmin Bao, Chong Luo, Zhibo Chen, and Baining Guo. Ccedit: Creative and controllable video editing via diffusion models. arXiv preprint arXiv:2309. 16496, 2023.
songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22930-22941, 2023.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image dif- fusion models without specific tuning. In The Twelfth International Conference Learning Representations, 2023.
Junlin Han, Filippos Kokkinos, and Philip Torr. Vfusion3d: Learning scalable 3d generative models from video diffusion models. arXiv preprint arXiv:2403. 12034, 2024.
Zekun Hao, Xun Huang, and Serge Belongie. Controllable video generation with sparse trajectories. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7854- 7863, 2018
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint 240.:22444.02101, 2024.
11

---

Published as a conference paper at ICLR 2025
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:22/0.023000 2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-vidoo generation via transformers. arXiv preprint 1rXi::2205.18666 2022.
Chen Hou, Guoqiang Wei, and Zhibo Chen. High-fidelity diffusion-based image editing. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 118422192, 2024.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation large language models. International Conference on Learning Representations, 2021.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2022.
Xin Li, Wenqing Chu, Ye Wu, Weihang Yuan, Fanglong Liu, Qi Zhang, Fu Li, Haocheng Feng, Errui Ding, and Jingdong Wang. Vidcogen: reference-guided latent diffusion approach for high definition text-to-vider generation. arXiv preprint 230900200000398 2023.
Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Vidco-p2p: Video editing with cross-attention control. arXiv preprint arXiv:2303.00761 047611 2023.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 11461-11471, 2022.
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang. Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pp. 10209-1033 10218, 2023.
Ze Ma, Daquan Zhou, Chun-Hsiar Ych, Xue-She Wang, Xiuyu Li, Huanrui Yang, Zhen Dong, Kurt Keutzer, and Jiashi Feng. Magic-me: Identity-speciinn video customized diffusion. arXiv preprint arXiv:2402.09336. 2024.
Jiafeng Mao, Xueting Wang, and Kiyoharu Aizawa. Guided image synthesis via initial image editing in diffusion model. In Proceedings of the 3t ACM International Conference Multimedia, pp- 5321-5329, 2023.
Luke Melas-Kyriazi Iro Laina, Christian Rupprecht, Natalia Neverova, Andrea Vedaldi, Oran Gafni, and Filippos Kokkinos. Im-3d: Iterative multiview diffusion and reconstruction for high-quality 3d generation. arXiv preprint arXiv: 240..08822.. 2024.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text- text-guided diffusion models. In International Conference Machine Learning, pp- 16784-16804, PMLR, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
12

---

Published as a conference paper at ICLR 2025
Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan. International 2020.
Journal of Computer Viston,
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 12512, 2023.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang. Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation generraatttggnetottooonn without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models video: new metric challenges. arXiv preprint arXiv: 11xv:1812.001117 2018.
Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Chris- tian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from single image using latent video diffusion. arXiv preprint 2rXiv:2403.122 /2008, 2024.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. arXiv preprint arXiv:2312.03641,, 2023.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104. 14806, 2021.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7623-7633, October 2023.
Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma. Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. arXiv preprint wrXiv:2402.03162 2024a.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint arXiv:2408.0600 2024b.
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Drag- nuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.0800 2023.
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9150-9161, 2023.
Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Wang Zhao, Shaohui Liu, Hengkai Guo, Wenping Wang, and Yong-Jin Liu. Particlesfi: Exploiting dense point trajectories for localizing moving cameras in the wild. European Conference on Computer Vision, pp. 523-542 Springer, 2022.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: 2018.
Learning view synthesis using multiplane images. arXiv preprint
13

---

Published as a conference paper at ICLR 2025
A
MORE RESULTS ON CAMERA CONTROL
We present additional qualitative results of CamTrol on our demo page:
https://liceeeodeeggthho.//aaaaa Trol/
This demo page includes Cam'Tro--generan videos including basic camera motions (including Zoom, Till, Pan, Pedestal, Truck, Roll, Rotate, Hybrid, Complicated, detailed definitions in Appendix B), hybrid motions (Zoom In first, then Pedestal Up. Zoom Out + Pedestal Up + Truck Left Tilr Down Pan Right), and complicated motions generated from precise camera extrinsics (extracted from RealEstatee 10k (Zhou et al. 2018)).
In addition, contains 3D rotation videos generated unsupervised from video base models (both objects and scenes). These outputs share similarities with those of 3D generation models, as they all exhibit turning-table-like way, with the camera rotating around objects. The difference here that the 3D model, as trained on specific datasets, can only generate outputs in certain styles, egg., single static object with no background. Instead, our model can handle arbitrary images as input and generate rotating videos with appropriate dynamics. From this perspective, be seen as an infinite source of 3D data. By employing our method with stronger backbones, video foundation models could truly become the largest source of 3D data, they should be.
method could
Furthermore, contains additional results discussed in ablation studies, e.g., controlling camera motions at different scales, and the effectiveness of using layout prior compared with the raw output given by the video base model.
B DEFINITIONS OF BASIC CAMERA MOTIONS
We refer to the terminology in cinematography to describe different camera motions, the definitions of each type are detailed in T
To obtain consistent images from multiple views, we set camera motion as a trajectory of extrinsic matrices (P ..., P each consisting of rotation matrix and translation matrix, representing the camera pose and position. For hybrid motions, CamTrol supports both spatial (i.e., performing several basic motions simultaneously) and temporal (i.e., concatenating basic motions sequentially) combinations. In the case of complicated trajectories, can directly precise parameters for camera's i extrinsic matrix as input to control video's 'ss motion. Additionally, if these parameters are not available, reference video with corresponding SFM algorithms (e.g.. COLMAP ParticleSFM (Zhao et al. 2022)), camera motion can be easily estimated and used for imitation. In this sense, Cam can also be seen unsupervised method to achieve video motion transfer.
also serve input. With the help of
C VISUAL COMPARISONS WITH STATE-OFFTTEE-ARTS
Sec. showcases some qualitative comparisons between CamTrrol and different state-of-the-an ap- proaches. For better visualization and comparison, we provide more video results the supplemen- tary materials.
D IMPLEMENTATION DETAILS
For text prompt input, use Stable Diffusion v2-1 1 Stable Diffusion XL generate the initial image. The inpainting model we apply is Stable Diffusion inpainting model proposed by Runway and the backward step of inpainting is set 25. We ZeoDepth [ as depth estimation model. The classifier-free guidance scale follows the default setting of the base models themselves. We use
https://github.... hhans://gthuuu.om/////o///////e wtwph//ww...cmm./ooooo////eeaatt
14

---

Published as a conference paper at ICLR 2025
Table 4: Computational analysis of on 576 1024.
SVD
MotionCtrl
CameraCtrl
CamTrol (to 15)
Max GPU memory(MB)
34236
71096
34236
Time (s)
pre-process
149
inference
32
54
22
Table 5: Definitions of camera motions. scribe different camera movements.
We adopt the terminology from cinematography to de-
Camera Motion
Directions
Definition
In
Zoom
Camera moves towards away from subject.
Out
Up
Tilt
Rotating the camera vertically from a fixed position.
Down
Left
Pan
Rotating the camera horizontally from a fixed position.
Right
Up
Pedestal
Moving the camera vertically in its entirety.
Down
Left
Truck
Moving the camera horizontally in its entirety.
Right
Clockwise
Roll
Rotating the camera in its entirety in a horizontal manner.
Anticlockwise
Clockwise
Rotate
Moving the camera around a subject.
Anticlockwise
Hybrid
Combinations
Spatial and temporal combinations of basic motions.
Complicated
Arbitrary
Camera extrinsic matrices or reference videos as input.
SVD's default setting of 6 fps for video generation, and process reference videos to 6 fps for FVD and FID calculations. The complete procedure of CamTrol is elucidated in Algorithm[]
For computational analysis, we set both the number of frames and the decoding size to 14, and the generation steps to 25. We do not apply xformers any of the approaches. The computational analysis at 576 x 1024 resolution is shown in Table 4 SVD-based CameraCtrl only supports resolution at 320 x 576.
E CHOICE OF 3D REPRESENTATION
In Sec. 3.1 we designate point cloud as the intermediate 3D representation for explicit camera motion modeling. Doubts may arise as why we do use more complex 3D representation which might be more useful. Here we take the most prevalent 3D representation-3D Gaussian Splatting as an example to elaborate on the benefits of using point cloud in three aspects:
Firstly, concerning the input, point cloud is able to construct the entire scene from one single input image combining techniques depth estimation and inpainting. GS, though also an explicit 3D representation, requires optimization based on images from different views, which means it is neither capable handling single input image, nor can leverage 2D inpainting to gradually complete the scene.
15

---

Published as a conference paper at ICLR 2025
Algorithm 1: Training-free camera control for video generation
Input: Text prompt p, camera motion P, input image Io (optional).
// Stage I: Camera Motion Modeling:
for N-1 do I, = inpainting(4.1,PPPP))))
D: :epth(I.))
while not converged do
- argming (Ers | ((],,k)),, - -P--
end
P. = oiL, P.,).. (P,, KP,,K,PPH. K.P.);
end
// Stage II: Layout Prior Ceneration:
Vo + (1))- n Sample random noise c ~ N(OI))
12 Motion inversion Vo + Vol, Vo + VI C:
for !=to.....! do
V--Vi-aei((V.)) voi
14
V-- + vas-1
+ 11-a: a: -1-0-110067 (V,) + +
15 end
Secondly, from the aspect of time, point cloud can directly lift 2D points onto 3D spaces, while 3DGS demands optimization for each scenario. As a training-free approach, our method takes al- most no time to generate video after multi-view images acquired, but would need more time if 3DGS were applied.
Lastly, from the task itself, the goal of stage is not to precisely reconstruct the 3D scene but rather to offer a rough layout guidance. this context, rendered images from point cloud are sufficiently qualified, and further refinement of the 3D reconstruction is necessary.
From the analysis, point cloud is quite qualified serving as a rough layout guidance in a relatively quick speed, without any further optimization redundant multi-view images as input. The use of point cloud allows our algorithm to be completely training-free, while still being able to produce camera-moving videos quickly with only one single image or text prompt input.
F DETAILS ABOUT GENERATING 3D ROTATION VIDEOS
3D generation models (oleti et al. 2024 Melas Melas-Kyriazi et al., 2024 Shi et al. 2023 Chen et al., 22044] [Han al. 2024) are trained on Highly-reeulltee 3D datasets, these datasets are hard to collect, and include only a limited variety of data types (e.g., single static objects with no background). As consequence, 3D generation models exhibit a very constrained output distribution and can only produce results in certain styles. CamTrol avoids these problems of arduous data collection, labori- ous finetuning, and output collapse by utilizing the layout prior hidden video foundation models. Not only does it require no training, but this advantage also benefits CamTrol from inheriting most of the prior knowledge inside video foundation models, such as the diversity of scenarios, the dy- namics of moving objects, temporal consistency, etc. Thus, Cam is able to generate dynamic 3D content, both objects and scenes, in a totally unsupervised and training- free manner, this is what other methods cannot achieve yet.
Compared with regulated datasets, the problem of processing wild pictures 3D is that some of the parameters unknown. In Sec. 3.1l mentioned that the camera intrinsic matrix K and the initial extrinsic matrix Po are set by convention, as they are usually intractable. Another crucial parameter concemiin 3D video generation is the distance between the camera and the content of
16

---

Published as a conference paper at ICLR 2025
input image (denoted as f), note that input image could be synthetic or real. Considering that most camera rotations are performed around the center point, we extract a patch from the very center of the input image, conduct depth estimations it, and define the distance f as the average depth. The rotation and translation matrices of camera movement can be formed as:
cos 0
0
sin 0,
J sin 0: cos 0g
Ry
sin Og
cos 0
(6)
where f
D(ro + J. 90 + k:).
P
GAAEPP
Here, E N 1] and 0, refer to the rotation angle around the y axis at step denotes depth estimation of the image, and P represents the patch around the central point (Io, Wo). In our experiment, we choose (j.k) E [-10, 10] as the size of the patch.
G MOTION BLUR, PROBLEMS AND SOLUTIONS
Videos produced by CamTrol need to satisfy certain camera movements, and some drastic per- spective changes might cause visible trails, recognized as motion blur of objects scenes. This phenomenon will appear to be more indispensable when the video base model has a relatively small generation length (e.g., 16 frames) well the motion scale becomes larger (e.g., tilt up by 60 degrees or more). To avoid blur issues when controlling video camera motion, we propose several solutions follows:
1. According the analysis above, the blur issue is caused by limited generation frames and large camera movements, thus the most intuitive solution is to either reduce motion scale or use more capable generation model. For severe perspective changes, the optimal approach would involve employing video foundation models that support larger generation lengths (e.g., VideoX (Yang et al., 2024b) supports generating videos with 49 frames). This allows the model to manage motions of equivalent magnitude while experiencing smaller moving range between adjacent frames, thereby bringing effective alleviation to the blur problem.
2.
One can also stack the results of multiple generations to form complete output, i.e., treat- ing the last frame the previous generation as the starting frame for the next and integrat- ing them as whole. This approach is more suitable when using an image-to-video (12V) base model. Since most open-source video foundation models are text-to-video (T2V), one may consider increasing the step of camera motion inversion to, which guarantees more fidelity towards input images' content (and motion).
3. Besides the above two approaches, applying frame interpolation to the output is also common and convenient choice. Many off-the-shelf frame interpolation models are open- source and can found on GitHub.
4. Lastly, the video length cannot be altered, it may be necessary increase the fps of the generated videos for better visual quality. Although larger frame rate leads to shorter video duration, it simultaneously makes the visual persistence brought by motions less pronounced, which reduces blur visually.
In experiments, we take the raw output in all settings.
17


Reviews:
[{"review_id": "5Dmx6irYDe", "reviewer": "Reviewer_4R7b", "summary": "The paper introduces a method named CamTrol to enable controllable camera movements for video generation using off-the-shelf video diffusion models. The innovation lies in its training-free approach—no need for additional supervised or self-supervised training. The method employs a two-stage process:\n1. 3D Camera Motion Modeling: It use a single view image as input to regressive a 3D point cloud, then gradually rendering, refining the 3D point cloud. The rendering process adopt the camera poses to simulate camera motion. The generated image sequence is later used to guide the perspective and layout in generated video sequences.\n2. Noise Prior Guidance: It utilizes the “layout prior” of noisy latents, first adding noise of the image sequence, injecting the camera motion prior to the noisy latents. Then, providing the noisy latents into a pretrained video generation model to generation a reasonable and camera viewpoint alignment video.", "questions": "1. What is the evaluation dataset for the Table.1 ? If the references are come from the RealEstate10K dataset (Line 322), it is strange that the Video quality metrics of MotionCtrl and CameraCtrl are worse than the CamTrol, since both MotionCtrl and CameraCtrl are fine-tuned on the RealEstate10K dataset.\n2. What is the base video generation models of MotionCtrl and CameraCtrl in Table.1 ? If the base video generators are not SVD, this comparison in Table.1 is unfair. Besides, can you provide the Video Quality of the vanilla SVD model for a reference?\n3. Since the ParticleSFM may fail sometimes, a natural question is: how accurate is the camera computed ParticleSFM on the ground truth videos? It would be good to have some reference number on the ground truth dataset to show the lower bounds of Motion Accuracy metrics in Table.1 \n4. In some cases of the provides webpage,  with the same camera trajectory (especially on complex trajectories), different videos do not have the same camera movement. For example, the complex trajectory II, the first video shows downward at the end of the video, the third video shows upward, the last video ignores the last camera movement. However, MotionCtrl and CameraCtrl do not have this kind of inconsistency. Can the author provide some quantitative results on complex camera trajectories?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. CamTrol is a training-free solution, it does not need additional training, making it computationally efficient and easy to integrate with existing video diffusion models.\n2. It investigate the feasibility to adopt the “noise prior of latent” technique to control the camera viewpoint without direct supervision.\n3. CamTrol outperforms competing methods in both perceptual quality and motion alignment, particularly in complex camera movements, as demonstrated in both quantitative and qualitative experiments.", "weaknesses": "1. Since the whole pipeline is complex (depth estimation -> point cloud lifting -> rendering -> inpainting -> depth coefficient optimization), CamTrol’s quality is vulnerable. A problem at each of the current steps will have a bad effect on the next step. For example, if the depth estimation model cannot give the precise results, the structure of the entire scene may be strange when viewed from the different perspectives. If the inpainting model is not good enough, the generated videos may have much inconsistency.\n2. For different videos (and different camera trajectories), the best timestep t_0 may be inconsistent. It is hard to design a rule to choose the best t_0.\n3. The inpainting model is designed for single images, which may lead to inconsistent content and difficulty in generating reasonable new content. Additionally, the noisy latent not only provides layout priors but also carries some semantic information, which is unfavorable for the video generation model when creating new content. This makes CamTrol potentially unsuitable for generating long videos.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "65Hh2rU2Xv", "reviewer": "Reviewer_WcRk", "summary": "This work proposes a training-free solution for camera control in video generation, requiring no supervised fine-tuning. A given image is first projected into 3D point cloud space and a sequence of views are then rendered given the camera trajectory. The latter serves as a structure condition for generation through inversion.", "questions": "Please check the weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The writing is pretty clear, starting from the observation that camera movement could be regarded as one latent layout rearrangement. \nThe two-stage framework is well-presented and easy to understand. \nThe ablation study is comprehensive for multiple designs of proposed methods.", "weaknesses": "There are multiple major concerns:\n- The novelty of this pipeline is limited which combines point cloud reconstruction and inversion. For example, Infinite Nature [a] also uses a RGBD image together with rendering to generate novel views and then refine. \n- The claim for training-free could be further clarified since depth coefficient optimization is also adopted (as mentioned at L190). Although the base model is not tuned, this optimization could be empirical and time-comsuming, preventing it from the practical applications. \n- As stated at L210, video should be encouraged with more dynamics, which is quite important. Yet, we do not find any design/modification for this.\n- From one side, inversion provides the trade-off between fidelity and diversity (L261). From the other, it is challenging to choose which timestep should be used for the generation with given camera conditions. \n- How the proposed method tackle text-to-video generation tasks? Fig.1 suggests that an input image is always given.\n\n\n[a] Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "utHg9fZ2RW", "reviewer": "Reviewer_9UGe", "summary": "This paper proposes a framework to control the camera viewpoints of the video generation process. This frame work is training free, utilizing some existing pre-trained models, like depth estimation model, and the depth coefficient optimization model. It can be used on many different base video generators. Given a real or T2I-generated images, this frame first generate a series of images according to the camera trajectories, with the point clouds as the middle 3d representation. Then, these images are noised to a certain degree. After that, these noised latents are denoised with the video generator, getting the videos with desired camera trajectory.", "questions": "Besides the first point in the weakness, I have the following questions.\n1. In Table 1, what are the base video generator for MotionCtrl and CameraCtrl? Can you use the same base video generator for Motion, CameraCtrl, and CamTrol, like SVD to quantitatively evaluate them?\n2. In the camera motion modeling step, from the same first image, it seems that the CamTrol can generate different image sequences, according to the camera trajectories. If seeding these image sequence into the second step (layout prior of noise). Can we get some videos with similar appearance but different camera trajectories? Can you have some comparisons with the CVD (Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control) method?\n3. How to find the best trade-off of t_0 for each videos?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The framework is totally training free, combining some pre-trained models.\n2. The written is easy to follow.\n3. Visualization results demonstrate the effectiveness of each part in this pipeline.", "weaknesses": "1. In Table.1 , some quantitative comparison between the SVD and CamTrol+SVD is lacked, using the FVD, FID, IS, and CLIP-SIM, metrics. This comparison can reflect the impact of using proposed pipeline on the pretrained video generators.\n2. In the demo video, there are some obvious consistency or unreasonable on objects in generated videos. One possible reason is the in painting model cannot handle some situation well, for example, when the camera motion is large, leading much holes in the images, the inpainting model cannot get enough context to generate reasonable contents for the holes.\n3. Continue with the second point, this pipeline may have some difficulty in properly use the pre-trained video generator to generate consistent and reasonable videos.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w8aMRfUAOG", "reviewer": "Reviewer_syyx", "summary": "This paper propose a training-free camera movement control for off-the-shelf video diffusion models. Firstly, the image is converted to 3D point cloud space using depth estimator, and the video frame is re-rendered using the camera motion trajectory. In this process, inpaint is used to repair the vacant part of the point cloud in the rendering result, and finally, generate videos using layout prior of noisy latents(inversion from the rendering result). The experiments and demo show good results.", "questions": "1. In the quantitative evaluation, could you clarify the differences in the experimental settings for MotionCtrl, CameraCtrl, and other related papers? I noticed that there does not appear to be a uniform experimental setup in the field of camera control video generation.\n2. The paper states that it compares CamTrol with state-of-the-art works, including AnimateDiff, MotionCtrl, and CameraCtrl. While it mentions that \"AnimateDiff is excluded from quantitative analysis due to its incapacity to handle complicated trajectories,\" I still have not seen a qualitative comparison with AnimateDiff. Additionally, could you consider comparing your work with other relevant studies, such as CamCo[1] and Vd3d[2]?\n3. The current results are excellent in terms of trajectory control, but they do not satisfy me in terms of video quality. Given the availability of a superior fundamental video model (with larger resolution and longer duration), and considering the combination with CogVideoX presented in this paper, I would appreciate it if you could provide some generated samples or comparisons for a more thorough evaluation. I will adjust the final score based on these results.\n\n[1] Xu, Dejia, et al. \"CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation.\" arXiv preprint arXiv:2406.02509 (2024).\n[2] Bahmani, Sherwin, et al. \"Vd3d: Taming large video diffusion transformers for 3d camera control.\" arXiv preprint arXiv:2407.12781 (2024).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The presentation of this paper is clear. The proposed method is reasonable, effectively generating videos with camera trajectories from a higher-dimensional space (3D point cloud). Notably, the proposed method is training-free, allowing for easy adaptation to new fundamental video generation models and related research.", "weaknesses": "1. While the overall pipeline is effective, it is relatively complex, including depth estimation, rerendering, 2D image inpainting, video fundamental model redrawing, and other processes. Additionally, the integration of these various components is not particularly seamless. This complexity can lead to reduced generation efficiency and may result in inconsistencies between the different stages, which could ultimately compromise the quality of the output.\n2. The presentation of the experimental part was poor, and the comparison with the relevant work and the combination with the latest video base model were not seen in the demo web page, although they mentioned it in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NTtbVLX2ZZ", "reviewer": "Reviewer_14Er", "summary": "This paper proposes CamTrol, a training-free method for camera control in pre-trained video diffusion models. While previous works mainly train a camera control module using a fine-tuning dataset with camera annotations, this work does not require any external dataset to control the camera. The pipeline uses 3D point clouds and inversion to guide the generation process, enabling flexible camera control. CamTrol outperforms previous works which rely on external data, while enabling metric scale control based on the metric depth estimator.", "questions": "I currently rate this paper below the acceptance threshold. I have some concerns about the result quality and moreover it is not clear how large the computational overhead at inference time is compared to just having some simple embedding module as in MotionCtrl and CameraCtrl.\nI would like authors to address following questions:\n- Why does the scene often change with camera movement?\n- Why is the camera often not that accurate even for simple movements?\n- What is the computational overhead at inference time?\n\nI am open to adjusting my rating based on the rebuttal.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Control over motion strength: The method can synthesize videos with different camera motion strengths. Previous methods struggle due to having no knowledge of scale. Because the method uses a metric depth estimator, this problem should be fixed.", "weaknesses": "- Scene inconsistency: The scene changes when the camera moves. When looking at the supplementary website, “Tilt Down” changes the top part of the scene or “Pedestal Down” changes the whole arrangement of the scene. But the point of camera control is that the scene remains intact, so in the best case the scene could be reconstructed from the video.\n- Inaccurate camera: When looking at the supplementary website, the cameras are not that accurate. E.g., “Pan Left” also moves forward in the scene and doesn’t look like panning but rotating on the spot.\n- Missing computational costs analysis: The approach seems very expensive. The authors motivate that previous works fine-tune some camera control modules on a camera annotated dataset which requires additional training. However, those approaches can do inference with basically zero computational overhead after training. This optimization-based approach requires computational overhead for each inference run and it is not clear how significant that is. E.g., if a user has to wait tens of minutes to hours to generate one result, there is a trade-off reached after a few prompts where it is more practical to train a camera control module once and then use it without overhead. It’s important to analyze both optimization time and memory usage.\n- Missing quantitative ablations: The ablations mentioned in Sec. 4.3 are missing quantitative comparisons. The depth optimization seems to change nearly nothing in Tab. 2. When looking at relative differences between metrics, the metrics differences are basically 0.\n- Writing quality: While the paper starts ok in terms of grammar, it gets worse throughout the paper and it makes the paper hard to read, especially the method section. I recommend putting the paper through some grammar correction tool.\n- Unclear motivation of generalizability: The paper mentions in the introduction that methods fine-tuned on RealEstate10K suffer from generalizability issues. But there are no experiments or visuals which demonstrate this. Does the method really work better when cameras totally different from the training distribution are used?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)