CameraCtrl: Enabling Camera Control for Video Diffusion Models
Authors: Hao He
Yinghao Xu
Yuwei Guo
Gordon Wetzstein
Bo Dai
Hongsheng Li
Ceyuan Yang
Venue: ICLR 2025 Poster (2025)
Keywords: camera viewpoints control in Video Generation
Primary area: generative models

Abstract: Controllability plays a crucial role in video generation, as it allows users to create and edit content more precisely. Existing models, however, lack control of camera pose that serves as a cinematic language to express deeper narrative nuances. To alleviate this issue, we introduce \method, enabling accurate camera pose control for video diffusion models. Our approach explores effective camera trajectory parameterization along with a plug-and-play camera pose control module that is trained on top of a video diffusion model, leaving other modules of the base model untouched. Moreover, a comprehensive study on the effect of various training datasets is conducted, suggesting that videos with diverse camera distributions and similar appearance to the base model indeed enhance controllability and generalization. Experimental results demonstrate the effectiveness of \method in achieving precise camera control with different video generation models, marking a step forward in the pursuit of dynamic and customized video storytelling from textual and camera pose inputs.

Published as a conference paper at ICLR 2025
CAMERACTRL: ENABLING CAMERA CONTROL FOR VIDEO DIFFUSION MODELS
Hao Hel.2-
Yinghao Xu
Yuwei Guo .2
Hongsheng Li 1 The Chinese University of Hong Kong
Gordon Wetzstein 3
Bo Dai
Ceyuan Yang31
2 Shanghaai Artificial Intelligence Laboratory
3 Stanford University
ABSTRACT
Controllability plays a crucial role in video generation, as allows users to create and edit content more precisely. Existing models, however, lack control of camera pose that serves as a cinematic language to express deeper narrative nuances. To alleviate this issue, introduce CameraCtrl, enabling accurate camera pose control for video diffusion models. Our approach explores effective camera tra- jectory parameterization along with plug-and-play camera pose control module that is trained on top of a video diffusion model, leaving other modules of the base model untouched. Moreover, comprehensive study on the effect of var- tous training datasets is conducted, suggesting that videos with diverse camera distributions and similar appearance to the base model indeed enhance controlla- bility and generalization. Experimental results demonstrate the effectiveness of Camer rattrl achieving precise camera control with different video genera- tion models, marking a step forward in the pursuit of dynamic and customized video storytelling from textual and camera pose inputs. The project website is at: attps://behaol: github. wt/pri//ppo/. http/pooeeccttt CameraCtr1/
INTRODUCTION
Recently, diffusion models have significantly advanced the ability to generate video from text or other input (Blattmann et al. 2023b] Xing et al., 2023] Wu et al. 2023 Ho et al.] 2022a| Guo et al. 2023b), with transformative impact digital content design workflows. In the applications of practical video generation applications, controllability plays a crucial role, allowing for better customization according
needs. This improves the quality, realism, and usability of the generated videos. Although text and image inputs are commonly used to achieve controllability, they may lack precise control over visual content and object motion. To address this, some approaches have been proposed,
leveraging control signals such as optical flow (Yin et al. 2023 Chen et al. [023b] (Sh et al. (024).
pose skeleton Ma et 2023 Ruiz et al., 2023), and other multi-modal signals (Wang et al. 2024
Ruan et al. 2023), enabling more accurate control for guiding video generation.
However, existing models lack precise control over adjusting or simulating camera viewpoints in video generation. The ability to control the camera viewpoints during the video generation process crucial in many applications, such as visual reality, augmented reality, and game development. Moreover, skillful management of camera movements enables creators to emphasize emotions, highlight character relationships, and guide the audience's 's focus, which holds significant value in the film and advertising industries. Recent efforts have been made to introduce camera control in video generation. For example, AnimateDiff (Guo et al. 2023b) incorporates a MotionLoRA module on top of its motion module, enabling some specific types of camera movement. Nevertheless, it struggles to generalize to the camera trajectories customized by users. MotionCtrl (Wang et al. (202)) offers more flexible camera control by conditioning video diffusion models sequence of camera pose parameters, but it relies solely on the numerical values of camera parameters without geometric cues
"This work was done while the author was an intern at Shanghai Artificial Intelligence Laboratory. "orrrsponniig author

---

Published as a conference paper at ICLR 2025
Text condition (AnimateDiff base model)
Camera Control Model O
D
Texl Condition (AnimateDiff with Realistic Vision)
Diffusion
Image Condition (Stable Video Diffusion)
+ Image & Text condition (SparseCtrl)
can control the camera trajectory for both general T2V (Guo et al. (023b) and personalized T2V generation (civitai), shown in the first two rows. Besides, illustrated in the third row, it can be used with I2V diffusion models, like Stable Video Diffusion (Blattmann et al. 2023a). The condition image is the first image of row 3. CameraCtin can also collaborate with other visual controllers, such as the RGB encoder from SparseCtrl (Guo et al.) 2023a to generate videos condition image and text conditions and manage camera movements.
Figure Illustration of CameraCtrl.
of camera poses, which insufficient in ensuring precise camera control. Additionally, MotionCtrl lacks the ability to generalize camera control across other personalized video generation models.
We thus introduce CameraCtrl, learning a precise plug-and-play camera pose control module that could control camera viewpoints in video generation. Considering that seamlessly integrating camera control module into existing video diffusion models is challenging, we investigate how to represent and inject the camera pose effectively. Concretely, adopt Plücker embeddings (Sitzmann et al. 2021) as the primary form of camera pose conditioning. This choice is attributed to their encoding of geometric interpretations for each pixel in video frame, offering comprehensive description of camera pose information. To ensure the applicability and generalizability of our CameraCtrl after training, we introduce a camera control module that only takes the Plücker embedding input and is thus agnostic to the appearance of the training dataset. To evaluate effective training strategies of the camera control model, comprehensive study is conducted to investigate the effects of various types of training data, ranging from photorealistic to synthetic data. Experimental results suggest that data (e.g., RealEstate (Zhou al. 2018) with similar appearance to the original base model and diverse camera pose distribution achieve the best trade-off between generalizability and controllability.
We first implement CameraCtrl on of AnimateDiff, enabling precise camera control in text- to-video (T2V) generation across various personalized T2V models (see Fig. IT] rows 1-2). We also integrate CameraCtrl with Stable Video Diffusion (Blattmann et al. 2023a) to achieve camera control in the image-to-vided (I2V) setting, illustrated in the third row of Fig. [ In addition, as shown in the last of Fig. Came r is also compatible with other plug-and-play modules, e.g. SparseCtrl (Guo et al. 2023a) to control video viewpoints under the conditions of both text and structure information, e.g., image.
In summary, our main contributions have three parts:
We introduce CameraCtrl, which empowers video diffusion models with flexible and precise controllability over camera viewpoints.
The plug-and-pe camera control module be adapted to various video generation models, producing visually appealing camera control.
We provide a comprehensive analysis datasets to train the camera control module. We hope this will be helpful for future research in this direction.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Video generation. Benefiting from the stability in training and well-established open-sourced communities, recent attempts at video generation, mainly leverage diffusion models (Ho al.)
2020:
Song et al.] 2020 Peebles & Xie, 2023b). Many recently video diffusion models are the
T2V models (Karras et al. 2023; Ruan et al.] 2023 Zhang et al., 2023b He al.) 2022 Chen et al. 2023a Hong et al. [022] Yang et al. 2024b). Some approaches seek to replace the guidance
signal from text to image, focusing on the 12V setting (Chen al. 2023b d Esser et al. (023). As
pioneering approach, Video Diffusion Model (Ho et al. 2022b) expands a 2D image diffusion architecture to accommodate video data and jointly train the model on image and video from scratch. To utilize powerful pre-trained image generators, such Stable Diffusion (Rombach al. 2022), later works inflate the 2D architecture by interleaving temporal layers between the pre-trained 2D layers and finetune the new model on large video dataset (Bain et al. 2021). Among them, Align-Your- Your- L Latents (Blattmann et al.) 2023b) efficiently turns a text-to-image (121) model into a video generator by aligning independently sampled noise maps. Stable video diffusion (SVD) (Blattmann al. 2023a) extends Align- Youur-Latettt by more elaborate training steps and data curation. AnimateDiff (Guo al. 2023b) utilizes a pluggable motion module to enable high-quality animation creation on personalized image backbones (Ruiz et al. (023). To enhance temporal coherency, Lumiere (Bar- Tal al. 2024) replaces the commonly used temporal super-resolution module and directly generates full- frame-rate videos. Other significant attempts include leveraging scalable transformer backbone (Ma ct al., 2024), operating in ppace-temporal compressed latent space, e.g., W.A.L.T. (Gupta al. 2023) and Sora (Brooks et al. 2024). and using discrete token with language model for video generation (Kondratyuk et al. 2023). Please see (Po al. 2023) for a comprehensive survey.
Controllable video generation. The ambiguity of text image conditioning alone often leads to weak control for video diffusion models. To provide enhanced guidance, some works adopt other signals, e.g., depth/skeleton sequence, to precisely control the scene/human motion in the generated videos Guoo et al. 2023a Chen et al. 2023c /Zhang et al. 2023c Khachatryan et al., 2023 Hu et al.
2023
Xu et al.] 2023). Another method (Guo al. 2023a) utilizes sketch images as the control signal, contributing to high video quality or accurate temporal relationship modeling. In contrast, our work focuses on camera control during the video generation process. AnimateDiff adopts efficient LoRA (Hu et al.) 202T) finetuning to obtain model weights specified for different camera movement types. Direct-a- Video Yang et 2024a) proposes a camera embedder to control the camera pose of generated videos, but only conditions three camera parameters, limiting its camera control ability to the most basic types, such as pan left. MotionCtri (Wang et al., 2023) takes more camera parameters as input to control the camera viewpoints. However, solely relying on the numerical values of camera parameters restricts the accuracy of camera control, and the necessity to fine-tune part of the video diffusion model's parameter can hamper its generalization ability across different video domains. In this study, we aim to control the camera poses during the video generation process precisely, and expect the corresponding model can be used in various video generation models.
3 CAMERACTRL
Introducing precise control of the camera into existing video generation methods is challenging, but holds significant value in terms of achieving desired results. To accomplish this, we address this problem by considering three key questions: (1) How can we effectively represent the camera condition to reflect the geometric movement in 3D space? (2) How can we seamlessly inject the camera condition into existing video generators without compromising frame quality and temporal consistency? (3) What type of training data should be utilized to ensure proper model training?
This section is thus organized as follows: Sec. 3.1 presents brief background discussion of video generation models; Sec. 3.2 introduces the camera representation used by CameraCt rl; Sec. 3.3 presents the camera model 4. for injecting camera representation into the video diffusion models. The data selection process is discussed in Sec. 3.4
3.1 PRELIMINARIES OF VIDEO GENERATION
Video diffusion models. T2V diffusion models have seen significant advancements in recent years. Some approaches (Singer et al. 2022 Ho et al. 2022b) train video generators from scratch, while
3

---

Published as a conference paper at ICLR 2025
others (Guo al. 2023b Blattmann et al. 2023b) utilize powerful T2I diffusion models as the pretrained model and train some temporal blocks on them. Besides, some methods jointly train the video generators using images and videos (Yang et al. 2024b). Despite with different training recipes, these models often adhere to the original formulation used for image generation. Concretely, sequence of N images (or their latent features) =/=h are first added noises gradually to normal distribution at '' steps. Given the noised input 1i: at the & step, neural network ég is trained to predict the added noises. During the training, the network tries minimize the mean squared error (MSE) between its prediction and the ground truth noise scale; the training objective function is formulated as follows:
(() -. .r Ille εo(2 1:N ct, (11)),
where C2 represents embeddings of the corresponding condition signal, like text prompts.
Controllable video generation. In addition to text conditioning, there have been further advance- ments in enhancing controllability. By incorporating additional structural control signals 8, (e.g., depth maps and canny maps) into the process, controllability for both image and video generation can be enhanced. Typically, these control signals are first fed into an additional encoder 4, and then
injected into the generator through various operations (Zhang et al.) [203aa] Mou et al.) [023 Ye ct al. 2023). Consequently, the objective of training this encoder can be deñned as follows:
[()) gg s,,cc.s II|e (=(= 1:N Cr, (st), ()))
In this work, we take the camera poses as an additional control signal to the video diffusion models and strictly follow the objective of Eq. 2 to train our camera encoder ee..
3.2 CAMERA POSE REPRESENTATION
Before diving into the architecture and training of the camera control module, we first investigate which kind of camera representation could precisely reflect the camera movement in 3D space.
Camera representation. Typically. the camera pose refers to the intrinsic and extrinsic parameters, denoted as K E 3x33 and E [R; t| E _3x4 respectively, where R E 3233 representations the rotation part of the extrinsic parameters, and t E R is the translation part.
To condition video generator camera pose, one straightforward choice is to feed raw values of the camera parameters into the generator. However, such a choice may not contribute to accurate camera control for several reasons: (1) While the rotation matrix R is constrained by orthogonality, the translation vector t is typically unconstrained in magnitude, leading to a mismatch in the learning process of camera control model.
(2) Direct use of raw camera parameters makes it difficult for the model to correlate these values with image pixels, limiting precise control over visual details. We thus choose Plücker embeddings (Sitzmann al. [021] as the camera pose representation. Specifically, for each pixel (u, v) in the image coordinate space, its Plücker embedding Pu,e (o du. d E R where o E R33 is the camera center in world coordinate space, and du, R3 is the direction vector in world coordinate space pointed from the camera center the pixel (u, v), it is calculated as,
d RK- [u, u, 11" + t.
Then, it is normalized to ensure it has a unit length. For the i-th frame in a video sequence, its Plücker embedding can be expressed P. 16xhxx where and w are the height and width for the frame.
Note that Eq. represents the inverse process of camera projection, which maps point from the 3D world coordinate space into the pixel coordinate system through the use of matrices E and K. Thus, compared with the numerical values of extrinsic and intrinsic matrices, the Plücker embedding has more geometric interpretation for each pixel of a video frame then can provide a more informative description of camera pose information to the base video generators. Consequently, it can better adopt the temporal consistency ability of base video generators to generate video clips with a specific camera trajectory. Besides, the value ranges of each item in the Plücker embedding are more uniform, which beneficial for the learning process of a data-driven model. The illustration of different camera representations are in the Fig. 6, both the camera matrices and the Euler angles are numerical values, while the Plücker embedding is a pixel-wise spatial embedding. After obtaining the Plücker embedding P, for the camera pose of the i-th frame, represent the entire camera trajectory of a video as a Plücker embedding sequence P e IR" xxxxxxxxx where n denotes the total number of frames in a video clip.
4

---

Published as a conference paper at ICLR 2025
Camera Encoder C
Ct
2t (xx
Ej.A1
ResNet
I [(bxhxw)xxxe
Temponal Attention
2D U-Net Block
(bxxooo)
Pitcter Enteeding
Pe P Bhsazeshoww
layer
Temporal Attention Layer
Fpred
Pre-trained vidco diffusion model
21
(b)
Figure 2: Framework of CameraCtrl. (a) Given a pre-trained video diffusion model (e.g. An- imateDiff (Guo et al., 2023b)) and SVD (Blattmann et al.] 2023a), CameraCtr] trains camera encoder on it, which takes the Plücker embedding input and outputs multi-scale camera represen- tations. These features are then integrated into the temporal attention layers of the U-Net at their respective scales to control the video generation process. (b) Details of the camera injection process. The camera features C and the latent features 2 are first combined through the element- wise addition. A learnable linear layer is adopted to further fuse representations which are then fed into the first temporal attention layer of each temporal block.
3.3 CAMERA CONTROLLABILITY INTO VIDEO GENERATORS
Due to the fact that a camera trajectory is parameterized by Plücker embedding sequence pixel-wise spatial ray map, we follow the literature (Zhang et al. 2023a Mou et al. (2023) by first using an encoder model to extract the features of Plücker embedding sequence and then fusing the camera features into video generators.
Camera encoder. Given specific camera encoder, we can take both the Plücker embedding sequence and the corresponding image features its input, like the ControlNet (Zhang et al. 2023a). Alternatively, we can input only the camera features into the camera encoder, as done by the T21- Adaptor (Mou al., 2023). Through empirical analysis, observe that the first approach tends to leak appearance information from the training dataset due to the use the input image's 's latent representation. This causes the model rely the appearance bias inherent in the training data, thereby limiting its ability to generalize camera pose control across various domains. Therefore, as illustrated in aa) our camera encoder &. only take the Plücker embedding input and delivers multi-scale features. Based on the encoder used in T2I-Adaptor, we introduce a camera encoder specifically designed for videos. This camera encoder includes a temporal attention module after each convolutional block, allowing it to capture the temporal relationships between camera poses throughout the video clip. The detailed architecture of camera encoder is Appendix [.I
Camera fusion. After obtaining the multi-scale camera features, we aim to integrate these features seamlessly into the U-Net architecture of the video diffusion model. Thus, we further investigate the different types of layers in the U-Net, aiming to determine which layer should be used to incorporate the camera information. Recall that the U-Net model contains both spatial and temporal attention. We inject the camera features into the temporal attention blocks. This decision stems from the capability of the temporal attention layer to capture temporal relationships, aligning with the inherent sequential and causal nature of camera trajectory, while the spatial attention layers always picture the individual frames. This camera feature fusion process is shown in Fig. b). The image latent features 2, and the camera pose features e are directly combined through pixel-wise addition. Then, the integrated feature is passed through a linear layer, whose output fed directly into the fixed first temporal attention layer of each temporal attention module.
3.4 LEARNING CAMERA DISTRIBUTION IN DATA- DATA-DRIVEN DRIVEN MANNER
Training the aforementioned camera encoder and the fusion linear layers video generator usually requires a lot of videos with camera pose annotations. One can obtain the camera trajectory through stracturefrommmmotion (SfM), egg. COLMAP (Schönberger & Frahm 2016) for realistic videos while
5

---

Published as a conference paper at ICLR 2025
others could collect videos with ground truth camera poses from rendering engines, such as Blender. We thus investigate the effect of various training data on the camera- ceneraccontollled generator.
Dataset selection. We aim to select dataset with appearances that closely match the training data of the base video diffusion models and have as wide a camera pose distribution as possible, We choose
three datasets as the candidates: Objaverse (Deitke et al.] 2023), ImaageeNet (Yu et al. 2023 and
RealEstate 10K (Zhou et al. 2018). Samples from the three datasets can be found in the Fig. 5
Indeed, datasets with cemmutter-gnnraatee imagery, such Objaverse (Deitke al.. 2023), exhibit diverse camera distributions since we can control the camera parameters during the rendering process. Yet, these datasets often suffer from distribution gap in appearance when compared to real-world datasets, such as WebVidIOM (Bain et al. 2021). which is used to train the base video diffusion model. When dealing with real-world datasets, such MVImageNet and RealEstate IOK, the distribution of camera parameters is often not very broad. In this case, a balance needs to be found between the complexity of individual camera trajectory and the diversity among multiple camera trajectories. The former ensures that the model learns to control complex trajectories during each training process, while the latter guarantees that the model does not overfit to certain fixed patterns. In reality, while the complexity of camera trajectories in MV MVmmageNet may slightly exceed than that of RealEstaae 10K for individual trajectories, the trajectories of MVImageNet are typically limited to horizontal rotations. In contrast, RealEstate 10K showcases wide variety of camera trajectories. Considering our goal is to apply the model to a wide range of custom trajectories, we ultimately selected RealEstate 10K as our training dataset. Besides, there some other datasets with characteristics similar to RealEstate 10K, such as ACID (Liu et al. 2021) and GannequinChallengg (Li et al. 2019). but their data volume is much smaller than RealEstaaee We tried combine them with RealEstate 10K to jointly train the CameraCtr] rl model, but found no benefit.
Measuring the camera controllability. To monitor the training process of our camera encoder, we design two metrics to measure the camera control quality by quantifying the error between the input camera conditions and the camera trajectory of generated videos. Concretely, we utilize COLMAP (Schönberger & Frahm 2016) to extract the camera pose sequence of generated videos, which consists of the rotation matrices Reen E R3*33x3 and translation vectors Ten E TR xgxx3x1 of camera trajectory. Furthermore, since the rotation angles and the translation scales are two different mathematical quantities, we measure the angle error and translation error separately and term them as RotErr and TransErr. Motivated from (Belousov), the RotErr is computed by comparing the ground truth rotation matrix Rgg and Rgen. formulated as,
tr tr R&T gen gf
-1
RotErr Σ
arccos
(4)
2
where R and R Rye. represent the ground truth rotation matrix and generated rotation matrix for the i-th frame, respectively. And tr is the trace of matrix. To quantify the translation error, we use the L2 distances between the ground truth translation vector gt and T gens that is,
TransErr = ΣIT j=1
(5)
where T and Té gen are the ground truth and generated translation vectors in the i-th frame. More discussions of RotErr and TransErr can be founded in Appendix D.5
EXPERIMENTS
In this session, we evaluate CameraCtr1 with other methods and show its applications in dif- ferent video generation settings. Sec. 4.1 presents the implementation details. Sec. 4.2 compares CameraCtr: rl with baseline methods AnimateDiff (Guo al. (023b) and MotionCtrl (Wang et al.) 2023). Sec. 4.3 shows the comprehensive ablation studies of CameraCt. Sec. 4. express the various applications of rameractri.
4.1 IMPLEMENTATION DETAILS
Base video diffusion model. In the T2V setting, AnimateDiff V3 (Guo et al. 2023b) is used as the base model. AnimateDiff can be integrated with various T21 LoRAs base models across

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparisons. MotionCtrlye and MotionClrlsvD represent MotionCtrl with VideoCrafter (Chen et al. 2023a) and SVD (Blattmann et al., 2023a) as base model, respectively. Correspondingly, CameraCurlap and CameraCtrlsyD denote base models of AnimateDiff and SVD with ra respectively.
Method
FVD 4
CLIPSIM 1 FC
ODD 1 TransErr + RotErr 4 User Preference Rate (%)
AiimateDiff
1022.4
0.298
0.930
56.4
Incapable
Incapable
19.4
MotionCutlye
1123,2
0.286
0.922
42.3
14.02
1.58
37.0
CameraCtrlAD
1088.9
0.301
0.941
49.8
12.98
1.29
43.6
SVD
371.2
0.312
0.957
47.5
Incapable 10.21
Incapable 1.41
Incapable 26.9
MotionCullsvn
386.2
0.303
0.953
41.8
CameraCtrlsvr
360.3
0.298
0,960
46.5
9.02
1.18
73.1
different genres. This feature helps evaluate the generalization ability of CameraCtri. When implementing Came raCtrl the 12V setting, SVD (Blattmann et al., 2023a) the base model.
model with constant learning rate of 1x10-1 (T2V) or xx 10-5 (12V). As stated in Sec. 3.4 we choose RealEstate 10K as the dataset, with around 65 K video clips for training. The camera encoder and the linear layers for camera feature injection are trained together with batch size of 32 for 50K steps. More details in Appendix D.2
Training. We the AdamW optimizer to train
Evaluation metries. To ensure that our camera model does not negatively impact the appearance quality of the original video diffusion models, we utilize the Fréchet Video Distance (FVD) (Un-
terthiner al. 2018 2019), CLIPSIM (Radford al. 2021). and Frame Consistency (FC) (Huang et al. (223)) to evaluate the video appearance quality. Besides, motivated from the Dynamic Degree metric in VBnnh (Huang al.) 2023), we propose an Object Dynamic Degree (ODD), details in Appendix D.4. to evaluate the extent of object motion. Additionally, the quality of camera control evaluated using the metries R rr and TransErr introduced in the Sec. [3.4] For the reference
videos and (or) the text captions for FVD, CLIPSim, FC, ODD, random sample 1,000 videos from Web Vid dataset. For the RotErr, and TransErr, we have randomly chosen 1,000 videos and the corresponding camera poses from the RealEstate test set.
4.2 COMPARISONS WITH OTHER METHODS
Quantitative comparison. To prove the effectiveness of CameraCtrl, we compare it with two alternative methods: AnimateDiff with MotionLoRA (Guo et 20236). and MotionCtrl (Wang et al.) 2023). Notably, AnimateDiff supports only eight basic camera movements and we do not have the ground truth camera trajectories of these camera movements. Thus, we cannot calculate the RotErr and TransErr for AnffateDiff. Instead, we conducted user study (details Appendix B to assess user preference for camera control capabilities between different models. Besides, we compare CameraCtrl with MotionCtrl both T2V and 12V settings. The quantitative results are shown in Tab. [ The results of middle block are in the T2V setting, while the results of T2V setting shown in the bottom block. Compared to Animate Diff with Motion LoRA and MotionCtrl, it is evident that our approach outperforms them terms of camera control accuracies (TransErr, Rrr and user preference). The lower bounds of TransErr and RotErr are listed in the Appendix F. Besides, compared with the base models (AnimateDiff and SVD), CameraCtr. does not sacrifice the visual quality and the dynamic degree of the gerneated videos, demonstrated by the better or comparable metries of FVD, CLIPSIM, FC, and ODD.
Qualitative comparison. We also provide qualitative comparisons between CameraCtrl and MotionCtrl both T2V and I2V settings Fig. 3 From the comparisons between the first two rows, we find that MotionCtri cannot follow the camera condition, it shows the scene rotation, not the camera movement. contrast, CameraCtr) 11 can distinguish the camera movement from the scene motion, strictly following the camera trajectory condition. Besides, MotionCtrl is insensitive to small camera movements. As illustrated in the third row, the result MotionCtrl shows only forward camera movement, ignoring small movement the left in the condition trajectory. By comparison, the CameraCt result in the last row accurately follows both the forward and left camera movements. More qualitative comparison results be found in Appendix G
7

---

Published as a conference paper at ICLR 2025
Rocky coustline with crushing waves.
Figure 3: Qualitative comparisons between CameraCtrl and MotionCtrl. The first two rows are in the T2V setting, representing MotionCtrl with VideoCrafter and CameraCtrl with AnimateDiffV3 as base model, respectively. The last two rows are MotionCtrl and CameraCtrl with SVD as base model taking the image condition signal. Condition images the first images of each row.
4.3 ABLATION STUDY
We break down the camera control problem into three challenges, regarding the selection of camera representation in Sec. 3.2 the architecture of camera control model in Sec. 3.3 and the learning process of camera control model in Sec. 3.4 In this session, we comprehensively ablate the design choices to each of them using the FVD, TransErr, and RotErr as the main metrics. All the CameraCt x1 models in this section are implemented with the AnimateDiffV3 model. Unless otherwise specified,
the same 1,000 video clips in RealEstate dataset as in Sec. 4.2
Plücker embeddings represent a camera precisely. Plücker embeddings naturally serve as a spatial, pixel-wise map with distinct values across locations. As an alternative, could directly use the numerical values for intrinsic matrix K and extrinsic matrix E, or convert the rotation matrix of E into Euler angles. Then, repeating them spatially form a pixel-wise map with identical content across locations. Another approach is to combine ray directions (varying across pixels) and repeated camera origin (constant across spatial positions) into spatial pixel map. Experiment results are illustrated in Tab. 2a using the Plücker embeddings as the camera representation yields the best camera control results. This stems from Plücker embedding's ability to provide a geometric interpretation for every pixel. By contrast, relying solely on numerical values might lead to numerical mismatches, adversely affecting the camera model's learning efficiency. For the representation using ray direction and camera origin, though it can provide the accurate camera origin information, the repeated the camera origin parameters adds redundancy, potentially misaligning features and hindering the model's 's understanding of camera motion.
Noised latents as input limit the generalization. In ablating the architecture of camera encoders, differentiate between ControlNer (Zhang et al., (023a)), whose input is the summation of image features and Plücker embedding sqeuence, and the T2I-Adaptor, solely taking Plücker embedding sequence as input. This distinction is crucial as the use of noised latent, mentioned in SparseCtrl (Guo et al. 2023a), has been associated with appearance leakage, effectively limiting the generalization capability of the camera control quality between different domains. Besides, enhance inter-frame camera coherence, we also consider adding temporal attention block to each encoder. Thus, when choosing the camera architecture of camera encoder, our experiment covers four configurations: ControlNet, T21-Adaptor, and their temporal intention-enhnneed variants. The ablation result is in the Tab. 2h With ControlNet as the camera encoder, the appearance quality suboptimal as shown by the FVD metric in the first two rows. For the models utilizing the T2I-Adaptor, it is observable that the model with additional temporal attention module exhibits better camera control ability. Therefore, camera encoder.
chose the T2I-Adaptor encoder with temporal attention module
Injecting camera condition into temporal attention. Next, we investigate where the extracted camera features should be inserted within the pre-trained U-Net architecture. For this purpose, conduct four experiments to insert the features into the spatial self attention, spatial cross attention, both spatial self and cross attention, and temporal attention layers of the U-Net, respectively. The
8

---

Published as a conference paper at ICLR 2025
Table 2: Ablation study on camera representation, condition injection and effect of various datasets.
Representation type FVD4 TransErr Jeceeer_RRoooo ERotErr
Encoder architecture type/VD] TransErr JootErr
Raw Values
230.1
13.88
1.51
ControlNet
295.8
13.51
1.42
Buler angles
221.2
13.71
1.43
ControlNet + Temporal
283.4
13.13
1.33
Direction + Origin
232.3
13.21
1.57
T21 Adaptor
223.4
13.27
1.38
Plücker embedding 222.1
12.98
1.29
T2I Adaptor Temporal
222.1
12.98
1.29
(a) How to represent camera parameters.
(b) Camera encoder architecture.
Attention
TransErr EootErr
Datasets
FVD, TransErr JootErr
Spatial Self
241.2
14.72
1.42
Objaverse
1435.4
Incapable
Incapable
Spatial Cross
237.5
14.31
1.51
MVImageNet
1143.5
13.87
1.52
Spatial Self + Cross
240.1
14.52
1.60
RealEstate 1OK ACID 1102.4
13.48
1.41
Temporal 222.1
12.98
1.29
RealEatate 10K
1088.9
12.99
1.39
(c) Where to inject camera representations.
(d) Effect of datasets.
results are presented in the Tab. 2c indicating that inserting camera features into the temporal attention layers yields better outcomes. This improvement can be attributed to the fact that camera motion typically induces global view changes across frames. Integrating camera poses with the temporal blocks of the U-Net resonates with this dynamic nature.
Videos with similar appearance distribution and diverse camera help controllability. To test our argument on dataset selection, discussed in Sec. we train CameraCtri using different datasets. The Objaverse (Deitke et al. (023) dataset, has the widest distribution of camera poses but has a significantly different appearance from WebVid1OM. For real-world datasets, compared to MVImageNet, RealEstate10K possesses a more diverse range of camera trajectories. We evaluate these models using diverse data sources, including WebVidI0M for FVD, and MannequinChal. lenge (Li et al. 2019) camera trajectories for TransErr and RotErr. The results, as shown in Tab. [d]] display that compared to RealEstate both FVD scores and camera errors are signifi- cantly higher with MVImageNet. For Objaverse, COLMAP struggles to extract sufficient number of camera poses to yield meaningful TransErr and RotErr metrics. One possible reason for this result is that the difference in the dataset appearance may prevent the model from effectively distinguishing between camera pose and appearance, making a difficult for COLMAP to estimate the with RealEstate 10K and similar dataset ACID,
camera pose. We also jointly train CameraCte but there is no improvement regarding the camera control ability. This result indicates that to improve CameraCtr) further, dataset with a larger camera distribution needed.
4.4 APPLICATIONS OF CAMERACIRL
camera control Applying CameraCtrl to different video generators. As detailed in Sec. 333
model exclusively uses Plücker embeddings as input, making it independent of the appearance of the training dataset. Besides, mentioned in Sec. 33.4] select dataset with an appearance closely resembling that of the training data of the base video generators. Benefiting from these design choices, camtraCtrl can focus solely on learning camera control-related information. This enables its application across various video generators. We demonstrate this in Fig. 4 Appendix H.. and Appendix H.2] In the T2V setting, implement CameraCtr] based on AnimateDiff. The result shown in the first row adopts the vanilla AnimateDiff model depicting a natural scene. Rows two and three showcase results generated by AnimateDiff embedded with other personalized image generators (Realistic Vision (civitai) and Toon You (BradCatt).. Row two represents video style divergent from the typical reality, the buildings of a cyberpunk city. The third row expresses video of cartoon character. Besides, implement Came raCtrl r1 with SVD and sample video in the 12V setting, shown in the last row. Across these varied video generation types, Camera aCt r consistently demonstrates effective control over the camera trajectories, showcasing its broad applicability and effectiveness in enhancing video narratives through dynamic camera trajectory control.
Integrating CameraCtrl with other video control methods. Thanks to the plug-and and-play nature of our method, not only can it be used during the generation processes of different base video generators, but it also be integrated with other video generation control techniques together

---

Published as a conference paper at ICLR 2025
Turtle swimming in ocean.
an acrial view of a eyberpunk city, night time, ncon lights, masterpicee, high quality
masterpiece, 1girl, cloudy sky, dundelion, contrapposio, alternate hairstyle
a back View a boy, standing on the ground, looking at the sky, sunlight,
Figure 4: Applications of CameraCtrl, The first represents video generated by the base DifmateDiff. The Following two rows showcase the results of two personalized T2V generators, Realistic Vision and Toon You. The fourth row expresses the video generated by Came a Cttrl integrated with another video control method, SparseCtrl (Guo et al. 2023a). The video of the last row is produced by I2V generator, SVD, taking the first image of last row condition.
produce videos. For example, utilize SparseCtrl (Guo al. 2023a recent approach that controls the overall video generation by manipulating a few sparse frames. This control can be based on RGB images, sketch maps, or depths. Here, we adopt the RGB encoder and sketch encoder of SparseCtrl, results are shown in the last row of Fig. I and the fourth row of Fig. 4 using RGB and sketch encoder of SparseCtrl, respectively. As illustrated in these two videos, this approach has high level of consistency between the scenes and objects in the generated video and those in the reference frame. Additionally, it is obvious that the camera movements of the generated videos possess a high level of alignment with the provided camera trajectories. The successful integration of CameraCte with SparseCtrl further demonstrates the generalization capabilities of CameraCtzz and enhances its application scenarios. More visual results of this part can be found in Appendix H.3
DISCUSSION
In this work, present CameraCtrl, method that addresses the limitations of existing models in precise camera control for video generation. By learning a plug-and-play camera control module, CameraCt r1 enables accurate control over camera viewpoints. Plücker embeddings adopted as the primary representation of camera parameters, providing comprehensive description of camera pose information by encoding geometric interpretations. Through comprehensive study on training data, it is found that using data with similar appearance to the base model and diverse camera pose distributions, such as RealEstate 10K, achieves the best trade-off between generalizability and controllability. Experimental results demonstrate the effectiveness of CameraCtrl.
Ethics Statement. CameraCtrl enhances video generation technology by providing precise control over camera viewpoints, significantly improving the realism and interactivity of many appli- cations. Conversely, CameraCtss could raise some ethical concerns, particularly privacy and the potential for creating some misleading contents. There is critical need for ethical oversight and more advanced deepfake detectors to manage these risks and ensure the proper usage of CameraCtri.
the realms of
Reproducibility Statement. We provide detailed implementation of our training method in the main text Sec. 4.1 and Appendix D.2 We also provide the model architecture of camera encoder in Appendix D.I
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENT
This project is funded in part by National Key R&D Program of China Project 2022ZD016 100, and by NSFC-RGC Project N_ N_CUHK498/24. Hongsheng Li of CPII under the InnoHK.
REFERENCES
Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin- Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al. Vd3d: Taming large video diffusion transformers for 3d camera control. arXiv preprint arXiv:2407. 12781, 2024.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1728-1738, 2021.
Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: space-time diffusion model for video generation. arXiv preprint arXiv:2401.12944 2024.
Boris Belousov. So3 roration distance. http://www.boiissaaloooorrreerrd -bw.ooooo..eee/0111---- 01/quat- - dist/]
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets, arXiv preprint arXiv:2311.15127 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis, Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
22563-225777 2023b.
BradCatt. Toonyou. https://eiviaaa..ccomooooo
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr. Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. http://ppaaa.ccoooea... com/ hwtp://eee//ceeceeeeeaeeeee research/ senerntioommmdeel---vvvrr--iiiii
2024. (ttthtttss///peeeea video- gereoo-geeratiion moneeeraaiioommmessllsssss models as wor horrdd-- simulators
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang. Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, and Ying Shan. Videocrafterl: Open diffusion models for high-quality video generation, 2023a.
Tsai-Shien Chen, Chieh Hubert Lin, Hung- Yu Tseng, Tsung- Yi Lin, and Ming-Hsuan Yang. Motion- conditioned diffusion model for controllable video synthesis. arXiv preprint arXiv:2304.1444 2023b.
Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305.13840, 2023c.
Xinyuan Chen, Yaohui Wang. Lingjun Zhang, Shaobin Zhuang. Xin Ma. Jiashuo Yu, Yali Wang. Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. In The Twelfth International Conference on Learning Representations. 2023d.
Soon Yau Cheong, Duygu Ceylan, Armin Mustafa, Andrew Gilbert, and Chun-Hao Paul Huang. Boosting camera motion control for video diffusion transformers. arXiv preprint arXiv:2410. 10.002. 2024.
SG 161222 civitai.
Realistic
vision.
kttps://civitaal.commmmbbbe
real iisticivisionneee
11

---

Published as a conference paper at ICLR 2025
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13 13142-13153, 2023.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7346-7356, 2023.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. arXiv preprint X.Xx:231111199: 2023a.
Yuwei Guo, Ceyuan Yang. Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023b.
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorcalistic video generation with diffusion models. arXiv preprint 1xXiv:2312.06662. 2023.
Yingging He, Tianyu Yang. Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffiusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.1322. 2022.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint 1xXix:2277.1299 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.023:0 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXivv2204.0344 2022b.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv: 15868, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0966 09685, 2021.
Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-vine video synthesis for character animation. arXiv preprint arXiv:2311.17117, 2023.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Champaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. arXiv preprint 2. 2023.
Johanna Karras, Aleksander Holynski, Ting-Chun Wang, and Ira Kemelmacher-Shhine Shhizerman..... Dream- pose: Fashion video synthesis with stable diffusion. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 22680-22699 2023.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 25:26565-26577 2022.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video zero: Text-to-image diffusion models are zero-shot video generators. IEEE International Conference Computer Vision (ICCV), 2023.
12

---

Published as a conference paper at ICLR 2025
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125, 2023.
Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hongsheng Li, Leonidas Guibas, and Gordon Wetzstein. Collaborative video diffusion: Consistent multi-video generation with camera control. arXiv preprint arXiv:2405. 17414, 2024.
Dongxu Li, Junnan Li, Hung Le, Guangsen Wang. Silvio Savarese, and Steven C.H. Hoi. LAVIS: A one-stop library for language-vision intelligence. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 31-
41, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https
Thallatthooooy.orr/222333
Zhengqi Li, Tali Dekel, Forrester Cole, Richard Tucker, Noah Snavely, Ce Liu, and William
Freeman. Learning the depths of moving people by watching frozen people. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4521-4530, 2019.
Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from a single image. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 14458-14467, 2021.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint 240.v22401.03048; 2024.
Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Ying Shan, Xiu Li, and Qifeng Chen. Fol- low your pose: Pose-guided text-to-video generation using pose-free videos. arXiv preprint arXiv:2304.01186, 2023.
Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, al. Snap video: Scaled spatiotemporal transformers for text-to-video synthesis. In Proceedings of the IEEE/CVE Conference on Computer Vision and Putlern Recognition, pp. 7038-7048, 2024.
Chong Mou, Xintao Wang, Liangiin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-imaae diffusion models. arXiv preprint arXiv:2302. 08453, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CV International Conference on Computer Vision, pp. 4105-4205, 2023a.
William Peebles and Saining Xic. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023b.
Ryan Po, Wang Yifan, and Vladislav Golyanik et al. Compasitional 3d scene generation using locally conditioned diffusion. ArXiv, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Tianhe Ren. Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
13

---

Published as a conference paper at ICLR 2025
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-moda diffusion models for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10219-10228, 2023.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510, 2023.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung. Simon See. Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling. arXiv preprint arXiv:240111597, 2024.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 1ee.1447499 14792, 2022.
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with Lingl-evaluation rendering. Advances in Neural Information Processing Systems, 34:19313-19325, 2021.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I1 16, pp. 402 419. Springer, 2020.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arX1v:18/2.01177. 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. hupps//ppenreiee..e//foomm didrrrguuidd,,,, 2019.
Karol Kurach, Raphaäl Marinier,
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Prucessing Systems, 36, 2024.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. arXiv preprint arXiv:2312.03641, 2023.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video. One-shot tuning of image diffusion models for texi-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. (623-7633, 2023.
Jinbo Xing, Menghan Xia, Yuxin Liu, Yuechen Zhang, Yong Zhang, Yingqing He. Hanyuan Liu, Haoxin Chen, Xiaodong Cun, Xintao Wang, et al. Make-your-video: Customized video generation using textual and structural guidance. arXiv preprint arXiv:2300.0099 2023.
Dejia Xu, Yifan Jiang, Chen Huang, Liangehen Song, Thorsten Gernoth, Liangliang Cao, Zhangyang Wang, and Hao Tang. Cavia: Camera-comee contraacontrollable multi-view video diffusion with view-integrated attention. arXiv preprint a10 10774, 2024a.
14

---

Published as a conference paper at ICLR 2025
Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Camco: Camera--cntrolllable 3d-consistent image-to- -ovideo generation. arXiv preprint wrXiv:2406.02509 2024b.
Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. arXiv preprint arXiv:2311.16449 2023.
Shiyuan Yang, Liang Hou, Haibin Huang. Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. arXiv preprini 03Xiv:2402.03 162, 2024a.
Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint 240.:22000000072,, 2024b.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308..6622 2023.
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308. 08089, 2023.
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9150-9161, 2023.
David Junhao Zhang, Roni Paiss, Shiran Zada, Nikhil Karnad, David E Jacobs, Yael Pritch, Inbar Mosseri, Mike Zheng Shou, Neal Wadhwa, and Nataniel Ruiz. Recapture: Generative video camera controls for user-provided vidcos using masked vidco fine-tuning. arXiv preprint arXiv:441.00500,, 2024.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 3836-3847, 2023a.
Shiwei Zhang, Jiayu Wang. Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin. Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-vides synthesis via cascaded diffusion models. arXiv preprint 0tdi::3311.04444. 2023b.
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian, Con- trolvideo: Training-fre free controllable text-to-video generation. arXiv preprint arX1v:2305.13077, 2023c.
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024. URL httpt/gtttttttttt///ee//ttehht com ctwp://paaaaeccc/eoeeeeeeeeo Soral
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely, Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv: 09810, 2018
15

---

Published as a conference paper at ICLR 2025
A APPENDIX / SUPPLEMENTAL MATERIAL
This supplementary material provides discussion of CameraCtri and other methods, more discus- sions on data selection, implementation details, details of user study, additional ablation experiments. more qualitative comparisons, and more visualization results of CameraCtrl.
In all visual results, the first image in each row represents the camera trajectory of a video. Each small tetrahedron on this image represents the position and orientation of a camera for one video frame. Its vertex stands for the camera location, while the base represents the imaging plane of the camera. The red arrows indicate the movement of camera pasition but do not depict the camera rotation. The camera rotation can observed through the orientation of the tetrahedrons. For clearer understanding of the camera control effects, we highly recommend that readers watch the videos provided in our supplementary file.
The organization of this supplementary material is as follows: Appendix B gives some discussions between CameraCtrl and concurrent works. Appendix [ presents more discussions on the dataset selection process. Appendix D gives more implementation details. The details of the user study are shown in Appendix E Appendix depicts extra experiment results on the model architecture, camera representation, and the lower bound of the RotErr and TransErr. Then, we provide more qualitative comparisons between CameraCtri with AnimateDiff and MotionCtrl in Appendix G After that, more visualization results showcased in Appendix H Finally, provide some failure cases in Appendix!!
B DISCUSSION WITH CONCURRENT CAMERA CONTROL WORKS
Recent works have explored camera control in video generation, addressing different aspects of the challenge. VD3D (Bahmani et al. 2024 integrates camera control into a DiT-based (Peebles & Xie 2023a; Menapace et al. 2024) model with a novel camera representation module in spatiotemporal transformers. CamCo Xu al. 2024b leverages epipolar constraints for 3D consistency in image-to- video generation. CVD Kuang et al. (024) uses the camera control method and extends it to enable multi-view video generation with cross- view consistency. Recaprture (Zhang al. 2024) enables video-to-video camera control, effectively modifying viewpoints in existing content. However, it's limited to simpler scenes and struggles with complex dynamic environments. Cavia (Xu et al. 2024a) enhances multi-view generation through training on diverse datasets, improving cross- view consistency. (Cheong al., 2024) improves camera control accuracy using a classifier free guidance Ho & Salimans| (022) like mechanism in a DiT-based /Zhng et al. (2024) model. Despite the numerous works addressing camera control in the video generation process, to our best knowledge, CameraCt r1 is among the early methods to achieve accurate camera control in video generation models. provides valuable insights and a solid foundation for future advancements related fields, such video generation, as well as 3D and 4D content generation.
C MORE DISCUSSIONS ON DATASET SELECTION
When selecting the dataset for training our camera control model, we first choose three datasets as candidates, they are Objaverse (Deitke al. 2023). MVImageNet (Yu al. 2023). and RealEstate 10K (Zhou et al. 2018).
For the Objaverse dataset, its images are rendered with software like Blender, enabling highly complex camera poses. However, as seen in row one to row three of Fig. [ its content mainly focuses on objects against white backgrounds.
contrast, the training data for many video diffusion
models, such as Web Vid-10M 10M (Bain et al.) (021), encompasses both objects and scenes against more intricate backgrounds. This notable difference in appearance can detract from the model's ability to concentrate solely on learning camera control. In our initial trial, We tried to train the CameraCtr] x1 with the Objaverse dataset, the resulting model can control the camera trajectory in the Objaverse-lie like video (single object with white background) well. In other domains, however, the camera control model cannot generalize well in controlling the camera viewpoints during the video generation process.
16

---

Published as a conference paper at ICLR 2025
Table 3: Output feature shapes of each layer (encoder scale) of camera encoder. And c 6 x 8 x 8 384, C1, C2, Cs, C4 are equal to the channels numbers of the corresponding U-Net output feature with the same resolution. For examble, with a stable video 1.5 model, C1, C22 C3 Cq equal to 320, 640, 1280, 1280.
input
b x n x 6 x h b xx X n X e X
Pixel unshuffle
3x 3 conv layer
X
Encoder scale
6 T X c1 X
X
Encoder scale
6x c2
Encoder scale
6x X
Encoder scale
64
For MVImageNet data, it has some backgrounds and complex individual camera trajectories. Nev- ertheless, as demonstrated in row four to row six Fig. 5 most of the camera trajectories in the MVImageNet are horizontal rotations. Thus, its camera trajectories lack diversity, which could lead the model to converge to a fixed pattern.
Regarding RealEstate data, shown in row seven to nine of Fig. it features both indoor and outdoor scenes and objects. Besides, each camera trajectory in RealEsstate is complex and there exists considerable variety among different camera trajectories. Therefore, RealEstate dataset to train our camera control model. There are other datasets that possess similar camera trajectory with the RealEstate 10K dataset, like the ACID (Liu et al. (02I) and MannequinChallenge data (Li et al.) 2019), but with fewer data samples. We tried to train the CameraCt 11 using the RealFitttee TOK and the ACID but did not find an improvement in the camera control accuracy, as shown in Tab.22 This result indicates that the current bottleneck of the camera control accuracy may lie in the complexity of the camera pose distribution.
choose the
D MORE IMPLEMENTATION DETAILS
D.1 CAMERA ENCODER &. ARCHITECTURE.
we take temporal attention-enhanced T2I Adaptor (Mou al. 2023) encoder as our camera encoder & to extract the camera features from Plücher embeddings. Generally, the camera encoder consists of pixel unshuffle layer, a convolution layer, and 4 encoder scales. It takes a batch of Plücker embedding sequence P E Rhxnx6xh
As stated in Sec. 3.3
where b, n, h, w represent the batch size, number of frames in video clip, the height and width of the video clip, respectively as input, and output multi-scale camera features. The output feature shapes of each layer are listed in Tab.
Besides, each encoder scale is composed of one downsample ResNet block (Mou al. 2023) (except for the encoder scale 1) and one ResNet block, each block is followed by one temporal attention block (Guo al., 2023b). More specifically, the temporal attention block consists of a temporal self-attention layer, layer normalizations and position-wise MLP as follows:
< + x + PosEmbb(z)
(6)
LayerNorm(()
(7)
C2 f MulhHHcadSSafAttenniooo// +
(8)
LayerNorm((2)
(9)
x + MLP((s) + C2.
(10)
(11)
where PosEmb is the temporal positional embedding.
D.2 TRAINING.
We use the LAVIS et al. 2023) to generate the text prompts for each video clip of the used dataset (Objaverse, MVImageNet, Realfistate 10K, and ACID). For the text-to-video (T2V) setting,
17

---

Published as a conference paper at ICLR 2025
converse chuck taylor all star hi black
wooden cart with wooden seat and wheels
vase with handle stripe pattern
blue and white floral couch sitting wooden floor
piece lettuce sitting on top of a conference table
wooden bench sitting in front of stone wall
bathroom with curfain toilet
empty with desk and chair in
group of people standing the deck of house
Figure 5: Samples of different datasets. Rows 1 to row are samples from the Objaverse dataset, which has random camera poses for each rendered image. Rows 4 to 6 show the samples from the MV MVImageNet dataset. Samples of the RealEstate 10K dataset are presented from rows to row 9.
we used AnimateDiffV33 as the base video generation model. To let the camera control model better focus on learning camera poses, similar to AnimateDiff (Guo et al. (0026), we first train an image LoRA on the images of the RealEstate 10K dataset. Then, based on the AnimateDiff model enhanced with LoRA, we train the camera control model. Note that, after the camera control model is trained, the image LoRA can be removed. For each training sample of the CameraCtr.] we sample 16 images from one video clip with the sample stride equal to 8, then resize their resolution to 256 x 384. For the data augmentation, we use the random horizontal flip for both images and poses with 50 percent probability. The Adam optimizer is adopted to train the models, with a constant learning rate le 0,=0.9, 0.99. weight decay equals 0.01. We Bxtart = 0.00085, Bend = 0.012, and 1000. We use 16 80G NVIDIA A100 GPUS to train the CameraCt +1 models with batch size of 2 per GPUS for 50K steps, taking about 25 hours.
linear beta noise schedule, where
18

---

Published as a conference paper at ICLR 2025
For the Image-to- Image-to-Video (12V) setting, use the Stable Video Diffusion (SVD) the base video generator. We directly train the camera encoder and the merge linear layer on top of the SVD. For each training sample, sample 14 images for one video clip with the sample stride equal to 8, then resize their resolution to 320 x 576. Then Adam optimizer is utilized with a constant learning rate 3e-5, 31 0099, $2=0.99, weight decay equals to 0.01. Following SVD, we used the EDM (Karras) et al. 2022) noise scheduler and set all the hyper-parameters equal to SVD. We used 32 80G NVIDIA A100 GPUS to train the models with batch size of I per GPUS for 50K steps, taking about 40 hours.
D.3 INFERENCE.
By utilizing structure-from frotum---ooommmtiionn methods such as COLMAP (Schönberger & Frahm 2016), along with existing videos, we can extract the camera trajectory within a video. This extracted camera trajectory can then be fed into our camera control model to generate videos with similar camera movements. Additionally, we can also design custom camera trajectories to produce videos with desired camera movement. During the inference, use different guidance scales for different domains' videos and adopt a constant denoise step 25 for all the videos.
D.4 OBJECT DYNAMIC DISTANCE (ODD) METRIC
We first utilize the Grounded-SAM- Ren et al. 2024 to segment the main object in a video. Then, following the dynamic degree in VBecch, we use RAFT (Teed & Deng] 2020) to estimate the optical flow, and only keeping the estimated flows belongs to the main object. Then, following the dynamic degree metric, these optical flows are taken as the basis to determination whether the video is static. The final object dynamic degree score is caleulated by measuring the proportion of nonstatic videos generated by the model.
D.5 MORE DETAILS OF ROTERR AND TRANSERR
When using the COLMAP extract the camera poses of generated videos, it is not very stable to generate the reliable camera pose sequence. Thus, when COLMAP fails, we have manually filtered these failed video clips and not added them in the calculation of RotErr and TransErr. Besides, since the COLMAP is scale invariant, there may have some scale issues of the generated camera poses. These scale issues only have some impact on the calculation of TransErr, not the Rrt To deal with with this scale issue, we have performed some postprocessings to the COLMAP results. Specifically, we first compute the relative poses of the ground truth and generated camera poses by setting the homogencous extrinsic matrix of first frame as a 4 x 4 identity matrix. Then, normalizing the scale of the COLMAP results with the ground truth camera trajectory. Concretely, we calculate the translation gap between the first two frames for both the generated and ground truth camera poses to obtain a rescale factor. We then normalize other generated camera poses with this rescale factor to align the scales of the two camera trajectories. This normalization helps alleviate the scale problem in COLMAP results, making our evaluation metrics more convincing.
E DETAILS OF USER STUDY
The camera error TransErr and Rot Err proposed in Sec. Sec. n..4need COLMAP to extract the camera poses of the generated videos. However, the COLMAP can not extract precise camera poses in short videos (16 frames in T2V, 14 frames in 12V) stably. To compare the camera control quality between CameraCtrl with AnimateDiff and MotionCtrl from another perspective, we conduct some user studies. Specifically, since the AnimateDiff is only able to generate videos with eight base camera movements in the T2V setting, we sample these three methods with base camera movements and let the user watch the video to decide which video is more in line with the condition camera trajectory. Then calculating the approving rate for each method, the results are in the User Preference Rate column in Tab. []'s middle block. Besides, we employ some complex camera trajectories extracted from the test set of RealEstate 10K to condition the MotionCtrl and CameraCtis in the T2V setting. The generated videos sent to users to decide which one has better camera trajectory alignment with the reference videos. The user preference rate for MotionCtrl and Came raCt are 27.6% and 72.4%. respectively.
19

---

Published as a conference paper at ICLR 2025
Table 4: Ablation study of the camera feature injection place.
Injection Place
FVD
TransErr
RotErr 4
U-Net Encoder
210.9
13.91 12.98
1.51 1.29
U-Net Encoder Decoder
222.1
K1
R
K1 a, B1 Y1 f
Repeet HxW
K1
R1
Repeat HxW
Kg " V1 f
Pf*
R2
K1 d4 P1 81 4
PfW
K2 R2 22
K2 a2 B2 Y2 t2
p2
Repest JXW
K2 Rg 12
Repeat HXW
K2 ag 12
P
pf E 1
K2 R2 12
K2 a2 B2
paw
Ku Rn In
Kn anBa Yn tn
P
Repest
Kn Rn In
Repeat HXW
Kn ag Yn En
HXW
Kn Rn tn
Kn Bn
pHW-
Figure 6: Different camera representation. The left subfigure row shows the camera represented using the intrinsic K; and the extrinsic matrices E; (composed of rotation matrix R; and the translation vector 4).. The middle subfigure give the camera representation of converting the rotation matrix R3 into Fuler angles ai, Bi, N. Plücker embedding are given in the right subfigure, the intrinsic and extrinsic matrices are converted into the Plücker embeddings to form pixel-wise spatial embedding. While the left and middle camera representations are not a pixel-wise camera representations naturally.
After that, in the 12V setting, we sample the MotionCtrl and CameraCtr] with complex camera trajectories extracted from the RealEstatc10K dataset. With these videos, another user study is conducted to let the user choose which video has the better camera trajectory condition performance. Results are shown in the User Preference Rate column of the bottom block of Tab. [ These user study results further demonstrate the the superiority of Came during the video generation process. We invite 50 users to conduct all the user studies. Considering the difference between the education levels of these users, we design the user studies easily as possible to get more reliable results.
in controlling the camera trajectory
F EXTRA EXPERIMENTS
F.1 EXTRA ABLATION STUDY
Injecting camera features into both encoder and decoder of U-Net. In the vanilla T2I- Adaptor (Mou et al. 2023), the extracted control features are only fed into the encoder of U-Net. In this part, we explore whether injecting the camera features into both the U-Net encoder and decoder could result in performance improvements. The experiment results are shown in Tab The improvements of TransErr and RotErr indicate that compared to only sending camera features to the U-Net encoder, injecting the camera features to both the encoder and decoder enhances camera control accuracy. This result could be attributed to the fact that similar to text embedding, Plücker embedding inherently lacks structural information. Such that, this integrating choice allows the U-Net model to leverage camera features more effectively. Therefore, we ultimately choose to feed the camera features to both the encoder and decoder of the U-Net.
20

---

Published as a conference paper at ICLR 2025
There upper floors and tables and chairs
Figure 7: Qualitative comparison of using different camera representations. The first row shows the result using the raw camera matrix values camera representation. Result of the second row adopts the ray directions and camera origin as camera representation. The last row exhibits the result taking the Plücker embedding as the camera representation. All the results use the same camera trajectory and the text prompt.
Table 5: Lower bound of TransErr and RotErr on RealEstate 10K test set.
TransErr
RotErr
Lower Bounds
6.93
1.02
F.2 QUALITATIVE COMPARISON ON DIFFERENT CAMERA REPRESENTATION
Here, we provide the qualitative comparison using different camera representations, results are shown in Fig. 7 The provided camera trajectory primarily moves forward, with rightward shift at the end. From the figure, can seen that when using the raw camera matrix values as camera representation, the model ignores the final rightward movement. With the hybrid camera pose representation, the model exhibits an abrupt shift in the last few frames to achieve the rightward movement. In contrast, using the Plücker embedding as the camera representation results smoother generated video, with the final rightward movement appearing natural and seamless. These results further demonstrate the effectiveness of using Plücker embedding as the camera representation.
F.3 LOWER BOUND OF TRANSERR AND ROTERR ON REALESTATE 10K TEST SET
Since the COLMAP not 100 percent accuracy, we need to know the lower bounds of the TransErr and RotErr metrics. With the sampled video clips (each has 16 frames) in the RealEstate 10K test set, we run the COLMAP on these video clips to get the estimated camera poses. Using these camera poses and the ground truth camera poses, we calculate the Transărr and RotErr, results are shown in the Tab.
G MORE QUALITATIVE COMPARISONS
In this section, we first provide more qualitative comparisons between CameraCtr] with Animate- Diff using the base camera trajectories the test-to-video (T2V) setting. Then, in the T2V setting, we also deliver more qualitative comparisons between CameraCtr] and MotionCtrl with the complex camera trajectories extracted from the RealEstate 1OK test set. Finally, more qualitative comparisons between CameraCtr1 and MotionCtrl in the image-to-vides (I2V) setting given.
G.1 QUALITATIVE COMPARISONS IN THE T2V SETTING
Comparisons between AnimateDiff and CameraCtrl. Results shown in Fig. 8 find that the generated video of AnimateDiff shows the camera movement of pan up not the given camera movement pan down. In contrast, the video generated by CameraCt in row 2 follows
In row
21

---

Published as a conference paper at ICLR 2025
pyramid the desert
A cute dog sitting the green grass.
Figure 8: Qualitative comparison between AnimateDiff and CameraCtr1 Resuls of rows 1, and 5 are from AnimateDiff.. Results of CameraCtrl are shown in rows 4, 6, 7. Rows and use the same camera trajectory, pan down. Camera trajectory pan left is adopted by rows 3 and 4. For rows 5 and the camera trajectory pan down utilized. The result of the last row generated with the camera trajectory pan left down. Rows and 2 condition on the same text prompt, while rows 3 condition on another text prompt.
the desired camera movement. Thus, we can conclude that in some situations, AnimateDiff cannot distinguish the object movement from the camera movement. Results rows and exhibit that, though sometimes AnimateDiff can generate the videos with the desired camera movement, it cannot keep the object consistent during the whole video. By comparison, Came CameraCtr] can generate videos with consistent contents and strictly follows the condition camera trajectory, illustrated in rows 4 and Besides, AnimateDiff only supports some simple camera trajectories. For other more complex camera trajectories, it does not support even a combination of two base camera trajectories, like the combination of pan left and pan down, while CameraCt r1 can support this trajectory, shown in the last row of Fig.
Comparisons between MotionCtrl and CameraCtrl. In Fig. we provide more qualitative comparisons between the MotionCtrl and CameraCtri in the T2V setting. For the trajectories with translation or rotation to small extent, like the left camera translation in the first trajectory (rows and 2) and the left rotation at the beginning of the second trajectory (rows 3 and 4), MotionCtrl does not very sensitive to them. It only focuses on the main camera movement, the forward translation. By contrast, the generated videos of CameraCtrl (rows 2 and 4) accurately obey these small camera trajectories. For the third (rows 5 and 6) and the fourth (rows 7 and 8) camera trajectories, they contain both camera rotation and translation. For the videos generated by MotionCtrl (rows 5 and 7), however, it focuses more the camera rotation and ignores the camera translation. In contrast, CameraCt xl can make good balance between the camera rotation and translation and generate satisfactory videos, shown in
6 and 8 of Fig. 9
22

---

Published as a conference paper at ICLR 2025
A horse is eating grass on the grassland.
sparse pine forest covered fresh
A villa in a garden.
Ripe apple on a wooden table
Figure 9: Qualitative comparison between MotionCtrl and CameraCtrl in T2V setting, Results of rows 1, 3, and 7 are generated by MotionCtrl, while results of Camera rl are shown in rows 2, 4, 6, and 8. Every two adjacent rows use the same text prompt and camera trajectory.
G.2 QUALITATIVE COMPARISONS THE I2V SETTING
Similar to the results of T2V, MotionCtrl still cannot handle the small camera movement well in the 12V setting.
In Fig. T0, for the results of the first two camera trajectories, compared to the results of our Cameractrl (rows 2 and 4), videos generated by MotionCtrl (rows and 3) ignore the small camera rotation the very beginning of the camera trajectories. For the third and the fourth camera trajectories, the camera movement extent of MotionCtrl results (rows 5 and 7) rather less than that of the CameraCtrl 1 results (rows and 8). The results of CameraCtrl camera movement extent of the trajectories and 4. We strongly recommend the readers to watch the provided videos in the supplementary file for more direct understanding.
reveal the practical
Note that, in the 12V setting, both MotionCtrl and CameraCtrl are implemented on the save video diffusion model, SVD (Blattmann et al.) 2023a), which excludes the influence stemming from the different base video generators. Thus, the better camera viewpoint controlling the generated videos benefits from the better design choices of Came raCtrl.
23

---

Published as a conference paper at ICLR 2025
Rocky coastline with crashing
Turtle swimming ocean.
Flowers in front of mountainside.
pyramid the desert.
Figure 10: Qualitative comparison between MotionCtrl and CameraCtrl in I2V setting. The condition images are shown the first images of each row. These images are generated with the SDXL (Podell et al.] 2023) taking the text prompts located below of every two rows input. Note that, both MotionCtrl and CameraCte r only condition on the conditioning images, not include the 5, and 7 are the results of MotionCtrl, while the results of Came raCtrl are in rows 2, 4, 6, and Every two adjacent rows are generated with the same condition image and the camera trajectory.
text prompts. The rows
H MORE VISUALIZATION RESULTS
This section provides additional visualization results CameraCtr Specifically, Appendix provides the various domain videos generated by integrating CameraCtrl with AnimateDiff (Guo
al., 2023b) in T2V setting. In Appendix HH.2 we exhibit the generated videos of CameraCt
in the 12V setting where the Stable Video Diffusion (SVD) (Blattmann et al. 2023a) is chosen as
base video generator. After that, video results of combining CameraCtr: with another video
control method, SparseCtrl (Guo et al. 2023a) is shown in Appendix H.3
Finally, Appendix H.4
shows the flexibility of CameraCtri.
H.1 VISUALIZATION RESULTS VARIOUS DOMAIN T2V VIDROS
Visual results of RealEstate10K domain. First, with the aforementioned image LoRA model trained on RealEstatelOK dataset, and using captions and camera trajectories from RealEstate IOK, CameraCtr) r1 is capable of generating videos within the RealEstate domain. Results shown in Fig. [T] the camera movement in generated videos closely follows the control camera poses, and the generated contents are also aligned with the text prompts.
Visual results of original T2V model domain. We choose the AnimateDiff V3 (Guo et al. 2023b) video generation base model, which is trained the WebVid-10M dataset. Without the
24

---

Published as a conference paper at ICLR 2025
bedroom with bed and dresser in
bodroom with bed, television
chair sitting in room next to window
kitchen with stainless steel appliances and wooden cabinets
kitchen with wooden cabinets and black stove
living room with leather couches and fireplace an empty room with desk and chair it
kitchen with wooden cabincts and stainless steel appliances
RealEstate10K visual results. The video generation results of CameraCtrl. The control camera trajectories and captions are both from RealEstate 10K test set.
Figure 11:
RealEstate 10K image LoRA, CameraCt can be used to control the camera poses during the video generation of natural objects and scenes. As shown in Fig. 12, with the same text prompts, taking different camera trajectories as input, CameraCt 1 can generate almost the same scene, and closely follows the camera trajectories. Besides, Fig. shows more visual results of natural objects and scenes.
Visual results some personalized video domain. By replacing the image generator backbone of T2V model with some personalized generator, CameraCt.ri can be used to control the camera poses in the personalized videos. With the personalized generator Realistic Vision (civitai), Fig [4 showcases the results of some stylized objects and scenes, like some uncommon color schemes in the landscape and coastline. Besides, with another personalized generator Toon You (BradCatt). CameraCt rl be used in the cartoon character video generation process. Some results shown in Fig. 15 In both domains, the camera trajectories in the generated videos closely follow the control camera poses.
25

---

Published as a conference paper at ICLR 2025
A horse is cating grass the grassland.
Using CameraCtrl on the same caption and different camera trajectories. The camera control results of CameraCtrl. Camera trajectories are from RealEstatel05 test set, all videos utilize the same text prompts.
Figure 12:
H.2 12V VISUALIZATION RESULTS OF CAMERACTRL INTEGRATED WITH SVD
By taking the SVD as the base video generator to implement our CameraCtrl, we can sample videos with desired camera trajectories the 12V setting. Fig. 16 shows some of them. The camera viewpoints of generated videos strictly follow the camera trajectory input, and video content also align with the condition images.
H.3 INTEGRATING CAMERACTRL WITH OTHER VIDEO CONTROL METHOD
gives some generated results by integrating the CameraCtrl with another video control method SparseCtrl (Guo et al.) (023a). The content of the generated videos follows the input RGB image or sketch map closely, while the camera trajectories the videos also effectively align with the conditioned camera trajectories.
Fig. 17
26

---

Published as a conference paper at ICLR 2025
The sunflower in the sineshine
A pair of worn leather boots porch
squiree cating pine nuts
massive, multi-ticred elven palace adorned with flowing waterfalls, cascades forming staircases betweer ethereal realms
A still life of vintage objects on a wooden table
A sparse pine forest covered in fresh snow
serene lake sunrise, with mist hovering the
balloon rising up into the air
Figure 13: Visual results of natural objects and scenes. The natural video generation results of CameraCtri rl. CameraCtrl can be used to control the camera poses during the video generation process of natural objects and scenes.
H.4 FLEXIBILITY OF CAMERACTRL
Different camera movement intensity. By adjusting the interval between the translation vectors of two adjacent camera poses, we can control the overall intensity the camera movement. shown in the Fig. 18 we can make the camera movement more intense more gradual.
Controlling camera movement by adjusting intrinsic. Since the Plücker embedding requires internal parameters during the computation, we can achieve camera movement by modifying the camera's intrinsic parameters. As shown in the Fig. 19 by changing the position the camera's 's principal point (ex, cy), adjusting the focal length (fx, fy), we can achieve a zoom-in and zoom-out effect, as shown the last two rows.
achieve camera translation (as shown in the first three rows). By
27

---

Published as a conference paper at ICLR 2025
photo of coastline, rocks, storm weather, wind, waves, lightning, soft lighting
landscape with Four-Point Perspective, perfect composition, high detail, masterpicce, grass, meadow, field
better mecha crushing an upside down weight, rust, nuked city, landscape, blinding light, fire, explosion
city street, neon, fog, volumetric. closeup portrait photo of young woman in dark clothes
close up photo of rabbit, forest, haze, halation, bloom, dramatic atmosphere, centred
Figure 14: Visual results of stylized objects and scenes. With the personalized generator Realis- Vision (civitai), CameraCtr1 can be used in the video generation process of stylized videos.
I FAILURE CASES
we provide some failure cases of CameraCtrl. The main problem for these cases is that when the rotation of the camera trajectory has large extent, CameraCtrl cannot properly generate videos with enough rotation. The first and second rows take the vertical uniform rotation 100 degrees input, but the generated videos cannot rotate 100 degrees. The same problem is kept for rows and 4, where we desire a horizontal uniform of 150 degrees. However, there is only about 90-degree rotation of the generated videos. The main reason for this failure situation may lie in that the training dataset (RealEstate I (OK does not contain enough camera trajectories with a large degree of rotation. Thus, improve the camera trajectory performance further, dataset possessing similar visual appearance to RealEstate I IOK and larger camera pose distribution is needed.
Fig. 20
28

---

Published as a conference paper at ICLR 2025
closeup face photo of man in black clothes, night city street, bokeh, fireworks in background
masterpiece, Igirl, looking al viewer, blurry background, upper body, contemporary, dress
masterpiece, Igirl, short hair, cafe, barista, dark skin, happy, looking viewer, hairband, apron, carrings, upper body
mkvd, Igirl, turtleneck sweater, sweater yellow, happy, looking at viewer
masterpicce, best quality, Igirl, hanami, pink flower, spring season, wisteria, petals, flower, outdoors, falling petals, black eyes
Figure 15: Visual results of cartoon characters. With the personalized generator Toon You (Brad- Catt), CameraCtr) rl can be used in the video generation process of cartoon character videos.
A boat sailing leisurely along the Seine River with the Eiffel Tower in background by Vincent van Gogh.
An exploding cheese house.
A fish is swimming the aquarium tank.
A serene mountain lake sunrise, with mist hovering the water.
Fireworks display illuminating the night sky.
Integrating CameraCtr1 with SVD in the I2V setting. The condition images are located in the right bottom corners of each rows first image. These images are generated by the text-to-image model SDXL with the text prompts down below each row input. The condition signals of generated videos are only images, not include the text prompts.
Figure 16:
29

---

Published as a conference paper at ICLR 2025
professional photo, photo of landscape, dramatic lighting, gloomy, cloudy weather
fat spiderman eating a burger
city street, neon, fog, volumetric, closeup portrait photo of young woman dark clothes
back boy, standing ground, looking sky, sunlight, masterpieces
a back view a boy, standing on the ground, looking at the sky, clouds, sunsct, orange sky, beautiful sunlight, masterpizoes
acrial view of'a cyberpunk city. night time, neon lights, masterpiece, high quality
Figure 17: Integrating CameraCtr1 with other video generation control methods. Row one to three express the results by integrating the CameraCtrl with RGB encoder of SparseCtrl (Guo et al. 2023a and row four to row six, shows videos produced with the sketch encoder of SparseCtrl. The condition RGB images and sketch maps are shown in the bottom right corners of the second images for each row. Note that, the camera trajectory of the last row is zoom-in.
30

---

Published as a conference paper at ICLR 2025
night, b&w photo of house, post apocalypse. forest, storm weather, wind, rocks, soft lighting. high quality, film grain
an ouldour lounge area with fire pit overlooking the city
Figure 18: Camera movement intensity. The first two rows taking the pan down camera trajectory as input, with the camera translation interval the second row being four times that of the first row. The camera trajectory for the third and fourth rows are zoom in, with the camera translation interval the fourth row being four times that of the third row.
masterpicoe, quality, !girl, solo, cherry blossoms, pink flower, white flower, spring senson, petals, plum blassoms, outdoors, falling petals
night, old post apocalypse,
lighting. high quality, grain
Firewarks
photo coastline, rocks, weather, wind, lightning, soft lighting
constline, rocks, weather, wind, lighning, soft lighting
Figure 19: Controlling camera movement by adjusting intrinsic. The first three rows show the generated results using camera pan left, left up, right down, respectively. The last two rows take the zoom in, zoom out camera trajectories the input. In each camera trajectory, all the camera poses have the same extrinsic matrix, the camera movement is implemented by adjusting the intrinsic parameters, cx and cy for the first three rows, fx and fy for the last two rows.
31

---

Published as a conference paper at ICLR 2025
a gated driveway leads to large house
outdoor lounge with fire pit overlooking the city
gated driveway leads to large house
an outdoor lounge area with a fire pit overlooking the city
Figure 20: Failure cases. All results are generated with the CaneraCtri implemented on Animate- DifIV3 the T2V setting. The camera trajectory for rows and 2 is the vertical uniform rotation of 100 degrees. The trajectory horizontal uniform rotation 150 degrees is used during the generation of rows 3 and 4.
32


Reviews:
[{"review_id": "4sw9cdpX9D", "reviewer": "Reviewer_FegY", "summary": "CameraCtrl enhances video generation by enabling precise camera pose control in video diffusion models, preserving narrative nuances through cinematic language. This approach integrates a plug-and-play camera pose control module and demonstrates improved controllability and generalization, with experimental results showcasing its effectiveness across different models.", "questions": "Please see the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The motivation is clear.\n\nThe results appear promising and solid.\n\nThe experiments are thorough.\n\nThe writing is easy to follow.", "weaknesses": "Applying a controlnet-like network to incorporate control signals into video diffusion isn’t particularly novel, though the use of Plücker embeddings adds a degree of originality.\n\nSince the RealEstate10K dataset mainly features static videos with few moving objects, could this lack of motion introduce a bias that undermines the base model’s natural grasp of object motion? The foreground objects in the examples provided show limited movement.\n\nThere is a need for comparisons and discussion with recent camera control methods, such as VD3D.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lnJGbNCNlz", "reviewer": "Reviewer_7FvC", "summary": "This paper proposes a plug-and-play module for control of camera pose. This module can be integrated to various video diffusion models. This module leverages Plücker embeddings to encode camera movements, offering detailed spatial representation compared to traditional numerical values (e.g., extrinsic matrix).", "questions": "* Why are static scene video datasets mainly used without general video datasets? (For example, one could use general video datasets for training along with estimated camera trajectories from various models like particleSFM.)\n* This didn't affect my rating, but I'm curious about the authors' opinions on whether this method can be generally applied to DiT-based methods that do not separate temporal and spatial aggregation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* This is one of the early works that enables adding camera control to existing video models. I believe it will have a positive impact on various downstream tasks.\n* While the way of achieving camera control itself isn't technically very novel, the paper presents an intuitive design and provides extensive experimental comparisons on various conceivable design choices (e.g., where to inject the control and different camera representations).\n* It is interesting that camera control can be achieved with a lightweight additional adapter without further tuning the video model.", "weaknesses": "* Since it is fine-tuned on static scene data, it tends to show somewhat reduced ability in generating dynamic motions compared to the original video diffusion baseline (as shown in ODD). But, it shows better performance compared to other methods.\n* In object-centric video generation, it appears that the movement of the object is minimal compared to the background. Ideally, models fine-tuned on Objaverse or MVImgNet should be able to generate camera trajectories covering 360 degrees of the object. I'm curious if this model can effectively generate videos when provided with such camera trajectories as input (e.g., generating a video from a frame showing the front of a person to one showing the back).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S5jMFpbwRO", "reviewer": "Reviewer_BKUF", "summary": "The authors explore the problem of conditioning a pretrained video generation model on a desired camera motion. The problem is of high relevance not only to the video generation community, but also in 3D and 4D generation. The authors propose the use of a frozen video generation backbone, coupled with a camera encoder block, injecting camera information into the backbone. The method is simple, sound, clearly written and accompanied by a comprehensive evaluation validating the importance of each of the main components: camera embeddings, encoder architecture, feature injection, and training datasets.", "questions": "- The choice of the Plucker representation appears effective when compared to Euler angles, quaternions and raw values, though some properties of the Plucker representation appear suboptimal for the purpose of achieving camera control. In particular, the set of points along a certain camera ray will have a constant embedding. Sitzmann (2021) introduces the use of Plucker coordinates in the context of LFNs, for which this property is well justified and beneficial. In the context of the current paper, though, Plucker embeddings appear to obscure the camera origin, a fundamental information for the model. With each Plucker embedding defining a set of points along a ray, the only way for the model to recover the camera origin is to solve a system of equations involving multiple rays. This is a necessary, but likely difficult task for the model to perform. Why is this property desirable for this work? Did the authors experiment with alternative representations not suffering from this drawback such as (o, d) or (normalized_o, d) (for some choice of normalization for o) which would make the camera origin readily available to the model?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The problem treated in the paper is of high significance to video generation, and 3D/4D generation.\n- The paper is very clearly written. Every section of the paper was easy to follow and it appears the information in the paper would be sufficient to replicate the presented method and results. In addition, code is available.\n- I appreciate the simplicity of the method. The authors identified the most crucial components to enable camera control: use of Plucker embeddings, camera encoder architecture, how to couple the encoder to the retrained backbone, and training data recipe. Without adding bells and whistles, the authors show good camera control performance.\n- Evaluation is solid and informative. Every of the most critical design dimensions (use of Plucker embeddings, camera encoder architecture, how to couple the encoder to the retrained backbone, and training data recipe) is validated in a respective ablation, providing useful insights. The model is demonstrated on diverse video generation backbones\n- Qualitative evaluation is rich. Limitations are shown and highlighted", "weaknesses": "- Discussion of the camera representation could be strengthened by an accompanying figure.\n- While Plucker embeddings were found more effective than a set of straightforward alternative representations, their properties do not appear to be completely aligned to the nature of the proposed approach. See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cipCFFndkm", "reviewer": "Reviewer_UQ23", "summary": "The paper introduces a general-purpose encode for a moving camera that can be used to add camera conditioning to different video generator architectures. The paper considers three technical problems: representing the camera, injecting the representation in a video generator, and developing training data sufficient for learning the model. The camera encoding is based on Plucker rays, it is injected in a video generator via cross-attention layers added to the temporal self-attention of the video generator, and the training data combines synthetic and real-life datasets of static contest where the camera is known.\n\nEmpirically, the new module is integrated AnimateDiff and in and Stable Video Diffusion (SVD), two different video generators prompted using text. Compared to alternative like MotionCtrl, the approach has better metrics in terms of camera reconstruction and image quality.", "questions": "* How is scale handled by your model? For example, what is the meaning/effect of translating the camera by, say, one unit to the right? What about two units?\n\n* How do you make sense of the camera motion if you don't know what is going to be generated in the video?\n\n* Does the choice of focal length (camera intrinsics) affect the generated video?\n\n* Can you comment on the potential issues on the evaluation discussed above?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The problem considered by the paper is practically important: camera control is an important feature in many potential applications of video generation\n\n* The approach is technically simple. The architecture design is reasonably well motivated. The performance is shown to be better than close competitors.\n\n* Several applications beyond text-to-video generation with camera control are demonstrated, showing that the method successfully combines with other control types.", "weaknesses": "I remain unconvinced about the formulation of the problem (which is a limitation shared by prior works).\n\nThe paper aims at controlling camera motion accurately, but camera motion can only be interpreted in *relation* to the content generated in the video, and this relationship is not well defined in this work. This has several manifestations, discussed next.\n\nFirst, there is the issue of scale: how a particular value for the camera translation should be interpreted depends on the scale of the generated scene, which is unknown and/or arbitrary. For example, a translation of one meter to our right is entirely negligible if we are looking at the sun, but if we are looking at the chair in front of us it is sufficient to take the object out of view entirely. There is no indication on whether the model is learning a \"metric\" (or canonical) interpretation of scale; this seems in fact unlikely given the nature of the training datasets which do not posses a well defined scale. The appendix briefly mentions scale as an issue of COLMAP, which requires scale normalisation for evaluation. But this is not a minor issue compared to the  much more serious concern that we cannot assume that the scale of the training data is know. Even if it has \"some\" value, the latter is often arbitrary and inconsistent (many real-life datasets like MVImageNet use COLMAP for camera annotation; synthetic ones like Objaverse have arbitrary scale).\n\nEven if the training data was \"metric\", so that one could at least attempt to learn a meaningful canonical notion of scale, and thus interpret camera translation accordingly, this would still be difficult to do for a model. This is a reflection o the fact that very similar content can exist at different physical scales in the world (e.g., model of a city vs city), so learning  a canonical reference scale, insofar as one can be defined, is difficult.\n\nSecond, along with scale there is the issue of framing and aligning content and camera motion. Suppose you wanted to move the camera around an object. You could specify a camera trajectory that pivots around a given arbitrary physical point in front of you, in a turn-table-like fashion. Whether the network decides to place an object in correspondence of the pivot point, thus obtaining a turn-table-like effect, is not clear. Camera motion is only useful when its relation to the content is meaningful, so that the subject of the video is framed in the intended way.\n\nThirdly, there is the issue of the intrinsic parameters of the camera. It is not clear how these are handled in the paper. Specifically, the Plucker ray should be able to encode any camera model/focal length, but it is not obvious whether the authors train the model with different sets of cameras, and if the model is accurate enough to capture focal length correctly. Note that focal length would affect the apparent size of an object in an image, so this also has a strong effect in attempting to learn a canonical/metric interpretation of scale, which seems necessary in this formulation of the problem (see above).\n\nNote that the issue of scale trickles down to evaluation. Even if the content of the video is not well matched to the camera motion (i.e., the framing and/or scale is not what one would expect), the model can still generate a video with the prescribed camera motion. The latter can then recovered using, e.g., COLMAP and compared to the input camera up to scale (for the translation component). The latter is described in appendix, but evaluation of camera reconstruction has been considered extensively int he literature; why not using one of the many standard scale-invariant metrics instead of proposing a new one (line 990)?\n\nMore worryingly, the authors state (line 983) that they manually discard badly generated videos where COLMAP fails in their evaluation (line 983). This can very well introduce a selection bias, where their method is favoured by simply discarding difficult videos more liberally than for others. While it is difficult to compare video generators directly due to the aleatoric nature of their task, nevertheless the authors should consider a metric that would at least remove this problem. For example, they could measure success rate, as in the percentage of generated videos where the camera is reconstructed at different levels of accuracy, generating corresponding curves. By prompting different models with the same textual prompts and the same camera motions, such curves would at least be more comparable. \n\nThe novelty of the paper is reasonable, but not outstanding. A very similar problem is considered in MotionCtrl and the paper is mostly an iteration that innovates on the three technical components discussed above (better camera encoding, better camera injection, and better data). There is nothing wrong with it, and the results are a bit better, but the conceptual innovation does not run very deep.\n\n# Minor issues\n\n* Line 195: the sentence structure is broken\n\n* Line 460: Do you mean that COLMAP cannot reconstruct the generated videos? Is this an issue with COLAMP or with the quality of generation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)