I2VControl-Camera: Precise Video Camera Control with Adjustable Motion Strength
Authors: Wanquan Feng
Jiawei Liu
Pengqi Tu
Tianhao Qi
Mingzhen Sun
Tianxiang Ma
Songtao Zhao
SiYu Zhou
Qian HE
Venue: ICLR 2025 Poster (2025)
Keywords: Video Generation
Camera Control
Primary area: generative models
TL;DR: Precise Video Camera Control with Adjustable Motion Strength

Abstract: Video generation technologies are developing rapidly and have broad potential applications. Among these technologies, camera control is crucial for generating professional-quality videos that accurately meet user expectations. However, existing camera control methods still suffer from several limitations, including control precision and the neglect of the control for subject motion dynamics. In this work, we propose I2VControl-Camera, a novel camera control method that significantly enhances controllability while providing adjustability over the strength of subject motion. To improve control precision, we employ point trajectory in the camera coordinate system instead of only extrinsic matrix information as our control signal. To accurately control and adjust the strength of subject motion, we explicitly model the higher-order components of the video trajectory expansion, not merely the linear terms, and design an operator that effectively represents the motion strength. We use an adapter architecture that is independent of the base model structure. Experiments on static and dynamic scenes show that our framework outperformances previous methods both quantitatively and qualitatively. Project page: https://wanquanf.github.io/I2VControlCamera.

Published as a conference paper at ICLR 2025
I2 12VCONTROL-CAMERA, PRECISE VIDEO CAMERA CONTROL WITH ADJUSTABLE MOTION STRENGTH
Wanquan Feng" 1 Jiawei Liu' Pengqi Tu
Tianhao Qi! Mingzhen Sun!..
Tianxiang Ma
Songtao Zhao'
Siyu Zhou' Qian He!
ByteDDncce China
University of Science and Technology of China (USTC)
Institute of Automation, Chinese Academy of Sciences (CASIA)
ABSTRACT
Video generation technologies are developing rapidly and have broad potential applications. Among these technologies, camera control is crucial for generating professional-gualiyy videos that accurately meet user expectations. However, ex- isting camera control methods still suffer from several limitations, including con- trol precision and the neglect of the control for subject motion dynamics. In this work, we propose 12VControl-Camera. a novel camera control method that sig- nificantly enhances controllability while providing adjustability over the strength of subject motion. To improve control precision, we employ point trajectory in the camera coordinate system instead of only extrinsic matrix information as our control signal. To accurately control and adjust the strength of subject motion, we explicitly model the higher-order components of the video trajectory expan- sion, not merely the linear terms, and design an operator that effectively repre- sents the motion strength. We use an adapter architecture that is independent the base model structure. Experiments on static and dynamic scenes show that our framework outperformances previous methods both quantitatively and quali- wttp://www.iiiibuooo//oooooooooo
tatively. Project page:
INTRODUCTION
Video generation technologies are explored to synthesize dynamic and coherent visual content, con- ditioned on various modalities including text (Blattmann et al. 2023c Wang al., 2024a] Gupta et al. 2023) and images (Blattmann et al.) 2023b] Feng et al. 2024). Video generation has broad application potential across various fields, such as entertainment, social media, and film production. Motion controllability is crucial for ensuring that generated videos accurately meet user expecta- tions, with camera control being one the most important aspects. Camera control is the process of adjusting the position, angle, and motion of camera, resulting in changes to the composition, perspective, and dynamic effects of a video. This technique is essential for generating professional- quality videos, it influences the attention of viewers and enhances the expressiveness of scenes.
Although precise camera control crucial for producing high-quality videos, existing methods still face challenges. The first challenge pertains the precision and stability of control. The lack of pre- cision would result in an inaccurate reflection of the user control intention, significantly degrading user satisfaction. The second challenge is ensuring the natural dynamics of the subjects themselves, independent of camera movements. Similar to the challenges in multi-view (Mildenhall et al.) 2020) Kerbl et al.) 2023) and 3D geometric algorithms (Wang al. (202I) 1) where static scenes are much easier to handle than dynamic ones (Pumarola et al. [020 Cai et al. (0022), generating plausible dynamics in videos proves to be more complex than managing static elements.
While AnimateDiff (Guo et al. 2024b) utilizes LoRA (Hu al. 2022) strategy for controlling cam- era movements, the motion-LoRAs are confined to limited set of fixed movement modes, lacking flexibility, and it only allows for coarse control, thus failing to provide precise scale adjustments. A direct and intuitive approach allowing for arbitrary camera movements is embedding the camera pose matrix, in MotionCtrl (Wang et al. 2023).. However, this method results in sparse input sig- nals that heavily rely on the training set distribution, which leads to poor generalization capability. Consequently, it may inadequately respond to less common camera parameters within the training

---

Published as a conference paper at ICLR 2025
Dolly out
Tilt down
Rotate clockwise
Pan right Move left
I Surround
Figure 1:
We propose IZVControl-Cames a novel camera control method for image-to-video generation, offering high control precision and adjustable motion strength.
dataset, and thus hinders precise control over the motion's amplitude. Although CameraCtrl (He al. 2024) attempts mitigate this sparsity issue by employing Plücker embeddings (Sitzmann et al. 202T), this parameterization lacks information of the input image, and it actually does not offer any additional information compared to the camera matrix used in MotionCtrl. Another nat- ural strategy is novel view synthesis, which uses 3D implicit representations that can be rendered from arbitrary views, such as Cat3D (Gao* et al. 2024). Unfortunately, this strategy cannot support subject motion well, thus undermining the goal of creating dynamic video content.
In this paper, we propose 22CControl-Camera, a camera control method (some examples shown in Fig. I to surmount these prevalent issues in image-to-video generation, enhancing the control precision and adding control over the dynamic strength of subject motion in video output. To en- sure control precision and stability, we point trajectories in the camera coordinate system as our control signals, instead of extrinsic matrix. From the point trajectory function, we extract the linear term serve proxy for camera control, ensuring high precision, stability and user friend- liness. To control the motion strength, we further represent object motions with higher-order terms in the trajectory function and explicitly model the degree of dynamics. Specifically, we employ the derivative of the high-order terms to compute the motion speed of each point and integrate them the image domain to obtain the entire motion strength as the control input of the network. This approach allows to accurately gauge and adjust the amplitude of subject motion dynamics.
We construct training data from regular RGB videos registering 3D tracking information and motion mask for them. Our approach features adapter architecture that remains agnostic to the under- lying base model structure. Experimentally, we conduct experiments in both static and dynamic scenes. For static scenes, can set the motion strength zero, resulting in significantly higher precision than previous methods. In dynamic scenes, we can configure a higher motion strength, which allows for both high control precision and vivid subject motion. Our approach outperforms previous methods both quantitatively and qualitatively. In summary, our contributions include:
We explicitly model decoupled motion representations: 3D rigid point trajectories and mo- tion strength for camera and subject motion controls.
We propose data pipeline to construct training control signals from RGB videos.
For both static and dynamic scenes, our method outperformances previous methods both quantitatively and qualitatively.
2
RELATED WORK
2.1 TEXT TO VIDEO SYNTHESIS
Text-to-video generation requires models to synthesize realistic videos based on given textual de- scriptions. Recent progress in diffusion models has boosted the quality of T2V generation to
2

---

Published as a conference paper at ICLR 2025
unprecedented degree, achieving both impressive visual quality and surprising text-video consis- tency (Brooks et al. 20224 Blattann et al. 2023b). Image Video (Ho et al. 2022) cascaded mul- tiple video generation and super-resolution diffusion models to generate long and high-resolution videos from textual descriptions. Make-A- Aideo (Singer et al. 2022) extended a diffusion-based
'21 model to T2V in a spatiotemporal factorized manner. Based the successful experiences of
image generation methods, several works (Wang et al., 2024a] Girdhar et al.) [023] Mei & Patel 2023] performed T2V by first generating images from texts and then synthesizing videos based on images. EMU VIDEO (Girdhar et al. 2023) introduced adjusted noise schedules and multi-stage training strategy for high-quality video generation. To reduce the computational complexity of video
generation, other works (Blattmann et al. 2023c He et al.) [022 Yu et al. [023 Gupta et al.) 2023) explored different designs of video auto-encoders, which can map a high-dimensional video into a low-dimensional latent space. LVDM (He et al.] (2022) compressed videos from both the spatial and temporal dimensions, obtaining a low-dimensional 3D latent for each video. In addition, Lumiere (Bar- Tal et al. (2024) and Latte (Ma et al., (024) explored different 3D model structures. Recently, Sora (Brooks et al. 2024) showed the power of DiT (Peebles & Xie 2022) in T2V task.
2.2
IMAGE TO VIDEO SYNTHESIS
Image-to-video task aims to generate videos with a static image as the condition. One classic strat- egy is integrating CLIP embeddings of the static image into DPMs. For instance, VideoCrafterl (Chen et al. 2023a) and 12V 12V-Adapter (Guo et al. (024a) utilized a dual cross-attention layer, similar
the IP-Adapter (Ye et al., 2023), fuse these embeddings effectively. However, due to the no- torious issue of CLIP image encoders losing fine-grained details, subsequent works (Hul 2024 Weil
al., 2024) have proposed using more expressive image encoders to capture finer image features. In addition, another strategy is to expand the input channels of DPMs to concatenate noisy frames and the static image. Notable works in this category include SEINE (Chen et al.) 20236). Pixel-
Dance (Zeng et al. 2024), AnimateAnything (Dai et al. 2023), and PIA (Zhang et al. 2024]) which have demonstrated superior results by enhancing the input channels to integrate image information more effectively. Finally, methods such as DynamiCrafter (Xing et al., 2023). I2VGen-XL (Zhang jet al. 2023), and SVD (Blattmann et al. 2023a) combined channel concatenation and attention mechanisms to simultaneously inject image features, aiming achieve consistency in both global semantics and fine- -en-ggrined details. This dual approach ensured that the generated videos maintained high level of fidelity to the original static images while introducing realistic and coherent motion.
2.3 VIDEO CAMERA CONTROL
While methods aiming to control video foundation models continue to emerge. relatively few works explore how to manipulate camera motions in generated videos. AnimateDiff (Guo et al.] 2024b) employed temporal motion LoRA (Hu et al., 2022) trained video datasets with similar camera motions, where one single trained LoRA can control a specific type of camera motion. MotionC- trl (Wang et al. 2023) proposed to employ an adaptor structure encode the extrinsic matrix of
each frame into the temporal attention layers. Further, CamereCtrl (He et al., 2024) utilized the
Plücker embedding to improve the controllability. Camtrol (Hou et al.j 2024) proposed a simple training- muining-free method to directly render static point cloud to multiview frames and construct the final output video in a video-to-video manner. CamCo (Xu et al. 2024) integrated an epipolar attention module in each attention block that enforces epipolar constraints to the feature maps, which keeps 3D-consistent well but causes small motion dynamic. In our work, we propose a method that can enhance the precision of camera control and add the control over subject motion strength.
METHOD
3.1 VIDEO REPRESENTATION AND NOTATIONS
In this section, we introduce the video representation and notation used in this paper. First, we stipulate that the coordinates of all points we study are in the camera coordinate system. Although both the camera and the captured scene may move, transfer all dynamics to the camera coordinate system, as in Fig. 2 Intuitively, the entire 3D world can be divided into the the static part and the dynamic part, where the static part corresponds linear motion in the camera coordinate system.
3

---

Published as a conference paper at ICLR 2025
Coordinate
System
Lift to 3D
Transform
Camera
Initial Camera
Initial Camera
Movement
Input Image
Point Cloud
Point Trajectory
Figure 2: We lift the input image from 2D to 3D as a RGBD point cloud. When the camera moves, the 3D points can be considered as moving in the camera coordinate system. Then we project them onto 2D according current camera pose obtain the 2D point trajectory.
Consider dynamic sequence F(P, A):
F(p. A) R [0, - -R, s.t. F(p.
where À E [0, A] represents a time moment during the video, and p E TR.S denotes point of the entire 3D world. Notice that specifically enforce [p,,00 0) p, ensure that F accurately defines the 3D motion trajectory originating from the first frame. Considering the physical properties of the macroscopic world, is reasonable to consider F smooth mapping function. Naturally, we can assert that for any given À E A], there exist unique RA E 3x3 and tx E R such that:
F(p. = RA 0().0) + tx + o(p).
(2)
where (p) denotes an infinitesimal of higher order than p. To simply prove we only need to perform a Maclaurin expansion of F(, A) and F(p, 0) at p 0:
F(p, A) = F(0, A) + (0, A) - o(p), F(p,0) = +J,(0,0) -p+
(3)
(4)
where J.r denotes the Jacobian matrix, representing the gradient for vector-valued functions. Sub- tracting the equations and performing simple calculation yields:
A(P, = [I Jo F(p, A)), o(p).
(5)
Evidently, we can define:
R I+ J) 2 F(O,A),
(6)
Subsequently, we further denote:
C A) A)
(RA F(P, + 6.) o
which actually represents the extent of nonlinearity, being higher-order infinitesimal with respect to than the linear term. Up to now, we have introduced the variables (R,, tx, G(p, to facilitate our forthcoming discussion on video camera control.
3.2 CONTROL SIGNAL CONSTRUCTION
While the most intuitive method is to directly employ RA and fx the control signals, we aim to overcome the previously mentioned challenges controllability and subject motion. Denote the region of 3D points captured by the first frame Ω C 3 We compute the linear translation for Ω and project to 2D, which defines point trajectory on the camera plane:
Tx II(R tx), [0, A]
(8)
where II is the projection operation. Compared to RA and tx. TA offers denser representation, thereby providing enhanced controllability and stability.
However, at the same time, this could further inhibit the motion of the nonlinear parts, which is undesirable. To address this issue, we proceed to model the motion of the nonlinear parts (dynamic regions in the world system) as well. Considering that have already defined the variable G(p, X) to measure the extent nonlinearity, we employ its first-order derivative with respect to time À to quantify the degree of motion dynamics time moment X. In our generative tasks, cannot control
4

---

Published as a conference paper at ICLR 2025
the motion of every individual point, instead resort to secondary strategy. We integrate the Le norm of the first-order derivative (physically represents the motion speed of the point, as shown in Fig across the entire domain f2, to characterize overall measure of motion strength:
MA [22] /
[g(p. 2) a%
1 [22
(
( ar
(0g(p. a2
12 dp
dp
(9)
Up to this point, we have fully defined the inputs of our camera control framework, xTx, m.). where the former enhances controllability and stability, and the latter effectively represents the extent of motion dynamics, thus fulfilling the original intent of our designed method.
In addition, we discuss some properties for the control signals. As discussed in Sec. [3.]] Ω can be divided into static part and dy- namic part, which we can denote as f2s and Qn respectively. No-
Motion Vector (g(p,)) A)
02
tably, Sög corresponds to the linear motion within the camera sys- tem. fact, we have:
Point P
Speed
00 A)
112
F(p, A) RA F(p, 0) Lx Vp 2g-
(10)
Value
0A
In other words, (p. A) = 0 on S2g. According to Eq. 10
if we can
G
obtain the partition = 2g Qdp, we can calculate (T A my) by
Figure 3: Illustration of mo-
simply linear fitting the point trajectory function A),
tion strength (speed value).
3.3 DATA PIPELINE
In Sec. 3.2 we theoretically analyzed how to derive the input signal (Tx, m2x ) for camera control. In this section, we show how compute them for the real-world video data gt. For the real-world video, the timesteps is a discrete sequence À j0, TIZ where À represents the timestep index. The region captured by the first frame can be organized on H W pixels, denoted Ω2 {Pt)) H.W H,W
Further, we divide the whole point set (Pty (p.y)tjj into the static part and the dynamic part:
H.W = (P ijj = 22 112p
(11)
where Ug denotes the static part, and 2D denotes the dynamic part. Different from the theoretical analysis discussed in above sections, there are several major gaps between real-world RGB video data and the continous trajectory function:
Lack of 3D Information: Real-world video data only contains 2D pixels without 3D information.
The raw video data does not explicitly involve the informa- tion about the temporal movement of dense points described by the continous trajectory function.
Lack of Temporal Correspondence:
Lack of dynamic/statie partition: real-world video data, discerning which regions are dynamic and which are static remains ambiguous, especially when the camera itself is also mobile. This introduces a coupling between the movement of objects and the motion the camera.
To address the first issue, employ metric depth estimation method, Unidepth (Piccinelli et al.) 2024). to bridge the gap between 2D and 3D data representation. For the second issue, we utilize tracking method, SpatialTracker (Xiao et al. 2024). to establish pixel correspondence between consecutive frames, so that we can obtain the discrete trajectory (still denoted as F(p, A) for ease of reading). For the third issue, need to extract U2s 2p S Ω from Ω. A key insight lies in Eq. IO] which means the trajectory on Ps can be linearly fitted well. We solve this problem in a iterative manner, described in Alg.
Iteratively, we the trajectory and extract the well-fitted region as the updated static region 2g, while the remaining part the dynamic part SDp Once obtain the result Qg and Ω2, like in each iteration, we can final compute (R,, tx) by addressing a nonlinear least squares problem using the L-BFGS (Luu & Nocedal 1989) algorithm:
(RA, tx) arg min| A)) I(R Us + t)1
(12)
R,t
Then, Ty can be calculated according Eq.
5

---

Published as a conference paper at ICLR 2025
Data: Whole region S2, point trajectory F(p, X), tolerable enror acceptable ratio a, maximum iterations Nmax
Result: The static region Gg and the dynamic region ?2p
initialization: n +- 0, S2g <- 22 Ap 0;
while n < Nax do
for each À [0, [] do
Solve with L-BFGS: (RA, tx) = arg min (a II(R 22 + t)l;;
R,t
end
Compute: Emaz maxpens
Σ
[[](F(p))) - II( p ++)l:
if Cmaz < then stop (solution found);
else
2s (p e 2
Σ A)) [R, p - tx)l1? < a (Gmas + c));
if 22 = Ω then
stop (solution found);
end end
?D
Q 923; 71 +- n1 + 1;
end
Algorithm 1: Static and dynamic region extraction based on trajectory analysis
For the motion strength, we empoly the difference between adjacent frames to replace the first-order derivative of G(p, A). Specifically, calculate:
if X = 0,
(13)
H.W
[p - -( 1)ll2 Σ
if > O.
HW
Thus, calculate the required control signals (T m)) from any RGB video, which allows us to train the model with a vast array of easy-acquired RGB video data.
3.4 NETWORK, TRAINING AND INFERENCE
To ensure method remains compatible with rapidly evolving base models, have implemented an adaptive structure. Our network design is illustrated in Fig. 4 Starting from the original control signal, adapter network generates control feature that can be integrated into any diffusion process, thereby allowing adaptation to various video generation base frameworks.
Considering that TA is 4-dim tensor with shape (T', H, W) and mx 2-dim tensor with shape 1), we use tiling method to expand m to the same shape as Tx, and then concatenate them along the channel dimension finally obtain (T, 3, H, WS-shaped tensor the input of the network. As shown in Fig. El (layers marked with flame are our adaptive layers), we first employ several convolutional layers convert the input the same size as the tokens used in the diffusion process. We then concatenate the features with the tokens before computing self-attention. After the self-attention computation, we restore the original shape by removing the additional parts added during concatenation, similar to Hu (202)).
During the training phase, we merely incorporate the insertion of the control signal, while all other training strategies remain unchanged. We adopt the same loss function and the same scheduler, with the sole modification being the introduction of the control signal condition. During testing, we first employ Unidepth (Piceinelli et al. 2024) to lift the input image into RGBD point cloud. When the user moves the camera, we convert it as the transformation of 3D points in the camera coordinate systemcan according to the camera poses. We finally project the transformed 3D points of each

---

Published as a conference paper at ICLR 2025
frame onto camera plane to compute the 2D trajectory, shown in Fig. 2 Additionally, the user can provide a scalar value for the motion strength control. As result, this conforms to the training paradigm and produces suitable camera movement effects. When the motion strength is set small, nearly static camera movements can be achieved, whereas a significant motion strength allows for more pronounced dynamics of the subject.
4 EXPERIMENTS
In this section, we show our experiments. Sec. 4.1 introduces our implementation details and experi- ment settings. Sec. 4.2 shows the results and some properties of our method. In Sec. [4.]] we compare our method with previous baseline methods.
User Input Controls
Temporal Self-Amention
Motion Strength
Point Trajectory
Tokens
Adaptive Self- Aelf-Attenioon
M
4.1 SETTINGS
Conv Layers e
Control Tokens
Tokens
(B,T, (B.T,C.M, W)
Implementation details. We employ a Image- to- vo-Video version Magicvideo-V2 (Wang et al.) 2024b) as our base model, where we set the frame number as 24 and the resolution as 704 x 448. We use 16 NVIDIA A100 GPUs to train them with batch size per GPU for 20K steps, taking about 36 hours. During training. fix the parameters of the base model and only train our adapter part.
Trainblee
Temporal Self-Atlention
Tokens
Figure 4: The adaptive network structure.
Datasets. Although previous methods trained the RealEstaae 10K (Zhou et al. (0188) dataset for train- ing, we do not choose as our training because the videos in this dataset are all nearly static scenes with very limited dynamic motion of objects, which is conflict with our goal of achieving controllable motion dynamics. Therefore, we collect dataset of 30K video clips as our training set, which contains not only camera movements but also natural motion. For validaition, we choose two testing sets. The first testing set comprises 500 random static scene clips from RealEstate where each clip only extracts the initial frames according to the generated frame number. To enrich the testing set, we randomly substitute the camera movements half these clips with one of eight basic camera movements (as in MotionCtrl (Wang et al., 2023)): ) pan left, pan right, pan up, pan down, zoom in, zoom out, anticlockwise rotation, and clockwise rotation. The second testing set consists of 480 samples generated by text-to-image model that feature movable objects including humans and animals, each equipped one the basic camera movements.
Metrics. To comprehensively evaluate the quality of the results generated by method and to facilitate a fair comparison with existing techniques, we employ the same metrics as in CameraC- trl (He al. 20244) Rotational Error (RotErr), Translational Error (TransErr) and Fréchet
Inception Distance (F (FID) (Seitzer 2020). To compute F FDD, we randomly select 2000 video frames from WebVid (Bain al. 2021). As for the calculation of Rot and TransErz, we refer to the formula in CameraCtrl (He et al. 2024). Furthermore, as our method supports adjusting the motion dynamics, we design a metric measure the motion dynamics in the generated videos, too. Specifically, we employ the open source RAFT (Teed & Deng. 2020) optical flow model to calcu- late a motion score, denoted as MSC. Specifically, we use optical flow to establish a correspondence relationship between any two adjacent frames, then perform 2D rigid alignment between adjacent frames (to appropriately remove the optical flow caused by camera movement), and compute the average of the L2 alignment errors as the metric MSC.
4.2 VISUALIZATION RESULTS
In this section, show the visualization results of our method, demonstrating both pixel-level controllability and the motion strength adjustment. Due to the format of the paper, the results shown below are all in frame-by-frane image format. Please project page for video results.

---

Published as a conference paper at ICLR 2025
Time
GT Preview
Ours
motion strength 0)
GT Preview
Ours
( (motion strength 600)
Figure 5: Visualization of our pixel-level controllability. The figure presents two samples: the top one demonstrates pan-left camera movement, while the bottom shows the camera sliding to the right. For each sample, we show a preview (directly render the RGBD point cloud on to 2D plane according to the extrinsic matrix) and our generated result. We can that the generated result can almost follow the control signal at the pixel level (can be seen in the green boxes) even when there exists movable object (the cat the red box).
4.2. PIXEL-LEVEL CONTROLLAABILITY
We demonstrate comprehensive pixel-level user controllability in our approach, as illustrated in Fig. Initially, we estimate the metric depth from the input image, and then directly manipulate the RGBD point cloud with control signals to render preview image. This provides users with an immediate and intuitive visual feedback, labeled as "Preview" in the figure. Below the direct rendering results, we display the outputs generated by framework. As observed, the camera pose in the generated results is largely consistent with the preview, indicating that our control system achieves precise pixel-level control. In the first sample, where set the motion strength to 0, the fox remains static, and the entire image aligns perfectly with the preview. In the second sample, with the motion strength set to 600, the cat is able to walk the floor. Despite the movement of cat, the camera positioning remains consistent with the preview across all static elements, such the fireplace. These examples underscore the ability of framework to maintain pixel-level alignment regardless of the motion strength. This high level of controllability ensures that users can interactively and effortlessly tailor their visual outputs with exceptional precision, epitomizing a truly user- friendri experience.
4.2.2 MOTION STRENGTH ADJUSTMENT
In Fig- 6 we illustrate the effects of varying motion strength values on the same input image with consistent camera movements. When the motion strength 0, the image content appears almost stationary. Conversely, as the motion strength is increased, the main objects within the scenes begin exhibit motion. For instance, in the first example, the camera performs pan-right movement, shifting the entire scene to the left. At a motion strength of 0, the polar bear remains static, moving uniformly with the background. However, increasing the motion strength allows the bear to move independently, walking naturally and vividly across the frame, giving an impression of freedom and animation. In the second example, with the camera moving downward. the scene seems to ascend. With motion strength as 0, the astronaut stands still, anchored to the ground. Increasing the motion strength causes the astronaut to walk toward the camera, thereby enhancing the dynamic interaction and realism within the scene. The third example features the camera rotating counterclockwise, which results in the scene rotating clockwise. Here, a motion strength 0 keeps the wolf stationary. Yet, upon intensifying the motion strength, the wolf begins run, infusing the scene with action
8

---

Published as a conference paper at ICLR 2025
Motion Strength 0
Motion Strength 600
Modion Strength 0
Motion Strength 600
Motion Strength -
Motion Sirength 600
Figure 6: Results under different motion strength values. We test the same camera control signal with different motion strength value. When the motion strength is set as O, the entire scene is nearly static when there movable objects in the figure (polar bear, astronaut, wolf); when the motion strength is large, the main objects moves obviously.
and enlivening the overall visualization. These demonstrations confirm the efficacy of controlled motion strength system, showcasing its capability to customize dynamic behaviors accordance with the desired camera movements and scene compositions.
4.3 COMPARISONS
In this section, we compare our results with previous baselines: MotionCtrl (Wang et al. (022)) and CameraCtrl (He al., 2024). It is important note that the original MotionCtrl and CameraCtri differ significantly from our training configurations, including differences in the base model, training set, image resolution, and even the number of frames. Fortunately, they both employ an adapter architecture, allowing their designs to be adaptable to various base models. Therefore, to ensure fair comparison, we choose retrain MotionCtrl and CameraCtrl using the same experimental settings and base model (Magicvideo-VD) as ours. In the subsequent text of this section, whenever we refer to MotionCtrl and CameraCtrl, we are referring to the version that have been retrained by us. Considering that the motion-LoRA A of AnimateDiff only supports a limited number of fixed camera movement patterns, we excluded from our comparison.
4.3.1 COMPARISON ON REALESTATEI 100K 10K DATASET
We compare method with MotionCtrl and CameraCtrl on the RealEstate 10K dataset. Consider- ing that data in this dataset are nearly all static scenes, we set our motion strength to O. Quantitative comparisons presented in Tab. [ Our method significantly outperforms the previous methods in both Rot and Trans consistent with the pixel-level precision control observed Sec- tion 4.2.1 For qualitative comparison, refer the left sample in Fig. 7 While our results largely consistent with the preview, the outputs from CameraCtrl and MotionCtrl exhibit noticeable deviations. The results from MotionCtrl has the right trend but with some extra zoom-in, while CameraCtrl, although correct in the direction of camera movement, applies excessive movement amplitude, resulting failure align at the pixel level with the ground truth. It can be seen that our method is closest to the preview image, which is consistent with the conclusion of quantita- tive comparison, further confirming the superiority of our controllability. Our results also show the smallest values in terms of F ID and MSC, indicating that our method not only produces the highest quality of generated images but also maintains the static nature static scenes.
4.3.2 COMPARISON ON DATASET OF MOVABLE OBJECTS.
We also evaluate method against MotionCtrl and CameraCtrl the movable object dataset. Considering it contains movable objects, we experimented with several motion strength values: 0, 200, 400, 600. Quantitative comparisons presented in Tab. 2 Our method performs best in both

---

Published as a conference paper at ICLR 2025
GT Preview
Ours
MotionCtrl
CameraCtrl
GT Preview
Ours
MotionCtrl
CameraCtrl
Figure 7: Qualitative comparison with previous methods. By comparing the preview with the gen- erated results of different methods, we see that control precision is the best.
Table I: Comparison on the RealEstate 10k dataset.
Methods
RotErri
TransErru
FID
MSC
MotinnCrl 2.66
12.70
164.62
13.71
CameraCtrl
1.26
21.60
156.69
15.52
Ours-O
0.53
9.72
155.01
12.93
Table 2: Comparison on the movable object dataset.
Methods
RotErr]
FIDL TransErry
MSC 1
MotionCtrl
2.10
8.08
98.54
42.28
CameraCtrl
1.56
11.32
99.59
32.69
Ours-O
0.76
6.97
100.36
18.96
Ours-200
1.03
7.53
93.28
38.23
Ours-400
1.12
7.23
91.93
47.13
Ours-600
1.18
8.16
91.86
47.70
RotErr and TransErr. Although Ours-200, Ours-400 and Ours-600 perform slightly worse on these two metrics than Ours-O, they are still better than the comparison methods. Ours-600 achieves the best FID and thus the best image quality. The FID of Ours-O is slightly higher than that of the other settings. A possible reason for this could be that the movable objects are forcibly held static, resulting in unnatural and insufficiently diverse frames, while diversity is crucial for FID. For MSC, our smallest value (Ours-0) is lower than the comparing methods, and our largest value (ours-600) higher than the comparing methods, which proves adjustable motion strength control abality again. Qualitative comparison is shown on the right sample in Fig. 7, where only our method is pixel-level aligned with the ground truth.
CONCLUSION
In this work, introduced 12V 12VControl-Camers a precise camera control method designed en- hance the controllability of video generation while maintaining robust range of subject motion. We successfully addressed the challenge of control stability by employing point trajectories in the cam- era coordinate system, rather than relying on extrinsic matrices. Additionally, our approach involved modeling higher-order components of video trajectory expansion, enabling the network to precisely perceive and adjust the amplitude of subject motion dynamics. Our method demonstrated superior performance over previous methods in both quantitative and qualitative assessments. Looking for- ward, possible future work includes extending our framework to include more control modalities, such as drag and motion brush controls. These enhancements will allow for even more detailed and varied manipulations of video content, enabling creators to achieve wider range of visual effeets and further personalize their video productions.
REFERENCES
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In IEEE International Conference Computer Vision,
10

---

Published as a conference paper at ICLR 2025
2021.
Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat. Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprini arXiv:2401. 12945, 2024.
Andreas Blattmann, Tim Dockhom,, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:231 2023a.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:231..15127 2023b.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563-22575, 2023c.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL https::/openai. http://pnnaa.cccrreeeeeeeeeeeeee www..o/wweeee./////oo//
vidde--generationnnee
s- aa--as-wee wor d- s mimuulll lators
Hongrui Cai, Wanquan Feng, Xuetao Feng, Yan Wang, and Juyong Zhang. Neural surface recon- struction of dynamic scenes with monocular rgb-d camera. In Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS), 2022.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high- quality video generation. arXiv preprint arXiv:2310.1951/ 2023a.
Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma. Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. In The Twelfth International Conference on Learning Representationss 2023b.
Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Animateanything -inengr open domain image animation with motion guidance. arXiv e- prints, pp. arXiv-2311, 2023.
Wanquan Feng, Tianhao Qi, Jiawei Liu, Mingzhen Sun, Pengqi Tu, Tianxiang Ma, Fei Dai, Songtao Zhao, Siyu Zhou, and Qian He. 12vcontrol: Disentangled and unified video motion synthesis control. arXiv preprint 4rX1v.241.117655, 2024.
Ruiqi Gao Aleksander Holynski*, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul P. Srinivasan, Jonathan T. Barron, and Ben Poole*. Cat3d: Create anything in 3d with multi-view diffusion models. arXiv, 2024.
Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Ramb- hatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709 2023.
Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Pengfei Wan, Di Zhang, Yufan Liu, Weiming Hu, Zhengjun Zha, al. 12v-adapter: general image-to-we to-vieoo adapter for diffusion models. In ACM SIGGRAPHI 2024 Conference Papers, pp. 1-12, 2024a.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang. Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image dif- fusion models without specific tuning. International Conference on Learning Representations, 2024b.
11

---

Published as a conference paper at ICLR 2025
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662 2023.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint 27Xi::2044.02101, 2024.
Yingqing He, Tianyu Yang. Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelijy long video generation. arXiv preprint arXiv:2211.33222 2022.
Jonathan Ho. William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2200.02300 2022.
Chen Hou, Guoqiang Wei, Yan Zeng, and Zhibo Chen. Training-fres camera control for video generation. CoRR, abs/2406. 10126, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation large language models. In International Con- ference on Learning Representations, 2022. URL. htttps://openreeeee
net. forum?
id=nZevKeeFYf9
Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character anima- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8153-8163, 2024.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics, 2023.
Dong C. Liu and Jorge Nocedal. On the limited memory BFGS method for large scale optimization. Math. Program., 1989.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024.
Kangfu Mei and Vishal Patel. Vidm: Video implicit diffusion models. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 9117-9125, 2023.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint a7Xiv:221..09748, 2022.
Luigi Piccinelli, Hsug-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. UniDepth: Universal monocular metric depth estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. arXiv preprint arXiv:2011.13961, 2020.
Maximilian Seitzer. pytorch-fic fid: FID Score for Py PoTchh. httta:://gituub.com///stt pytorch- fid August 2020. Version 0.3.0.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation nennnrratoioogenetttto without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Frédo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Process- ing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, 2021.
12

---

Published as a conference paper at ICLR 2025
Zachary Teed and Jia Deng. RAFT: recurrent all all-pairs field transforms for optical flow. In Andrea Vedaldi. Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference. Glasgow, UK. August 23-28, 2020, Proceedings, Part II, 2020.
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurlPS, 2021.
Weimin Wang, Jiawei Liu, Zhijie Lin, Jiangqiao Yan, Shuo Chen, Chetwin Low, Tuyen Hoang, Jie Wu, Jun Hao Liew, Hanshu Yan, et al. Magicvideo-v2: Multi-stage high-aesthetic video generation. arXiv preprint arXiv:2401 04468, 2024a.
Weimin Wang, Jiawei Liu, Zhijie Lin, Jiangqiao Yan, Shuo Chen, Chetwin Low, Tuyen Hoang, Jie Wu, Jun Hao Liew, Hanshu Yan, al. Magicvideo-v2: Multi-stage high-aesthetic video generation. arXiv preprint arXiv:2401.0444 2024b.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Yin Shan. Motionetrl: unified and flexible motion controller for video generation. 2023.
Yujie Wei, Shiwei Zhang. Zhiwu Qing, Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hongming Shan. Dreamvideo: Composing your dream videos with customized sub- ject and motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6537-6549) 2024.
Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXi:2230..21190 2023.
Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Camco: Camera-controlllblee 3d-consistent image-to-video generation. arXiv preprint 2rXiv:2406.02509, 2024.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint vrXiv:2308.06721. 2023.
Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 18456-18466, 2023.
Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8850-8860, 2024.
Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-viee video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.04144 2023.
Yiming Zhang, Zhening Xing, Yanhong Zeng, Youqing Fang, and Kai Chen. Pia: Your personal- ized image animator via plug-and-play modules in text-to-image models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7747-7756, 2024.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: learning view synthesis using multiplane images. ACM Trans. Graph., 2018
13


Reviews:
[{"review_id": "JaJTUZONfA", "reviewer": "Reviewer_iTMq", "summary": "The submission pintroduces a new framework for training camer-controllable text-to-video model. To achieve the goal, the authors propose to disentangle the motion into two components: one is 3D rigid transformation of the majority of the underlying 3D scene - presumably this motion component is induced by the camera movement during capturing the scene; and another component that describe the non-rigid transformation of the rest of the scene, which, on the other hand, is supposed to be induced by the movement of the subjects in the scene. \n\nFollowing that, a data collection pipeline is proposed to extract associated control signals that represent the camera movement and the subjects movement of each video clip, respectively. Then a didecated adaption network is trained on top of a existing T2V model to learn a camera- and subject movement-controlled model.", "questions": "Breaking the scene motion into ‘global’ and ‘local’ movements makes sense. However, to what extent can the proposed data pipeline work? For instance, how would it handle a fly-through video over the ocean?\n\nThe point trajectory mentioned at Line 170 is confusing when it first appears. It can easily be mistaken for 3D trajectories.\n\nThe authors claim compatibility with various base models, but there are no experiments to support this claim.\n\nHow are the users supposed to provide the dense point trajectories?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The formulation that decouple the motion behind a video clip into 'global' and 'local' motions is sound.\n\nA practical solution, that utilize several existing computer vision techniques, has been introduced to extract the associated control signals as described in that formulation.\n\nThe results produced by the proposed method demonstrate good visual quality.", "weaknesses": "Lack of sufficient experimental results. The submission contains a limited set of qualitative results. More video results and comparative visual results are needed to justify the effectiveness of the proposed method.\n\nSmall training dataset: The training dataset is relatively small, with only 30k video clips. While I appreciate the authors’ focus on data preparation rather than altering model structures, the limited dataset results in noticeable artifacts, such as flat character bodies in the video results. I am curious why the authors did not use a larger dataset. Is the proposed data pipeline robust and efficient enough for obtaining training data at scale?\n\nExposition: The paper is somewhat poorly written, leading to an unsmooth reading experience. The writing often complicates the ideas rather than simplifying them. Some expositions are even misleading. For instance, the statement “the entire 3D world can be divided into the static part and the dynamic part” at Line 161 is confusing. I believe “rigid and non-rigid motions” would be more appropriate terminology. The majority of the 3D scene remains relatively static only when the camera is fixed in the 3D scene, but the scope of this work obviously goes beyong fixed cameras.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pHZs9drmfd", "reviewer": "Reviewer_k3TV", "summary": "This paper initially defines motion strength as the motion remaining after eliminating camera rotation and translation. Subsequently, adapter layers are incorporated and trained to adhere to the parameters of camera motion and motion strength. These parameters are extracted from captured videos through metric depth estimation, pixel tracking correspondence, and the separation of static and dynamic components via linear motion fitting. The experimental results validate the efficacy of the proposed method.", "questions": "Two questions:\n\n1. What is the range of the 2D coordinates in 3D point projection?  Are they the normalized device coordinates?\n2. Which video generation model is used in the proposed method?  I did not find this information. Maybe I miss something.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The definition of motion strength aligns with the concept of elastic deformation in physics, referring to the shape deformation that remains after removing rigid motion, which holds theoretical significance. The algorithm for recovering motion strength from videos is meticulously designed.\n\n2. The quality of the generated videos is impressive. The objects in the scene exhibit natural motions, and artifacts such as flickering and inconsistent pixels are rare, even during camera movements.", "weaknesses": "1. Although the paper title emphasizes precise video camera control, the main body primarily discusses motion strength. The camera trajectory control capabilities of the proposed method are comparable to those of SOTA papers, like MotionLORA.\n\n2. The camera trajectory representation is based on the projection of 3D points. Firstly, this makes the representation reliant on an additional input: 3D points. Secondly, the projection of these points onto the camera image plane results only in 2D pixel locations, meaning the projections are largely similar across different viewpoints. I am confused by the projection operation used for representing the camera trajectory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jsQWBtlSQE", "reviewer": "Reviewer_UqgM", "summary": "In this work, the authors propose a novel method to control cameras to generate a video with dynamic camera motion from a given image. The existing methods suffer from coarse controllability of cameras and difficulties to control dynamic motion of the subject, this paper resolves the aforementioned challenges by introducing point-based controlling and new training data pipeline.\n\nFrom the quantitative and qualitative results provided by the authors, this paper significantly outperforms the existing methods. With the pleasant qualitative results, I think this new approach can bring the significant contribution to the community and thus, I recommend its acceptance.", "questions": "Given the proposed formulation (2), the method should be generalized to any camera motion as rotational and translational movements of the camera are function of time. But both the paper and the supplementary videos seem to include one component of the camera motion (only zoom-in/out, only rotation, only translation). I would like to hear authors' perspective of this limited results.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) They propose point-based control signals instead of the camera extrinsics. This method can provide dense representation to control the model to generate temporally and spatially coherent video.\n\n2) They resolved the limitations of the in-the-wild video dataset by leveraging the off-the-shelf depth estimation (Piccinelli et al., 2024), point tracking method (Karaev et al., 2023). While this is a bit straightforward, combining with their dynamic/static segmentation, the proposed pipeline can be effective for the future training.\n\n3) The video representation is pleasant and the paper is well written.", "weaknesses": "I do not observe any hard limitation to tackle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3Bnpkl7tOO", "reviewer": "Reviewer_whzD", "summary": "This paper addresses the limitations of existing camera control methods in achieving precise control and managing subject motion dynamics in video generation. The authors introduce a novel approach, \"I2VControl-Camera,\" which uses point trajectories in the camera coordinate system to improve control precision and stability. The method incorporates higher-order trajectory components, allowing for adjustable motion strength to achieve natural and dynamic subject movements. Experiments show that \"I2VControl-Camera\" surpasses state-of-the-art methods in both static and dynamic scenes, demonstrating precise control and realistic subject motion even in complex video generation scenarios.", "questions": "1. The author should provide more details about the experimental setup, e.g., L-BFGS tolerable error, acceptable ratio, iterations, and video length of training.\n2. It would be beneficial if the authors could furnish a detailed analysis of the time complexity for each component of the algorithm, as well as the memory requirements during the training process. This information is essential for assessing the algorithm's efficiency and scalability.\n3. Despite the authors providing an anonymous GitHub link, it remains difficult to effectively validate the algorithm's actual effectiveness. Does the algorithm support the generation of multiple dynamic objects? What is the finest level of control achievable in terms of precision? We request that the authors provide more practical application examples to validate the algorithm's actual effectiveness, including fine-grained pixel-level control, controllable dynamic range, and sequence length of video generation. Furthermore, we ask the authors to provide additional evidence to support the claims made in the introduction section.\n\nCurrently, I would rate this paper as borderline. However, I am willing to reconsider my evaluation if the authors can provide additional clarifications and engage in a more in-depth discussion to address the concerns I have raised.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed method derives camera control precision by using point trajectories in the camera coordinate system, enhancing control tractability compared to traditional extrinsic matrix-based methods. This approach enables more accurate motion handling by explicitly modeling subject dynamics.\nThe method is designed to integrate with forward warp techniques and is compatible with various video generation models due to its adapter-based architecture, which maintains adaptability without modifying base model structures.\nIn practical applications, the method demonstrates the ability to handle deformations in complex, dynamic scenes with minimal camera movement. This adaptability allows for natural, adjustable subject dynamics and accurate control in diverse environments.\nThe paper is well-structured and concise, with clear figures and formulas that effectively support the proposed method's technical clarity and readability.", "weaknesses": "1. The authors’ approach of using point cloud poses to control video generation notably enhances control accuracy and dynamism. However, this method faces a key challenge: dense point clouds greatly increase computational cost, which impacts training stability and limits practical application. A potentially more effective control mechanism could involve using sparse control points—akin to the sparse point clouds generated by Structure-from-Motion (SfM)—which could provide a balanced trade-off between precise dynamic control and computational efficiency. I would highly encourage the authors to include a discussion comparing their method with existing approaches, particularly in terms of computational overhead, inference time, and final output quality.\n\n2. The authors simultaneously consider 3D point motion and camera motion and utilize a unified point cloud trajectory as the control representation for video generation. However, the authors seem to only consider motion as a second-order nonlinear motion described by a Jacobian matrix in Eq.5. Compared to linear motion, what benefits does this second-order representation bring? What computational overhead does it incur? Are there further gains to be had from higher-order representations? It would make sense if the authors were to delve deeper and elucidate the reasons behind their choice of second-order representation.\n\n3. From my perspective, the training data is expensive, and the process is redundant. It involves depth estimation and iterative L-BFGS algorithm, which are computationally costly. For complex video generation models, handling depth and solving for Rλ and tλ are expensive, limiting the algorithm's practical applicability.\n\n4. The authors should provide a more comprehensive explanation of their implementation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)