I2VControl-Camera: Precise Video Camera Control with Adjustable Motion Strength

Authors: Wanquan Feng, Jiawei Liu, Pengqi Tu, Tianhao Qi, Mingzhen Sun, Tianxiang Ma, Songtao Zhao, SiYu Zhou, Qian HE
Primary area: generative models
Keywords: Video Generation, Camera Control

Abstract:
Video generation technologies are developing rapidly and have broad potential applications. Among these technologies, camera control is crucial for generating professional-quality videos that accurately meet user expectations. However, existing camera control methods still suffer from several limitations, including control precision and the neglect of the control for subject motion dynamics. In this work, we propose I2VControl-Camera, a novel camera control method that significantly enhances controllability while providing adjustability over the strength of subject motion. To improve control precision, we employ point trajectory in the camera coordinate system instead of only extrinsic matrix information as our control signal. To accurately control and adjust the strength of subject motion, we explicitly model the higher-order components of the video trajectory expansion, not merely the linear terms, and design an operator that effectively represents the motion strength. We use an adapter architecture that is independent of the base model structure. Experiments on static and dynamic scenes show that our framework outperformances previous methods both quantitatively and qualitatively. Project page: https://wanquanf.github.io/I2VControlCamera.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730023782785

thread_type:
official_review

<thread_object>
creator: reviewer_whzd

<content>
summary:
This paper addresses the limitations of existing camera control methods in achieving precise control and managing subject motion dynamics in video generation. The authors introduce a novel approach, "I2VControl-Camera," which uses point trajectories in the camera coordinate system to improve control precision and stability. The method incorporates higher-order trajectory components, allowing for adjustable motion strength to achieve natural and dynamic subject movements. Experiments show that "I2VControl-Camera" surpasses state-of-the-art methods in both static and dynamic scenes, demonstrating precise control and realistic subject motion even in complex video generation scenarios.

strengths:
The proposed method derives camera control precision by using point trajectories in the camera coordinate system, enhancing control tractability compared to traditional extrinsic matrix-based methods. This approach enables more accurate motion handling by explicitly modeling subject dynamics.
The method is designed to integrate with forward warp techniques and is compatible with various video generation models due to its adapter-based architecture, which maintains adaptability without modifying base model structures.
In practical applications, the method demonstrates the ability to handle deformations in complex, dynamic scenes with minimal camera movement. This adaptability allows for natural, adjustable subject dynamics and accurate control in diverse environments.
The paper is well-structured and concise, with clear figures and formulas that effectively support the proposed method's technical clarity and readability.

weaknesses:
1. The authors’ approach of using point cloud poses to control video generation notably enhances control accuracy and dynamism. However, this method faces a key challenge: dense point clouds greatly increase computational cost, which impacts training stability and limits practical application. A potentially more effective control mechanism could involve using sparse control points—akin to the sparse point clouds generated by Structure-from-Motion (SfM)—which could provide a balanced trade-off between precise dynamic control and computational efficiency. I would highly encourage the authors to include a discussion comparing their method with existing approaches, particularly in terms of computational overhead, inference time, and final output quality.

2. The authors simultaneously consider 3D point motion and camera motion and utilize a unified point cloud trajectory as the control representation for video generation. However, the authors seem to only consider motion as a second-order nonlinear motion described by a Jacobian matrix in Eq.5. Compared to linear motion, what benefits does this second-order representation bring? What computational overhead does it incur? Are there further gains to be had from higher-order representations? It would make sense if the authors were to delve deeper and elucidate the reasons behind their choice of second-order representation.

3. From my perspective, the training data is expensive, and the process is redundant. It involves depth estimation and iterative L-BFGS algorithm, which are computationally costly. For complex video generation models, handling depth and solving for Rλ and tλ are expensive, limiting the algorithm's practical applicability.

4. The authors should provide a more comprehensive explanation of their implementation.

questions:
1. The author should provide more details about the experimental setup, e.g., L-BFGS tolerable error, acceptable ratio, iterations, and video length of training.
2. It would be beneficial if the authors could furnish a detailed analysis of the time complexity for each component of the algorithm, as well as the memory requirements during the training process. This information is essential for assessing the algorithm's efficiency and scalability.
3. Despite the authors providing an anonymous GitHub link, it remains difficult to effectively validate the algorithm's actual effectiveness. Does the algorithm support the generation of multiple dynamic objects? What is the finest level of control achievable in terms of precision? We request that the authors provide more practical application examples to validate the algorithm's actual effectiveness, including fine-grained pixel-level control, controllable dynamic range, and sequence length of video generation. Furthermore, we ask the authors to provide additional evidence to support the claims made in the introduction section.

Currently, I would rate this paper as borderline. However, I am willing to reconsider my evaluation if the authors can provide additional clarifications and engage in a more in-depth discussion to address the concerns I have raised.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730500779933

thread_type:
official_review

<thread_object>
creator: reviewer_uqgm

<content>
summary:
In this work, the authors propose a novel method to control cameras to generate a video with dynamic camera motion from a given image. The existing methods suffer from coarse controllability of cameras and difficulties to control dynamic motion of the subject, this paper resolves the aforementioned challenges by introducing point-based controlling and new training data pipeline.

From the quantitative and qualitative results provided by the authors, this paper significantly outperforms the existing methods. With the pleasant qualitative results, I think this new approach can bring the significant contribution to the community and thus, I recommend its acceptance.

strengths:
1) They propose point-based control signals instead of the camera extrinsics. This method can provide dense representation to control the model to generate temporally and spatially coherent video.

2) They resolved the limitations of the in-the-wild video dataset by leveraging the off-the-shelf depth estimation (Piccinelli et al., 2024), point tracking method (Karaev et al., 2023). While this is a bit straightforward, combining with their dynamic/static segmentation, the proposed pipeline can be effective for the future training.

3) The video representation is pleasant and the paper is well written.

weaknesses:
I do not observe any hard limitation to tackle.

questions:
Given the proposed formulation (2), the method should be generalized to any camera motion as rotational and translational movements of the camera are function of time. But both the paper and the supplementary videos seem to include one component of the camera motion (only zoom-in/out, only rotation, only translation). I would like to hear authors' perspective of this limited results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730552140775

thread_type:
official_review

<thread_object>
creator: reviewer_k3tv

<content>
summary:
This paper initially defines motion strength as the motion remaining after eliminating camera rotation and translation. Subsequently, adapter layers are incorporated and trained to adhere to the parameters of camera motion and motion strength. These parameters are extracted from captured videos through metric depth estimation, pixel tracking correspondence, and the separation of static and dynamic components via linear motion fitting. The experimental results validate the efficacy of the proposed method.

strengths:
1.The definition of motion strength aligns with the concept of elastic deformation in physics, referring to the shape deformation that remains after removing rigid motion, which holds theoretical significance. The algorithm for recovering motion strength from videos is meticulously designed.

2. The quality of the generated videos is impressive. The objects in the scene exhibit natural motions, and artifacts such as flickering and inconsistent pixels are rare, even during camera movements.

weaknesses:
1. Although the paper title emphasizes precise video camera control, the main body primarily discusses motion strength. The camera trajectory control capabilities of the proposed method are comparable to those of SOTA papers, like MotionLORA.

2. The camera trajectory representation is based on the projection of 3D points. Firstly, this makes the representation reliant on an additional input: 3D points. Secondly, the projection of these points onto the camera image plane results only in 2D pixel locations, meaning the projections are largely similar across different viewpoints. I am confused by the projection operation used for representing the camera trajectory.

questions:
Two questions:

1. What is the range of the 2D coordinates in 3D point projection?  Are they the normalized device coordinates?
2. Which video generation model is used in the proposed method?  I did not find this information. Maybe I miss something.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661663488

thread_type:
official_review

<thread_object>
creator: reviewer_itmq

<content>
summary:
The submission pintroduces a new framework for training camer-controllable text-to-video model. To achieve the goal, the authors propose to disentangle the motion into two components: one is 3D rigid transformation of the majority of the underlying 3D scene - presumably this motion component is induced by the camera movement during capturing the scene; and another component that describe the non-rigid transformation of the rest of the scene, which, on the other hand, is supposed to be induced by the movement of the subjects in the scene. 

Following that, a data collection pipeline is proposed to extract associated control signals that represent the camera movement and the subjects movement of each video clip, respectively. Then a didecated adaption network is trained on top of a existing T2V model to learn a camera- and subject movement-controlled model.

strengths:
The formulation that decouple the motion behind a video clip into 'global' and 'local' motions is sound.

A practical solution, that utilize several existing computer vision techniques, has been introduced to extract the associated control signals as described in that formulation.

The results produced by the proposed method demonstrate good visual quality.

weaknesses:
Lack of sufficient experimental results. The submission contains a limited set of qualitative results. More video results and comparative visual results are needed to justify the effectiveness of the proposed method.

Small training dataset: The training dataset is relatively small, with only 30k video clips. While I appreciate the authors’ focus on data preparation rather than altering model structures, the limited dataset results in noticeable artifacts, such as flat character bodies in the video results. I am curious why the authors did not use a larger dataset. Is the proposed data pipeline robust and efficient enough for obtaining training data at scale?

Exposition: The paper is somewhat poorly written, leading to an unsmooth reading experience. The writing often complicates the ideas rather than simplifying them. Some expositions are even misleading. For instance, the statement “the entire 3D world can be divided into the static part and the dynamic part” at Line 161 is confusing. I believe “rigid and non-rigid motions” would be more appropriate terminology. The majority of the 3D scene remains relatively static only when the camera is fixed in the 3D scene, but the scope of this work obviously goes beyong fixed cameras.

questions:
Breaking the scene motion into ‘global’ and ‘local’ movements makes sense. However, to what extent can the proposed data pipeline work? For instance, how would it handle a fly-through video over the ocean?

The point trajectory mentioned at Line 170 is confusing when it first appears. It can easily be mistaken for 3D trajectories.

The authors claim compatibility with various base models, but there are no experiments to support this claim.

How are the users supposed to provide the dense point trajectories?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732144706070

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Rebuttal

comment:
We sincerely thank all the reviewers for their reviews and constructive feedback. Taking into account each concern and question posed by the reviewers, we have given thorough responses within our rebuttal. It is our hope that our responses will be kindly considered during the evaluation of our submission by the reviewers and the AC. Here, we summarize the primary concerns raised by the reviewers and provide a consolidated response.

---
## More Visual Results/Comparisons (@Reviewer iTMq; @Reviewer UqgM; @Reviewer whzD)

We have provide more visual results and comparisons in our anonymous repository for rebuttal: https://github.com/iclr2025sub1844/iclr2025sub1844/blob/main/rebuttal.md. 

We explain our newly added results here:

- @Reviewer iTMq : As suggested, we show [more comparative visual results](https://github.com/iclr2025sub1844/iclr2025sub1844/blob/main/rebuttal.md#pixel-level-control--visual-comparisons) in the newly added page to show the effectiveness of the proposed method.

- @Reviewer UqgM : We show [the results of multiple camera motion combinations](https://github.com/iclr2025sub1844/iclr2025sub1844/blob/main/rebuttal.md#combinations-of-multiple-camera-movements). For easier understanding, we annotate each example with the camera motion mode, such as "move left + pan right" or "move left + pan right + move up + tilt down".

- @Reviewer whzD : We show the [pixel-level controllability](https://github.com/iclr2025sub1844/iclr2025sub1844/blob/main/rebuttal.md#pixel-level-control--visual-comparisons) by previewing the ground truth control trajectory in the page. It can be seen that compared with the comparison method, our results are closest to the ground truth. We also show the results of [different motion strength](https://github.com/iclr2025sub1844/iclr2025sub1844/blob/main/rebuttal.md#multiple-motion-strength) and [multiple dynamic objects](https://github.com/iclr2025sub1844/iclr2025sub1844/blob/main/rebuttal.md#multiple-dynamic-objects).
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a method for allowing more precise camera control in video generation. The reviewers appreciated the approach, which decomposes scene motion into global motion, attributed to camera movement and local motion induced by objects. This decomposition enables more accurate motion handling by explicitly modeling subject dynamics. The proposed method demonstrates strong results, with natural motions and minimal artifacts. The primary weakness of the original submission was the lack of sufficient experiments. The rebuttal addressed this by providing additional experiments on pixel-level control, varying motion strength, handling multiple dynamic objects, and testing with different base models. The paper presents an effective solution for precise camera motion control in video generation, supported by impressive results.