ObjCtrl-2.5D: Training-free Object Control with Camera Poses
Authors: name: Zhouxia Wang; name: Yushi Lan; name: Shangchen Zhou; name: Chen Change Loy
Published: 2024-12-10 18:14:30

Abstract: This study aims to achieve more precise and versatile object control in
image-to-video (I2V) generation. Current methods typically represent the
spatial movement of target objects with 2D trajectories, which often fail to
capture user intention and frequently produce unnatural results. To enhance
control, we present ObjCtrl-2.5D, a training-free object control approach that
uses a 3D trajectory, extended from a 2D trajectory with depth information, as
a control signal. By modeling object movement as camera movement, ObjCtrl-2.5D
represents the 3D trajectory as a sequence of camera poses, enabling object
motion control using an existing camera motion control I2V generation model
(CMC-I2V) without training. To adapt the CMC-I2V model originally designed for
global motion control to handle local object motion, we introduce a module to
isolate the target object from the background, enabling independent local
control. In addition, we devise an effective way to achieve more accurate
object control by sharing low-frequency warped latent within the object's
region across frames. Extensive experiments demonstrate that ObjCtrl-2.5D
significantly improves object control accuracy compared to training-free
methods and offers more diverse control capabilities than training-based
approaches using 2D trajectories, enabling complex effects like object
rotation. Code and results are available at
https://wzhouxiff.github.io/projects/ObjCtrl-2.5D/.