SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation
Authors: name: Koichi Namekata; name: Sherwin Bahmani; name: Ziyi Wu; name: Yash Kant; name: Igor Gilitschenski; name: David B. Lindell
Published: 2024-11-07 18:56:11

Abstract: Methods for image-to-video generation have achieved impressive,
photo-realistic quality. However, adjusting specific elements in generated
videos, such as object motion or camera movement, is often a tedious process of
trial and error, e.g., involving re-generating videos with different random
seeds. Recent techniques address this issue by fine-tuning a pre-trained model
to follow conditioning signals, such as bounding boxes or point trajectories.
Yet, this fine-tuning procedure can be computationally expensive, and it
requires datasets with annotated object motion, which can be difficult to
procure. In this work, we introduce SG-I2V, a framework for controllable
image-to-video generation that is self-guidedx2013offering
zero-shot control by relying solely on the knowledge present in a pre-trained
image-to-video diffusion model without the need for fine-tuning or external
knowledge. Our zero-shot method outperforms unsupervised baselines while being
competitive with supervised models in terms of visual quality and motion
fidelity.