Cinemo: Consistent and Controllable Image Animation with Motion
  Diffusion Models
Authors: name: Xin Ma; name: Yaohui Wang; name: Gengyu Jia; name: Xinyuan Chen; name: Yuan-Fang Li; name: Cunjian Chen; name: Yu Qiao
Published: 2024-07-22 14:00:03

Abstract: Diffusion models have achieved great progress in image animation due to
powerful generative capabilities. However, maintaining spatio-temporal
consistency with detailed information from the input static image over time
(e.g., style, background, and object of the input static image) and ensuring
smoothness in animated video narratives guided by textual prompts still remains
challenging. In this paper, we introduce Cinemo, a novel image animation
approach towards achieving better motion controllability, as well as stronger
temporal consistency and smoothness. In general, we propose three effective
strategies at the training and inference stages of Cinemo to accomplish our
goal. At the training stage, Cinemo focuses on learning the distribution of
motion residuals, rather than directly predicting subsequent via a motion
diffusion model. Additionally, a structural similarity index-based strategy is
proposed to enable Cinemo to have better controllability of motion intensity.
At the inference stage, a noise refinement technique based on discrete cosine
transformation is introduced to mitigate sudden motion changes. Such three
strategies enable Cinemo to produce highly consistent, smooth, and
motion-controllable results. Compared to previous methods, Cinemo offers
simpler and more precise user controllability. Extensive experiments against
several state-of-the-art methods, including both commercial tools and research
approaches, across multiple metrics, demonstrate the effectiveness and
superiority of our proposed approach.