Still-Moving: Customized Video Generation without Customized Video Data
Authors: name: Hila Chefer; name: Shiran Zada; name: Roni Paiss; name: Ariel Ephrat; name: Omer Tov; name: Michael Rubinstein; name: Lior Wolf; name: Tali Dekel; name: Tomer Michaeli; name: Inbar Mosseri
Published: 2024-07-11 17:06:53

Abstract: Customizing text-to-image (T2I) models has seen tremendous progress recently,
particularly in areas such as personalization, stylization, and conditional
generation. However, expanding this progress to video generation is still in
its infancy, primarily due to the lack of customized video data. In this work,
we introduce Still-Moving, a novel generic framework for customizing a
text-to-video (T2V) model, without requiring any customized video data. The
framework applies to the prominent T2V design where the video model is built
over a text-to-image (T2I) model (e.g., via inflation). We assume access to a
customized version of the T2I model, trained only on still image data (e.g.,
using DreamBooth or StyleDrop). Naively plugging in the weights of the
customized T2I model into the T2V model often leads to significant artifacts or
insufficient adherence to the customization data. To overcome this issue, we
train lightweight Spatial Adapters that adjust the features produced
by the injected T2I layers. Importantly, our adapters are trained on
"frozen videos" (i.e., repeated images), constructed from image
samples generated by the customized T2I model. This training is facilitated by
a novel Motion Adapter module, which allows us to train on such
static videos while preserving the motion prior of the video model. At test
time, we remove the Motion Adapter modules and leave in only the trained
Spatial Adapters. This restores the motion prior of the T2V model while
adhering to the spatial prior of the customized T2I model. We demonstrate the
effectiveness of our approach on diverse tasks including personalized,
stylized, and conditional generation. In all evaluated scenarios, our method
seamlessly integrates the spatial prior of the customized T2I model with a
motion prior supplied by the T2V model.