Lumiere: A Space-Time Diffusion Model for Video Generation
Authors: name: Omer Bar-Tal; name: Hila Chefer; name: Omer Tov; name: Charles Herrmann; name: Roni Paiss; name: Shiran Zada; name: Ariel Ephrat; name: Junhwa Hur; name: Yuanzhen Li; name: Tomer Michaeli; name: Oliver Wang; name: Deqing Sun; name: Tali Dekel; name: Inbar Mosseri
Published: 2024-01-23 18:05:25

Abstract: We introduce Lumiere -- a text-to-video diffusion model designed for
synthesizing videos that portray realistic, diverse and coherent motion -- a
pivotal challenge in video synthesis. To this end, we introduce a Space-Time
U-Net architecture that generates the entire temporal duration of the video at
once, through a single pass in the model. This is in contrast to existing video
models which synthesize distant keyframes followed by temporal super-resolution
-- an approach that inherently makes global temporal consistency difficult to
achieve. By deploying both spatial and (importantly) temporal down- and
up-sampling and leveraging a pre-trained text-to-image diffusion model, our
model learns to directly generate a full-frame-rate, low-resolution video by
processing it in multiple space-time scales. We demonstrate state-of-the-art
text-to-video generation results, and show that our design easily facilitates a
wide range of content creation tasks and video editing applications, including
image-to-video, video inpainting, and stylized generation.