Emu Video: Factorizing Text-to-Video Generation by Explicit Image
  Conditioning
Authors: name: Rohit Girdhar; name: Mannat Singh; name: Andrew Brown; name: Quentin Duval; name: Samaneh Azadi; name: Sai Saketh Rambhatla; name: Akbar Shah; name: Xi Yin; name: Devi Parikh; name: Ishan Misra
Published: 2023-11-17 18:59:04

Abstract: We present Emu Video, a text-to-video generation model that factorizes the
generation into two steps: first generating an image conditioned on the text,
and then generating a video conditioned on the text and the generated image. We
identify critical design decisions--adjusted noise schedules for diffusion, and
multi-stage training--that enable us to directly generate high quality and high
resolution videos, without requiring a deep cascade of models as in prior work.
In human evaluations, our generated videos are strongly preferred in quality
compared to all prior work--81% vs. Google's Imagen Video, 90% vs. Nvidia's
PYOCO, and 96% vs. Meta's Make-A-Video. Our model outperforms commercial
solutions such as RunwayML's Gen2 and Pika Labs. Finally, our factorizing
approach naturally lends itself to animating images based on a user's text
prompt, where our generations are preferred 96% over prior work.