VideoPoet: A Large Language Model for Zero-Shot Video Generation
Authors: name: Dan Kondratyuk; name: Lijun Yu; name: Xiuye Gu; name: José Lezama; name: Jonathan Huang; name: Rachel Hornung; name: Hartwig Adam; name: Hassan Akbari; name: Yair Alon; name: Vighnesh Birodkar; name: Yong Cheng; name: Ming-Chang Chiu; name: Josh Dillon; name: Irfan Essa; name: Agrim Gupta; name: Meera Hahn; name: Anja Hauth; name: David Hendon; name: Alonso Martinez; name: David Minnen; name: David Ross; name: Grant Schindler; name: Mikhail Sirotenko; name: Kihyuk Sohn; name: Krishna Somandepalli; name: Huisheng Wang; name: Jimmy Yan; name: Ming-Hsuan Yang; name: Xuan Yang; name: Bryan Seybold; name: Lu Jiang
Published: 2023-12-21 18:46:41

Abstract: We present VideoPoet, a language model capable of synthesizing high-quality
video, with matching audio, from a large variety of conditioning signals.
VideoPoet employs a decoder-only transformer architecture that processes
multimodal inputs -- including images, videos, text, and audio. The training
protocol follows that of Large Language Models (LLMs), consisting of two
stages: pretraining and task-specific adaptation. During pretraining, VideoPoet
incorporates a mixture of multimodal generative objectives within an
autoregressive Transformer framework. The pretrained LLM serves as a foundation
that can be adapted for a range of video generation tasks. We present empirical
results demonstrating the model's state-of-the-art capabilities in zero-shot
video generation, specifically highlighting VideoPoet's ability to generate
high-fidelity motions. Project page: http://sites.research.google/videopoet/