VideoPoet: A Large Language Model for Zero-Shot Video Generation
Authors: name: Dan Kondratyuk, name: Lijun Yu, name: Xiuye Gu, name: José Lezama, name: Jonathan Huang, name: Rachel Hornung, name: Hartwig Adam, name: Hassan Akbari, name: Yair Alon, name: Vighnesh Birodkar, name: Yong Cheng, name: Ming-Chang Chiu, name: Josh Dillon, name: Irfan Essa, name: Agrim Gupta, name: Meera Hahn, name: Anja Hauth, name: David Hendon, name: Alonso Martinez, name: David Minnen, name: David Ross, name: Grant Schindler, name: Mikhail Sirotenko, name: Kihyuk Sohn, name: Krishna Somandepalli, name: Huisheng Wang, name: Jimmy Yan, name: Ming-Hsuan Yang, name: Xuan Yang, name: Bryan Seybold, name: Lu Jiang

We present VideoPoet, a language model capable of synthesizing high-quality
video, with matching audio, from a large variety of conditioning signals.
VideoPoet employs a decoder-only transformer architecture that processes
multimodal inputs -- including images, videos, text, and audio. The training
protocol follows that of Large Language Models (LLMs), consisting of two
stages: pretraining and task-specific adaptation. During pretraining, VideoPoet
incorporates a mixture of multimodal generative objectives within an
autoregressive Transformer framework. The pretrained LLM serves as a foundation
that can be adapted for a range of video generation tasks. We present empirical
results demonstrating the model's state-of-the-art capabilities in zero-shot
video generation, specifically highlighting VideoPoet's ability to generate
high-fidelity motions. Project page: http://sites.research.google/videopoet/