Emu3: Next-Token Prediction is All You Need
Authors: name: Xinlong Wang; name: Xiaosong Zhang; name: Zhengxiong Luo; name: Quan Sun; name: Yufeng Cui; name: Jinsheng Wang; name: Fan Zhang; name: Yueze Wang; name: Zhen Li; name: Qiying Yu; name: Yingli Zhao; name: Yulong Ao; name: Xuebin Min; name: Tao Li; name: Boya Wu; name: Bo Zhao; name: Bowen Zhang; name: Liangdong Wang; name: Guang Liu; name: Zheqi He; name: Xi Yang; name: Jingjing Liu; name: Yonghua Lin; name: Tiejun Huang; name: Zhongyuan Wang
Published: 2024-09-27 16:06:11

Abstract: While next-token prediction is considered a promising path towards artificial
general intelligence, it has struggled to excel in multimodal tasks, which are
still dominated by diffusion models (e.g., Stable Diffusion) and compositional
approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a
new suite of state-of-the-art multimodal models trained solely with next-token
prediction. By tokenizing images, text, and videos into a discrete space, we
train a single transformer from scratch on a mixture of multimodal sequences.
Emu3 outperforms several well-established task-specific models in both
generation and perception tasks, surpassing flagship models such as SDXL and
LLaVA-1.6, while eliminating the need for diffusion or compositional
architectures. Emu3 is also capable of generating high-fidelity video via
predicting the next token in a video sequence. We simplify complex multimodal
model designs by converging on a singular focus: tokens, unlocking great
potential for scaling both during training and inference. Our results
demonstrate that next-token prediction is a promising path towards building
general multimodal intelligence beyond language. We open-source key techniques
and models to support further research in this direction.