MIO: A Foundation Model on Multimodal Tokens
Authors: name: Zekun Wang; name: King Zhu; name: Chunpu Xu; name: Wangchunshu Zhou; name: Jiaheng Liu; name: Yibo Zhang; name: Jiashuo Wang; name: Ning Shi; name: Siyu Li; name: Yizhi Li; name: Haoran Que; name: Zhaoxiang Zhang; name: Yuanxing Zhang; name: Ge Zhang; name: Ke Xu; name: Jie Fu; name: Wenhao Huang
Published: 2024-09-26 09:57:16

Abstract: In this paper, we introduce MIO, a novel foundation model built on multimodal
tokens, capable of understanding and generating speech, text, images, and
videos in an end-to-end, autoregressive manner. While the emergence of large
language models (LLMs) and multimodal large language models (MM-LLMs) propels
advancements in artificial general intelligence through their versatile
capabilities, they still lack true any-to-any understanding and generation.
Recently, the release of GPT-4o has showcased the remarkable potential of
any-to-any LLMs for complex real-world tasks, enabling omnidirectional input
and output across images, speech, and text. However, it is closed-source and
does not support the generation of multimodal interleaved sequences. To address
this gap, we present MIO, which is trained on a mixture of discrete tokens
across four modalities using causal multimodal modeling. MIO undergoes a
four-stage training process: (1) alignment pre-training, (2) interleaved
pre-training, (3) speech-enhanced pre-training, and (4) comprehensive
supervised fine-tuning on diverse textual, visual, and speech tasks. Our
experimental results indicate that MIO exhibits competitive, and in some cases
superior, performance compared to previous dual-modal baselines, any-to-any
model baselines, and even modality-specific baselines. Moreover, MIO
demonstrates advanced capabilities inherent to its any-to-any feature, such as
interleaved video-text generation, chain-of-visual-thought reasoning, visual
guideline generation, instructional image editing, etc.