Multimodal Latent Language Modeling with Next-Token Diffusion
Authors: name: Yutao Sun; name: Hangbo Bao; name: Wenhui Wang; name: Zhiliang Peng; name: Li Dong; name: Shaohan Huang; name: Jianyong Wang; name: Furu Wei
Published: 2024-12-11 18:57:32

Abstract: Multimodal generative models require a unified approach to handle both
discrete data (e.g., text and code) and continuous data (e.g., image, audio,
video). In this work, we propose Latent Language Modeling (LatentLM), which
seamlessly integrates continuous and discrete data using causal Transformers.
Specifically, we employ a variational autoencoder (VAE) to represent continuous
data as latent vectors and introduce next-token diffusion for autoregressive
generation of these vectors. Additionally, we develop sigma-VAE to address
the challenges of variance collapse, which is crucial for autoregressive
modeling. Extensive experiments demonstrate the effectiveness of LatentLM
across various modalities. In image generation, LatentLM surpasses Diffusion
Transformers in both performance and scalability. When integrated into
multimodal large language models, LatentLM provides a general-purpose interface
that unifies multimodal generation and understanding. Experimental results show
that LatentLM achieves favorable performance compared to Transfusion and vector
quantized models in the setting of scaling up training tokens. In
text-to-speech synthesis, LatentLM outperforms the state-of-the-art VALL-E 2
model in speaker similarity and robustness, while requiring 10x fewer decoding
steps. The results establish LatentLM as a highly effective and scalable
approach to advance large multimodal models.