Beyond Language Modeling: An Exploration of Multimodal Pretraining
Authors: name: Shengbang Tong, name: David Fan, name: John Nguyen, name: Ellis Brown, name: Gaoyue Zhou, name: Shengyi Qian, name: Boyang Zheng, name: Théophane Vallaeys, name: Junlin Han, name: Rob Fergus, name: Naila Murray, name: Marjan Ghazvininejad, name: Mike Lewis, name: Nicolas Ballas, name: Amir Bar, name: Michael Rabbat, name: Jakob Verbeek, name: Luke Zettlemoyer, name: Koustuv Sinha, name: Yann LeCun, name: Saining Xie
Organization: AI at Meta

The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.