MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal
  Dataset with One Trillion Tokens
Authors: name: Anas Awadalla; name: Le Xue; name: Oscar Lo; name: Manli Shu; name: Hannah Lee; name: Etash Kumar Guha; name: Matt Jordan; name: Sheng Shen; name: Mohamed Awadalla; name: Silvio Savarese; name: Caiming Xiong; name: Ran Xu; name: Yejin Choi; name: Ludwig Schmidt
Published: 2024-06-17 07:21:36

Abstract: Multimodal interleaved datasets featuring free-form interleaved sequences of
images and text are crucial for training frontier large multimodal models
(LMMs). Despite the rapid progression of open-source LMMs, there remains a
pronounced scarcity of large-scale, diverse open-source multimodal interleaved
datasets. In response, we introduce MINT-1T, the most extensive and diverse
open-source Multimodal INTerleaved dataset to date. MINT-1T comprises one
trillion text tokens and three billion images, a 10x scale-up from existing
open-source datasets. Additionally, we include previously untapped sources such
as PDFs and ArXiv papers. As scaling multimodal interleaved datasets requires
substantial engineering effort, sharing the data curation process and releasing
the dataset greatly benefits the community. Our experiments show that LMMs
trained on MINT-1T rival the performance of models trained on the previous
leading dataset, OBELICS. Our data and code will be released at
https://github.com/mlfoundations/MINT-1T.