xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
Authors: name: Le Xue, name: Manli Shu, name: Anas Awadalla, name: Jun Wang, name: An Yan, name: Senthil Purushwalkam, name: Honglu Zhou, name: Viraj Prabhu, name: Yutong Dai, name: Michael S Ryoo, name: Shrikant Kendre, name: Jieyu Zhang, name: Can Qin, name: Shu Zhang, name: Chia-Chih Chen, name: Ning Yu, name: Juntao Tan, name: Tulika Manoj Awalgaonkar, name: Shelby Heinecke, name: Huan Wang, name: Yejin Choi, name: Ludwig Schmidt, name: Zeyuan Chen, name: Silvio Savarese, name: Juan Carlos Niebles, name: Caiming Xiong, name: Ran Xu

This report introduces xGen-MM (also known as BLIP-3), a framework for
developing Large Multimodal Models (LMMs). The framework comprises meticulously
curated datasets, a training recipe, model architectures, and a resulting suite
of LMMs. xGen-MM, short for xGen-MultiModal, expands the Salesforce xGen
initiative on foundation AI models. Our models undergo rigorous evaluation
across a range of tasks, including both single and multi-image benchmarks. Our
pre-trained base model exhibits strong in-context learning capabilities and the
instruction-tuned model demonstrates competitive performance among open-source
LMMs with similar model sizes. In addition, we introduce a safety-tuned model
with DPO, aiming to mitigate harmful behaviors such as hallucinations and
improve safety. We open-source our models, curated large-scale datasets, and
our fine-tuning codebase to facilitate further advancements in LMM research.
Associated resources will be available on our project page above.