xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
Authors: name: Le Xue; name: Manli Shu; name: Anas Awadalla; name: Jun Wang; name: An Yan; name: Senthil Purushwalkam; name: Honglu Zhou; name: Viraj Prabhu; name: Yutong Dai; name: Michael S Ryoo; name: Shrikant Kendre; name: Jieyu Zhang; name: Can Qin; name: Shu Zhang; name: Chia-Chih Chen; name: Ning Yu; name: Juntao Tan; name: Tulika Manoj Awalgaonkar; name: Shelby Heinecke; name: Huan Wang; name: Yejin Choi; name: Ludwig Schmidt; name: Zeyuan Chen; name: Silvio Savarese; name: Juan Carlos Niebles; name: Caiming Xiong; name: Ran Xu
Published: 2024-08-16 17:57:01

Abstract: This report introduces xGen-MM (also known as BLIP-3), a framework for
developing Large Multimodal Models (LMMs). The framework comprises meticulously
curated datasets, a training recipe, model architectures, and a resulting suite
of LMMs. xGen-MM, short for xGen-MultiModal, expands the Salesforce xGen
initiative on foundation AI models. Our models undergo rigorous evaluation
across a range of tasks, including both single and multi-image benchmarks. Our
pre-trained base model exhibits strong in-context learning capabilities and the
instruction-tuned model demonstrates competitive performance among open-source
LMMs with similar model sizes. In addition, we introduce a safety-tuned model
with DPO, aiming to mitigate harmful behaviors such as hallucinations and
improve safety. We open-source our models, curated large-scale datasets, and
our fine-tuning codebase to facilitate further advancements in LMM research.
Associated resources will be available on our project page above.