Molmo and PixMo: Open Weights and Open Data for State-of-the-Art
  Multimodal Models
Authors: name: Matt Deitke; name: Christopher Clark; name: Sangho Lee; name: Rohun Tripathi; name: Yue Yang; name: Jae Sung Park; name: Mohammadreza Salehi; name: Niklas Muennighoff; name: Kyle Lo; name: Luca Soldaini; name: Jiasen Lu; name: Taira Anderson; name: Erin Bransom; name: Kiana Ehsani; name: Huong Ngo; name: YenSung Chen; name: Ajay Patel; name: Mark Yatskar; name: Chris Callison-Burch; name: Andrew Head; name: Rose Hendrix; name: Favyen Bastani; name: Eli VanderBilt; name: Nathan Lambert; name: Yvonne Chou; name: Arnavi Chheda; name: Jenna Sparks; name: Sam Skjonsberg; name: Michael Schmitz; name: Aaron Sarnat; name: Byron Bischoff; name: Pete Walsh; name: Chris Newell; name: Piper Wolters; name: Tanmay Gupta; name: Kuo-Hao Zeng; name: Jon Borchardt; name: Dirk Groeneveld; name: Jen Dumas; name: Crystal Nam; name: Sophie Lebrecht; name: Caitlin Wittlif; name: Carissa Schoenick; name: Oscar Michel; name: Ranjay Krishna; name: Luca Weihs; name: Noah A. Smith; name: Hannaneh Hajishirzi; name: Ross Girshick; name: Ali Farhadi; name: Aniruddha Kembhavi
Published: 2024-09-25 17:59:51

Abstract: Today's most advanced multimodal models remain proprietary. The strongest
open-weight models rely heavily on synthetic data from proprietary VLMs to
achieve good performance, effectively distilling these closed models into open
ones. As a result, the community is still missing foundational knowledge about
how to build performant VLMs from scratch. We present Molmo, a new family of
VLMs that are state-of-the-art in their class of openness. Our key innovation
is a novel, highly detailed image caption dataset collected entirely from human
annotators using speech-based descriptions. To enable a wide array of user
interactions, we also introduce a diverse dataset mixture for fine-tuning that
includes in-the-wild Q&A and innovative 2D pointing data. The success of our
approach relies on careful choices for the model architecture details, a
well-tuned training pipeline, and, most critically, the quality of our newly
collected datasets, all of which will be released. The best-in-class 72B model
within the Molmo family not only outperforms others in the class of open weight
and data models but also compares favorably against proprietary systems like
GPT-4o, Claude 3.5, and Gemini 1.5 on both academic benchmarks and human
evaluation.
  We will be releasing all of our model weights, captioning and fine-tuning
data, and source code in the near future. Select model weights, inference code,
and demo are available at https://molmo.allenai.org.