Molmo and PixMo: Open Weights and Open Data for State-of-the-Art
  Multimodal Models
Authors: name: Matt Deitke, name: Christopher Clark, name: Sangho Lee, name: Rohun Tripathi, name: Yue Yang, name: Jae Sung Park, name: Mohammadreza Salehi, name: Niklas Muennighoff, name: Kyle Lo, name: Luca Soldaini, name: Jiasen Lu, name: Taira Anderson, name: Erin Bransom, name: Kiana Ehsani, name: Huong Ngo, name: YenSung Chen, name: Ajay Patel, name: Mark Yatskar, name: Chris Callison-Burch, name: Andrew Head, name: Rose Hendrix, name: Favyen Bastani, name: Eli VanderBilt, name: Nathan Lambert, name: Yvonne Chou, name: Arnavi Chheda, name: Jenna Sparks, name: Sam Skjonsberg, name: Michael Schmitz, name: Aaron Sarnat, name: Byron Bischoff, name: Pete Walsh, name: Chris Newell, name: Piper Wolters, name: Tanmay Gupta, name: Kuo-Hao Zeng, name: Jon Borchardt, name: Dirk Groeneveld, name: Jen Dumas, name: Crystal Nam, name: Sophie Lebrecht, name: Caitlin Wittlif, name: Carissa Schoenick, name: Oscar Michel, name: Ranjay Krishna, name: Luca Weihs, name: Noah A. Smith, name: Hannaneh Hajishirzi, name: Ross Girshick, name: Ali Farhadi, name: Aniruddha Kembhavi

Today's most advanced multimodal models remain proprietary. The strongest
open-weight models rely heavily on synthetic data from proprietary VLMs to
achieve good performance, effectively distilling these closed models into open
ones. As a result, the community is still missing foundational knowledge about
how to build performant VLMs from scratch. We present Molmo, a new family of
VLMs that are state-of-the-art in their class of openness. Our key innovation
is a novel, highly detailed image caption dataset collected entirely from human
annotators using speech-based descriptions. To enable a wide array of user
interactions, we also introduce a diverse dataset mixture for fine-tuning that
includes in-the-wild Q&A and innovative 2D pointing data. The success of our
approach relies on careful choices for the model architecture details, a
well-tuned training pipeline, and, most critically, the quality of our newly
collected datasets, all of which will be released. The best-in-class 72B model
within the Molmo family not only outperforms others in the class of open weight
and data models but also compares favorably against proprietary systems like
GPT-4o, Claude 3.5, and Gemini 1.5 on both academic benchmarks and human
evaluation.
  We will be releasing all of our model weights, captioning and fine-tuning
data, and source code in the near future. Select model weights, inference code,
and demo are available at https://molmo.allenai.org.