Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
Authors: name: Shengbang Tong; name: Ellis Brown; name: Penghao Wu; name: Sanghyun Woo; name: Manoj Middepogu; name: Sai Charitha Akula; name: Jihan Yang; name: Shusheng Yang; name: Adithya Iyer; name: Xichen Pan; name: Austin Wang; name: Rob Fergus; name: Yann LeCun; name: Saining Xie
Published: 2024-06-24 17:59:42

Abstract: We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a
vision-centric approach. While stronger language models can enhance multimodal
capabilities, the design choices for vision components are often insufficiently
explored and disconnected from visual representation learning research. This
gap hinders accurate sensory grounding in real-world scenarios. Our study uses
LLMs and visual instruction tuning as an interface to evaluate various visual
representations, offering new insights into different models and architectures
-- self-supervised, strongly supervised, or combinations thereof -- based on
experiments with over 20 vision encoders. We critically examine existing MLLM
benchmarks, addressing the difficulties involved in consolidating and
interpreting results from various tasks, and introduce a new vision-centric
benchmark, CV-Bench. To further improve visual grounding, we propose the
Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector that
integrates high-resolution vision features with LLMs while reducing the number
of tokens. Additionally, we discuss the curation of high-quality visual
instruction-tuning data from publicly available sources, emphasizing the
importance of data source balancing and distribution ratio. Collectively,
Cambrian-1 not only achieves state-of-the-art performance but also serves as a
comprehensive, open cookbook for instruction-tuned MLLMs. We provide model
weights, code, supporting tools, datasets, and detailed instruction-tuning and
evaluation recipes. We hope our release will inspire and accelerate
advancements in multimodal systems and visual representation learning.