Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
Authors: Shengbang Tong
Ellis L Brown II
Penghao Wu
Sanghyun Woo
ADITHYA JAIRAM IYER
Sai Charitha Akula
Shusheng Yang
Jihan Yang
Manoj Middepogu
Ziteng Wang
Xichen Pan
Rob Fergus
Yann LeCun
Saining Xie
Venue: NeurIPS 2024 oral (2024)
Keywords: Multimodal LLM
Visual Representation Learning
Evaluation Protocol
Data Mix
Open Science
Primary area: machine_vision
TL;DR: Cambrian-1 is a vision-centric study of MLLM design—spanning visual representation choice, connector design, instruction tuning, and benchmarking.

Abstract: We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. While stronger language models can enhance multimodal capabilities, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning research. This gap hinders accurate sensory grounding in real-world scenarios. Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations, offering new insights into different models and architectures—self-supervised, strongly supervised, or combinations thereof—based on experiments with over 15 vision models. We critically examine existing MLLM benchmarks, addressing the difficulties involved in consolidating and interpreting results from various tasks. To further improve visual grounding, we propose spatial vision aggregator (SVA), a dynamic and spatially-aware connector that integrates vision features with LLMs while reducing the number of tokens. Additionally, we discuss the curation of high-quality visual instruction-tuning data from publicly available sources, emphasizing the importance of distribution balancing. Collectively, Cambrian-1 not only achieves state-of-the-art performances but also serves as a comprehensive, open cookbook for instruction-tuned MLLMs. We provide model weights, code, supporting tools, datasets, and detailed instruction-tuning and evaluation recipes. We hope our release will inspire and accelerate advancements in multimodal systems and visual representation learning.

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
Shengbang Tong'
Ellis Brown"
Penghao Wu*
Sanghyun Woo Manoj Middepogu
Sai Charitha Akula
Jihan Yang Austin Wang
Shusheng Yang
Adithya Iyer
Xichen Pan
Rob Fergus
Yann LeCun Saining Xiel
New York University
Abstract
We introduce Cambrian-l, family of multimodal LLMs (MLLMs) designed with a vision-centric approach. While stronger language models can enhance multimodal capabilities, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning research. This gap hinders accurate sensory grounding in real-world scenarios. Our study uses I.LMs and visual instruction tuning as an interface evaluate various visual representations, offering new insights into different models and architectures- self- supervised, strongly supervised, or combinations thereof-baseed on experiments with over 20 vision encoders. We critically examine existing MILM benchmarks, addressing the difficulties involved in consolidating and interpreting results from various tasks, and introduce a new vision-centric benchmark, CV-Bench. To further improve visual grounding, we propose the Spatial Vision Aggregator (SVA), connector that integrates high-resolution vision
dynamic and spatially- features with LLMs while reducing the number of tokens. Additionally, we discuss the curation of high-qualie visual instruction-tuning data from publicly available sources, emphasizing the importance of data source balancing and distribution ratio. Collectively, Cambrian-1 not only achieves state-of-the-art performance but also serves as a comprehensive, open cookbook for instruction-tuie provide model weights, code, supporting tools, datasets, and detailed instruction- tuning and evaluation recipes. We hope our release will inspire and accelerate advancements in multimodal systems and visual representation learning. Project page:
MI.LMs. We
an 1m. github..t.... io/
Introduction
There is long-standing debate in philosophy about whether understanding and meaning in language require sensory grounding, Aristotle's s emphasis on acquiring knowledge through sensory experience and empirical observation was central his ancient Peripatetic school and remains influential to this day [8]: Aquinas famously formalized these ideas in the 13th century with the Peripatetic axiom: "Nihil est in intellectu quod non sit prius in sensu" (Nothing
in the intellect that was not first in the senses) [7]. Though many philosophers disagree [23], it is evident that having robust and highly least beneficial. Consider the Cambrian explosion, during which the emergence of vision is believed [105] to have been crucial for early animals to not only find food and avoid predators but also to evolve and improve. In fact, most human knowledge (and nearly
capable sensory grounding
Project Lead Corresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Visual Representation Learning Evaluation Protocols
ImageNet-Ik
Linear Probing
Vision Model
or End-to-End Tuning
COCO
ADFZUK
SDG
DE7
Multimodal Large Language Models
Pretrined
Connector
shifl CherQA SDAVP MRD
Vision Model
Visual Instruction Tuning w/ LLMs
NIVE Mathibat SQA
VQA
dlo
Visual Representations
Connector Design
Instruction Tuning Data
Instruction Tuning
Evaluation Protocol
Recipe
We draw parallels between traditional protocols and use of MLLLMs for evaluating visual represen- tations. ML ML employ visual question answering to address a diverse array of real-world perception tasks. The bottom section highlights the five key pillars studied in Cambrian-1.
Figure I:
all animal knowledge) is acquired through sensory experiences like sight, hearing, touch, taste, and smell, through interactions with the physical world [107]. These sensory experiences are fundamental to understanding the world around us and are crucial for real-world actions and decision-making.
Beyond philosophical dehates, recent advances in multimodal large language models (MLLMs) have brought the topic of visual representation learning vs. language understanding into practical focus. Language models have shown strong scaling behaviors [55], and recent advancements in multimodal learning are largely driven by the development better, larger LLMs [81]. On the other hand, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning research. For instance, many pioneering frameworks such as LLaVA [82] use vision transformer- based CLIP models [109, 145]. which are strongly supervised by language*, as the vision feature extractor. While other visual representations, such as self-supervised DINO [103], being explored [126], there is lack of comprehensive and systematic study in this domain. This gap exists primarily because such studies are challenging: MILMs involve a complex training and evaluation pipeline with numerous design decisions consider. In this work, aim to bridge the gap by exploring MLLMs from a vision-centric perspective. More specifically, we use MILM instruction tuning an evaluation protocol for various visual representations (illustrated in Fig. 1).
Our motivation for this study also stems from two potential concerns of the current multimodal learning research: 1) relying too heavily too early on language act shorteut [47, 144], com- pensating for the deficiencies in learning effective visual representations, and 2) existing benchmarks may not provide adequate guidance for real-world weenarios-when visual grounding is crucial for robust multimodal understanding. These concerns are not unfounded, as researchers have started to notice that visual grounding is becoming bottleneck for applying MLLMs in some challenging real-world applications, despite significant progress in improving general capabilities [41, 126, 136].
From another perspective, traditional evaluation protocols for visual representation learning (e.g.. linear probing and end-to-end fine-tuning on datasets like ImageNet- 1K [113], COCO [79], and ADE20K [154]) are becoming saturated and do not reflect the diverse perception challenges found in real-world distributions. On the other hand, using language in the form of visual question answering (VQA) offers flexible and robust evaluation protocol. Our study aims to explore this new protocol design, setting it up to gain insights that will guide the development of better visual representations in the future. Furthermore, to better evaluate visual representations in this integrated setting, we develop a vision-centric MILM benchmark, CV-Bench, by transforming traditional vision benchmarks into VQA format (Section 2.2).
Cambrian-1 is structured around five key pillars, each offering important insights into the design space of MLLMs:
Visual Representations: We explore various vision encoders and their combinations. $2.4
We emphasize that CLIP training should be considered as strongly supervised, as language provides significantly richer supervision than class labels.

---

buy
Superied
Mi Oaikedta
MiLL Hankin
Depi Reprrted Iaaee1
Segineaatio Sgeeited 5OM
Figure 2: Examples of various vision models, objectives, and architectures studied. Image from [48].
Connector Design: We design a new dynamic and spatially- connector that integrates vision features with LLMs while reducing the number of tokens. $3
Instruction Tuning Data: We curate high-quality visual instruction-tuniin data from public
sources, emphasizing the importance of distribution balancing. $4
Instruction Tuning Recipes: We discuss instruction tuning strategies and practices. $2.3 Benchmarking: We analyze existing MILM benchmarks, cluster them into 4 intuitive groups, and introduce new vision-centric benchmark "CV-Bench". $2.1, $2.2
We defer detailed review of the fundamental components and methodologies that underpin MLLM research to Appendix B
Evaluating Visual Representations through MLLMs
Current MLLMs predominantly rely CLIP [109] the visual encoder due to its pre-alignment with language and case adaptation to the LLM token space. However, strong language priors can be double-edged sword-they compensate for deficiencies in learning effective visual represen- tations [126] and diminish insights gained from extensive visual representation learning research. In this section, systematically evaluate how various visual encoder choices (see Fig. 2) impact the multimodal capabilities of MLLMs. We also advocate for using MLLM evaluation as a robust framework for assessing visual representation methods, moving beyond traditional protocols like linear probing and end-to-end fine-tuning to more faithfully reflect the diverse perception challenges in real-world scenarios and better guide the development of improved visual representations.
2.1 Analyzing the Benchmarks
To effectively evaluate visual representations and MLLMs, we first need select benchmarks that accurately assess the multimodal capabilities of these models. We use a suite of commonly used benchmarks [24, 45, 54, 57, 83, 84, 91, 92, 96, 97, 120, 126, 137, 143]. which is the intersection of those used in recent MILLM research [75, 77, 137]. To help interpret our results, we begin by analyzing the benchmarks themselves. Here, we train MI.LMs with 23 different vision backbones (see Table 6) from variety of model families (see Fig. 2) using 2-stage instruction tuning process initially proposed in [82]: first training connector on 1.2M adapter data from ShareGPT-4V [27] followed by fine-tuning both the connector and LLM on 737K instruction tuning data (see more details in Appendices G.5 and H). Full benchmark results Table 9.
Who's answering the question: the LLM or MLLM? Determining whether benchmark truly needs visual input to be solved has been a persistent challenge in vision-language research [2, 26, 50, 94]. In this study, we compare the performance of MI.LMs with and without visual input and also calculate the expected score via randomly guessing. These three conditions are visualized in Fig. 3-lefit, with benchmarks sorted by the difference between the average score with vision enabled and disabled. SQA-I', MMMU, Math Vista, and A12D display less than a 5% gap between vision enabled and disabled, suggesting that these benchmarks may not significantly depend on visual input and rather heavily rely on the base LLM. TexiVQA and GQA both demonstrate a nearly 40% positive gap between random guessing and dision-disable benchmarks. On the other hand, the vision- -isoondd performance on benchmarks like MMVP is notably worse than random guessing, suggesting that strong visual grounding is particularly crucial.
scores, implying strong language bias these
Clustering the Benchmarks To better understand the different aspects of MLLM performance, we analyze the correlations between the performance of our 23 MLLMS on each benchmark. A
'ee note that our instruction-tuniigg data includes text-only data, so text-only questions are not OOD.
'The subset of SQA [91] with images.
3

---

DwYYQN
TevRIDA
Recduc Cuess
Drabled Diebled
Diubted Sandon
Gouded
Voku Daakel
02-
ChariiA
Real Porld
MMN
Scomo
SERDI
300v2
Verrepo
-02
Xathrata
CVBaxds
spad
Madithes
Reviels
chargi
actared
Deevigie
Gga
MAR
Red Worl
-08
04
-01
02
03
64
Figure 3: Left: Performance comparison of MLLMs with visual input enabled and disabled across various benchmarks. Benchmarks are sorted by the difference between the average score with vision enabled and disabled. Right: Principal component analysis displaying clusters of benchmarks based performance metrics, with bubble size corresponding to benchmark size. We label the clusters "General" in green, "Knowledge" in yellow, "Chart & OCR" in red, and "Vision-Centric" in blue.
confusion matrix (Fig. 10) reveals that certain benchmarks, such as MMMU, are largely uncorrelated with the others. We perform principal component analysis the benchmark scores and observe the formation of clusters corresponding to "General," "Knowledge," "Chart & OCR," " and "Vision-Centric" categories (Fig. 3-right). We assign MMMU to the knowledge category based on the types of questions it includes (see Appendix D). We also find that existing vision-centric benchmarks [126, 137] are of insufficient size (see Fig. 3-right), challenging the robustness of evaluating such capabilities. These benchmarks do not cover crucial visual elements such as depth and spatial awareness.
Finding 1: Most benchmarks do not properly measure vision-centric capabilities, and the ones that do have very few samples.
2.2 Cambrian Vision-Centric Benchmark (CV-Bench)
To address the limitations of existing vision-centric benchmarks, we introduce the Cambrian Vision- Centric Benchmark (CV-Bench). With 2638 manually-inspectee examples, CV-Bench provides significantly more examples than other vision-centric MI.LM benchmarks- -.55x more than Real- WorldQA [137] and 8.8x more than MMVP [126]. By repurposing standard vision benchmarks [18, 79, 154]', we can assess models at classic vision tasks within multimodal context. Leveraging the rich ground truth annotations from the benchmarks, we formulate natural language questions that probe the fundamental 2D and 3D understanding of the models.
As visualized in Fig. 11, CV-Bench evaluates 2D understanding via spatial relationships & object counting, and 3D understanding via depth order & relative distance. We refer details Appendix
Existing vision benchmarks can be effectively repurposed into VQA questions, enabling the assessment of vision-centric MLLM capabilities.
Finding 2:
2.3 Instruction Tuning Recipes
MLLMs start with pre-trained LLM and vision backbones, connecting these modules with a connector such projector (MLP). The original LLaVA [80, 82] proposes 2-stage frozen training process: first, pre-training a connector between frozen L.M and vision backbones using adapter data, and then fine-tuning both the connector and LLM with instruction tuning data while leaving the vision encoder frozen. Various studies [27, 63, 81, 98] have drawn different conclusions regarding the optimal training methodology for MLLMs. Here, we revisit this topic with extensive experiments.
For our experiments, we tune a set of MILMs using Vicuna-1.55 1.5-7B as the LLM backbone and each of our 23 vision models (Table 6) as the visual encoder. We use 737K instruction tuning data mix for all experiments here (see Appendix H). All hyperparameters are matched across each experimental setting- highhighingiiiinnnngg the impact of different tuning strategies with each visual encoder. All experimental settings and results are tabulated in Appendix F.2.
Ommi3D assets sourced from [3. 13, 20, 46, 111, 121].

---

Language Supervised
Supervised
DM
Gmeral
0.5M 0-
+
12M
1.2M V
90
50
35
Knowledge
0.5M
12M
12M
46
44.5
450
45.5
66.0
46.5
44.0
44.5
45.0
45.5
46.0
0.5
OCR &
1.2M
1.2M d
18.0
185
18.0
18.5
Visten Centric
0M
0.5M
1.2M
1.2M
50
36
Figure 4: Effect of Training Recipe on Model Performance. Boxplots display the distribution of benchmark scores across benchmark categories for different training recipes and types of visual encoders. The four training recipes include freezing the visual encoder with various amounts of adapter data (0M 0.5M @ 1.2M as well unfreczing with I. 1.2M & adapter data. Amount of Adapter Data: model types show increased performance on general and seionccentric benchmarks; knowledge benchmarks show mixed results; OCR chart benchmarks benefit from more data for sanguage-superiig models. Unfreezing: Unfreezing the visual encoder with 1.2M adapter data generally benefits all categories.
One Stage vs Two Stage Training Recent work [63] advocates for skipping connector pre-training, claiming this "reduces compute cost without harming downstream performance." To explore whether this claim holds-especialyy when using aoo-langgaag--uuperviee experiments using 0, 0.5M, and 1.2M adapter data. Following LLaVA's recipe [82], tune only the connector on the adapter data during this first phase, before unfreezing the LLM and connector during instruction tuning on the 737K mix. Fig. 4 shows that pre-training the connector first enhances model performance and that more adapter data further improves performance across all domains. Thus, we subsequently adopt stst training with 1.2M adapter data as our standard setup.
visual encoders- we conduct
Finding 3: Two-stage training is beneficial; more adapter data further improves results.
Freeze Unfreeze Vision Encoder There are also mixed practices in freezing [63, 80, 82] or unfreezing [44, 81] vision backbones during fine-tuning. Some argue that unfreezing the vision backbone significantly degrades performance [63]. Our experiments demonstrate that unfreezing benefits performance across all benchmarks except for marginal change in knowledge benchmarks (Fig, 4). We suspect this due to the composition of the 737K instruction tuning data and the LLM- heavy focus of these benchmarks (see Section 2. 1). We note that unfreezing the vision backbone introduces additional computational overhead, which prohibits testing on some larger vision models under current sharding strategies (see more details in Appendix H).
Unfreezing the vision encoder is widely beneficial. Language-superrise models always benefit; SSL models particularly benefit on vision-centric benchmarks.
Finding 4:
2.4 MILMss Visual Representation Evaluator
As discussed in earlier sections, MILMs provide new interface to explore aspects vision models beyond traditional benchmarks like ImageNet- 1k linear probing. We study the 2-stage instruction tuning setting using 1.2M adapter data, 737K fine-tuning data, and frozen visual encoders allow comparison of the widest range of models.
5

---

CpenttiP Canchalt-La112
Average Scooe
OpeneTiP Caechedi www.helitattee Laider
CyeaeTJP Cerciel XXL-0044
Sell SellGieeiiie
60-
2DND-2V/17:/46600
DINDSEVI
Mario a Aiteeto
Matio Lilkeet
30
IJBPA VEAU/N
MAB WTT/
Coler
SAM VIT RVSGENN
20
SIDUSA VIT L/T6sta
5223929112
Pxplit WILL00024
Gareral
Knowledge
OCR& Chart
Vision-Centric
Figure 5: Evaluating Visual Representations with MLLMs While mngguage-superiiseee models outperform self supervisdd other models, well-trained supervisd model like DINOv2 can also achieve competitive performance on asion-centric tasks.
General
Knowledge
OCR & Chart
Vision-Centie
50.0
55.0
54.0
Avenge Performence
72.0
50.0
70.0
69.0
45.0
52.0
68.0
40.0
50.0
66.0
48.0
35.0
64.0
30.0
48.0
62.0
47.0
60,0
20.0
46.0
58,0
0.7M
0.7MM
5M
5M
0.7M
0.7MA
5M @
SMA
0.7M @ 0.7M&
5M
5MA
0.7M 05 0.7MM
5M @ SM
OpenAI CLIP V-T-L/144336
DINOv2 VIT-L/14H336
Figure 6: Continued Fine-Tuning Narrows the Gap Between CLIP and DINOv2. Performance is compared with 0.7M and 5M instruction tuning data in both frozen (8) and unfrozen (+) settings. DINOv2 shows significant performance improvement with increased data and unfreczing surpassing the 0.7M @ CLIP model several benchmarks and narrowing the gap to the 5M model knowledge and visian-centric tasks.
We evaluate on benchmarks detailed Section 2.1, calculating the average performance for each category and visualize the results in Fig. 5 (full results in Appendix F). Our findings highlight the advantages of aanguage-superiise models over non-CLIP models across all benchmark categories, with significantly better performance chart and OCR-related benchmarks. We hypothesize that this is due to CLIP's training data, such as LAION [115], containing abundant OCR and text-heavy data, whereas SSL and other vision models primarily train on natural images with significantly less text content. It is also noteworthy that innuage--superiise
models are typically trained with very large pool of data, ranging from 400 million [109] to 10 billion [28] samples, whereas the largest vision self-supervised training dataset, like DINOv2, consists of only 142 million samples [103].
Additionally, observe that higher-resolution models particularly enhance performance chart and vision-centric benchmarks while remaining neutral on general VQA and knowledge-basee VQAs. While the majority of the backbones examine ViT-based [39], Conv-Net-based architectures (such as OpenCLIP ConvNeXt [86]) are inherently well-suited for high-resolution image processing [130] and can produce superior results on OCR & Chart and Vision-Centric benchmarks. In vision-centric benchmarks, the gap between language-supervised and other types of vision models is smaller, with well-trained self-supervised DINOv2 model even outperforming some supurveesupervised models.
High-res encoders greatly enhance performance on chart & vision-centric benchmarks, and ConyNet-based architectures are inherently well-suited for such tasks.
Finding 5:
Narrowing the gap between Language- and Self-Supervised models DINOv2 stands midway between SSL models and knowledge benchmarks, even outperforming some waaguage-superviiin models on vision-centric benchmarks. Here, study whether the continued finetuning of an MLLM based on SSL model
Above, we observe that models on general and
"Before averaging, divide the MME Perception score by 20 have the same scale other benchmarks.

---

model. Specifically, we scale up the instruction tuning data from 737K to 5M (see more details in Appendix G.5), and instruction tune MLLMs with DINOv2 ViT-L/144 @ 336 and OpenAl CLIP VTT-I/14@ 336 encoders in both frozen and unfrozen settings. In Fig. 6, we observe that by unfreezing the vision backbone, the DINOv2-based MILM fine-tuned with 5M data surpasses the MILM trained with CLIP model on 0.7M data. Additionally, the gap between DINOv2 and the CLIP models is reduced under the 5M setting.
can achieve performance similar to that of a
Finding 6: Language supervision offers strong advantages, but the performance gap can be narrowed with SSL methods given enough data and proper tuning.
2.5 Combining Multiple Vision Encoders
As observed in Fig. 5, different vision encoders excel in different aspects of MLLM performance. In this study, we explore the potential of combining multiple vision encoders to leverage their distinctive representations, aiming to build a more capable MI.LM. Given that different vision encoders use varying architectures and image resolutions, we interpolate to a fixed number of visual tokens (576) in this subsection (see details in Appendix F.3). We then concatenate these tokens along the feature dimension, following method similar to A-MoF proposed in [126].
Our study indicates that adding non-language-superve mark performance, especially in vision-centric tasks. Notably, even OCR benchmarks benefit from incorporating DINOv2. This highlights the importance of self-supervised learning models comple- menting aunguag--upervisng models to achieve robust multimodal understanding. Detailed results and configurations are available in Appendix F.3.
model (DINOv2) can improve bench-
However, this naive strategy has two limitations: 1) it employs interpolation, which can lead to information loss, especially with vision encoders with high-resolution feature maps, and it treats each model equally via simple concatenation. Therefore, we seek a more effective strategy that can more flexibly leverage model combinations with less information loss.
Combining multiple vision encoders, including SSL models, can enhance MILM performance across various benchmarks, particularly in vision-centric tasks.
Finding 7:
3 Spatial Vision Aggregator (SVA): A New Connector Design
To effectively aggregate features from multiple vision encoders and prevent the information loss introduced by interpolation, we use a set of learnable latent queries that interact with multiple vision features via cross-attention layers [37]. In particular, our approach incorporates two new vision-centric design principles:
1. We introduce spatial inductive bias by explicitly defining the aggregation space for each token in the query.
2. We aggregate vision features multiple times across the LLM layers, enabling the model to repeatedly access and integrate necessary visual information.
To facilitate information aggregation via cross-attention, we create C-dimension learnable latent token x E RCC that is repeated
times to form 2D grid, serving as the query X E R The set of visual features F from N vision encoders serve as the context (i.e., key and value). We ensure the output resolution of every vision encoder is multiple of L. Formally, the feature map of the k-th vision encoder (F&) has resolution of m x m L x C, where m, is positive integer multiplier, and L is the height/width of the learnable 2D grid with hidden dimension C.
Spatial inductive bias To maintain the spatial structure during cross- attenntt,,,,on, we align each token in the query with a specific sub-region of the feature maps in all vision encoders. Formally, a token at row i and column j in the query Xi,j corresponds to the sub-region
F i: my (i+ 1), mi m) J j:mk (j + 1)] Rmixc
of the k-th vision feature map. As a result, token xt, aggregates total of Ee mf features from N vision encoders through cross-attention (see Fig. 7-left).
Specifically, the updated query vector q'1 ij E RIxC position (i,j) is computed as
7

---

Spatial Vision Aggregator
LLM Transformer
Transformer Block k
Enc N
SVA
Enc
Transformer Block
Visual Takms
Erec 1
Latent Tokens
Text Tokens
Figure 7: Spatial Vision Aggregator (SVA). We propose SVA, a dynamic and spatially-aware connector that integrates multiple vision features with LLMs while reducing the number of tokens.
[kiga, 2.......... " [V..s.
1. sottmax
())
VC
where
9., x !x
k.,,A W i me (f 1), me-d me +++1) e V.J.A Fx ti:m 1), j ( + 1]] R=
Here, Qr is the query vector at position (i.j), calculated using the query projection matrix WQ RCxC The key vectors ji,, and value vectors Vi,j& are computed for each vision encoder h using their respective key and value projection matrices W K RCxC and W RCxC, Since Σn m features are aggregated into single token, we effectively reduce the number of tokens.
Multi-layer vision aggregation Although our proposal effectively aggregates features from multiple vision encoders, there is still potential information loss with high-resolution input (large my) or multiple vision encoders (large N). Here, a single token would have handle a larger amount of context information during aggregation. To prevent this, allow cross-attention occur multiple times by inserting our proposal throughout the LLM layers-allowing consistent access to the uncompressed visual information (see Fig. 7-right).
Hyperparameters To flexibly modulate capacity, we introduce two hyperparameters D and G, which indicate the number of cross-attention layers and distinct groups of learnable queries used between the vision models and the LLM, respectively. D and G are always set to for cross-attention layers within LLM layers. We provide ablation studies on the selection of D and G in Appendix H.
Connector
General
Knowledge
OCR & Chart
Vision-Centric
Concat. [126]
67.2
48.9
50.1
52.6
Resampler [58]
63.1
46.5
27.1
42.6
SVA-no-multi-agg
68.0
49.5
55.2
52.6
SVA
68.5
49.7
55.5
53.2
Table 1: Comparison between SVA and other aggregation approaches. The SVA module consistently
outperforms other baselines excels in aggregating high-resolution vision information.
We demonstrate the efficacy of SVA module using the best vision model combination results from the previous section and Vicuna-Le 1.5-7B base LLM. Specifically, we employ combination of four vision encoders: OpenAl CLIP ViT-L/14 336, SigLIP ViT-S0400M/14 @ 384, OpenCLIP ConvNeXt-XXLe 1024, and DINOv2 ViT-L/14 518. We compare our method with two strong baselines: 1) concatenation based [126] and 2) Re-sampler [11, 72]. Here, we include two variants of our SVA module. The standard one, "SVA", uses 3,
1, and inserts cross-atie coos-atttntionn blocks inside the LLM with layer stride of 3. To isolate the advantages of spatial inductive biases, we include another SVA variant, "SVA-no-multi-a" the L.LM and sets D = 3 and G 3. Table 1 shows that SVA outperforms both baselines, with a significant improvement in the OCR chart category. contrast, the Resampler- lacks spatial inductive biases strugless to condense concatenated tokens from various vision towers into
that does not add cross-attention blocks inside

---

RandandToo (14) (MOO
RoCOCO (20 (3
CLEVER (DSUR RJ
FJaned DVGA (LSSAO K3
TALLOA (1) 220 Kg
Connnny Code
DVQA (nl) (755
FRigA [-11 (1.0 ())
VinaaTe (199)0443))
Syakkigg (6 K)
IaSANA [16) RD
LARON GPE-V (10 (100
Filared Weshight (7000
OCR
AMARQA (Mg)
TATAQA (:50 (2.0
IDS [23] (1.00
Moblightt 110.0
DevignE
DOOANOA (159)80
Low-Land (2ao (00
Skexay (120112//18
Cambrian-7M
Due (7)05
Vozdere (12 (u0k)
KAVES (42))
Language
LVVE Iastas/V [171] (2
Glvertion [14] (Mi)
Chenss (S)
Jl 19K (63) (1500 KD
OperOrol (96L0
Matiiiss (1520
IoaeAA (S): (2T.
Vanaklemen (07] 1988
Naditarra [142] (2CL))
tace-crs (1.8)
ChastTen (5210655))
(2000
General
GON
DyewCoaknorppen (157)
Engae
DKVIG [1H ion
Dodly Esgge DXy
Pattigg (S7]
SE-VAN (11 KD
Allwaald (4500)
ADD (-1E550)
StautirT (Qea
Flband CLEVE 130.
Figure 8: Cambrian-7M: Large-Scale Curated Instruction Tuning Dataset for MLLM. Left: The inner
circle shows the original distribution of Cambrian- 10M. The outer circle shows the curated Cambrian-7M Right:
All the data sources in the Cambrian datasct well as the ones filtered in data curation.
Vidoo Centric
OCR& & Chart
Knowlegge
DCR&&Clar Charr
Vidont-Comi Centric
General
Knowlelge
Average
150k
53.7
68.0
51.3
45.2
50.5
250k
54.3
68.1
51.5
45.3
52.2
LLaVA
350k
54.3
67.4
51.4
46.0
52.3
Cambriun 10M
54.8
68.7
51.5
47.3
51.4
450k
54.2
68.0
52.2
45.5
50.7
Combriin-7M
55.9
69.6
52.6
47.3
54.1
Table 2: value between 250k and Figure 9: Exploring instruction tun- Table Performance improves with
350k obtains better performance. ing data mixture ratios.
better instruction tuning data curation.
a limited number of learnable queries via global cross-attention. We also compare SVA with other connectors Appendix H and show clear advantages.
Spatial inductive bias and deep interaction between LLM and vision feature help to better aggregate and condense vision features.
Finding 8:
4
Instruction Tuning Data for Training MLLMs
4.1 Data Collection
Collecting Instruction Tuning Data from existing data sources Unlike language data, multimodal (visual) Intructtionttunngg data is much rarer and harder to collect. To address this, we use existing multimodal benchmarks and datasets involving visual interaction data, such as Visual Question Answering (VQA) and OCR data. To help maintain conversational abilities [147], we also collect small volume of high-quality language-only instruction-follont General conversation, OCR, Counting, Code, Math, Science, and Language-only data. We list the data sources in Fig. 8, and the details of data preparation in Appendix G.
data. We categorize data into
Targeted Internet Data Collection Engine As observed in Fig. 8, there is an unbalanced distribution of data. Some categories, such science, have very few data sources, and each source has limited samples. Inspired by previous works [73], we introduce a data engine to create large-scale, reliable, high-quality knowledge-based instruction tuning data (see Fig. 15). Details are in Appendix G.3. Our data engine produces a large volume of reliable scientific data, increasing the diversity in the data pool. We generate 16lk science-related data points-4000 more than the previous combined data sources.
We create large pool of instruction tuning data, which we refer as Cambrian- 10M. This pool contains approximately 9784k data points, offering a diverse range of data for our work and future research. We visualize its composition in Fig.
Cambrian-10M.
4.2 Data Curation
Cambrian 10M is large pool of instruction tuning data sourced from a variety of data sources, with an unbalanced data ratio between categories. Here, we take a preliminary step to study data curation by improving data balancing and adjusting data ratios.

---

Model
General
Knowleige
OCC & Chor
Vison-Cooni Centii
CW-Beach 20
CW-Heechh
Tuk.
MMI
SEED
GAAA
MIMMEL
AvZ
Charg4
TosVV4
DovVON
Avg
MMPP
Mechod
UNK.
14094
75.8
36.5
75.7
56.8
49.9
17.4
785
06.5
7800
18.4
500
01.4
64.
Cemine11.01Pm
UNK
1696.6
75.0
10.7
79.5
-7.9
45.2
Gemini-1 Pm
UNK
58.5
52.1
80.3
81.3
73.5
86.5
67.
1r
UNK
85.0
MM-1-98
1697.6
72.1
$1.0
44.7
Bure ZLM Elomo-Fdtoodd
Gemisi
1606.0
12.7
13.2
64.5
70.2
74.6
62.1
LLaVA-NeXTaB 2590
1903.7
72.1
72.7
65.2
55.5
72.5
54.6
72.6
56.6
35.7
60.1
Cambhiin-1-3 1-3B
S76
73.1
1,547.1
75.9
74.7
64.6
61.3
80.4
42.7
49.0
73,0
71.3
73.3
62.4
71.7
77.5
65.0
51.3
64.2
72.3
72.8
Bare ZZMC. Veuna- 1.5-130
Gemini 13B
1597.0
68.6
70.6
63.7
54.3
37.3
70.2
09.3
49.4
57.5
1
2550
1575.0
70.U
53.7
73.5
36.2
15.1
51.4
57.1
70.9
55.9
59.1
Cumbeian-1 1-13B
575
73.7
1,610.4
75.7
74.4
64.3
60.2
79.3
40.0
48.0
73.6
71.3
73,8
61.9
72.8
76.5
62.3
41.3
63.0
72.5
71.
Bure LLM. Hernosr 343
Min--Gemim--B 5415
2550
1659.0
3116
15.3
03.5
624
77.7
48.0
43.4
51.8
78.9
63.
37.3
672
71.5
1J.VA-NoNT:544
Combrian 1, 34B 575
76.8
1689.3
91.4
75.3
03.5
67.0
85.6
49.7
53.2
79.7
71.9
75.6
60.0
76.7
75.5
05.5
52.7
67.8
74,0
79.7
Table 4: Comparison of Cambrian-1 with other leading MLLMs. Cambrian- I outperforms other open-source models and achieves competitive performance, compared to proprietary models such as GPT-4V, Gemini, and Grok-1.5, Despite using only 576 visual tokens, Cambrian- 1 performs better on OCR & Chart and Vision-Centric benchmarks compared to Mini-Gemini-HHi and LLxVA-NeXT, which use 2880 tokens.
Data Balancing We follow previous work [109, 138] to set thresholds t for the number of data points from single data source. We choose f = 150k, 250k, 350k, and 450k in this section and observe an elbow effect in Table 2-finding that threshold between 250k and 350k work the best for Cambrian- 10M. We also plot in Appendix G.4 the cumulative sum of counts for entries sorted by counts from tail to head and we see this intermediate threshold prevents explosive heavy tail.
Data Ratio Cambrian- -0Mbrian-1OMMM is designed for visual instruction tuning. Given the various capabilities of different types of data, it is essential to balance the ratio of these data types. We conduct pilot experiments with fixed dataset size of 1350k, examining the impact of different data ratios. We visualize the results in Fig. 9 and summarize our findings as follows: (i) Balancing General, OCR and Language data is crucial. (ii) Performance on knowledge-intensive tasks is influenced by multiple factors, often requiring a mix of OCR, chart, reasoning, and general perception.
Cambrian-7M By applying data filtering Cambrian-10M with our identified data ratio, we create smaller but higher-quality dataset called Cambrian-7M. Table 3 showcases the benefits of a well- balanced and carefully curated dataset. Despite having fewer samples, Cambrian- 7M demonstrates improved performance. We additionally apply system prompts in Cambrian to avoid the "answer machine phenomenon", see more details Appendix G.2.
State of the Art Performance
Finally, we leverage the insights from all our previous studies to train family of MLLMs we call Cambrian- 1. We train models using LLM backbones of various scales: LLaMA-3-Instrum
[4], [139]. Our vision component combines four models- OpenAl CLIP V/T-L/144 336, SigI.IP ViT-50400M/1400 384, OpenCIIIP ConyNeXI-XXI.e 1024, and DINOv2 ViT-L/14@511 (Section 2.5) -vi the Spatial Vision Aggregator (Section 3). We pre-train the connector using 2.5M adapter data and instruction tune using our Cambrian-7M data mix (Section 4.2). Our models are evaluated on the benchmarks categorized in Section 2.1, with results presented in Table 4.f1.
ficuna-1.5-133 [151], and Hermes-2-Yi-e
Cambrian-1 surpasses open-source models like LLaVA-NeXT and Mini-Gemini. Thanks to the SVA, Cambrian-1 excels in tasks requiring high-resolution image processing, even with only 576 image tokens lokens-about 1/5 of the tokens used by LLaVA-NeXT and Mini-Gemini. Cambrian-1 1 also achieves comparable performance to the best proprictary models, such as GPT-4V, Gemini-Pro and MM-1, on several benchmarks. We provide model weights, open-source code, datasets, and detailed recipes for model training and evaluation. We hope our work will strengthen the open research community and accelerate research in both visual representation learning and multimodal systems.
For the General Average, we note that GPT-4's performance on the GQA test set is low, possibly because other models are trained on the GQA training set, whereas the training used for GPT-4 is unclear.
10

---

Acknowledgements
We are grateful to LLaVA [82] for their excellent codebase, which served the launching point for our research. Special thanks to Hexu Zhao for extensive discussions and knowledge-sharing around FSDP and large-scale training techniques, and Jiasen Lu for helpful discussions on TPU and JAX distributed training infrastructure. We also appreciate the assistance and responses from the PyTorchXLA team via GitHub.
We are thankful to Kaiming He for early discussions on multi-modal large language models. We also thank Zhuang Liu, Junlin Han, Yuexiang Zhai, Tianzhe Chu, Daohan Lu, Weiyang Jin, Boyang Zhang, and Jiayi Pan for reviewing this manuscript. We also acknowledge DeepSeek [87] for the paper template inspiration.
This work was primarily supported by the Google TPU Research Cloud (TRC) program and the Google Cloud Research Credits program (GCP19980904). Additional support provided by the NYU IT High Performance Computing resources, services, and staff expertise. S.X. would like to thank the OpenAl Researcher Access program, Open Path AI Foundation, and an Amazon Research award for their support. S.T. is supported by the OpenAl SuperAlignment Fellowship, and E.B. is supported by the NDSEG Fellowship.

---

References
[1] M. Acharya, K. Kafle, and C. Kanan. "TallyQA: Answering complex counting questions", In:
AAAI. 2019.
[2] A. Agrawal et al. "Don't just assume; look and answer: Overcoming priors for visual question answering". In: CVPR. 2018.
[3] A. Ahmadyan et al. "Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild
with Pose Annotations". In: CVPR (2021).
[4] Al@Meta. "I.lama Model Card". In: (2024).
[5] H. A. Alawwad et al. "Enhancing Textbook Question Answering Task with Large Language Models and Retrieval Augmented Generation" In: arXiv preprint urXiv:2402.05122 (2024).
[6] J.-B. Alayrac al. "Flamingo: visual language model for few-shot learning". In: NeurIPS, 2022.
[7] T. Aquinas. Quaestiones Disputatae de Veritate. q.2 a.3 arg.19, 1259.
[8] Aristotle. Metaphysics. Ed. by T. by W. D. Ross. The Internet Classics Archive, 350BCE.
[9] M. Assran al. -Sper-supe learning from images with joint-emeedding predictive architecture". In: CVPR. 2023.
[10] J. Bai et al. "Qwen Technical Report". In: arXiv preprint arXiv:2309.1660 (2023).
[11] Bai al. "Qwen-vl: A versatile vision-language model for understanding. localization, text reading, and beyond". In: (2023).
[12] M. E. Banani et al. "Probing the 3D Awareness of Visual Foundation Models". In: arXiv preprint arXiv:2404. 08636 (2024).
[13] G. Baruch et al. "ARKitScenes A Diverse Real-World Dataset for 3D Indoor Scene Under- standing Using Mobile RGB-D Data". . In: NeurIPS Datasets and Benchmarks Track (Round 2021.
[14] Belouadi, A. Lauscher, and S. Eger. "Automatikz: Text-guided synthesis of scientific vector graphics with tikz". In: ICLR. 2024.
[15] R. Birkl, D. Wofk, and M. Müller. "Midas v3. 1-a model zoo for robust monocular relative depth estimation". In: arXiv preprint arXiv:2307. 14460 (2023).
[16] A. F. Biten al. "Laatr Layout-aware transformer for scene-text vqa". In: CVPR. 2022.
[17] A. Biten al. "Scene text visual question answering". In: ICCV, 2019.
[18]
G. Brazil al. "Omni3d: A large benchmark and model for object detection in the wild".
In: CVPR. 2023.
[19] J. Buchner. imagehash (fork). https: : // github. com/ hooo://ooammeeBBcchee//gggggoae
2021.
[20]
H. Caesar et al. "nuscenes: A multimodal dataset for autonomous driving". ". In: CVPR. 2020.
[21]
J. Cha et al. "Honeybee: eenhaity-enhanee projector for multimodal 1lm". In: CVPR. 2024.
[22]
S. Cha et al. "Visually Dehallucinative Instruction Generation: Know What You Don't Know".
In: arXiv preprint arXiv:2402.0977 (2024).
[23] D. J. Chalmers. "Does Thought Require Sensory Grounding? From Pure Thinkers to Large Language Models". In: Proceedings and Addresses of the American Philosophical Association 97 (2023), pp. 22-45.
[24] Y. Chang al. "A survey evaluation of large language models". In: ACM Transactions Intelligent Systems and Technology 15.3 (2024), pp. 1-45.
[25] G. H. Chen et al. "ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision-Language Model". In: arXiv preprint arXiv:2402.14 1164 (2024).
[26] L. Chen et al. "Are on the Right Way for Evaluating Large Vision-Language Models?"
In: arXiv preprint arXiv:2403. 240.:223002 (2024).
[27] L. Chen et al. "Sharegpt4v: Improving large multi-modal models with better captions". In:
arXiv preprint arXiv:231 1.17993 (2023).
[28]
X. Chen et al. "Pali: A jointly-scaled multilingual tanguage-image model". In: ICLR. 2023,
[29] X. Chen, S. Xie, and K. He. "An empirical study of training self-supervised vision transform-
ers". In: ICCV. 2021.
12

---

[30] Z. Chen et al. "How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites". In: arXiv preprint 168221 (2024).
[31] Z. Chen et al. "Finqa: A dataset of numerical reasoning over financial data". In: EMNLP.
2021.
[32] Z. Cheng al. "HiTab: A hierarchical table dataset for question answering and natural language generation". In: ACL. 2022.
[33] M. Cherti et al. "Reproducible scaling laws for contrastive language-image learning". In: CVPR. 2023.
[34] W.-L. Chiang et al. "Chatbot arena: An open platform for evaluating llms by human prefer-
ence". In: arXiv preprint arXiv:2403.0 04/32 (2024).
[35] X. Chu et al. "Mobilevim v2: Faster and stronger baseline for vision language model". In: arXiv preprint arX1::2402.03766 (2024).
[36] M. Conover ct al. Free Dolly: Introducing the World's First Truly Open Instruction-Tunee
LLM. 2023. URL: https: http:::/wwwwwwwwbadbeeeeeae www.bricks com wwtpcoo///////ccccccc/cc//oooo/t 2023///22//////// 20023/04/ /04/10////2 dolly-firrst first-
open- www.eroiici.......m............e wow.eriallll-iiiiiiiiiiaaaaa wtt.aaallly..aaa---aaatttooooooo
11m (visited on 06/30/2023).
[37] W. Dai al. "Instructblip: Towards general-purpose vision-language models with instruction tuning". . In: NewrIPS. 2024.
[38] H. Dong et al. "Rlhf workflow: From reward modeling to online rlhf". In: arXiv preprint arXiv:2405.077 07866 (2024).
[39] A. Dosovitskiy et al. "An image is worth 16x 16 words: Transformers for image recognition
at scale". In: ICLR. 2021.
[40] A. Fang et al. "Data filtering networks". In: ICLR. 2024.
[41] X. Fu et al. "BLINK: Multimodal Large Language Models Can See but Not Perceive". In: arXiv preprint arXiv:2404. 12390 (2024).
[42] Gao et al. "G-llava: Solving geometric problem with multi-modal large language model". In: arXiv preprint arXiv:23122 22. 1370 (2023).
[43] P. Gao et al. (Llama-adapter v2: Parameter-efficien visual instruction model". In: arXiv preprint arXiv:2304.1 15.0044.110000 (2023).
[44] P. Gao et al. "SPHINX-X. Scaling Data and Parameters for a Family of Multi-modal Large Language Models", In: arXiv preprint arXiv:2402.05940 05935 (2024).
[45] Y. Ge al. "Planting seed of vision in large language model". In: arXiv preprint
arXiv:2307.08041 (2023).
[46] A. Geiger, P. Lenz, and R. Urtasun. "Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite". In: CVPR. 2012.
[47] R. Geirhos et al. "Shortcut learning in deep neural networks". In: Nature Machine Intelligence (2020).
[48] R. Girshick et al. "Rich feature hierarchies for accurate object detection and semantic segmentation". In: CVPR. 2014.
[49]
Google. Gemini. 2023.
[50]
Y. Goyal al. "Making the in vqa matter: Elevating the role of image understanding in visual question answering". In: CVPR. 2017.
[51] D. Gurari et al. "Vizwiz grand challenge: Answering visual questions from blind people". In: CVPR. 2018
[52]
K. He et al. "Masked autoencoders are scalable vision learners". In: CVPR. 2022.
[53]
X. He et al. "Path 30000+ Questions for Medical Visual Question Answering". In: CoRR abs/2003. 10286 (2020).
[54]
T. et al. "A12D-RST: multimodal corpus of 1000 primarr school science diagrams".
In: Language Resources and Evaluation 55 (2021), pp. 661-688.
[55]
J. Hoffmann al. "Training compute-optimal large language models". In: NeurIPS (2023).
[56]
Y.-C. Hsiao, Zubach, M. Wang, et al. "Screenqa: Large-scale question-answe pairs over mobile app screenshots". In: arXiv preprint 27XX092209008999 (2022).
[57] D. A. Hudson and C. D. Manning. "GQA: A New Dataset for Real-World Visual Reasoning
and Compositional Question Answering". In: CVPR. 2019.
[58] A. Jaegle et al. "Perceiver: General perception with iterative attention". In: JCML. 2021.
13

---

[59] J. Johnson et al. "Clevr: A diagnostic dataset for compositional language and elementary visual reasoning". In: CVPR. 2017.
[60] N. Jouppi et al. "Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings". In: Proceedings of the 50th Annual International Symposium on Computer Architecture. 2023.
[61] K. Kafle et al. "Dvqa: Understanding data visualizations via question answering". In: CVPR.
2018.
[62] S. Kantharaj et al. "Chart-to-text: A large-scale benchmark for chart summarization". In: ACL. 2022.
[63] S. Karamcheti et al. "Prismatic vims: Investigating the design space of visually-conddtioned language models". In: arXiv preprint arXiv:2402.07865 (2024).
[64] M. Kazemi et al. "Geomverse: A systematic evaluation of large models for geometric reason- ing". In: 2023.
[65]
A. Kembhavi et al. "A diagram worth dozen images". ECCV. 2016.
[66]
D. Kiela et al. "The hateful memes challenge: Detecting hate speech in multimodal memes".
In: NeurIPS. 2020.
[67]
G. Kim al. "Donut: Document understanding transformer without ocr". In: ECCV. 2022.
[68]
A. Kirillov et al. "Segment anything". In: ICCV. 2023.
[69]
R. Krishna et al. "Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations". In: IJCV (2016).
[70]
LAION. laion//gn44vduttaee.. 2023.
[71] H. Laurençon, L. Tronchon, and Sanh. "Unlocking the conversion of Web Screenshots
into HTML Code with the WebSight Dataset". In: arXiv preprint arXiv:2403.0900 (2024).
[72] H. Laurençon al. "What matters when building vision-language models?" In: arXiv preprint
#Xiv:2405.02244 (2024).
[73]
A. C. Li et al. "Internet Explorer: Targeted Representation Learning on the Open Web". In:
ICML. 2023.
[74]
A. C. Li et al. "Your diffusion model is secretly a zero-shot classifier". In: ICCV. 2023.
[75]
B. Li et al. LLaVA-NeXT: Stronger LLMs Supercharge Multimodal Capabiliries in the Wild.
2024.
[76]
L. Li et al. "Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Languag Models". In: arXiv preprint arXiv:2403.00231 (2024).
[77] Y. Li al. "Mini-gemini: Mining the potential of multi-modality vision language models".
In: arXiv preprint arXiv:2403, 18814 (2024).
[78] W. Lian et al. OpenOrca: An Open Dataset of GPT Augmented FLAN Reasoning Traces.
https: : / htt///tttts/////// ///tttts:/huuginggggee htt.//uggggggaeee.c// co/Open Orca/ /ocaa//ppOOraa.... 2023.
[79]
T.-.. Lin et al. "Microsoft coco: Common objects in context". In: ECCV. 2014.
[80] H. Liu et al. "Improved baselines with visual instruction tuning". In: arXiv preprint
arXiv:2310.03744 (2023).
[81]
H. Liu et al. LLVVA-NeXT: Improved reasoning, OCR, and world knowledge. 2024.
[82]
H. Liu et al. "Visual Instruction Tuning". In: NewrlPS. 2023.
[83]
Y. Liu et al. "Mmbench: Is your multi-modal model an all-around player?" In: arXiv preprint arXiv:2307.0628/ (2023).
[84]
Y. Liu et al. "On the hidden mystery of ocr in large multimodal models". In: arXiv preprint arXiv:2305.0780 07895 (2023).
[85]
Z. Liu and K. He. "A Decade's Battle on Dataset Bias: Are We There Yet?" In: arXiv preprint
urXiv:243300863. (2024).
[86]
Z. Liu et al. "A convnet for the 2020s". In: CVPR. 2022.
[87]
H. Lu al. "DeepSeek- VL: towards real-world dision-languae understanding". In: arXiv
preprint XXXX:224400005255 (2024).
[88] P. Lu et al. "Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning". " In: ICLR. 2023.
[89] Lt al. "Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning". In: NeurIPS. 2021.
14

---

[90] P. Lu et al. "Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning". In: ACL. 2021.
[91] P. Lu et al. "Learn to explain: Multimodal reasoning via thought chains for science question answering". In: NeurIPS. 2022.
[92] P. Lu al. "Mathvista: Evaluating mathematical reasoning of foundation models in visual
contexts". In: ICLR (2023).
[93] Z. Luo et al. "Wizardcoder: Empowering code large language models with evol-instruct", In: ICLR. 2024.
[94] A. Majumdar et al. "OpenEQA: Embodied Question Answering in the Era of Foundation Models". In: 2nd Workshop Mobile Manipulation and Embodied Intelligence at ICRA 2024. 2024.
[95] K. Marino et al. "OK-VQA- A Visual Question Answering Benchmark Requiring External Knowledge". In: CVPR. 2019.
[96] A. Masry et al. "Chartqa: A benchmark for question answering about charts with visual and logical reasoning". In: ACL. 2022.
[97] M. Mathew, D. Karatzas, and C. Jawahar. "Doevqa: A dataset for vqa on document images".
In: WACV. 2021.
[98] B. McKinzie et al. "Mml: Methods, analysis & insights from multimodal Ilm pre-training". arXiv preprint arXiv:2403. 09611 (2024).
[99] A. Mitra et al. Orca-Math: Unlocking the potential of SLMs in Grade School Math. 2024.
arxiv: 2402. 14830 [cs. CL].
[100]
"OCR-VQA: Visual Question Answering by Reading Text in Images". In: 2019.
[101] OpenAl. ChaiGPT. 2022.
[102] OpenAl. gpi4o. 2024.
[103] M. Oquab et al. "Dinov2: Learning robust visual features without supervision". In: TMLR
(2023).
[104] L. Ouyang et al. "Training language models to follow instructions with human feedback". In: NeurIPS. 2022.
[105]
A. Parker. In the blink of an eye: how vision sparked the big bang of evolution. 2003.
[106] P. Pasupat and P. Liang. "Compositional semantic parsing on semi-structured tables".. In:
ACL. 2015.
[107] J. Piaget, M. Cook, et al. The origins of intelligence in children. Vol. 8. 5. International Universities Press New York, 1952.
[108] J. Pont-Tuset et al. "Connecting Vision and Language with Localized Narratives". In: ECCV. 2020.
[109] A. Radford et al. "Learning transferable visual models from natural language supervision". In: ICML. 2021.
[110] R. Rafailov al. "Direct preference optimization: Your language model is secretly reward model". In: NeurIPS. 2024.
[111] M. Roberts et al. "Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding". In: /CCV. 2021.
[112] R. Rombach et al. Resilu-Resotttion Image Synthesis With Latent Diffusion Models". In: CVPR. 2022.
[113] O. Russakovsky al. "Imagenet large scale visual recognition challenge". In: IJCV (2015).
[114] O. Sanseviero. LLM Evals and Benchmarking. 2022.
[115] C. Schuhmann et al. "Laion-Sb; An open large-scale dataset for training next generation
image-text models". In: NeurIPS. 2022.
[116] D. Schwenk al. "A-OKVOA: A Benchmark for Visual Question Answering using World
Knowledge". In: ECCV. 2022.
[117] M. Shridhar et al. "ALFWorld: Aligning Text and Embodied Environments for Interactive Learning". In: ICLR. 2021.
[118] C. Si et al. "Design2Code: How Far Are We From Automating Front-End Engineering?" In: arXiv preprint 4xX31:2003.33133 (2024).
15

---

[119] O. Sidorov et al. Tex:Caps: Dataset for Image Captioning with Reading Comprehension.
2020. arXiv: 2003. 12462 [cs. CV].
[120] A. Singh et al. "Towards vqa models that can read". In: CVPR. 2019.
[121] S. Song, S. P. Lichtenberg, and J. Xiao. "Sun rgb-d: A rgb-d scene understanding benchmark
suite". In: CVPR. 2015.
[122] Q. Sun et al. "Eva-clip: Improved training techniques for clip scale". In: arXiv preprint arXiv:2303. 15389 (2023).
[123] R. Tanaka, K. Nishida, and S. Yoshida. "VisualMRC: Machine Reading Comprehension
Document Images". In: AAAI, 2021.
[124] B. J. Tang, A. Boggust, and A. Satyanarayan. "Vistext: A benchmark for semantically rich chart captioning". In: Xrviv preprint wrxiv:2307.05366 (2023).
[125] S. Tong, E. Jones, and J. Steinhardt. "Mass- phass-prouucin failures multimodal systems with language models". In: NeurIPS. 2024.
[126] S. Tong al. "Eyes wide shut? exploring the visual shortcomings of multimodal Ilms". In: CVPR. 2024.
[127]
H. Touvron et al. "LLaMA 2: Open foundation and fine-tuned chat models". In: (2023).
[128] H. Touvron al. "LLaMA: Open and efficient foundation language models". In: arXiv preprint arXiv:2302.139) (2023).
[129] H. Tu et al. "How many unicorns are in this image? safety evaluation benchmark for vision Ilms". In: arXiv preprint arXiv:2311.16100 (2023).
[130] K. Vishniakov, Z. Shen, and Liu. "ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy". In: ICML. 2024.
[131] J. Wang et al. "To see is to believe: Prompting gpt-4y for better visual instruction tuning". In: arXiv preprini arXiv:231 12:33/107577 007574 (2023).
[132] K. Wang et al. "Measuring Multimodal Mathematical Reasoning with MATH- Dataset".
In: arXiv preprint arXiv:2402. 14804 (2024).
[133] Wei et al. 'Chain-of-thought prompting clicits reasoning in large language models". In:
NeurIPS. 2022.
[134] C. Wendler. wendlerr/RendereeTen. 2023.
[135] H. Wu et al. "Q-instruct: Improving low-level visual abilities for multi-modality foundation models". In: arXiv preprint arXiv:231 1.06/330783 (2023).
[136] P. Wu and S. Xie. "V*: Guided Visual Search Core Mechanism in Multimodal LLMs".
In: CVPR. 2024.
[137] xAI grok. 2024.
[138]
H. Xu et al. "Demystifying clip data". In: ICLR. 2024.
[139]
A. Young al. "Yi: Open foundation models by 01. ai". In: arXiv preprint arXiv:2403.04652 (2024).
[140]
L. Yu et al. Modeling Context in Referring Expressions. 2016. arXiv: 1608. 1608.00272 00272 [cs. cv].
[141]
T. Yu et al. "Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback". In: arXiv preprint arXiv:2312.00849 (2023).
[142] X. Yue et al. "Mammoth: Building math generalist models through hybrid instruction tuning".
In: ICI.R. 2024.
[143] X. Yue al. "Mmmu: massive multi-diseipline multimodal understanding and reasoning
benchmark for expert agi". In: CVPR. 2024.
[144] M. Yuksekgonul et al. "When and why vision-language models behave like bags-of-words.
and what to do about it?" In: ICLR. 2022.
[145]
X. Zhai et al. "Sigmoid loss for language image pre-training". In: ICCV. 2023.
[146]
Y. Zhai et al. "Fine- Tunnng Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning". In: arXiv preprint arXiv:2405. 10292 (2024).
[147]
Y. Zhai et al. "Investigating the catastrophic forgetting in multimodal large language models".
In: CPAL. 2024.
[148]
C. Zhang al. "Raven: A dataset for relational and analogical visual reasoning". In: CVPR.
2019.
16

---

[149] Y. Zhang et al. "Llavar: Enhanced visual instruction tuning for text-rich image understanding".
In: arXiv preprint arXiv:2306.17107 (2023).
[150] Y. Zhao et al. "Pytorch fidp: experiences on scaling fully sharded data parallel". In: arXiv preprint arXiv:2304. 1277 (2023).
[151] I.. Zheng et al. "Judging llm-as-a-judge with mt-bench and chatbot arena". In: NeurlPS. 2024.
[152] T. Zheng et al. "OpenCoadentteppreeer Integrating Code Generation with Execution and Refinement". In: arXiv preprint arXiv:2402. I 14658 (2024).
[153] V. Zhong, C. Xiong, and R. Socher. "Seq2sql: Generating structured queries from natural language using reinforcement learning". In: 2017.
[154]
B. Zhou et al. "Semantic understanding of scenes through the ade20k dataset". In: IJCV (2019).
[155]
K. Zhou et al. "Don't Make Your LLM an Evaluation Benchmark Cheater". In: arXiv preprint arXiv:2311.0196 (2023).
[156]
B. Zhu al. Starling-7b: Improving Ilm helpfulness harmlessness with rlaif. 2023.
[157]
D. Zhu al. "Minigpt-4: Enhancing vision-language understanding with advanced large language models". In: arXiv preprint arXiv:2304. 200::3044.0592 (2023).
[158]
F. Zhu et al. "TAT-QA: A question answering benchmark on hybrid of tabular and textual content in finance". In: ACL. 2021.
[159]
Y. Zhu et al.
"Visual7w: Grounded question answering in images", In: CVPR. 2016.
17

---

A Broader Discussion
We advocate for using MILMS as an interface to evaluate visual representations, as previous bench- marks are becoming saturated and do not adequately reflect the diverse and complex perception challenges of the real world. Our work highlights the current gap between sunguage-supervised models and self-supervised learning models and demonstrates the potential of bridging this gap. How- ever, it is known that features of language-suppervssed models behave like a bag-of-wordhh [125, 1 144], underscoring the need for advancements in vision-only models to ensure better visual understanding. We hope to inspire future research into developing better vision-only models intended to be adapted into the MILM setting, that more effectively leverage large-scale datasets [85] and preserve the advantages in visual grounding [126].
As we observe in Table 4, well-trained open-source model such as Cambrian-1 can match or even outperform proprietary models on many existing benchmarks. However, the use and evaluation of MLLMs extend far beyond the current scope of benchmarks conversational ability, creativity, reliability, and overall user experience. Developing models solely based on benchmark results can result in an "answer machine", over-optimized for benchmarks but lacking in practical interaction capabilities. Therefore, the development MILMs that better align with human and societal needs is a continuously evolving process, both in terms of evaluation and model development.
Our current Cambrian-1 model uses a moderate number of visual tokens and does not adopt the any- resolution strategy [30, 77, 81] to handle ultra high-resolution images or those with extreme aspect ratios, which require larger number of visual tokens. For specialized tasks like V*Bench [136], which require processing ultra high-resolution images, increasing the resolution and number of visual tokens could lead to HD version of the Cambrian-1 model.
One promising direction for post-training alignment is through reinforcement learning rather than supervised fine-tuning. Many MILM studies, including Cambrian, primarily focus on supervised fine-tuning. Yet, recent advancements in LLMs [38, 104, 110, 156] and some in MLLMS [141, 146] suggest that reinforcement learning from human or environmental feedback can further improve models, potentially surpassing the limits of supervised fine-tuning, especially in decision-making abilities.
Cambrian- 10M (Fig. 8 and Section 4) provides a rich pool of data for studying data curation in fine-tuning MLLMs. Our work takes initial step in curating higher-quality data to enable more efficient and effective instruction tuning. We believe there is room for further improvement in the data curation pipeline, and hope this work can serve as a foundation for future research.
Additionally, training large-scale models requires careful design of model sharding, data sharding, and infrastructure adaptations. In this work, we train our model TPU-V4 [60] with FSDP [150] using TorchXLA. We share our experiences, technical challenges, and solutions in Appendix C. We also open-source our implementation and provide tutorials to help the community undertake large-scale training more efficiently.
To conclude, Cambrian-1 introduces family of tfate-of-the-arr MLLM models that achieve top performance across diverse benchmarks and excel in visual-centric tasks. We provide model weights, open-source code, datasets, and detailed recipes for model training and evaluation. We hope our work will strengthen the open research community and accelerate future advancements both visual representation learning and multimodal systems.
B Multimodal LLMs: Preliminaries and Related Work
The key components of MILM research include the Large Language Model, Visual Encoder, Multimodal Connector, Data Curation Pipeline, Instruction Tuning Strategy, and Evaluation & Benchmaarking. Each component has its intricacies, and understanding their interactions presents significant challenges. Our study investigates these aspects from a vision-centric perspective.
Large Language Model Advanced LLMs [4, 101, 127, 128] are the foundation of an MLLM. After instruction-tuning on multimodal data, these models can prompted to solve a variety of complex tasks and generate free-form responses leveraging input from visual encoder. Recent MILM research focuses on enhancing the L.LM backbone [10, 75, 81], resulting in improved performance on benchmarks like MMMU [143] and A12D [54]. However, this improvement raises the concern
18

---

that our current multimodal evaluation is biased by the development of LLMs, neglecting true assessment of visual perception. For example, some benchmarks such as MMMU [143] are dominated by L.M capabilities, underscoring the need for evaluations that genuinely assess multimodality (see Section 2.
Visual Encoder Most MLLMs utilize lunguage-Supervised models like CLIP [109, 122, 145], which benefit from the massive scale of noisy web image-text data. However, there is a much broader pool of visual models that learn representations using only visual signals such self-supervised models [9. 103], segmentation [68], suppttiseeeeeiee [15], and diffusion models [74, 112] (see Fig. 2). Recent work [87, 126] advocates for incorporating these diverse vision models into MILMs. In this study, we systematically examine the impact of various vision backbones on MILMM performance (Section 1) and explore the benefits of model ensembles (Section 2.5).
Multimodal Connector Representations from a visual encoder cannot be natively processed by an LLM they must be mapped into the LLM token space by a connector. There are three primary approaches to connector design: Resamplers [6], Q-Formers [11, 37], and MLP Projectors [43, 80, 82. 157]. We begin our exploration using an MLP projector, which highly effective but presents challenges: the visual token count grows quadratically with image resolution, inhibiting scaling context length input resolution. For example, LLaVA--Next [81] requires 2880 visual tokens to process one 672px image. To address this, we explore new vision connector designs that process high-resolution images while maintaining a smaller number of visual tokens (Section 3).
Instruction Tuning Data Visual instruction tuning data is crucial but hard to collect, as it rarely naturally exists on the internet. Previous work [37, 80, 98] transforms existing VQA benchmarks [50, 69] into instruction tunigg data, showing marked MILM performance improvements. With this inspiration, we collect all VQA benchmarks and visual interaction data that we can find (Fig. 8), study data balancing and category mixtures (Section 4.2), and develop internet data collection engine to fill in the gaps (Section 4.1).
Instruction Tuning Most current MLLMs leverage pre-trained LLMs and visual encoders, fine- tuning the LLM and connector using visual instruction tuning data. Some aspects of the tuning recipe are up for debate, including whether to pre-train the connector before joint fine-tuning with the LLM, and whether to freeze or unfreeze the vision encoder during fine-tuning [63, 98]. Additionally, some recent proprietary models explore end-to-end training from scratch [49, 102]. this work, we use pre-trained models and revisit the debated recipe aspects with extensive studies, providing more insights for future MLLM research (Section 2.3).
Evaluation & Benchmarking There is an extensive set of benchmarks that evaluate various aspects of MLLMs, such as perception [45, 83). knowledge [91, 92], chart interpretation [84, 96], and visual capabilities [126, 136]. Instead of over-optimizing for specific benchmarks, we advocate for examining aggregates of benchmarks that focus on specific capabilities. To achieve this, we analyze existing benchmarks, categorize them, and assess the extent to which they measure maltimodality (Section 1). Additionally, we find there are currently few benchmarks focused on vision-centric evaluation, and those that do exist contain relatively few images, leading to higher variance during evaluation. To address this issue, we propose a new vision-centric benchmark by reformulating classic vision tasks (Section 2.2).
C Training, Infrastructure, and Implementation
All models in this paper were trained using TPU-V4 pods [60]; we evaluate using NVIDIA A6000, A100, and H100 cards. The experiments in Section 2.4 require less than 24 hours on a TPU-V4-128, while our final Cambrian-1 models are trained in less than 4 days a TPU-V4-512.
To enable and facilitate large-scale parallel training on TPUs, we employ TorchXLA with FSDP [150] to handle training sharding and parallelism. Training large-scale multimodal model with TorchXLA on TPU is a challenging journey, as there are no are not supported in the TorchXLA or TorchXLA FSDP libraries. To provide a brief taste of the difficulties: TPUs require a static graph throughout the program, which requires ground-up rewrites of dynamically-writeer open-source PyTorch codebases; model resuming is not implemented in TorchXLAA which especially crucial when training on preemptable TPUs; existing TorchXIA FSDP tutorials fail to compile due to version changes in TorchXLA, updates in Hugging Face
codebases and many critical features
19

---

Transformers & Accelerate, or simply inherent issues with the tutorial; loading very large models (over 30 billion parameters) with the TorchXLA FSDP library is natively impossible due to the 100GB memory constraints of TPU-V4s, and requires extensive workarounds.
To this end, we have rewritten developed many new functions to make this research possible. For instance, we rewrote the TorchXLA FSDP Sharding API to load very large models; we implemented model resuming on TorchXLA; we rewrote parts of the Hugging Face Transformers FSDP and gradient checkpointing implementations to enable large-scale FSDP training. We are committed to open-sourcing our codebase and publishing comprehensive tutorial to share our insights, with the hope of inspiring and supporting future research and open-source contributions to the TPU and TorchXLA ecosystem.
D Analyzing the Benchmarks
MLLM Benchmark Performance Confusion Matrix
We evaluate the benchmark scores for our one-stage, two-stage finetune-only and hybrid models, and then plot the correlation matrix for the pool of MLLM benchmarks. The correlation plot displays in Fig. 10. The result demonstrates that MMMU is less correlated in measuring model performance to other benchmarks. Nonetheless, we acknowledge it is widely used and therefore cluster it into knowledge-based QAs based on the nature of their questions.
0.65
0.78
MNH
0.97
1.03
0.85
0.73
0.75
0.87
0.87
0.81
3.00
0.79
SEED
0.96
0:57
1.00
0.57
0.79
0.70
0.77
0.85
0.78
0.53
0:84
0.79
0.91
9.51
0.83
CQA
8.65
0:55
0.97
0.70
0.64
0.71 0.76
0.74
0.69
0.78
0.73
8.70
1.00
0.57
0.55
0.65
0.60
0.62
MathMssta
-075 0.75 0.77
0.71
0.75
0.57
1.00
0.65
0.78
0.78
0.80
0.78
0.74
0.74
0.52
ADD 0.82 0.04
0.85
0.76
0.78
0.54
0.00
1.00
0.83
0.06 0.86
0.83
0.81
0.85
0.75
ChartQA
0.78 0.80
0.76
0.55
0.09
0.34
0.70
0..53
01:0
0:5e
0.98
0.05
0.7
TeatvOA
0.87
0.84
0.95
0.53
0.30
1.00
0.99
0.79
0.89
0.04
DocHOA
0.79
0:81
0.79
0.69
0.93
0.58
0.78
0.53
0:98
0.55
0:38
1.00
0.73
0.85
0.70
S8RPP
0.87 0.87
0.91
0.87
9.71
0.51
0.74
0.81
0.75
0.76
0.79
0.73
1.00
3.90
0.79
TW
0.K4
0.88
0.91
0.80
0.99
0.85
0.09
0.84
0.85
2.00
0.34
MNB
GOk
ADD
oCRBarch-
D:cVOA
Figure 10: Correlation matrix for MLLM benchmarks.
The correlation matrix for MLLM benehmarks with
respect to different vision backbones. The correlation matrix helps us to analysis and group benchmarks.
E Cambrian Vision-Centric Benchmark (CV-Bench)
CV-Bench Curation Below we describe the procedure for programmatically constructing questions for each task. To ensure reliability, we also manually inspect each question, removing those that are unclear, ambiguous, erroncous.
Spatial Relationship (2D). We consider images with two distinct ground-truth object categories and visual prompts (bounding boxes) avoid ambiguity when multiple instances present. In these questions, we designate an anchor object, and the question asks for the direction of the other object relative to this anchor.
Object Counting (2D). This tests the model's s ability to count objects. When generating options for these questions, construct multiple-choice options that are similar to the correct answer. For
20

---

Spatial Relationship
Objoct
Depth Relative
How ctamy sacce Source benchmurk: ADE20K [154] and COCOII
is doser to camere, pillowe?
refnigerator dhair,
reepect
image!
Source benchmarkk: Omnil [18]
Figure 11: Cambrian Vision-Centri Benchmark (CV-Bench), We repurpose standard vision henchmarks to evaluate the fundamental 2D and 3D visual understanding of MI..Ms. See Section 2.2 for more details.
Type
Task
Description
Sources
# Samples
Spatial Relationship
Determine the relative position an object w.r.t. the anchor object. Consider left-right or top- bottom relationship.
ADE20K COCO
650
2D
Object
Determine the number of instances present the image.
ADE20K
788
Count
COCO
Depth Order
Determine which of the two distinct objects is closer to the camera.
Omni3D
600
3D
Relative Distance
Determine which of the two distinct objects is closer to the anchor object.
Omni3D
600
Table Breakdown of the 2D and 3D tasks evaluated in the Cambrian Vision-Centrie Benchmark (CV-Bench). examples are sourced from ADE20K [154], COCO [79], and Omni3D [18].
example, if the correct answer is 4, the options might be 3, & 6. We also include existence check examples where the correct count is O.
Depth Order (3D). consider images with two distinct categories (i.e., object A and object B) and visual prompts (e.g., bounding boxes with two different colors) avoid ambiguity. We define "closer" as follows: object A closer to the camera than object only if the farthest vertex object A is cl the camera than the nearest vertex of object B by specified offset.
Relative Distance (3D). We consider images with three distinct categories (i.e., anchor, object A, and object B), and use visual prompts (e.g.. bounding boxes with three different colors) to avoid ambiguity. Object A is closer than object only if the farthest distance from A's vertices shorter than the shortest distance from B's vertices to the anchor object by certain offset.
Curation Procedure We provide an overview of the data curation process Fig. 12, which is conducted in semi-automatic manner. The procedure consists of two main steps:
First, using the original benchmarks and their associated ground truth annotations, we generate query and answer pairs. These pairs are tailored to specific tasks: 2D-related tasks with COCO and ADE20K datasets, and 3D-related tasks with Omni3D.
Second, after generating the query and answer pairs, we engage human experts to manually filter any incorrect ambiguous queries enhance the quality of benchmark. Fach query is assigned one of three statuses: accepted (used as is), modified (where the incorrect answer is modified), and rejected (queries that are ambiguous, such as those too small or difficult discern, even for human experts).
Following this two-stage process, finalize the benchmark, which results in total of 2638 image queries with improved accuracy and reliability. Subsequently, we will discuss the methods human verification and the evaluation metries used in this process.
Human verification
We use the Euclidean distance.
21

---

Spatial Relations
Dats
ADEZIK
Count
Manual Filter
COCO
Filter Cenerate
All Q&A
Modified
Benchmark Datasel
Relelive Distance
OMNI3D
Rejected
Depth
Figure 12: CV-CB Benchmark Filtering. We reformulate classic 2D and 3D CV benchmarks into Q&A questions to evaluate MLLM's visual capabilities.
There are multiple reasons for the above generated data to be inaccurate. One of the main reasons is sparse annotations, but occasionally there could be wrong annotations well.
Thus, we need manual inspection to change/remove these examples generated. Here are a few criteria followed while manually filtering COCO and ADE20K data.
For Counting question types, if instances of category are not annotated, the ground truth would have lower count than the actual number of instances appearing in the image. In few cases where the image distinctly has different countable instances of the object, we change the options/answer. In case the count is ambiguous, we reject the data sample altogether.
For Relative Distance question types without annotation, the question is asked about two objects A and B and if there are two instances of specific category (say A), the relative location of A w.r.t B can be have multiple correct answers. We reject the sample in this case. We also reject cases with clear incorrect annotations.
Benchmark Evaluation To ensure that equal importance is given to both 2D and 3D tasks, we use an evaluation metric that is the average of the accuracies obtained from these tasks. Specifically, the overall performance is calculated as follows:
Accuracy + Accuracy ADE20
Accuracy
Accuracy,2 + Accuracy 3D
Overall Accuracy
2
F
Vision Models in MILMs
As mentioned in Section 2.4, we use MILM as interface to evaluate vision model's different capabilities. Here, we list details terms of the model selection, full results, and data split.
F.1 Details of Vision Models
In our exploration of versatile vision models, we select thirteen models and group them into four categories: danguage-superviser models (i.c.. OpenAl CLIP [109], SigLIP [145], DFN-CLIP [40], EVA-CLIP [122] and OpenCLIP [33]), self-supervised models (i.c., DINOv2 [103], I-JEPA [9]. MAE [52]. MoCo v3 [29]), elas-supeeriiedd models (ImagetNet22K ViT (39]) and other models such as stable diffusion [112], segmentation models like SAM [68], and depth estimation models like MiDaS [15]. To provide clear understanding of the specific variant evaluated,
meticulously detail their backbone architectures, resolution, number of tokens, and hidden dimension sizes in Table 6. For models that output large number of patches in the last layer (e.g.. SAM and ConvNeXt) we interpolate to the number of tokens specified in Table 6, and denote interpolation with?
We extract features following the practice in [12]
22

---

Supervision
Method
Architecture
Patch Size
Res.
# Tok.
Hidden
Type
Size
Laaguage-Superveee Supervised
Language
OpenAl CLIP
VIT-L
14
336
576
768
DEN-CLIP
ViT-L
14
224
256
1024
DFN-CLI
ViT-H
14
378
729
1280
EVA-CLIP-02
VIT-L
14
336
576
1024
SigLIP
ViT-L
16
384
576
1024
SigLIP
ViT-S0400M
14
384
729
1152
OpenCLIP
ConvNeXT-L
512
1576
1536
OpenCLIP
ConvNeXT-L
1024
1576
1536
OpenCLIP
ConvNeXT-XXL
1024
576
3072
Self-Supervised
Contrastive
DINOv2
VIT-L
14
336
576
1024
DINOv2
ViT-L
14
518
576
1024
MoCo v3
VIT-B
16
224
196
768
MoCo v3
ViT-L
16
224
196
1024
Masked
MAE
ViT-L
224
1024
MAE
ViT-H
224
256
1280
JEPA
I-JEPA
ViT-H
14
224
256
1280
Other
Segmentation
SAM
ViT-L
16
1024
576
1024
SAM
ViT-L
16
1024
1766
1280
Depth
MiDaS 3.0
ViT-L
384
1024
MiDaS 3.1
ViT-L
16
518
1024
1024
Diffusion
Stable Diffusion
VAE+UNet
16
512
1024
3520
2.1
Class Labels
SupVIT
VIT-L
16
224
196
1024
SupViT
ViT-H
1280
Table 6: Catalog of all vision backbones tested. denotes that the visual tokens have been interpolated down to the specified length.
Method
Architecture
Patch Size
Resolution
Tokens
Linear Probing (%)
EVA-CLIP-02
VIT-L
14
336
576
85.0
DEN-CLIP
VIT-L
224
256
83.6
DINOv2
VIT-I
336
576
83.1
OpenCLIP
ConvNeXI-L
512
576
82.9
OpenAl CLIP
VIT-L
14
336
576
80.3
I-JEPA
VIT-H
14
224
256
77.0
Supervised
VIT-L
224
196
MoCo
ViT-B
224
196
71.9
MiDaS
VIT-L
16
384
576
70.1
MAE
VIT-L
16
224
196
68.3
Table 7: Linear Probing Results of Different Vision Backbones
F.2
Full Results of Different Vision Backbones
Here, we also show a ranking version of Fig. 5. We observe a clear advantage of CLIP models over non-CLIP models. We also observe that within the family of CLIP models, each model perform differently in different domains. This provide insight into both vision model development and data curation in training large vision models.
For the above-listed vision models Table they are integrated as the vision encoder of the MLLMs.
These MILMs are trained various adapter adapter data splits (i.e., O, 0.5 and 1.2 million), and subsequently fine-tuned on 737K instruction tuning dataset provided V.L. For the adapter data splits, the OM split indicates that initial adapter pertaining phase is employed for the
23

---

Language Supervised
Self-Supervised & Other
Model
Architecture
All
G
K
O
V
Model
Architecture
All
G
K
O
V
SigLIP
ViI-S0400M/14@3344
DINOw2
ViT-L/I4@518
1
OpenCLIP
ConyNeXIIXXXLC 1024
6
8
3
DINOv2
VTT-L/14@336
2
DFN-CLIP
ViT-H/14@378
3
4
2
5
4
MAE
ViT-L/16@222 224
3
5
2
2
OpenCLIP
ConvNeXt-LL@@ 1024
4
8
7
3
8
I-JEPA
VIT-H/14@224
4
3
6
8
3
SigLIP
IT-L//6@@384
5
5
4
4
6
SD2.1
VAE+UNeu/16@55 $12
5
7
9
9
5
OpenAI CLIP
ViT-I/14 333
6
3
6
6
7
MiDaS 3.0
ViT-I/166 384
6
6
8
EVA-CLIP-02
ViT-L/14@
7
2
5
8
2
SupViT
NTT-L/16@222
7
9
OpenCLIP
ComvNexX--L511
8
7
3
7
9
MoCo v3
ViT-B/16@222
8
DEN-CLIP
ViT-L/14@224
9
9
MoCo v3
ViT-L/16 @ 224
9
DINOv2*
ViT-L/14@518
10 10
10
10
5
SAM
ViT-H/16@ 1024
10
10
10
Table 8: Benchmark performance rankings for MLLMs built upon language- supervised and self- supersupee vision encoders across all benchmarks (AlI), and across general (G), knowledge (K), OCR & chart (O), and vision-centric (V) benchmark categories. Full results for models on each benchmark are tabulated Table 11. *We add DINOv2 here to show its standing amongst the CL.IP models.
MLLM. The 0.5M data split utilizes the 558K adapter data from LLaVA-1.5180), while the 1.2M variant uses ShTPeGPT4V-PT dataset [27].
OM Adapter Data 737K Instruction Tuning Data As shown in Table 9, we provide 20 results for different variants of the above-mentioned thirteen vision backbones. Among them, language- supervised models show superior performance. Especially, OpenCLIP ConvNeXT-XXL @ 1024 model surpasses all other models on DocVQA with over 12%, indicating its potential to handle OCR-related benchmarks.
Vision Rackhone
General
Kaowledge
OCR & Chart
Vision-Cantric
RealWarM
Math Visua
OCRBesch
CV <enchh
Average
MMDE
MMB
SEED
6QA
SQW
MMMU
A2D
Charqa
TexVQQ
DneVQA
MMVP
Model
Architecrure
Language Supervised
OpenAICI. CLIP
VIT-/1148 336
48.37
1,419.45
61 45
59.85
62:26
69.87
34.50
27 80
59
33.73
31.70
55.39
28.00
11.33
53.46
56.44
57.40
DEN-CLIP
NTT-L/448224
58.78
1.172.5
50
49.53
49.74
52.94
67. 74
34.00
27.30
56.99
16.75
4.87
44.81
11.19
6.67
46.97
40.29
52.08
DEN-CLIP
NIT:H/188
36.79
76
10.15
4.67
EVA-C1JP-02
WTT/114
1,325
62.99
20.00
22.50
52.68
SigLIP
VTTL/168 384
48.80
1,383.42
61 02.02
63.56
61.85
68
35 29
29.70
57.87
34.96
29.60
56.73
28.31
23.33
52.68 52.95
5483
SigLIP
WIT-00400//160
47,57
1,576.75
58,
60.59
60.92
69.01
34.40
26.50
58
30.72
28.60
55.10
28.31
19.33
50.71 52.33
58.67
OpenCLIP
NowNcci-L@@112
47.38
1,405.01
57.62
61.90
60.34
69.06
33.90
29.10
58 39
28.01
25.20
55:45
28.11
21.00
51,12 53.46
18.91
OpenCLIP
ComNeXt-L@@10 1024
39.02
1,139.60
49.59
65.7)
12.05
9.67
47.45
OpcnCLIP
ConNNeXiXXLe 1024
41,83
1,219.47
48.00
49.88
55.09
66
.4
35.09
27.60
56.67
16.92
5.00
46.90
40.98
16.00
47.32
43.40
52.75
Self Supervised
DINOv2
VIT-L/146 336
41.18
1,262.66
56
49.62
56.80
60 30
K5
35.00
26.40
56.41
16.48
3,10
44.04
11.90
18.67
50.20
49.43
52.25
DINOY2
40.60
1,242.48
15.11
10:95
48.63
M3
NIT-8/166221
34.94
966.45
36.77
33.00
47.35
62
96
32.80
26.20
55.05
16.01
2.60
43.81
10.31
6.67
45.36
39.03
52.83
M3Cov3
VTT-L/168 224
34.70
1010.18
34.64
41.71
47.46
64 70
33.70
26.30
55.05
16:24
2.70
42.60
10.39
4.00
45.36
44.67
35.16
MAE
NIT-L/168 224
37.69
1,114.07
42.30
35.93
20
63
51
34.60
26.00
56,
16.11
2,70
43.63
10.83
14.00
44,80
45.81
55.75
MAE
NITH1/11
1,063.35
50.99
51.90
IJIEPA
2TT-H/148224
58
1,152,07
44.68
51.74
55.37
66
06.04
34.20
26.40
56.09
15.80
3.00
43.66
11.48
10.67
46.01
46.74
53.50
Uther
SAM
VTT-L/168 1024
31.74
585.78
20.34
36.34
39.85
65
34.50
25.10
53.92
16.16
2.70
42.37
9.25
2.00
41.44
35.65 50.50
SAM
NTT-6/66 1024
22
55.25
MiDaS
35.65
981.5 36
38.57
40:93
49.04
63
25.70
16:36
2.60
11:24
6.67
38.78 53.40
MiDaS 3.1
WT--1165118
$5.44
983. 54
34.79
40.20
48.53
64
33.90
25.00
55 18
14
2.60
42.76
12.08
6.66
43.66
39.63
52.58
DifTosion
SD2.1/168 512
36.59
1,044,2
28
37.71
42,00
48 38
64
55
33.40
25.70
56.99
15.56
3.10
43,14
10.40
9,33
45.88
44.68 52.40
Sup
WTT 0TE//18220 224
16.44
11.83
46.67
$2.75
SupViT
WIT-H/188224
37.45
1,082.45
42.61
48.4
52.98
63
36.29
26.50
55.78
3.30
44.16
11.49
4.66
43.79
44.55 52.91
Table 9: All Benchmark Results for 0M Adapter Data + 737K Instruction Tuning Data
0.5M Adapter Data + 737K Instruction Finetune As shown in Table 10 and Table 9, the inclusion of alignment stage with 0.5M data split results in notable increase in performance for DFN-CLIP ViT-H/1 (1H//11@11) 378, from 36.21 to 49.94. This substantial improvement highlights the value of the alignment stage for enhancing certain vision backbones, suggesting its importance in harnessing the full potential of vision models.
1.2M Adapter Data 737K Instruction Finetune As we increase the amount of data in the align- ment phase, we observe a consistent performance improvement for SigLIP ViT-S0400M/144 384
24

---

Vision Backboon
General
Knowledge
OCR Chart
Vision-Centric
Average
MMD
MMB
SEED
604
SQN
MMMU
Vinn
AED
ChanQA
OCRBench
TeaVQA
DoeVOA
MMVP
ReaRWoMMA
CV-Bench 3o
CV-Bend)
Model
Architecrure
Langaage-Suun Supervvi-d
CLIP VIT-L/148 336
49.03
1,413.51
60
60.81
69 76
29
48
30.98
51.63
52.57
54.75
DEN-CIJP
45.59
1,382
57.3
63.26
60.54
29.20
23.45
52.27
1888
51.31
49.59
DEN-CLIP
50.62
1,500.45
02
66.44
70:25
35. 69
30.30
58. 78
20
29.80
56.98
31.39
29.33
53.59
54 54.96
52.58
EVA-CLIP-02
ViT:L/148 336
27,13
1,362.07
62
64 63.96
10
09
09.46
35.89
27.90
56.96
20.96
26.
53.93
19.00
20.00
53.07
56.28
58.16
SigL.P
VIT//168 384
48
59 45
70.25
76
28
25 60
26.00
56.33
SigLIP
1,327.79
36.99
33.20
36.58
OpenCLIP
Con:NeX-Lo0512
48.01
1,366.85
59. 39.66 62.89
61.31
68.77
36.99
28.50
59.29
27 88
28.50
57.57
29:48
16.00
53.20
53.81
55.91
OpeaCLIP
ConyNeXi- 1021
40.29
1,084.82
12.94 51.02
89.78
65.47
34,20
27.60
56.36
29.92
13.25
50.37
03.67
13.33
19.08
51.85 41.58
OpeaCLIP
Cun-NcXI-XXLe 1024
50.45
1,405.65
57.60
63.58
62.41
68.02
34.30
29.
39.62
42.96
96
26.20
61.82
42.67
28.67
55.16
49.92
51,16
Sule-Superiieed
DINOv2
VIF-L/1883 336
42.64
1,283.95
54.61
59.03
60.19
66.39
35,29
25.70
58.03
16.00
3.20
45.39
11.79
20.00
50.59
53.51
558.33
McCo
NIT-B/166:24
58.50
1,159.10
40.00
51.37
54.97
65 25.25
33.70
27.20
55.51
16.36
3.30
44.42
11.42
10.67
46.14
45,80
52.00
M3 NTT 224
37.71
1,074.13
49.19
49.46
53.61
63.66
33.70
27.40
55.83
17.04
3.40
43.84
11.96
8.00
46.27
48.69
45.59
VIT:L/6682 224
1.138.35
47.32
I-IEAA
XTB-L/448-24
39.91
1,180.12
44.26
52.86
55.32
65.94
34,40
27,00
57.16
15 88
3.20
44.36
11.61
13.33
46.27
52.19
55.83
Other
SAM
VTT-V668 1024
52.18
619.99
22.47
36.37
40.16
61.60
32.50
25.80
$4.66
15.80
2.70
42:40
8.89
0.00
45.62
37.02
53.08
MiDaS
VIT-L/168 384
1,183.95
19
27
17.00
11.55
47.32
54.58
Diffusion
SDD.1/118 512
58.26
1,123.46
42.04
50.66
53.63
65.74
33.30
24.50
14:52
3.30
43.95
10.62
10.00
43.53
48.24 $4.50
SupVIT
NIT-L/168:24
39.66
1,186.88
48.43
51.28
56.35
65:49
33.00
28.10 57.16
17.56
2.80
44.92
12.23
12.67
47.06 43.59 51.67
Table 10: All Benchmark Results for 0.5M Adapter Data + 737K Instruction Tuning Data
from 46.79 to 49.72 to 53.09 across OM, 0.5M to 1.2M data splits as shown in Table 9, Table 10 and
Table 11.
Vision Rackhore
General
Knowlddge
OCR & Chart
Centric
OCHllach
CV-Boad
Avcrage
MME
MMB
SEED
6QA
SQA
MMMM
CharQA
ToxVVO
DoVVQN
MMVP
Model
Architecture
Emguage-Supernen
OpenAl WT-LL1448366
50.49
1,476.85
61.96
62.78
29.50
58.21
32.11
$4.90
DEN-CLIP
VIT-L/141222
16.01
1,341.14
56.68
63.74
60.75
66
33.80
28.65
57.04
23.20
52.85
18.97
26.67
51.44
52.91
52.08
DFN-CLIP
378-
51.17
1.426.32
62.38
67.29
62.89
69
35.89
30.00
60.01
41
48.08
30.60
57.53
31 69
32.67
55.95
55,46 55.00
EVA-CLIP-02
1/1-1/143366
49.71
1,429. 06.99.788
64.00
67.53
63.80
69
99.91 35.49
25.40
59.16
24.76
27.10
55.39
21 63
34.67
55.69
57.83
57.75
StgLIP
VIT-L/166 394
50.87
1,424.20
59.
65.18
62.56
68
35.99
29.70
29
40.52
33.50
59.59
35 20
28.00
53.33
55.42 56.08
SigLIP
WTT-S0000//44
1.455.64
63.70
56.99
OpenCLIP
CoesNeXi-1,0511
49.16
1.416.87
60.60
63.87
61.87
69
92
35.79
29 50
50 56
28
36
78,
27.33
51
54.64 31,80
OpenCIJP
CowNeett-1@@ 60 1024
$1.00
1,392.92
58.21
65.47
62.89
67
43
34.90
20.90
59.15
46.08
25.50
50
62.
44
26.67
55.29
53.57 55.08
OpenCLIP
XX 1024
52.18
1,402.94
59.40
65.21
62.75
68
27
35.10
29.
59.84 84
48.00
28.00
663.27
4%
34.67
55.95
53.83
55.08
Sele-Superviied
DINOV2
336
41.85
1,190.81
51.83
60.38
34.20
27
3.50
45.12
49.67
53.91
v3
VIT-B/160224
38.88
1,129.32
41.62
52.19
55,05
68 39
35.30
28.30
56.44
16.48
3.00
44.09
11.47
12.00
47.58
45.17
$3.00
MoCo v3
VIT-L/16822 224
37.07
1015.20
37.28
48.31
52.63
65
49
34,50
22.60
$5.41
16.92
3.10
43.57
11.50
14.67
45.49
45.17 40.75
MAE
VIT-L/166 224
10.39
1.132.80
43.40
55.67
57.42
66.04
35.59
27.80
$6.48
17.36
3.30
41.53
12.30
16.00
47.51
49.24
56.75
IJEPA
VIT-H//118224
40.27
1,207.88
27.10
43.45
OTher
SAM
WT-H/1688 1034
32.54
682.81
23.32
36. 16
40.32
68 20
33.20
26.50
$4.21
15.68
2.50
41.76
8.98
1.33
46.80
37.66 $2.90
MID&S 3.0
WT-L/168 384
39.15
1,132,18
46.21
51.75
55.57
66.30
33.70
26,
56.06
17.08
3.10
43.65
11:66
15.30
45.75
44.44
52.58
Diffusion
SU2.1/166551
39.51
1.168.52
40.00
53.80
55.33
35.00
26.10
57.16
15.36
3.10
44.23
11.06
18.67
47.32
48.04 53.90
Supervised
VTT-L/168224
39.12
1,216.11
45.28
51.46
55.88
64.15
34.70
26.80
55.76
2.80
44.42
11.61
11.33
47.97
51.60
Table 11: All Benchmark Results for 1.2M Adapter Data + 737K Instruction Tuning Data
1.2M Adapter Data 737K Instruction Finetune with Unfrozen Vision Model Here, we present the results of different vision models trained with 1.2m adapter data and 737K instruction tuning data in Appendix F.2. Comparing to Appendix I.2, we observe nearly all the models see improvement on most of the benchmarks, especially on the OCR & Chart and Vision-Centric benchmarks.
1.2M Adapter Data + 5M Instruction Finetune We present the results of 5M instruction tuning experiments in Fig. 6 here. In Table 12, we observe that after 5m instruction tuning, the gap between DINOv2 and CLIP models continue to bridge on general, knowledge and vision-centric benchmarks.
F.3 Model Ensemble
Model Ensemble Details We introduce the implementation details of the model ensemble in Section 2.5. For given image, the image passes through each vision encoder to obtain the features
25

---

Vision Bucckhone
General
Knowlodge
OCR Chart
Visioe-Centric
RealooddOS
MathVisn
OCRBench
CY-Rench,3
CV-Rench
Averuge
MME?
MMB
60A
SQN
MMMU
AED
ChunQA
TexVVQ
DoA
MMVP
Model
Architscture
Other
OpenAT CLIP
VIT-1/148
52.90
1.477.15
68.49
69.21
67.68
35.22
26.00
59.65
56.16
DEN-CLIP
VIT-L/148224
45.80
1,364.91
55.15
63.05
59.99 94
67.
35.50
55.99
80
58.65
20.48
23 40
52
23
18.29
22.67
50.85
53.59
53.50
EVA-CLP--02
VIT-L/148338 336
51.30
1,392.35
65.53
69.75
65.18
68.02
35.00
29 50
60.78
29 19.32
29.90
56.87
21.59
44.67
18.95
54,66
55.91
Sigl
1,429.11
63.57
56
70
59
20
57.00
SipLIP
VIT-S04D0M/14G
1,489.05
66:55
64.50
62
51
40
61.58
53.91
OpenCLIP
Con-NeXi-L@512
52.76
1,467.38
63.40
66.
63.17
69
34,90
29
70
$8.45
52.004
35
40
87
38,
79
30.67
56.21
54.24
55.91
Other
DINOv2
VIT-L/18
1,261.43
$3.06
McCo V3
VTT-B/166:224
39.51
1.175.34
41,70
53.31
55.15
65
33,40
28.
38.30
55.76
15.48
3.20
44.42
11.10
18.00
46.67
45.28
55.25
McCo v3
VIF-L/168 224
37.59
1075.
39.15
50.14
$3.65
65.49
34,60
27
55.44
17.28
3.00
14.21
11.70
14.67
11.58
15.38
41.12
MAE
VIT-L/168 224
41443
1.181.51
45.53
58.92
58. 75
64.65
35.00
29
20
57.12
16.85
3.10
14.67
11.74
18.67
19.67
53.04
56.83
IJEPA
21T-
1,175.70
48.00
27.60
11.40
Other
MiDaS 3.0 Supervised
VIT-L/168384
38.28
1,065.26
26
42.64
50.95
56.10
65.39
35.00
27 27.20
52.98
15.96
2.80
43.49
11:23
12.00
46.14
43.41 53.90
21T-1/168224
40.01
1,222.41
47.40
34,40
26.40
56.09
16.20
11.74
Table 12: All Benchmark Results for 1.2M Adapter Data 737K Instruction Tuning Data with Unfrozen vision model.
Vision Backbone
General
Knowledge
OCR & Chart
Vision Centric
Mail Vestd"
RCReesch
RealWoraOO
CV-Benay
Avtnge
MMDB
S6E3
6QA
30N
MMMU
ChaiQQ
VouVOA
DnvVO4
Method
Rockhone
Unfreeze
OpenA CLIP W1-D1433%
19
59:75
DINO VIT-L/1498336
1.373.14
63.13
19.04
3
$7.91
OpenAl CLIP VIT-L/1489336
57:44
1,585.34
68.68
771.47
63.96
77
36.09
(37
30
62.12
59.36
48.00
62.99
45.24
31.33
56.21
61.09
56.16
DINOv2
VIE-L/148336
42.40
1,366.65
61.62
69.72
63.68
68.
68.72 36.29
35.50
60.38
18.64
4.40
47.92
14.00
34,07
54.64
60.98 57.83
Table 13: All Benchmark Results for 1.2M Adapter Data 5M Instruction Tuning Data
from the last layer. The shape of each model's output differs depending on the resolution and patch size of each vision model. To resolve these differences, we interpolate the output of each model to a fixed number of tokens, using 576 tokens in our implementation, as described in Section 2.5. Our example code for interpolation can be seen below.
Eraeple in code interpolatti on
b, zun, coleens, diz image_featuure fewtures. chape
imago,
tunttoeess 1- selt. wellf..mmaoo.tee toleo.ttboo__em len:
target_h terget_v (nt. h x int cqutt tokens))
testares innge_reatture reaturees.veec(,, Viee(b, h, dim)
inage_festares festurws inage, fenturreetttteeppr Ca 3, 2). no..cont 2).contigumux(
image taage_festures i.tterpolateettaaee.eeoo testurree, size-(target_h, target_u), mode- 'bilssorr,,, cosp__oonnereFPllee image. srage_ image_ fextures permste 2, S, 1). l..ottggoune O fiatomni, 2)
We then concatenate the model outputs along the feature dimension and a larger MLP project the concatenated visual tokens into the LLM token space.
Full results on Model Ensemble We present all the benchmarks from the model ensemble ex- periment in Section 2.5 in Table 14. As discussed in Section I and Section 2.4, this comprehensive view of benchmarks provides better understanding of the model's performance compared to simply averaging across benchmarks. Adding a vision-only SSL model enhances the MLLM's performance in vision-centric benchmarks while maintaining strong capabilities in other categories.
G Data
G.1
Catalog of Visual Instruction Data
Here, we provide a comprehensive catalog of visual instruction datasets utilized in our study. The datasets are categorized based on their primary focus, including general conversation and VQA data, relltel data, counting data, knowledge-basse data, and language-only data. Table 15 summarizes these datascts and their respective references.
26

---

Videe Coarli
00
MMBSe
MMB
Kovetedte
SQK4
AHD
OCR
TooVQ4
DaoVQA
MMVP
CV/Boochh
Aovrrngy
0:17
Language- Supersiaed Avcruge
3,77
225
1.39
1.82
2.97
2.90
3.48
-0.1
0.05
0.92
2.16
9.67
19.32
13.99
1.92
8.39
3.05
6.21
6.63
4.74
1.55
OpenAl CLIP VITLL/14/33
4.62
2.16
235 0J4
1.92 4.54 0.81 0.22 0.29 12.71 3.90
14.43
56.00
50.00
1.22
9.00
3,68
99.88
4.25
7.14
6.50
SigLIP
SigLIF VITLL/166184
3.28
2.77
1.41
0.34
7,02
2.54
3.59
-0.5
0.27 0.00
3.52
5.54
15.20
6.57
0.45
2.07
4.44
16:08
2,39
3.67
0.65
30
OpETTUIP DensNXX-1.1112
7.51
5.56
2.10
3.56
4.78
-1.31
09
-2.00 0.68
-1.91
16100-4
05:41
16.51
0.87
5.06
8.10
5.49
EVA-LLP M2 VIFL/14g11
2.48
2.54
0.49
3.87
18.42
10.33
0..3844.5
5.85
DEN-CLIP VII-L/449224
-0.00
-0.00-0..25
-0.74
-1.33
1.27
-2.70 -1.01
2.61
0.48 5.50 4.0 2.42
-3.35
12.15
0.86
-1.19
33.55
-3.26-
15.00
-4.99
-1.15
2.00
Sdt Supersised Average
3.37
3.44
2.91
9.12
1.99
3.779
33.54
7.97
10:82
5.13
2.84
DINOG2 VIF-L/14111
4.88
5.00
5.99
0.39
-085
1.00
0.47
-2.10
-6.35
5.05
40.55
+4.0
11:47
11:90
9:48
01200
MAE VIT-L/1692244
3.25
4.29
232
4.30
4.91 5.34
0.68
-2.10 16
5.04
1.13
-1.42 -2.76
-6.06
0.31
44.55
8.12
16.59
15:42
4.11
3.65
0.00
66007
2.51
IEB VIEE11622
3,92
4.10
5.70
-2.65
4.83
9:34
0.81
-129
2.01 1:85 2.01 1.24 0.00
66.25
4.72
8.31
04
4.16
8.83
5.00
Other Acerage
-0.19
4..99
1.71
-2.70
-155 1:44
.0.5 1.09 0.19 -2445 -3.37 55006 2.10
0.16
-1.28
0.00
1:00
-2.05
-1.30
5.51
VIT-L/1653644
-342
0.55
-5.1
-1.55
1.87
6,556
3.29
0.85
0.30
SUyVI VITL/166/224
7.00
3.05
2.47
0.52
4.69
5.23
0.09
0.52
4037
-1.49
0.59
0:31
3.55
14.29
0.69
1.12
2.26
23:55
2.67
-3.66
6.77
Figure 13: Percentage (%) Change in Benchmark Performance (Frozen Unfrozen) Heatmap depicting the percentage change in performance across multiple benchmarks when visual encoders unfrozen compared to when they are kept frozen during fine-tuning. The color gradient indicates the magnitude of the performance change after unfreezing visual encoders-white indicates no change, red is a positive change, and blue negative change. Notably, unfreezing leads to significant gains in OCR Chart tasks for most Language- Language-Supervised Models, as reflected by the deep red cells. ConyNeXt, in particular, shows substantial improvements, demonstrating the benefits of updating this visual encoder during fine-tuning.
Vision Backhone
General
Knowledge
OCR & Chart
Vision Centric
Real WorMMA
Benchh To
Average
XME2
MMB
SEED
60
S0N
MMMM
MothVista
CbaiQA
TenVVO
DAc
MMVP
Method
DINOI
51.61
1,432.02
65.09
68.82
69
43,00
60:40
30,00
53.99
S.g..I--DDND244CmmWeee Next
$4.52
1,503.51
63.83
67.97
63.95
70.40
35.99
29.
30
61.69
48.20
36
90
64.97
45.53
34.67
58.69
55.74
60.
(51.71
1.179.46
63.32
67.63
61.04
71.39
35.49
29,10
59.88
50.24
39.60
61.55
46.12
32.67
58.95
58.54
60.12
Sim
54.53
64.60
50.64
64.53
32.00
57.91
CC.IP.CoomNNst
$4.45
1,511:08
63.83
67.41
63.63
70,80
35.09
30.40
59.91
51.32
35.00
64
47.8%
33.33
57.25
56.32 59.08
53.78
1.40064
63.57
67.79
63.63
71.34
34,8 80
30.20
61.0
49.32
70
64.05
45.83
30.00
56.21
08 51.33
SigL. SggLUPP-CLLP--CoNNNtLL
1,507.24
28.67
55.92
Table 14: All Benchmark Results for Model Ensemble with 1.2M Adapter Data + 737K Instruction Tuning Data. Here, "SigLIP" ViT-S0400M/14@:366 "DINOv2" ViT-L/14@518, "ConvNext" = OpenCLIP ConyNeXI-XXI.@@ 1024, and "CLIP" = OpenAl CLIP ViT-L/14@336.
G.2 Additional System Prompts used in Cambrian Data
Here, we investigate phenomenon we term the "answer machine phenomenon". We observe that well-trained MLLM may excel at VQA benchmarks, but lack basic conversational abilities and default to outputting short, curt responses (see examples in Fig. 14). This discrepancy arises because benchmark questions typically require responses that are limited to single option, choice, or word- diverging from the more broad and realistic use cases of MI..Mss Similar phenomena have been discussed in other LLM studies [114, 151, 155].
27

---

Category
Datasets
General Conversation
LVIS-Instruct44 [131], SketchyVVQ [129], OODVQA [129], VizWiz [51],
VQA Data
ALLaVA [25], IDK [22], Q-Instract [135], LAION GPT-4V [70], Hateful-
Memes [66], Visual7W [1591, Visualmre [123], AlfWorld [117], LNQA [108],
LLaVA [82]. ShareGPT [27]. VQAv2 [50], GQA [57]. OKVQA [95], A-
OKVQA [1] ResCoco [140], VisualGenome [69], GPT-4V recorded chat
OCR Related Data
LLAVAR [149), ChartQA [96], DocVQA 197], DVQA [61], ArxivQA [76],
A12D [65], ScreenQA [56]. SynthDog [67]. IconQA [89], WTQ [106],
WikiSQL [153], FinQA [31], HiTab [32], TAT-QA [158], TabMWP [88],
Chart2Text [62], VisText [124]] InfoVQA [16], ST-VQA [17], Rendered-
Text [134], OCRVQA [100], TextCaps [119), ShareGPTOCRData [27]
Counting Data
TallyQA [J.. CLEVR [59]
Knowledge-Baid Data
Code: Design2Code [1188] WehSight [71], Datikz [14]
Math: MathVision [132], Geol 70K [42], TQA [5]. Inter-GPS [90],
RAVEN [148]. Geom Verse [64]
Science: ScienceQA [91]. PathVQA [53]
Language Only Data
Dolly [36], MathInstructt [142], WizardCoder [93], OrcaMath [99], OpenCodeIn-
terpreter [152], OpenOrca [78]
Table 15: Visual Inatruction-Tuningg Data Catalog
We suspect that this issue stems from instruction tuning data containing an excessive number of short-response VQA tasks, leading to catastrophic forgetting in LLMs. To address this, we incorporate additional system prompts during training. We append prompts such as "Answer the question using a single word or phrase." before questions that generate a single word or phrase in the response. Full details of the system prompts used are provided in Appendix G.2. After integrating these system prompts, we observe that while the model's benchmark performance remains unchanged, its conversational ability improves dramatically. For example, in Fig. 14, models with system prompts produce longer and more engaging responses while answering questions correctly. The system prompts also enhance the model's 's performance on reasoning-related tasks, such as math problems, by encouraging a chain of thoughts [133] followed by the answer.
This underscores the necessity of developing evaluation protocols like the Chatbot Arena [34] for MLLMs, despite the challenges in collecting large-scale, real-world interaction data. While performing well on benchmarks is important, it is equally crucial to ensure the model can engage meaningful and natural interactions. The overall user experience and the model's conversational model that excels in benchmarks but fails to converse effectively cannot
abilities are paramount, meet the needs of practical applications.
As our Cambrian data includes instractions/questions and responses of different types and formats (e.g., Short response with a single word regular response complete sentence), it is important to specify the required response format in the instruction prompt to avoid ambiguity and possible conflicts. Some of the datasets already include such prompts and we add proper prompts for the remaining datasets. The detailed response formatting prompts we additionally add are listed in Table 16.
G.3 Data Engine
Comprehensive Implementation Details of the Data Engine
The data engine is designed to generate instruction tuning data for knowledge-brat fields, where previous works rarely covers and MLLMs not reliable distill for from. The data engine takes in a given field, such as "Physics", utilizing reliable web sources like Wikipedia. Below are the various stages involved in the process. We also visualize this process in Fig. 15:
Stage Topic Generation: We start by compiling a list of fields and subfields and subsequently generate topics for each field using Large Language Model (LLM), such GPT-4. In this stage,
28

---

prompt
prompt
Hooe many people in this imzge?
Unced on this chart, whatl the most popular tea?
people trage.
Usaed black
tar the sigratt calas viluus acrves als years claplayed.
image please.
anound lendmaric pieturn, should
York
The Inagr renunn lo bathrra clmucied ewimtyy oily on gare imm liette uette sall ire petre stad clee
This dity, and the ar wit Jansuur, Lsage
stul suntte around
wighte
you
eccivon tim oe luttig withing parte etutte and tn aistrtiad batttinn,, divm matton
certil soire suppers
question slep by stop.
image?
trape unarad becaus chanch destin
ratae rautioe Clearttedy new
on the pe
use "palie art
(1ert/02)
menettin dyte
soul settiin liiee. cemerc
oppriirr
Amite Soecifiietts the ne udiccis twer are destgne to
tkts
(1-4
Fiasiby sustrct faw with cass
loce witee mails trott inttil culle be to and plantt or ortivet clmant the ballding.
233
commrrty
Figure 14: Incorporating System Prompt in Instruction Tuning Data alleviates the "Answer Machine Phenomenon" By adding system prompts in Cambrian-7M the model exhibits better chat ability while retaining strong question answering abilitics. The model without system prompts requires additional prompting clicit longer responses.
Web Knowledge
Search Engine
Image
Filtered Web Data
VQA Data
Fields
LLM
Topics
Relevant Text
LLM
Q&A
Figure 15: Targeted Internet Data Collection Engine. We build a targeted internet data engine to collect high-quality and large-scale multimodal instruction tuning data for domains like knowledge.
we processed 30 fields, resulting in 3660 topics. We then post-process the output of LLMs into json formats. For example, the topic data for Physics looks like below.
Physicn
Classical Mechanica":
"Wecton's 'a Lave of Mation",
"Oosservation of Energy"
"Harmonic Notion',
"Rotational Dynasica"
"Graritation and Orbite", "Fluid
"Elasticity Plasticity", "Priction".
"Navex and Bound", "Velocity Acceleratson', "Angelar Monestun", "Statica and Rguilibriun', "Kinemstice of Particles'
"Bycamica Systara Particles",
"Gentripetal Perce and Acceleration" "Lagrangian and ioriltonian Kechanica", Theory',
"Equations one of Notioon
].
"llectrosgneeienns "Electric
Potentinl",
"Ganso's Las"
"Capasitance Dielectrice", Besistance",
29

---

Index
Response formatting prompts
Answer the question using single word or phrase.
2
Answer the question using a single number or phrase.
3
Answer with the option's ss letter from the given choices directly,
4
Give the short answer directly.
5
Answer the question using single word phrase.
6
When the provided information is insufficient, respond with <no answer>. Directly provide the HTML code.
7
8
First show your reasoning process and then give the final answer.
9
When the provided information is insufficient, respond with 'Unanswerable'. Answer the question using single word or phrase.
10
Answer with the letter.
Dataset
Prompts added
Sketchy VQA
OODVQA
VizWiz
9
Q-Instruct
1,3
ChartQA
2
VQA
DVQA
A12D
ScreenQA
I,
CLEVR
TallyQA
PathVQA
MathInstruct
Design2Code
IconQA
10
HiTab
WTQ
WikiSQL
Inter-GPS
Visual7W
3
TQA
RAVEN
Table 16: Response formatting prompts for Cambrian Data
Circuita", "Eagottic Fields Magustic Forces", 'Axporo's i Las", "Faraday's Lav of Induction", "Inductance", "Altatntinng "Electroragnatic Havea",
Circuits",
"Faxvoll's Equations",
"Electrorageelic Radiation", "Optica Light",
"Quastua Electrodynazics",
"Special Theory of Belativity Ieplication', "Kagaatostatica"
"Electrostatica" "Bloohactroragaatiia""
Stage 2 Filtering Web Data: For each generated topic, we utilize search engine APIs fetch relevant high-quality web pages. For each topic, we query for relevant links. Thus, we get 36,600 webpages post this stage. Here an example of the data retrieved for the topic "Electric Field and Electric Potential":
"Elestric Elastric Potentinl
"nospe://en, wilipedia, wttp///eii/llleerr....tt
postnttall
"hespe://en. sikipedia www.ppeis..org/ess///llll.......
"botpe:/ //cp xikipodia, wtt//ci///Eeeetrreecpcoooo htwp://etttttttttttllleeeeeoe
30

---

"Mecpa: wikigedia, http////////////////////////////
"haspe: / pwd /ttpla...////ii//////eeee/ae/
"Iasspa //en. wikipedia. wwtpeee.t////aaaeeeeoo
"hespa //n wikipedia wrg/vizi// /Electric, _ipecl_.xoennt.,
"haapa wik.kkkiiiii wtw.di..g///eeae../ www. /Mtggtttiiii io..ettttttetorrrrooooooo so_seccor_poteniial
"hespa wikigedin /tg//a/// /ale///rriio- fecerri-oieddddaae acremincn,, "nespa: wwkipodia azg/ /ee//eeerrio_euu
Stage 3 Parsing: this stage, we parse each web page to extract image-caption-exx tuples. We
aim identify the blocks containing image, the image's caption, and relevant textual content. Below is an example of the parsed data for the same topic, "Electric Field and Electric Potential".
'Electric Field and Electric Potential",
"section": "Elestrontatica"
"Lezt": "An electric potential paint in static electric Field E. given the 1ine intagral where C arbitrary path from: some rufersnce point 2; uniqualy deternined up cocatant... gemeralization electric potential this described the sectioon Generalization electrotynanicon
"Irages":
"ar2": https://upload wikinedia, otg/vikipedia/ccomee wwtp://aaaccoo//eeeeeaaeeaeeeeeo http:///ww...aeee//u//////////// www..uu.....tttaaaaaaoeeee.e pugg/
102pz pept..puu, www.:tt.pp.eaiilleottteeeeo .ug.. pag,
"caption": "Electric potential of separate pocitive and negative point charges chown as color rango froz nagenta aquipotancial lines. Electric field 1inea positive
(+), through yellow eyen Circular
charge and enter the eagative charge."
httpe://upload vikinedsa, http:///seepeeeaa/ccoooo/o/eeea wwwpreett_ccrpppppppceoce wtw.p//www..ooooootttaalaao avg/ 260p WPPt, chargerr.ggg pttppersppo__nmms_..o mimes_ooesstin woreetiilllooooooooooooooooo roonoour evg. pig,
"caption": pobential vicinity apposita point charges."
}
"link"
"ritlet:
"Electric potestinl",
"fieldt: "Physiea",
"aubrield":
"Electronagwtiaa".
"sopict: "Electric Field and Electric Potential"
Stage 4 Data Generation: We generate dataset in this stage, ensuring high quality. We first filter out data samples with fewer than 50 words in the text. Then, instead downloading images directly from the links retrieved during web parsing, download high-resolution images from the original sources. We then convert formats like SVG or GIF into a common standardized format, PNG.
Question-Answer pairs are generated by using LLM such as GPT-3.5 from the image metadata, caption, and contextual text. These Q& pairs and the image form our VQA dataset. We generated 165k data samples. Here is an example of the generated data:
"id": "96232.png"
" "image_id": 5550px-Magnetic_. Mag.attte.eettorrooraaaaaaaaaaaa "image_uri" "...", "text":
"caption": "Representing the Coulomb gauge magnetic vector potential A, magnetic flux density B, and current density J fields around a toroidal inductor of circular cross section. Thicker lines indicate field lines of higher average intensity. Circles in the cross section of the core represent the B-field coming out the picture, plus signs represent B-field going into the picture. \u2207 \u22c5
31

---

6.0
"Question": "What do the thicker lines signify?", Figure 16: Dataset Image Id: 96232. png
General
Knowledge
OCR & Chart
Ceiion-Centric
Real WorldO
CV-Bench 3D
MathVista Vista
OCRBench
Ce-Bench
Average
MMMU
ChanQA
TexIVQA
DocVQA
MMVP
MME
MMB
SEED
GOA
SQA
A12D
Data
150k 4013k
53.74
1,1223
67.0
61.
73.4
34.3
62.4
44.6
39.1
38.5
30.0
55.2
61.98
34.7
= 250k 5218k
54.31
1,475.9
67.3
69.2
61.6
73.4
35.9
34.5
62.4
46.5
36,9
59.2
38.6
32.0
56.6
63.68
57.5
350k 5883k
54.27
1,4119
66.2
68.9
61.6
73.8
36.4
32.8
62.5
46.8
38.3
59.3
39.3
31.3
54.9
62.68
60.4
450k 6383k
54.15
1,34.1
67.6
66.3
61.9
72.9
35.1
36.9
63.8
45.8
38.6
58.4 39.4
28.0
53.6 64.60 56.8
Table 17: All Benchmark Results for Data Balancing Experiments
has been assumed",
"section"; "Depiction of the A-field"
"Answer": "The thicker lines in the image represent paths higher average intensity in the magnetic vector potential field. Shorter paths have higher intensity, ensuring that the path integral remains constant. These lines are drawn to visually convey the general behavior and intensity distribution of the A-field around the toroidal inductor" }
This data engine designed for scalability and efficiency and is capable of handling extensive data generation tasks using multithreading techniques.
G.4 Full results on data curation experiment
Data Balance via Fitlering / As discussed in Section 4.2, if left unfiltered, the data pool is dominated by noisy, unbalanced data sources such as CLEVR and DVQA, leading to pronounced exponential tails. However, we apply different values to filter data from each source, the exponential tails become less pronounced, resulting in more balanced dataset. We also present all the results in Table 17. É value 250k has the highest average across all benchmarks; 250k and 350k also have the highest performance across many individual benchmarks.
Here, plot the cumulative Fig. 17, we see this intermediate threshold prevents explosive heavy tail.
of counts for entries sorted by counts from tail head. From
Data Ratio Studies We present the full results of our data ratio study in Table 18. The table highlights the importance finding an optimal data ratio that balances different aspects MLLM. Experiment 5 achieves well-rounded performance with its selected data ratio.
32

---

Cumulative
Data Mix lundittene)
Data Mix (1-9%))
Data (t-356))
Data 4ix (9-2)))
Data Mix (b-106)
Cumullatve Sum Go)
4m
2m
10
20
30
60
70
Sorted Entries
Figure 17: Data Balancing via Applying Thresholds on Data Sources. Applying threshold alleviates the
exponential Cambrian-10m
General
Knowledge
OCR & Chart
Vision-Centric
RealworldQA
Average
MME
MMB
SEED
GQA
SQA
MMMU
MahhVista
A12D
ChartQA
OCRBench
TeRVQA
Doe
CV-Bench
CV-Bench
expl
47.49
1,309.10
58.00 60.10
54.00
72.40 34 34.80
31.20
59.10
34,20 34.50
54.20
33.00
13.30
60
57.40
50.58
exp2
47.78
1.351.70
60.30
61.20
55.40
80
35.20
29.50 59.10
31.20
33.40
54.20
30.50
15.30
48.20
58.40
52.25
exp3
48.28
1,299.53
60.56
61.79
55.74
72.04
34.90
32.10 59.40
33.
20 33,90
54
31.90
21.30
48.60
58.52
49.41
exp4
47.47
1,288.
98 58.16 61.47
55.00
71.05
17 28.20 58.50
33.72 34.50
55.07 31.69
20.66
47.06
56.30 46.58
exp5
48.96
1,363.26 60.48 63.18 55.92
70.35
35.70
31.40
57.19
32.88 34.60
54.74 32.10
22.70
47.30
(5.83 57.75
Table 18: All Benchmark Results for Data Ratio Experiments with fixed 1350k data
G.5
737K and 5M Mixes
0.7M For the 0.7M data we used Section 2.4, We add a small number of OCR and chart data to LLaVA 665K, specifically 15,501 A12D, 14,999 Doc VQA, and 13,000 DVQA data points. This results in a 737K mix, which covers all categories in training MLLMs. This data mix allows us to study visual representations efficiently.
5.0M For the 5M data mixes we use in Section 2.4, we apply data filtering discussed in Section 4.2 and apply 1=150k on all multimodal instruction data in Cambrian- 1OM.
G.6 Test Image Leakage in Visual Instruction Training Data
One potential concern with our targeted data engine (Section 4.1) is that instruction-tuning data collected from the open web could introduce data leakage. To address this, we systematically analyze the extent of direct image matches between our training data and our test sets. Using difference hashing (dHash) [19], we compute hashes for all images in the training data and test sets. We then compare these hash sets to determine how many test images overlap with our training data, reporting the number of collisions Table 19.
Across all fifteen datasets, our targeted data engine finds only 32 test images in total, amounting to just 0.06% of the test data. This low overlap percentage dispels concerns that our data engine inadvertently targets specific test sets. When analyzing the full Cambrian10M dataset-which is 15x larger than i.LaVA665kk-we observe only 6x more matching test images (7,244 compared to 1,034 in L.LVVA-655k. This discrepancy suggests that Cambrian 10M's scale does not inherently result in excessive overlap with test sets. Instead, any overlap likely arises from the natural reuse of training images across benchmark datasets rather than targeted duplication.
33

---

Category
Test Set
# Images
Data Eng.
Cambrian10M
LLaVA-665k
General
MME
2,374
0 (0.00%)
332 (13.98%)
82 (3.45%)
MMB
4.377
7 (0.16%)
1,122 (25.63%)
533 (12.18%)
SEED
17,990
6 (0.03%)
26 (0.14%)
0 (0.00%)
GQA
398
(0.00%)
(0.25%)
0 (0.00%)
Knowledge
SQA!
2,017
0 (0.00%)
1,263 (62.62%)
0 (0.00%)
MMMU
900
(0.11%)
3 (0.33%)
0 (0.00%)
Math Vista
1,000
2 (0.20%)
259 (25.909) (0%%
15 (1.50%)
A12D
3,088
0 (0.00%)
1,458 (47.22%)
0 (0.00%)
OCR & Chart
ChartQA
2,500
14 (0.56%)
670 (26.80%)
0 (0.00%)
OCRBench
1,000
(0.00%)
177 (17.70%)
59 (5.90%)
TextVQA
5,000
2 (0.04%)
1,122 (22.44%)
9 (0.18%)
DOA
5,188
(0.00%)
53 (1.02%)
0 (0.00%)
Vision-Centric
MMVP
300
0 (0.00%)
0 (0.00%)
0 (0.00%)
Realdo
765
(0.00%)
0 (0.00%)
0 (0.00%)
CV-Bench 2,638
0 (0.00%)
758 (28.73%)
336 (12.74%)
Total
49,535
32 (0.06%)
7,244 (14.62%) 1,034 (2.07%)
Table 19: Number of leaked test set images. Using image hashing, we assess the overlap of test images across three training datasets: Cambrian10M Data Engine 161k subset ("Data Eng.") "). CambrianIOM, and of matching images and percentage of overlap for each training set in blue. Our Data Engine finds a neglible 0.06% of test images, dispelling any concerns that it is targeting the test sets. The full Cambrian 10M training set contains 7.244 test images, whereas LLaVA-665k contains 1,034. Despite being a 15x larger dataset, Cambrian 10M only has 6x more overlapping images. Such overlap inevitable since many test sets validation images from standard benchmarks (like COCO). is worth highlighting: although exact image matches are found, this does not mean that exact image-questiee pairs have been found. Unlike in prior unimodal paradigms of computer vision research, in the multimodal setting, a single data point is composed of an image-text (question) pair, not just the image itself. Thus, seeing test image during training is not equivalent "training on the test set" so long as the training image does not have the same text pair as the test data point.
We list the number of images in each test set, as well as the number
It is important to emphasize that while some exact image matches are found, this does not imply that the exact image-question pairs have been encountered during training. Unlike traditional unimodal computer vision research, where an image alone constitutes data point, the multimodal paradigm treats each image-text (question-answer) pair as unique. Consequently, seeing a test image during training is not equivalent to "training on the test set" long the associated text (question-answer) annwer)) pairs differ. This distinction ensures that Cambrian I0M respects the integrity of test evaluations, even in cases where images might appear in both training and test sets.
We encourage future research exploring the impact of image-only leakage on the performance of MLLMs. Understanding this influence may yield insights into the boundaries of model generalization and guide future best practices for dataset construction in multimodal learning.
G.7 Broader Impacts
We conducted a preliminary analysis of the Cambrian dataset, focusing on the distribution male, female, and neutral pronouns. Our findings show the following distribution: 38.35% male pronouns, 17.99% female pronouns, and 43.66% neutral pronouns.
We recognize that training models on biased data can perpetuate these biases. Addressing bias by artificially modifying data distributtion--sude as through rebalancing or applying fairness con- straints- can help mitigate this issue, but it also presents challenges. These include the potential loss of generalization and the risk of introducing new biases. Additionally, identifying and mitigating bias in Multimodal Large Language Models (MLLMs)) is particularly complex, given the interaction
34

---

between different data modalities. We believe that openness in model development and data curation will accelerate research aimed at understanding and mitigating these potential harms.
H Implementation Details
Cambrian Models For our final Cambrian models, we use 2.5M adapter data which is comprised of 1. 1.2M captioning data from shareGPT4V [27] and 1.2M captioning data used in MiniGemini [77].
SVA We provide here ablation studies of SVA module.
D OCR Chart
G OCR & Chart
Multi-agg OCR & Chart
52.1
52.4
No
52.4
52.4
2
52.6
Yes
53.3
52.8
53.1
(a) # layers
(b) groups
Multi-layer aggregation
Table 20: Ablations hyperparameter choices for SVA. Enlarging the model capacity of the SVA module can further improve the performance.
We further conduct ablation experiments using OpenAI CLIP ViT-L/14 OpenCLIP ConvNeXt- L@ 1024 as our base model combination. We focus on the OCR & chart categories to assess the impact on high-resolution visual understanding. The results show that increasing capacity via D or G improves performance and that allowing vision aggregation across multiple layers by adding cross-attention layers within the LLM also enhances performance.
Compared with other spatial-based connectors like C/D-Abstractor [21] which are designed for single vision feature maps, our SVA module can dynamically combine visual features from multiple vision models with varying resolutions. Besides, our spatial inductive bias in SVA can better compress spatial information compared with such methods. To isolate the effect of spatial inductive bias, we consider the case of token reduction using a single vision encoder. Specifically, we use OpenAl CLIP ViT-L as the vision model and compress its original 576 tokens 36 tokens using SVA module and other connectors. We compare our SVA module with three baselines: 1) Direct interpolation + MLP, 2) C-Abstractor [21], and DPv2 Projector [35] (similar C-Abstractor but more lightweight). For fair comparisons, we do not include multi-layer aggregation inside the LLM for our SVA baseline, and the results are shown in Table 21. Compared with the simple MLP baseline, C-Abstractor performs better on General and Vision-Centria tasks but inferior on Knowledge and OCR & Chart tasks. LDPv2 performs similarly the MLP baseline. Our SVA consistently demonstrates superior performance across all categories, especially in OCR & Chart and Vision-Centro tasks, demonstrating its effectiveness in information compression.
Method
General
Knowledge
OCR & Chart
Vision-Centric
Interpolate + MLP
63.4
43.8
28.1
43.7
C-Abstractor [2]]
64.4
42.8
26.1
44.3
I.Pv2 [35]
62.5
43.9
28.7
43.9
SVA
65.5
44.5
31.4
46.9
Table 21: Comparison between SVA and other spatial-based connectors vision token compres-
sion. The SVA module with spatial inductive bias more effectively compresses the vision information.
We introduce learnable km x km positional encodings in the vision features when km > 1. Besides, during -rossaattention, the query is augmented with a global feature obtained by global pooling over the vision features, which is concatenated with qı to better guide the aggregation process. In our experiments, the feature maps of all vision encoders except for ConvNext are interpolated to 576 (m,- = for I, 24). For ConvNext, we first interpolate the feature maps from its 4 stages to x 96 (mg = for L = 24) and then channel-wise concatenate them to form its final vision feature map similar to [77].
35

---

For experiments in Section 3, we set = G = 1 and add cross- aoossatttnnioon layers between the layers of LLM with stride equal For our final Cambrian models, we set D 3, G 1 and insert multiple cross-attention layers in LLM considering the tradeoff between performance and efficiency. For Cambrian-8B, Cambrian-131, and Cambrian-34B, the strides of cross-attention layers inside the LLM are 3, 4, and respectively.
To study the importance of visual features from different vision models to different image categories, we further investigate the attention score distribution in our SVA module. We evaluate our Cambrian- 8b model on GQA, VQA, and ScienceQA (representing three different benchmark categories), and the attention distribution results are shown in Table 22. We can see that on real-world images (GQA), the contribution different vision models is relatively uniform, in part due the similar characteristies of SigLIP and CLIP. On document-tye and often high-resolution, the influence of SigI.P increases and that of ConvNext greatly increases to aid in high-resolution information processing. For scientific images (ScienceQA) composed of illustrations and diagrams about different science categories, the contribution of SigLIP further increased while the portion of DINOv2 decreases compared to GQA.
images (DocVQA) which are text-heavy
Model
GQA
Doc ScienceQA
SigLIP
29.7%
31.1%
35.2%
CLIP
18.5%
13.4%
16.3%
DINOV2
24.1%
11.0%
17.6%
ConvNext
27.7%
44.5%
30.9%
Table 22: Attention distribution studies. The attention distribution among different vision encoders varies with different image categories.
Unfreezing While unfreezing is largely beneficial (Section 2.3 and Fig. 13), it has a significant speed drawback. Given fixed computational resources, unfreezing visual encoders slows down the fine-tuning process by approximately 50-55%. For initial explorations or when computational overhead is concern, leaving the visual encoders frozen can be a practical strategy. This allows for quicker iterations and tuning, especially during early research phases, while still providing valuable insights. Ultimately, unfreezing is recommended for achieving the best performance once the setup has been optimized.
I Evaluation Details
L1 System Prompts Used in Evaluation
To ensure the reproduction of results, also include the system prompts used in this work. The system prompts for our models can be found in Table 25. Additionally, we release the prompts used while evaluating our models the various benchmarks Table 26. We hope this sets precedent for future research to improve the reproducibility of benchmark results.
I.2 Ablation Study on Fuzzy Matching Vs LLM Judgement
We use fuzzy matching evaluate responses in benchmarks, since MILMs answer questions with auxillary phrases. To study the effectiveness of our fuzzy matching, we compare our model accuracy through fuzzy matching with the model accuracy obtained when we use LLM grader.
The LLM grader is sensitive to the prompt given to it while grading, and we prompt the LLM (we use OpenAl GPT-3.5-turf 5-tubb and GPT-4-turbo as our graders) with few shot grading examples, which notice significantly improves grading accuracy. An example of such prompt is given below.
LLM Grader Prompt
You are a reliable grader. Reply with only either the following 2 words: CORRECT or INCORRECT.
36

---

Backbone
Data
Adupter wd
Instruction
Tuning
Experiment
LLM
Vision
Adapter
Instruction Tuning
Ir
bs
Ir
wd
bs
vision
BM Adaptere 737K II
Vicuna-1.53 1.576
OpenAT CLIP V11-L/1401366
757k
2e-5
512
Vicuns-
OpenAI
1.2M Adkpta=737K IT
Vicuna-1 1.5-B
OpenAl CLIP VIF-L/141336
1.2M
737k
le-3
0 512
2x-5
0
512
Unfreese Vision
Vieuna- 15778
OpenAl CLIP VIT-L/140336
1.2M
737k
les
512
2e-5
512
To-5
Ensemble
Data Balance
Vicuma11.54 1.5-B
OpenAl CLIP VIT-L/144:366
Mix Based three
2e-5
512
Data Ratio
Vicuna-1 1.57B
OpenAl CLIP VIF-L/146:336
1350k Based on Ratio
2e-5
0
512
LLVV 665K
Vicuns-1..-77
OpenAl CLIP VIT-L/140:156
IJaVA 663K
26-5
512
(Dam)
OpenAl
Cambriin-7M (Dana)
Vicuna-1.5-88 1577
OpenAl CLIP V11-1/1463336
Cambrian-7M
26-5
d
512
Cumbrian- 1-8B
Llama-3-Im-SB SVA with encoders"
2.5M
Cambrian77 7M
lc-1
512
2c-5
512
Cambrian-
encoders
Cambrian- 3-34B
Hermes-2 Y::4BB
SVA with encoders"
2.5M
Cambtian
le-4
2e-5
1024
Table 23: Implementation details and hyperparameters for all experiments. "4 encoders are: OpenAl CLIP ViT-L/14 336, SigLIP V DINOv2 ViT-L/14@518, Open- CLIP ConwNeXt-XXL 1024
You will be given 'answer' and 'gt_answer' (ground truth answer) ,nd you must reply with either CORRECT or INCORRECT based on the response. Tolerate 0.05 relative error for numerical answers. answer: 25
gt_answer:
evaluation: INCORRECT
answer: Yes
gt_answer: Yes
evaluation: CORRECT
answer: 80
gt_answer: 80
evaluation: CORRECT
answer: Ireland
gt_answer: Italy
evaluation: INCORRECT
answer: UK
gt_answer: UK
evaluation: CORRECT
answer: 2019
gt_answer: 2011
evaluation: INCORRECT
answer: (answer)
st_answer: {gt_ansver)
evaluation:
We conduct an ablation study on the benchmarks that require fuzzy matching and present the results in Table 24. We discover that fuzzy matching provides reliable results compared an LLM grader. We recommend using a more capable model (such as GPT-4-urbo) for grading benchmarks that have more subjective responses (such as numbers and words).
37

---

RealWorIdQA
OCRBench
SEED
GQA
SQA!
MMMU
A12D
ChariQA
TexvQQ
MMVP
Method
Eval
Cambrian-1 8B
Fuzzy Matching
74.7
64.6 80.4
42.7
73.0
73.3
62.4
71.7
51.3
64.2
Cambrian-1 8B
GPT3.5 Grading
78.4
65.8
82.0
38.9
78.1
71.2
67.0
69.2
49.3
63.5
A
+3.7 +1.2 +1.6 -3.8
45.1
221
+4.6
2.5
-20
-0.7
Cambrian-1 I 13B Fuzzy Matching
74.7 64.6 80.4 40.4
73.0
73.3
62.4
71.7
51.3
64.2
Cambrian-1 13B
GPT3.5 Grading
77.3 64.7 81.3 37.2
78.2
71.4
67.1
75.6
46.0
64.3
A
+2.9 +0.4 +2.0 -3.2 +4.6
-244
+5.2
+2.8
+4.7 +1.3
Table 24: Comparison between Fuzzy Matching Accuracy and LLM Judged Accuracy. Fuzzy matching and LLM referee yield similar accuracies for the benchmarks that require matching.
LLM
Backbone
System Prompt
Vicuna 1.5
chat between curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's 's questions.
LLAMA-3 8B
You Cambrian, a highly intelligent multimodal AI trained NYU Vision X. As multimodal AI, you have ability to process and analyze images. Whenever an image present the conversation, very carefully examine and consider its content when formulating your response. You should give concise responses very simple questions, but provide thorough responses
complex and open-ended questions.
Nous- Yi 34B
You are Cambrian, a highly intelligent multimodal AT trained by NYU Vision X. As a multimodal AI, you have the ability to process and analyze images. Whenever image is present in the conversation, very carefully examine and consider its content when formulating your response. You should give concise responses very simple questions, but provide thorough responses more complex and open-endee questions.
Table 25: LLM Backbone System Prompts
38

---

Table 26: Listing the prompts used in the evaluation of each benchmark
Benchmark
Prompt
Example
A12D
'nAnswer with the option's letter from the given choices di- rectly.
USER: <image>lnwhich of these define dairy itemln(A) cln(B) Din(C) bin(D) alnAnswer with the option's letter from the given choices directly. ASSISTANT:
ChartQA
\nAnswer the ques- tion using a single number or phrase.
USER: <image>InHow many food item is shown in the bar graph?InAnswer the question using a single number or phrase.
ASSISTANT:
DocVQA
\nGive the short an- swer directly.
USER: <image>inWhat is the dividend payout in 20127nnivve the short answer directly. ASSISTANT:
GQA
\nAnswer the ques- tion using single word or phrase.
USER: <image>InIs overcast" the question using single word or phrase. ASSISTANT:
Vista
inFirst show your rea- soning process and
USER: <image>inwhat is the total volume of the measuring cup?InFirst show your reasoning process and then give the final answer. ASSISTANT:
then give the final an- swer.
MM-Bench
InAnswer with the option's letter from the given choices di- rectly.
USER: <image>inWhich of the following was a dependent variable in this experiment?'n(A) cocoon'n(B) chrysalisIn(C) nanin(D) nnswer with the option's letter from the given choices directly. ASSISTANT:
EN
MME
inPlease answer the question using a sin- gle word or phrase. AnAnwwr with the option's letter from the given choices di- rectly.
USER: <image>Ans python code shown in the picture? Please answer yes or no. woo.nnn the question using a single word or phrase. ASSISTANT:
MMMU
USER: <image>inWhat causes these unusual formations on Mountain papaya? Options:InA. Abiotic'nB. ConfusedInC. Bi- oticInD. Answer with the option's 's letter from the given choices directly. ASSISTANT:
MMVP
'nAnswer with the option's letter from the given choices di- rectly.
USER: <image>AmAre the butterfly's wings closer to being open or closed? Options:\n(a) Open'n(b) Closedhn with the option's 's letter from the given choices directly. ASSISTANT:
OCR Bench
'nGive the short an- swer directly. 'nAnswer the ques- tion using single word or phrase.
USER: <image>Inwhat is written in the image? the short answer directly. ASSISTANT:
Real World
USER: <image>\nIn which direction is the front wheel of the car on the right side facing"AnnnA. Left'nB. StraightInC.
QA
RightlnAnsser the question using a single word or phrase. AS- SISTANT:
SQA-I
InAnswer with the option's letter from the given choices di- rectly.
USER: <image>inWhat is the name of the colony snown?'nA. MarylandinB. New HampshireinC. Rhode IslandanD. Ver- mont'nAnswer with the option's letter from the given choices directly. ASSISTANT:
SEED-I
\nAnswer with the option's letter from the given choices di- rectly.
USER: <image>InHow many towels are in the image? Op- tions:AnA. OneYnBB TwolnC. ThreeinD.. Fouren with the option's 's letter from the given choices directly. ASSISTANT:
TQX
InAnswer the ques- tion using a single word or phrase.
USER: <image>inwhat is the time?"nReefrrenee OCR tokens: N. u, go'nAnswer the question using a single word or phrase.
ASSISTANT:
ADE
AnAnswe with the option's letter from the given choices di- rectly.
USER: <image>innCnssdeeing the relative positions of the cush- ion and the sofa in the image provided, where is the cushion lo- cated with respect to the sofa? Select from the following choices. \n(A) rightin(B) leftnnAnswer with the option's 's letter from the given choices directly. ASSISTANT:
39

---

General
Knowledge
OCR Chart
Vision- Centic
603
775
262563
717
61A
713
213
72.
(27
70.0
554
12.9
669
68
$1.5
Mats-Carirs-122
LLaVA-NoTT
Cambrian-
GFT-4V Pirbrmana
Figure 18: Comparison of model average performances on category. Cambrian- outperforms other open-source models across all sizes. The lead is especially large on OCR & Chart and Vision-Centric benchmarks, highlighting the advantage of our vision-centric design.
Benchmark
Prompt
Example
COCO
\nAnswer with the op- tion's letter from the given choices directly. 'nAnswer with the option's letter from the given choices di- rectly.
USER: <image>inloow many trains are image? Select from the following choices. \n(A) 3ln(B) 0 An(C)) 1 \n(D) 2 \n(E) 4aaAnswer with the option's 's letter from the given choices directly. ASSISTANT:
Omni3D
USER: <imtge>-nEsstmaaee the real-world distances between ob- jects in this image. Which object is closer to the traffic cone (highlighted by red box). the motorcycle (highlighted by a blue box) or the bus (highlighted by green box)?)n(A) motor- cyclein(B) bus'nAnswer with the option's 's letter from the given choices directly. ASSISTANT:
More results of Cambrian-1 Model
In Fig. 18, we also plot our Cambrian-1 1 performance well as GPT-4 performances. In the plot, is clear that Cambrian-1 offers competitive performance compared to proprietary models in most categories. We also showcase some examples in Fig. 19, demonstrating that the model effectively attends details in images despite using only 576 tokens.
K Potential Misuse & Mitigation Strategies
We recognize that there ethical concerns regarding the potential misuse of multimodal large language models like Cambrian-1, particularly in generating misleading content or spreading misin- formation. Below, outline the main risks and provide strategies to address them:
Misinformation Cambrian-1 could be used to create misleading text descriptions of images, leading to false narratives misrepresentations. For instance, such models might be leveraged by social media bots to manipulate public opinion during elections or other critical events.
2. Hallucination Similar to any large language model, Cambrian- I may produce information that not based facts actual input data. This phenomenon, often called "hallucination." can be dangerous if users assume the model's output is entirely accurate without verification.
To mitigate these risks, users should exercise caution and critical thinking when interpreting outputs generated by Cambrian-1. 1. It important to verify the information produced by the model, particularly if the results are intended for sensitive high-stakes applications. Users must aware of the potential for hallucinations, where the model produces information not grounded in facts, and take steps to cross-check and validate any critical outputs. Additionally, implementing content filtering safeguard can help flag potentially harmful or misleading content before it is disseminated.
40

---

NOSES ARE RED
Whe
playfcc
usssancted tabst poo of nosutie
and creative oulisary station,,
tnapired we'ld
each place regresents thaught cn coakteg challingg thened
cligatly sceaarle, toe stics with comert createss a conctii offecct.
<kase Kurs.
sepliisant
BUB IN YO BED
chanacten
charecter ancoeted
Stapory,
Those solice the peotlen cop by dep
serzee Strpsees
rasell oftir banoress besiceer.
Daghtr Sinng the "epooded
solve this pretlem, reed
completes
conerne on
convertiors Standers tegitih. aesterce
athrrss
1507. Cstoss Amriicn actor Hay Hess, per pettaget nusenaa villades
Lis the arantia in disrinegu Be derrat pen (t:2 21:2..o)) Why
dlys Adbes
reear herelwe, firally get staving stanghat, out
the
scrme-
metmttte the seject-weed agrameet optioe a: "critic, stona Chen, clains" consa after the ram, uiich la nut
tption i: "critie, scass ctyn, chaine, sess coma after the nas and catio crasted suglich.
perticiaacing conttries In the sand 630
senoyyy bock flag cornespadds to
1h1s eptire coneer lecavon it yess not
porticisue the exest, Irdicored
<asyet that theee ceustntes witien tareawet.
This eptioe becane cone sertence, which Tharefore, connect "critic
Optson
Figure 19: Examples of Simmbrian-1-34B Cambrian-1 showcases impressive abilities in visual intersection.
The model demonstrates follooii ability such as output in json format, illustrated in the bottom- left example. Cambrian-1 also demonstrates remarkable OCR ability (See model handles different Comma "" in the right down example).
41

---

L
Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We have detailed all claims mentioned in the abstract and introduction in the remaining part of the paper and Appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [NA]
Justification:
We have discussed the limitations of our work in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
42

---

Justification:
We do not include any theoretical assumptions and proofs in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We will publicly release all code, hyperparameters, model checkpoints, and datasets for reproducing our experimental results.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
43

---

Answer: [Yes]
Justification: Yes, as highlighted in the abstract, we will release our model weights, code, tools, dataset, and all recipes.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https: /CodeSubmiiise //n /nips. ctips.cc/public//uuu htt.a///iiggddeeeddduuuuiiiiiiia wwwiibimsssioni ssionnolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes. We include all training and evaluation recipes in the main paper and Appendix, and will additionally include them in the code release.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We do report error bars and other statistical results for our experiments. Part of these results are included in Fig. 3, Fig. 4 and Fig. 6.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
44

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification:
We will include these in our code release.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //te:r//n.u...///eee
Answer: [Yes]
Justification:
Our research conform NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work has no societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
45

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: We used pre-trained LLM, which already has safeguards. As for datasets, we have conducted filtering to avoid safety risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have credited and mentioned the assets used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
46

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have explained the data engine and new benchmark in our paper.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
47


Reviews:
[{"review_id": "YfWtp6ScHL", "reviewer": "Reviewer_BwuE", "summary": "The paper explores Multimodal Large Language Models (MLLMs) and constructs the Cambrian-1 series models. This approach builds a series of advanced MLLMs through five key pillars, achieving exceptional performance in vision-centric tasks and diverse benchmark tests. By exploring different visual encoders, the method designs a novel spatial-aware connector, SVA, to reduce the number of annotations and enhance the integration of visual and language models. The method also curates high-quality visual instruction fine-tuning data from public sources, emphasizing the importance of balanced data distribution and discussing various instruction fine-tuning strategies. Additionally, this paper critically analyzes and supplements existing MLLM benchmarks, introducing the vision-centric benchmark CV-Bench to more accurately evaluate the models' visual-centric capabilities. This approach achieves top performance across diverse benchmarks and excels in visual-centric tasks.", "questions": "(1) As the number of cross-attention layers (D) and distinct groups of learnable queries (G) increases, the performance does not show continuous improvement (lines 256-258). It is worth exploring whether performance saturation occurs with the increase of (D) and (G).\n\n(2) Instruction tuning data collected from open web maybe raise the potential data leakage. And provide a statistical analysis of the data categories.\n\n(3) Table 2 suggests that increasing the value t does not continuously enhance performance. The proportion of data varies across different tasks. Additionally explore the data scaling law.\n\n(4) From Cambrian 10M to 7M, whether higher data quality results in better model performance.\n\n(5) Statistical analysis of the response length and the difficulty, diversity distribution of the instruction data.\n\n(6) Compare the performence of current MLLMs like LLaVA and BLIP-2 using the same data and other models (i.e., Minicpm v2.5) with Cambrian-1.\n\n(7) Evaluate model performance on high-resolution images or tasks with extreme aspect ratios, (i.e., V*Bench).\n\n(8) Determine whether training the visual encoder in all tasks outperforms freezing it, and compare the convergence speed of end-to-end training versus two-stage training.\n\n(9) Table 11 indicates that integrating more vision encoders does not necessarily lead to higher performance improvements, as seen with models like MMB, VStar, and MMEP.\n\n(10) Provide detailed information of the parameter counts, training duration, and  training hyperparameters for different model backbones.\n\n(11) The paper improves performance across various tasks by integrating most of the current vision encoders. Could a unified visual encoder be used instead?\n\n(12) Some related work needs to be included and discussed.\n\nZhu D, Chen J, Shen X, et al. Minigpt-4: Enhancing vision-language understanding with advanced large language models[J]. arXiv preprint arXiv:2304.10592, 2023.\n\nWang W, Chen Z, Chen X, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks[J]. Advances in Neural Information Processing Systems, 2024, 36.\n\nChen G, Liu X, Wang G, et al. Tem-adapter: Adapting image-text pretraining for video question answer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 13945-13955.\n\nHuang X, Wang J, et al. Segment and Caption Anything[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024\n\nLiu Y, Zhang C, et al. Universal Segmentation at Arbitrary Granularity with Language Instruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nGao P, Han J, Zhang R, et al. Llama-adapter v2: Parameter-efficient visual instruction model[J]. arXiv preprint arXiv:2304.15010, 2023.", "limitations": "The authors have discussed the limitations of this paper, and this paper has no direct negative societal impact.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper aims to bridge the gap in visual understanding by exploring Multimodal Large Language Models (MLLMs) from a vision-centric perspective. By investigating various visual encoders, this method introduces an innovative spatial-aware connector, SVA, which minimizes the need for annotations and improves the synergy between visual and language models.  Additionally, the paper offers a critical assessment and enhancement of current MLLM benchmarks, presenting a new vision-centric benchmark, CV-Bench, to more precisely measure the models' visual-centric abilities. Cambrian-1 achieves top performance across diverse benchmarks and excels in visual-centric tasks. The paper is well written and the experiment is well solid.", "weaknesses": "No Weaknesses. See questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and acknowledgments. We appreciate that you find our work “bridges the gap in visual understanding”, “offers critical assessment and enhancement of MLLM benchmarks”, “introduces an innovative spatial-aware connector”, and “achieves top performance”. We summarize and respond to your questions below: \n\n> **Q1: Does performance saturate with the increase of (D) and (G) in SVA**\n\nA: We recognize the importance of this point and have conducted experiments to further investigate it. We tabulate results below. We observe performance improves with increasing D & G and saturates with D > 4 or G >3.\n\n |D|G|OCR & Chart|\n|:-:|:-:|:-:|\n|2|1|52.1|\n|3|1|52.4|\n|4|1|52.8|\n|5|1|52.1|\n|3|1|52.4|\n|3|2|52.6|\n|3|3|53.1|\n|3|4|52.8|\n\n> **Q2: Data, Leakage, and effectiveness** \n\nA: Our data is collected from existing open-source works. Therefore, we prevent data leakage problems by carefully choosing only the training set of data sources. For our Internet Data Engine, we focus on only Wikipedia in our project, which does not overlap with our benchmarks. \nIn the revision, we will add a section reporting the number of test images in each benchmark present in our final Cambrian-10M dataset by checking image hashes.\n\n> **Q3&4: Increasing data threshold t will not enhance the performance* and *whether higher data quality results in better performance**\n\nA: As we show in §5.2, data quality matters more than quantity. In Tab. 2, intermediate t values result in better performance. This result echoes observations in the data curation pipelines of CLIP and MetaCLIP\nLikewise, we observe that our higher-quality 7M subset results in better benchmark performance compared to training on all 10M data (Tab. 3). We believe this is a result of better balancing the dataset sources (Tab. 2, Fig. 14) and adjusting their relative sizes (Fig. 10).\n\n> **Q5: response length, difficulty, diversity of instruction-tuning data**\n\nA: Thank you for raising these questions! We have conducted further analysis on Cambrian-7M and -10M regarding data composition, length of questions, length of responses, and number of rounds in the instruction tuning data and summarized the results in Tab. 1 of the rebuttal material. As a result of data curation, the Cambrian-7M data are distributed similarly to the best data ratio we found in the data ratio experiment (Fig. 10). \n\n> **Q6: Compare current MLLMs using the same data**\n\nA: Like many previous studies in (M)LLMs, we argue that data is crucial in distinguishing different works. We conduct additional experiments with the LLaVA model trained using LLaMA-3-8b and Cambrian-7M data. Due to the short rebuttal period, we use the conventional 576 visual tokens of LLaVA-1.5, not the high-resolution approach of LLaVA-Next. Despite fewer tokens, this version performs comparably or better than LLaVA-Next in general, knowledge, and vision tasks. Adding our SVA further improves performance, especially in OCR & Chart tasks, while still using only 576 tokens.\n\n|Data|# Vis Token|General|Knowledge|OCR & Chart|Vision-Centric|\n|:--|--:|--:|--:|--:|--:|\n|LLaVA-Next|2880|72.5|55.6|61.0|55.0|\n|LLaVA w/ Cambrian Data|576|72.0|58.1|54.3|55.6|\n|Cambrian-8B|576|74.4|60.1|66.2|60.3|\n\n> **Q7: Evaluate on high resolution benchmarks such as V*Bench**\n\nA: In our work, we adopt V*Bench in our evaluation—see “V*Star” in Tabs. 4, 11, & 13. On V*Star, Cambrian-1 is competitive with LLaVA-NeXT and GPT-4V. \n \n> **Q8: Determine if unfreezing visual encoder outperforms freezing in all tasks, convergence speed**\n\nUnfreezing most visual encoders outperforms freezing in most tasks. We have added a visualization to the rebuttal material (Fig. 1) that shows the %change from Frozen to Unfrozen for each model on each benchmark. Note: full Frozen & Unfrozen benchmark results are in Tables 9 & 10.\n\nThanks for raising the point about convergence speed. Given fixed compute, unfreezing is approximately 50-55% slower during fine-tuning. We will emphasize this drawback in the revision.\n\n> **Q9: Integrate more vision encoders does not lead to higher performance on every benchmark**\n\nA: Indeed, more vision encoders does not lead to higher performance on *every* benchmark. We believe this is expected, as different vision encoders have different strengths/weaknesses—as studied in §3.4 and Fig. 6—and thus different combinations of vision encoders inherit combinations of these strengths/weaknesses.\nWe will amend the 7th Finding to clarify that combining multiple vision encoders usually enhances performance, *but not necessarily on every benchmark*.\n\n> **Q10: Provide more detail about parameter count and training hyperparameters**\n\nA: Tab. 14 provides training hyperparameters, including learning rate, batch size, weight decay, etc. We have added compute resources and training durations in the Rebuttal Tab. 2.\n\n> **Q11: Could a unified vision encoder be used**\n\nA: As studied in this work, we do not have a “perfect” encoder that excels in all areas (visual grounding, language understanding, high-resolution features, etc). Therefore, we pursue hybrid encoders, which can leverage the different strengths of several pretrained visual encoders. We acknowledge that hybrid vision encoders are a work-around solution to take advantage of various pretrained models—unified vision encoders could be trained from scratch with superior performance, but that would require much more compute and data.\nOverall, we advocate to use MLLMs as a vision model evaluator and hope to inspire the development of a unified and powerful vision model. \n \n> **Q12: More Related Works**\n\nA: We thank the reviewers for providing these new references. We will add these references to the revision. Specifically, we will add MiniGPT-4 and LLama-adapter V2 to our discussion of developing MLLMs, and we will add Visionllm, Tem-adapter, Segment and Caption Anything, and Universal Segmentation to our discussion on the downstream applications of MLLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UVbJGbsq2a", "reviewer": "Reviewer_xvQn", "summary": "The paper conducts a comprehensive study of multimodal LLMs from a vision-centric perspective. Different from the lines of previous literature which aim to propose new architectures/algorithms for multimodal LLMs, this paper carefully splits the design space of visual parts of multimodal LLMs into several individual parts and diagnoses each with controlled experiments. This leads to several innovative conclusions about the visual aspects of multimodal LLMs, including the validity of standard benchmarks, the choices of visual encoders, etc.", "questions": "- What's your opinion of \"native\" multimodal LLMs? Do you think the findings in the paper will in a way transfer to more advanced models?", "limitations": "The authors have discussed the limitations quite adequately.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Generally, the paper is a pleasure to read.\n\n- The paper is well-motivated. MLLMs are differentiated from pure LLMs by the visual components. Thus it is quite natural to study the visual aspects of MLLMs.\n- The paper draws rich connections from the visual representation learning literature, which I see as an original perspective, putting the paper into the appropriate position and delivering more valuable information to a broader community.\n- The controlled analysis is precise and rigorous with carefully designed experiments. Particularly, the experiments start with an examination of existing benchmarks, which is a prerequisite of all following experiments.\n- The conclusions are insightful and can be valuable for the future development of multimodal LLMs.", "weaknesses": "- The experiments only consider one particular LLaVa-like formulation of MLLMs built upon a pretrained LLMs, while Sota MLLMs like GPT-4o and Reka are more likely to have completely different training diagrams and architectures, e.g., treating images and texts equally and training a native multimodal LLMs from scratch, or training with interleaved visual and text contents instead of fixed image-first formulations. The value of the paper is thus limited.\n\n-  A minor point: There is no analysis of why the findings hold.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and acknowledgments. We appreciate that you find our work “insightful and well-motivated”, contains “rigorous and carefully designed experiments”, and “can be valuable for the future development of multimodal LLMs”.\n\n> **W1 & Q1: Discussion around native MLLMs**\n \nA: We thank the reviewer for raising this question! We share our thoughts on these models below and show our findings in Cambrian are **very transferable** to these native models. We will add the discussion below to the revised version of our draft.  \n\n**Thoughts about native MLLMs** \n\nWe find native Multimodal LLMs that do not use a pre-trained vision encoder, like GPT-4o or Reka, to be an intriguing and promising approach. These native models have only recently been explored and are predominantly developed by proprietary companies such as OpenAI. As a result, their actual implementation, architecture, and training methods remain largely undisclosed. Additionally, there is insufficient evidence to assert that native MLLMs can overcome the limitations of current MLLMs, such as their visual deficiencies. On the other hand, vision-only representation learning itself is a significant and meaningful objective. Our study connects this goal with multimodal learning, providing scientific insights that complement future advancements in multimodal systems, whether they are native or not.\n\nWe note that one major downside of native MLLMs is that they require *much* more data and computational resources, as they do not rely on the knowledge embedded in pretrained encoders.\n\n**Findings in Cambrian transfer to native MLLMs** \n\nWe believe the findings and contributions in our work will continue to hold and guide the development of future models, including “native” MLLMs. Our findings regarding Data, Connectors, Evaluation, and Vision Backbones can be transferred in the following ways: \n- **Data**: The pool of instruction tuning data and our data curation studies can be very useful for supervised fine-tuning “native” MLLMs. Training native MLLMs is likely to still consist of pretraining, supervised fine-tuning, and RLHF. The data collection and curation insights can play an important role in both the pretraining and supervised fine-tuning stages.\n- **Connector**: The SVA module we proposed can be part of, or inspiration for, future native MLLM designs. The conflict between high-resolution features and a constrained number of tokens is likely to continue in native MLLMs. Therefore, the SVA module could be a competitive candidate for resolving this issue in such native MLLMs.\n- **Evaluation**: Our study on the “Multimodality” of benchmarks can help native MLLMs better assess their visual capability. The categorization of benchmarks also provides more organized and interpretable evaluation protocols for future works, especially in vision-centric domains.\n- **Vision Backbones**: Our study compares current visual representations, uncovering insights about training data (e.g., CLIP vs. SSL), training methods (Encoding vs. Generative), network architecture (ViT vs. ConvNext), and image resolutions. These insights can better guide developers when designing architecture, data, and methods for training native MLLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wt9GZQs0F9", "reviewer": "Reviewer_K2Un", "summary": "This paper introduces a multimodal large language models (MLLMs), named Cambrian-1, designed with a vision-centric approach. In current MLLM researches, the choices of visual encoder are not sufficiently explored. This study utilizes MLLM performance as a visual representation evaluator, showing different characteristics over differently trained vision encoders and revealing that various widely-used MLLM benchmarks are disconnected from visual understanding capability but connected to language capability. Furthermore, this study proposes spatial vision aggregator (SVA) to effectively connect vision and language models with spatial inductive bias. Additionally, curation of high-quality visual instruction-tuning dataset and its distribution balancing are discussed. As a result, Cambrian-1 achieves state-of-the-art performances and provides an open cookbook for MLLMs.", "questions": "In Table 11, why are there two entries for \"SigLIP+DINOv2+ConvNext\" with different numbers?", "limitations": "This study construct a new dataset based on web search, but it does not appear to address any privacy issues. It would be better to address this issue.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper is notably well-written and easy to follow.\n- Section 3.1 shows the limitations of MLLM benchmarks. The finding that several existing benchmarks like MMMU, which were considered important benchmarks in the MLLM field, do not properly evaluate multimodal capabilities is very interesting.\n- This study releases model weights, code, tools, datasets, and detailed recipes, which is a great contribution to this field.", "weaknesses": "- There exists a previous work about vision-language connectors with spatial inductive bias [1]. The comparison or at least discussion between the proposed SVA and C/D-Abstractor [1] is essential but lacks.\n- There are many overlapping findings with existing studies. For example, language-supervised models are effective [2], high-res encoders are beneficial [3], increasing data size and spatial inductive bias are advantageous for connectors [1], and so on. I believe that re-examining these aspects and analyzing them in different settings has its own contribution due to the empirical nature of this field. Nevertheless, it is difficult to attribute high value to the overlapping findings.\n- Findings 7 (the second Findings 6 in the paper, seems to be a typo) is not consistent with the results. The finding claims that performance improves with the vision encoder ensemble, but Table 11 does not seem to support this. For example, SigLIP+DINOv2 performs worse than sole SigLIP.\n\n[1] Cha, Junbum, et al. \"Honeybee: Locality-enhanced projector for multimodal llm.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.  \n[2] Chen, Xi, et al. \"Pali-3 vision language models: Smaller, faster, stronger.\" arXiv preprint arXiv:2310.09199 (2023).  \n[3] Liu, Haotian, et al. \"Improved baselines with visual instruction tuning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and acknowledgments. We appreciate that you find our work “notably well-written”, “shows the limitations of MLLM benchmarks”, and “a great contribution to this field” via our fully-open approach. We summarize your questions and provide our explanations below: \n\n> **W1: Comparison and Discussion between SVA and C/D-Abstractor**\n\nA: We appreciate the valuable suggestion. We provide such discussion and analyses here, and will incorporate this into our revision.\n\nWe first emphasize that our SVA module is distinct from other spatial-based connectors (e.g., C/D-Abstractor) in its ability to dynamically combine visual features from *multiple* vision models with *varying* resolutions.\n\nHowever, to isolate the effect of spatial inductive bias, we consider the case of token reduction using a single vision encoder. Specifically, we use OpenAI CLIP as the vision model and compress its original 576 tokens to 36 tokens using our SVA module and other connectors. We include three baselines:\n\n1. Direct interpolation + MLP\n2. C-Abstractor\n3. LDPNetV2Projector (similar to C-Abstractor but more light-weight)\n\nWe conduct experiments with the 1.2M alignment + 0.7M instruction tuning data setting with Vicuna-1.5-7b as the language model. For fair comparison, we do not include multi-layer aggregation inside the LLM for our SVA baseline.\n\nWe tabulate results below:\n\n| Method            | General | Knowledge | OCR & Chart | Vision-Centric |\n|-------------------|:-------:|:---------:|:-----------:|:--------------:|\n| Interpolate + MLP |  63.4   |   43.8    |    28.1     |      43.7      |\n| C-Abstractor      |  64.4   |   42.8    |    26.1     |      44.3      |\n| LDPNetV2          |  62.5   |   43.9    |    28.7     |      43.9      |\n| SVA               |**65.5** | **44.5**  |  **31.4**   |    **46.9**    |\n\nCompared with the simple MLP baseline, C-Abstractor performs better on General and Vision-Centric tasks but inferior on Knowledge and OCR & Chart tasks. LDPNetV2 performs similarly to the MLP baseline. Our SVA consistently demonstrates superior performance across all categories, especially in OCR & Chart and Vision-Centric tasks, demonstrating its effectiveness in information compression.\n\nOne possible explanation for SVA's data efficiency compared to C-Abstractor is that the SVA module performs local attention on all positions in the grid with the same parameters, so our SVA module receives more supervision.\n\n\n> **W2: Overlapping findings with existing studies**\n \nA: We believe the study of the Multimodal Large Language Model contains many moving parts, and previous works provide isolated studies on each component. This potentially leads to contradictory results in different studies (e.g., freeze or unfreeze vision in Prismatic VLM vs. Deepseek VL). \n\nOur work undertakes a systematic study, combining different moving parts together. We hope to draw more robust and reliable conclusions and clarify the contradicting conclusions that exist in the MLLM domain. In the meantime, we aim to push the study of these modules to the extreme, especially in the fully open-source setting. For example, we carefully compare 15+ vision models, hoping to provide insights to both the MLLM and visual representation learning communities. We also collect and curate, to our knowledge, the largest open-source instruction tuning datasets. These efforts turn the findings in our work into pieces that narrow the gap between open-source and proprietary models.\n\n> **W3: Finding 7**\n\nA: Regarding Finding 7, we draw this conclusion based on multiple experiments rather than a single data point. We observed that, compared to ensembling only CLIP models, adding the DINOv2 model improves performance, especially on vision-centric benchmarks like RealWorldQA and CV-Bench. We appreciate the reviewer's feedback and will include more clarification on this finding in our revision.\n\n\n> **Q1: Question about model combination and two entries for \"SigLIP+DINOv2+ConvNeXt\"**\n\nA: Thank you for reviewing our work so carefully and catching this typo! The second instance of “SigLIP+DINOv2+ConvNeXt” uses a ConvNeXt-L not an XXL.\nWe will correct this in the revised version of the draft.\n\n> **L1: Privacy of Internet Web Search** \n\nA: Thank you for raising this question! We absolutely value the importance of data privacy and copyright. We respect and approach this issue in the following two ways:\n\n*Collect Data from Licensed Websites*: Our web agent collects data from Wikipedia, which is licensed under CC BY-SA (https://en.wikipedia.org/wiki/Wikipedia:Copyrights). We attribute the data source in §5.1 and Appx. E.2. \n*Fully Open Source Data Collection*: We also fully open-source our data collection pipeline in Appx. E. This transparency allows for thorough inspection and verification, ensuring that our methods do not violate any copyright or data privacy regulations.\n\nData privacy is a collective challenge faced by the community, and proprietary models often do not disclose details about their data pipeline. One of the aims of our project is to raise awareness and inspire new research on this topic by being *fully* transparent. We will release all details of the data collection pipeline and plan to include a section in the revision to discuss this further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bqjua7O5JZ", "reviewer": "Reviewer_ED7N", "summary": "This paper presents extensive studies on existing MLLM benchmarks addressing the difficulties involved in consolidating and interpreting results from various tasks for MLLM designs. Moreover, the authors also propose Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector to fuse vision features with LLMs while reducing vision tokens. In addition, authors also collect high quality visual instruction-tuning data. The proposed model, Cambrian-1, achieves the state-of-the-art performance on multiple MLLM benchmarks.", "questions": "See the weakness part.", "limitations": "None", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1, Overall writing is good and easy to follow. The motivation is clear. \n\n2, The goal of this work is interesting. A good study for the effects on Large Language Model, Visual Encoder, Multimodal Connector, Data Curation Pipeline, Instruction in the MLLM system.\n\n3, Experiments are extensive, and several findings are useful when designing MLLM models.\n\n4, A new dataset Cambrian-7M is proposed, which may benefit the MLLM fields for further research. \n\n5, The performance looks good compared with LLaVA-Next. If open-sourced, this model can definitely benefit for the community.\n\n6, Open-sourced code and model.", "weaknesses": "1, The technical novelty is limited, combining multiple vision experts into MLLM is not new. Moreover, fusing visual tokens dynamically is also not new in dynamic network design. \n\n[-] Sphinx: the joint mixing of weights, tasks, and visual embeddings for multi-modal large language models, arixv-2023.\n\n[-] Mova: Adapting mixture of vision experts to multimodal context, arxiv-2024.\n\nI think there are more close related works than [1][2]. The authors should cite all these works as respect.\n\n\n2, I also have one question, What if you do not use the dynamic token but only using proposed tuning datasets, compared with LLaVA-1.6.\n\n3, Several findings are well known and also verified from previous works. For example, Finding-6 is common knowledge using high resolution visual encoder. \n\n[-] Vary: scaling up the vision vocabulary for large vision-language models\n\n\n4, Several figures are useless and common knowledge for MLLM community. For example, Fig.1 and Fig.2 can be merged into one figure. \n\n5, The ablation studies for SVA are not enough. For example, which tokens are more important in which datasets? This needs further analysis. Moreover, the effect of increasing instruction tuning data size is not well explored.\n\nGiven these, I rate this work as weak accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and acknowledgments. We appreciate that you find our work is “well-written”, contains “extensive experiments” and “good performance”, and will “definitely benefit the community” when open-sourced. We summarize your questions and provide responses below: \n\n> **W1&3: Findings are well-known and verified from previous work**\n\nA: We believe the study of the Multimodal Large Language Model contains many moving parts, and previous works provide isolated studies on each component. This potentially leads to contradictory results in different studies (e.g., freeze or unfreeze vision encoder in Prismatic VLM vs. Deepseek VL). Our work undertakes a *systematic study*, isolating and studying the numerous moving parts, which we believe allows more long-standing conclusions that benefit the community. \n\nFurther, we respectfully argue that our work differs from previous works in the following ways:\n\n- *More Comprehensive Experiments*: Our study considers more vision backbones and organized benchmarks than previous works, yielding new insights. For example, we find that high-resolution ConvNets benefit OCR & Chart tasks, and self-supervised learning models can potentially compete with language-supervised ones. Our experiments also reveal the properties of different CLIP models (e.g., SigLIP, EVA-CLIP) beyond ImageNet accuracy, providing valuable insights for both MLLM and Vision model developers. For instance, we observe that EVA-CLIP performs well in most domains but struggles with Chart tasks. This highlights the need for CLIP developers to focus more on OCR & Chart data collection during training.\n- *Findings on high-res encoders*: While concurrent works emphasize the importance of high-resolution features, we take a step further to pinpoint the potential of using ConvNets, such as the ConvNext OpenCLIP model, to efficiently and effectively process high-resolution images. \n- *New Vision Model Aggregation Strategy*: Compared to previous works that focus on fusing vision models, we study both *which models to combine* (§3.5) and *strategies for combining them* (§4). Our SVA approach preserves the resolution, maintains the spatial inductive bias in images, and uses a fixed number of visual tokens.\n\nWe also thank the reviewer for raising this point and suggesting these works. We will discuss each of them in our revision. \n\n> **W2: Training only using proposed tuning datasets, compare with LLaVA 1.6**\n\nA: We first want to clarify that LLaVA 1.6 (LLaVA-Next) proposes a dynamic resolution approach using *2880* tokens, while our SVA module uses only *576* fixed tokens. We conduct additional experiments with the LLaVA model trained using LLaMA-3-8b and Cambrian-7M data. Due to the short rebuttal period, we use the conventional 576 visual tokens in LLaVA and LLaVA-1.5, not the dynamic high-resolution proposal of LLaVA-Next. Despite fewer tokens, this version matches or outperforms LLaVA-Next in General, Knowledge, and Vision-Centric tasks. Adding our SVA module further improves performance, especially in OCR & Chart tasks, while still using only 576 tokens.\n\n|Data|# of Vis Tokens|General|Knowledge|OCR & Chart|Vision-Centric|\n|:--|--:|--:|--:|--:|--:|\n|LLaVA-Next|2880|72.5|55.6|61.0|55.0|\n|LLaVA w/ Cambrian Data|576|72.0|58.1|54.3|55.6|\n|Cambrian-8B|576|74.4|60.1|66.2|60.3| \n\n> **W4: Several figures are useless and common knowledge for MLLM (e.g, Figs. 1 & 2)**\n\nA: We hope our work provides a systematic study around MLLMs and can serve as informational for audiences both within and beyond the MLLM community. Especially now, as MLLM is becoming an ever-growing community, the introduction and figures serve as preparation and context-setting for a broader audience. We make no claims of novel findings in the initial figures and reserve such insights for the later sections after providing the audience requisite context. Nevertheless, we thank the reviewer for raising this concern, and we will consider condensing our presentation in the revised version.\n\n> **W5: Study of SVA Module is not enough**\n\nA: We thank the reviewer for raising this crucial point. We have added a study of the importance of visual features from different vision models to different image categories by investigating the attention score distribution in our SVA module.\n\nWe evaluate our Cambrian-8b model on GQA, DocVQA, and ScienceQA (representing three different benchmark categories), and tabulate attention distributions below. We can see that on real-world images (GQA), the contribution of different vision models is relatively uniform, in part due to the similar characteristics of SigLIP and CLIP. On document-type images (DocVQA) which are text-heavy and often high-resolution, the influence of SigLIP increases and that of ConvNext greatly increases to aid in high-resolution information processing. For scientific images (ScienceQA) composed of illustrations and diagrams about different science categories, the contribution of SigLIP is further increased while the portion of DINOv2 decreases compared to GQA.\n\n|Model|GQA|DocVQA|ScienceQA|\n|:--|--:|--:|--:|\n|SigLIP|29.7%|31.1%|35.2%|\n|CLIP|18.5%|13.4%|16.3%|\n|DINOv2|24.1%|11.0%|17.6%|\n|ConvNext|27.7%|44.5%|30.9%|\n\nWe also study the performance of our Cambrian-8b model with SVA modules on different sizes of alignment and instruction tuning data. The results are shown below. We can see that increasing the size of alignment data leads to improvement in all benchmark categories. Increasing the size of instruction tuning data leads to notable overall improvement, and the instruction tuning data is especially helpful for Knowledge, OCR & Chart, and Vision-Centric tasks.\n\n||General|Knowledge|OCR & Chart|Vision-Centric|\n|:--|--:|--:|--:|--:|\n|1.2M alignment + 0.7M instruction|72.3|54.8|58.3|57.2|\n|2.5M alignment + 0.7M instruction|72.7|55.8|58.9|58.3|\n|2.5M alignment + 7M instruction|74.4|60.1|66.2|60.3|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their thorough review and valuable feedback on our paper. We appreciate that you find our work "bridges the gap in visual understanding" (Reviewers BwuE, xvQn), "offers assessment and enhancement of MLLM benchmarks" (Reviewers BwuE, K2Un,  ED7N), "well-written" (Reviewers K2Un, BwuE), “release a new dataset that benefits the community” (Reviewer ED7N), “proposes an innovative connector” (Reviewers BwuE, ED7N), “contains extensive and rigorous experiments” (Reviewers xvQn, ED7N), and "achieves top performance" (Reviewers BwuE, ED7N).                            

In the responses below, we address each reviewer’s questions individually. We encourage the reviewers to refer to the attached rebuttal PDF for a detailed review, including additional figures and experiment results encouraged by the reviews. We hope our responses address your questions. We look forward to engaging with you during the reviewer-author discussion period if you have any further questions.

Meta-review: The paper presents Cambrian-1, a family of multimodal large language models with a vision-centric approach. The authors explore the use of various visual encoders and propose a novel spatial vision aggregator (SVA) to enhance the integration of vision and language models. The work critically evaluates existing benchmarks and curates a large dataset for visual instruction tuning. Cambrian-1 achieves state-of-the-art performance across multiple benchmarks and provides an open-source toolkit for further research in multimodal systems.

The contributions of the paper are solid. There are comprehensive evaluation of multimodal benchmarks, identifying gaps in visual understanding. The introduction of the SVA module, shows notable performance improvements in specific tasks like OCR and chart understanding. The open-source contributions including model weights, datasets, and code, will likely to benefit the research community considerably. Extensive experiments with various visual encoders offers valuable insights into their performance. Despite some limitations in novelty, the paper makes a valuable asset for the research community. The paper's performance on multiple tasks and its rigorous experimentation further support its acceptance.
Decision: Accept (oral)