Title: Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
Venue: NeurIPS 2024
Accepted: yes
Keywords: Multimodal LLM

Visual Representation Learning

Evaluation Protocol

Data Mix

Open Science
Abstract: We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. While stronger language models can enhance multimodal capabilities, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning research. This gap hinders accurate sensory grounding in real-world scenarios. Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations, offering new insights into different models and architectures—self-supervised, strongly supervised, or combinations thereof—based on experiments with over 15 vision models. We critically examine existing MLLM benchmarks, addressing the difficulties involved in consolidating and interpreting results from various tasks. To further improve visual grounding, we propose spatial vision aggregator (SVA), a dynamic and spatially-aware connector that integrates vision features with LLMs while reducing the number of tokens. Additionally, we discuss the curation of high-quality visual instruction-tuning data from publicly available sources, emphasizing the importance of distribution balancing. Collectively, Cambrian-1 not only achieves state-of-the-art performances but also serves as a comprehensive, open cookbook for instruction-tuned MLLMs. We provide model weights, code, supporting tools, datasets, and detailed instruction-tuning and evaluation recipes. We hope our release will inspire and accelerate advancements in multimodal systems and visual representation learning.

Cambrian -1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Abstract

Weintroduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. While stronger language models can enhance multimodal capabilities, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning research. This gap hinders accurate sensory grounding in real-world scenarios. Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations, offering new insights into different models and architectures-selfsupervised, strongly supervised, or combinations thereof-based on experiments with over 20 vision encoders. We critically examine existing MLLM benchmarks, addressing the difficulties involved in consolidating and interpreting results from various tasks, and introduce a new vision-centric benchmark, CV-Bench. To further improve visual grounding, we propose the Spatial Vision Aggregator (SV A), a dynamic and spatially-aware connector that integrates high-resolution vision features with LLMs while reducing the number of tokens. Additionally, we discuss the curation of high-quality visual instruction-tuning data from publicly available sources, emphasizing the importance of data source balancing and distribution ratio. Collectively, Cambrian-1 not only achieves state-of-the-art performance but also serves as a comprehensive, open cookbook for instruction-tuned MLLMs. We provide model weights, code, supporting tools, datasets, and detailed instructiontuning and evaluation recipes. We hope our release will inspire and accelerate advancements in multimodal systems and visual representation learning.

Project page:

https://cambrian-mllm.github.io/

1 Introduction

There is a long-standing debate in philosophy about whether understanding and meaning in language require sensory grounding. Aristotle's emphasis on acquiring knowledge through sensory experience and empirical observation was central to his ancient Peripatetic school and remains influential to this day [8]; Aquinas famously formalized these ideas in the 13th century with the Peripatetic axiom: ' Nihil est in intellectu quod non sit prius in sensu ' (Nothing is in the intellect that was not first in the senses) [7]. Though many philosophers disagree [23], it is evident that having robust and highly capable sensory grounding is at least beneficial. Consider the Cambrian explosion , during which the emergence of vision is believed [105] to have been crucial for early animals to not only find food and avoid predators but also to evolve and improve. In fact, most human knowledge (and nearly

Data

Recipe

Visual

Representations

or

End-to-End Tuning

Vision Model

Pretrained

ImageNet-1k

COCO

ADE20K

Visual Representation Learning - Evaluation Protocols

Multimodal Large Language Models

Vision Model

Connector

Visual Instruction Tuning

w/

LLMs

Pretrained

MMB

OCRBench

AI2D

ChartQA

MME

SEED

MathVista

MMVP

SQA

VQA

SEG

DET

CLS

SEG

DET

…

V*Bench

all animal knowledge) is acquired through sensory experiences like sight, hearing, touch, taste, and smell, through interactions with the physical world [107]. These sensory experiences are fundamental to understanding the world around us and are crucial for real-world actions and decision-making.

Beyond philosophical debates, recent advances in multimodal large language models (MLLMs) have brought the topic of visual representation learning vs. language understanding into practical focus. Language models have shown strong scaling behaviors [55], and recent advancements in multimodal learning are largely driven by the development of better, larger LLMs [81]. On the other hand, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning research. For instance, many pioneering frameworks such as LLaVA [82] use vision transformer-based CLIP models [109, 145], which are strongly supervised by language ‡ , as the vision feature extractor. While other visual representations, such as self-supervised DINO [103], are being explored [126], there is a lack of comprehensive and systematic study in this domain. This gap exists primarily because such studies are challenging: MLLMs involve a complex training and evaluation pipeline with numerous design decisions to consider. In this work, we aim to bridge the gap by exploring MLLMs from a vision-centric perspective. More specifically, we use MLLM instruction tuning as an evaluation protocol for various visual representations (illustrated in Fig. 1).

Our motivation for this study also stems from two potential concerns of the current multimodal learning research: 1) relying too heavily too early on language can act as a shortcut [47, 144], compensating for the deficiencies in learning effective visual representations, and 2) existing benchmarks may not provide adequate guidance for real-world scenarios-where visual grounding is crucial for robust multimodal understanding. These concerns are not unfounded, as researchers have started to notice that visual grounding is becoming a bottleneck for applying MLLMs in some challenging real-world applications, despite significant progress in improving general capabilities [41, 126, 136].

From another perspective, traditional evaluation protocols for visual representation learning (e.g., linear probing and end-to-end fine-tuning on datasets like ImageNet-1K [113], COCO [79], and ADE20K [154]) are becoming saturated and do not reflect the diverse perception challenges found in real-world distributions. On the other hand, using language in the form of visual question answering (VQA) offers a flexible and robust evaluation protocol. Our study aims to explore this new protocol design, setting it up to gain insights that will guide the development of better visual representations in the future. Furthermore, to better evaluate visual representations in this integrated setting, we develop a vision-centric MLLM benchmark, CV-Bench, by transforming traditional vision benchmarks into VQA format (Section 2.2).

Cambrian-1 is structured around five key pillars, each offering important insights into the design space of MLLMs:

Visual Representations : We explore various vision encoders and their combinations. §2.4

Sorrel

Barrel

Revolver

A cowboy rides a

horse at a rodeo.

Pos.

Neg.

Class Label Supervised

ImageNet-1K [113]

Language Supervised

CLIP [109]

SSL-Contrastive

DINOv2 [103]

SSL-Masking

MAE [52]

Diffusion

Stable Diffusion [112]

Depth Supervised

MiDaS [15]

Segmentation Supervised

SAM [68]

Connector Design : We design a new dynamic and spatially-aware connector that integrates vision features with LLMs while reducing the number of tokens. §3

Instruction Tuning Data : We curate high-quality visual instruction-tuning data from public sources, emphasizing the importance of distribution balancing. §4

Instruction Tuning Recipes : We discuss instruction tuning strategies and practices. §2.3

Benchmarking : We analyze existing MLLM benchmarks, cluster them into 4 intuitive groups, and introduce a new vision-centric benchmark 'CV-Bench'. §2.1, §2.2

We defer a detailed review of the fundamental components and methodologies that underpin MLLM research to Appendix B

2 Evaluating Visual Representations through MLLMs

Current MLLMs predominantly rely on CLIP [109] as the visual encoder due to its pre-alignment with language and ease of adaptation to the LLM token space. However, strong language priors can be a double-edged sword-they compensate for deficiencies in learning effective visual representations [126] and diminish insights gained from extensive visual representation learning research. In this section, we systematically evaluate how various visual encoder choices (see Fig. 2) impact the multimodal capabilities of MLLMs. We also advocate for using MLLM evaluation as a robust framework for assessing visual representation methods, moving beyond traditional protocols like linear probing and end-to-end fine-tuning to more faithfully reflect the diverse perception challenges in real-world scenarios and to better guide the development of improved visual representations.

2.1 Analyzing the Benchmarks

To effectively evaluate visual representations and MLLMs, we first need to select benchmarks that accurately assess the multimodal capabilities of these models. We use a suite of commonly used benchmarks [24, 45, 54, 57, 83, 84, 91, 92, 96, 97, 120, 126, 137, 143], which is the intersection of those used in recent MLLM research [75, 77, 137]. To help interpret our results, we begin by analyzing the benchmarks themselves. Here, we train MLLMs with 23 different vision backbones (see Table 6) from a variety of model families (see Fig. 2) using a 2-stage instruction tuning process initially proposed in [82]: first training connector on 1.2M adapter data from ShareGPT-4V [27] followed by fine-tuning both the connector and LLM on 737K instruction tuning data (see more details in Appendices G.5 and H). Full benchmark results in Table 9.

Who's answering the question: the LLM or MLLM? Determining whether a benchmark truly needs visual input to be solved has been a persistent challenge in vision-language research [2, 26, 50, 94]. In this study, we compare the performance of MLLMs with and without visual input § , and also calculate the expected score via randomly guessing. These three conditions are visualized in Fig. 3-left, with benchmarks sorted by the difference between the average score with vision enabled and disabled. SQA-I ¶ , MMMU, MathVista, and AI2D display less than a 5% gap between vision enabled and disabled, suggesting that these benchmarks may not significantly depend on visual input and rather heavily rely on the base LLM. TextVQA and GQA both demonstrate a nearly 40% positive gap between random guessing and vision-disabled scores, implying a strong language bias in these benchmarks. On the other hand, the vision-disabled performance on benchmarks like MMVP is notably worse than random guessing, suggesting that strong visual grounding is particularly crucial.

Clustering the Benchmarks To better understand the different aspects of MLLM performance, we analyze the correlations between the performance of our 23 MLLMs on each benchmark. A

confusion matrix (Fig. 10) reveals that certain benchmarks, such as MMMU, are largely uncorrelated with the others. We perform principal component analysis on the benchmark scores and observe the formation of clusters corresponding to 'General,' 'Knowledge,' 'Chart & OCR,' and 'Vision-Centric' categories (Fig. 3-right). We assign MMMU to the knowledge category based on the types of questions it includes (see Appendix D). We also find that existing vision-centric benchmarks [126, 137] are of insufficient size (see Fig. 3-right), challenging the robustness of evaluating such capabilities. These benchmarks do not cover crucial visual elements such as depth and spatial awareness.

Finding 1: Most benchmarks do not properly measure vision-centric capabilities, and the ones that do have very few samples.

2.2 Cambrian Vision-Centric Benchmark (CV-Bench)

To address the limitations of existing vision-centric benchmarks, we introduce the Cambrian VisionCentric Benchmark (CV-Bench). With 2638 manually-inspected examples , CV-Bench provides significantly more examples than other vision-centric MLLM benchmarks3 . 5 × more than RealWorldQA [137] and 8 . 8 × more than MMVP [126]. By repurposing standard vision benchmarks [18, 79, 154] || , we can assess models at classic vision tasks within a multimodal context. Leveraging the rich ground truth annotations from the benchmarks, we formulate natural language questions that probe the fundamental 2D and 3D understanding of the models.

As visualized in Fig. 11, CV-Bench evaluates 2D understanding via spatial relationships & object counting, and 3D understanding via depth order & relative distance. We refer details to Appendix E.

Finding 2: Existing vision benchmarks can be effectively repurposed into VQA questions, enabling the assessment of vision-centric MLLM capabilities.

2.3 Instruction Tuning Recipes

MLLMsstart with pre-trained LLM and vision backbones, connecting these modules with a connector such as a projector (MLP). The original LLaVA [80, 82] proposes a 2-stage frozen training process: first, pre-training a connector between frozen LLM and vision backbones using adapter data, and then fine-tuning both the connector and LLM with instruction tuning data while leaving the vision encoder frozen. Various studies [27, 63, 81, 98] have drawn different conclusions regarding the optimal training methodology for MLLMs. Here, we revisit this topic with extensive experiments.

For our experiments, we tune a set of MLLMs using Vicuna-1.5-7B as the LLM backbone and each of our 23 vision models (Table 6) as the visual encoder. We use a 737K instruction tuning data mix for all experiments here (see Appendix H). All hyperparameters are matched across each experimental setting-highlighting the impact of different tuning strategies with each visual encoder. All experimental settings and results are tabulated in Appendix F.2.





















































































































































One Stage vs Two Stage Training Recent work [63] advocates for skipping connector pre-training, claiming this ' reduces compute cost without harming downstream performance .' To explore whether this claim holds-especially when using non-language-supervised visual encoders-we conduct experiments using 0, 0.5M, and 1.2M adapter data. Following LLaVA's recipe [82], we tune only the connector on the adapter data during this first phase, before unfreezing the LLM and connector during instruction tuning on the 737K mix. Fig. 4 shows that pre-training the connector first enhances model performance and that more adapter data further improves performance across all domains. Thus, we subsequently adopt 2-stage training with 1.2M adapter data as our standard setup.

Freeze vs Unfreeze Vision Encoder There are also mixed practices in freezing [63, 80, 82] or unfreezing [44, 81] vision backbones during fine-tuning. Some argue that unfreezing the vision backbone significantly degrades performance [63]. Our experiments demonstrate that unfreezing benefits performance across all benchmarks except for a marginal change in knowledge benchmarks (Fig. 4). We suspect this is due to the composition of the 737K instruction tuning data and the LLMheavy focus of these benchmarks (see Section 2.1). We note that unfreezing the vision backbone introduces additional computational overhead, which prohibits testing on some larger vision models under current sharding strategies (see more details in Appendix H).

2.4 MLLMs as a Visual Representation Evaluator

As discussed in earlier sections, MLLMs provide a new interface to explore aspects of vision models beyond traditional benchmarks like ImageNet-1k linear probing. We study the 2-stage instruction tuning setting using 1.2M adapter data, 737K fine-tuning data, and frozen visual encoders to allow comparison of the widest range of models.



































































































































































We evaluate on benchmarks detailed in Section 2.1, calculating the average performance ** for each category and visualize the results in Fig. 5 (full results in Appendix F). Our findings highlight the advantages of language-supervised models over non-CLIP models across all benchmark categories, with significantly better performance on chart and OCR-related benchmarks. We hypothesize that this is due to CLIP's training data , such as LAION [115], containing abundant OCR and text-heavy data, whereas SSL and other vision models primarily train on natural images with significantly less text content. It is also noteworthy that language-supervised models are typically trained with a very large pool of data, ranging from 400 million [109] to 10 billion [28] samples, whereas the largest vision self-supervised training dataset, like DINOv2, consists of only 142 million samples [103].

Additionally, we observe that higher-resolution models particularly enhance performance on chart and vision-centric benchmarks while remaining neutral on general VQA and knowledge-based VQAs. While the majority of the backbones we examine are ViT-based [39], ConvNet-based architectures (such as OpenCLIP ConvNeXt [86]) are inherently well-suited for high-resolution image processing [130] and can produce superior results on OCR & Chart and Vision-Centric benchmarks. In vision-centric benchmarks, the gap between language-supervised and other types of vision models is smaller, with a well-trained self-supervised DINOv2 model even outperforming some language-supervised models.

Finding 5: High-res encoders greatly enhance performance on chart & vision-centric benchmarks, and ConvNet-based architectures are inherently well-suited for such tasks.

Narrowing the gap between Language- and Self-Supervised models Above, we observe that DINOv2 stands midway between SSL models and language-supervised models on general and knowledge benchmarks, even outperforming some language-supervised models on vision-centric benchmarks. Here, we study whether the continued finetuning of an MLLM based on a SSL model

can achieve performance similar to that of a language-supervised model. Specifically, we scale up the instruction tuning data from 737K to 5M (see more details in Appendix G.5), and instruction tune MLLMs with DINOv2 ViT-L/14@336 and OpenAI CLIP ViT-L/14@336 encoders in both frozen and unfrozen settings. In Fig. 6, we observe that by unfreezing the vision backbone, the DINOv2-based MLLM fine-tuned with 5M data surpasses the MLLM trained with a CLIP model on 0.7M data. Additionally, the gap between DINOv2 and the CLIP models is reduced under the 5M setting.

Finding 6: Language supervision offers strong advantages, but the performance gap can be narrowed with SSL methods given enough data and proper tuning.

2.5 Combining Multiple Vision Encoders

As observed in Fig. 5, different vision encoders excel in different aspects of MLLM performance. In this study, we explore the potential of combining multiple vision encoders to leverage their distinctive representations, aiming to build a more capable MLLM. Given that different vision encoders use varying architectures and image resolutions, we interpolate to a fixed number of visual tokens (576) in this subsection (see details in Appendix F.3). We then concatenate these tokens along the feature dimension, following a method similar to A-MoF proposed in [126].

Our study indicates that adding a non-language-supervised model (DINOv2) can improve benchmark performance, especially in vision-centric tasks. Notably, even OCR benchmarks benefit from incorporating DINOv2. This highlights the importance of self-supervised learning models in complementing language-supervised models to achieve robust multimodal understanding. Detailed results and configurations are available in Appendix F.3.

However, this naive strategy has two limitations: 1) it employs interpolation, which can lead to information loss, especially with vision encoders with high-resolution feature maps, and 2) it treats each model equally via simple concatenation. Therefore, we seek a more effective strategy that can more flexibly leverage model combinations with less information loss.

Finding 7: Combining multiple vision encoders, including SSL models, can enhance MLLM performance across various benchmarks, particularly in vision-centric tasks.

3 Spatial Vision Aggregator (SVA): A New Connector Design

To effectively aggregate features from multiple vision encoders and prevent the information loss introduced by interpolation, we use a set of learnable latent queries that interact with multiple vision features via cross-attention layers [37]. In particular, our approach incorporates two new vision-centric design principles:

We introduce spatial inductive bias by explicitly defining the aggregation space for each token in the query.

We aggregate vision features multiple times across the LLM layers, enabling the model to repeatedly access and integrate necessary visual information.

To facilitate information aggregation via cross-attention, we create a C -dimension learnable latent token x ∈ R C that is repeated L × L times to form a 2D grid, serving as the query X ∈ R L 2 × C . The set of visual features F from N vision encoders serve as the context (i.e., key and value). We ensure the output resolution of every vision encoder is a multiple of L . Formally, the feature map of the k -th vision encoder ( F k ) has a resolution of m k L × m k L × C , where m k is a positive integer multiplier, and L is the height/width of the learnable 2D grid with hidden dimension C .

Spatial inductive bias To maintain the spatial structure during cross-attention, we align each token in the query with a specific sub-region of the feature maps in all vision encoders. Formally, a token at row i and column j in the query x i,j corresponds to the sub-region

of the k -th vision feature map. As a result, a token x i,j aggregates a total of ∑ k m 2 k features from N vision encoders through cross-attention (see Fig. 7-left).

Specifically, the updated query vector q ∗ i , j ∈ R 1 × C at position ( i, j ) is computed as

Transformer Block

1

Text Tokens

!

L

!

×	C

SVA

Transformer Block

k

SVA

•

•

•

•

•

•

G

× D

Enc

1

• • •

• • •

Enc

N

Enc

k

Cross

Attention

!

Visual Tokens

Spatial Vision

Aggregator

%

%

%

&

"

',)

LLM Transformer

where

Here, q i,j is the query vector at position ( i, j ) , calculated using the query projection matrix W Q ∈ R C × C . The key vectors k i,j,k and value vectors v i,j,k are computed for each vision encoder k using their respective key and value projection matrices W K k ∈ R C × C and W V k ∈ R C × C . Since ∑ k m 2 k features are aggregated into a single token, we effectively reduce the number of tokens.

Multi-layer vision aggregation Although our proposal effectively aggregates features from multiple vision encoders, there is still potential information loss with high-resolution input (large m k ) or multiple vision encoders (large N ). Here, a single token would have to handle a larger amount of context information during aggregation. To prevent this, we allow cross-attention to occur multiple times by inserting our proposal throughout the LLM layers-allowing consistent access to the uncompressed visual information (see Fig. 7-right).

Hyperparameters To flexibly modulate capacity, we introduce two hyperparameters D and G , which indicate the number of cross-attention layers and distinct groups of learnable queries used between the vision models and the LLM, respectively. D and G are always set to 1 for cross-attention layers within LLM layers. We provide ablation studies on the selection of D and G in Appendix H.

We demonstrate the efficacy of SVA module using the best vision model combination results from the previous section and a Vicuna-1.5-7B base LLM. Specifically, we employ a combination of four vision encoders: OpenAI CLIP ViT-L/14@336, SigLIP ViT-SO400M/14@384, OpenCLIP ConvNeXt-XXL@1024, and DINOv2 ViT-L/14@518. We compare our method with two strong baselines: 1) concatenation-based [126] and 2) Re-sampler [11, 72]. Here, we include two variants of our SVA module. The standard one, ' SVA ', uses D = 3 , G = 1 , and inserts cross-attention blocks inside the LLM with a layer stride of 3. To isolate the advantages of spatial inductive biases, we include another SVA variant, 'SVA-no-multi-agg', that does not add cross-attention blocks inside the LLM and sets D = 3 and G = 3 . Table 1 shows that SVA outperforms both baselines, with a significant improvement in the OCR & chart category. In contrast, the Resampler-which lacks spatial inductive biases-struggles to condense concatenated tokens from various vision towers into

Science

Math

Cambrian-7M

OCR (27.6%)

RenderedText [134] (10.0 K)

RefCOCO [140] (30.0 K)

CLEVR [59] (350.0 K)

Filtered DVQA (1550.0 K)

VisText [124] (9.0 K)

VizWiz [51] (20.0 K)

TallyQA [1] (250.0 K)

DVQA [61] (775.0 K)

FinQA [31] (6.0 K)

Visual7W [159] (14.0 K)

Code (0.8%)

SynthDog [67] (500.0 K)

InfoVQA [16] (2.0 K)

LAION GPT-4V [70] (11.0 K)

Filtered WebSight (790.0 K)

ArxivQA [76] (100.0 K)

TAT-QA [158] (2.0 K)

IDK [22] (11.0 K)

WebSight [71] (10.0 K)

OCRVQA [100] (80.0 K)

HiTab [32] (2.0 K)

OKVQA [95] (9.0 K)

DaTikz [14] (47.0 K)

ScreenQA [56] (79.0 K)

General (33.3%)

HatefulMemes [66] (8.0 K)

Design2Code [118] (0.5 K)

WIkiSQL [153] (74.0 K)

ALLaVA [25] (700.0 K)

OODVQA [129] (8.0 K)

Math (3.2%)

Low-Level Vision [27] (50.0 K)

Q-Instruct [135] (400.0 K)

SketchyVQA [129] (8.0 K)

Geo170K [42] (170.0 K)

DocVQA [97] (39.0 K)

LNQA [108] (302.0 K)

Visualmrc [123] (3.0 K)

RAVEN [148] (42.0 K)

WTQ[106] (38.0 K)

LVIS-Instruct4V [131] (220.0 K)

Language (23.8%)

GeomVerse [64] (9.0 K)

ChartQA [96] (28.0 K)

LLaVA150K [82] (150.0 K)

OpenOrca [78] (994.0 K)

MathVision [132] (3.0 K)

IconQA [89] (27.0 K)

VisualGenome [69] (86.0 K)

MathInstruct [142] (262.0 K)

Inter-GPS [90] (1.0 K)

Chart2Text [62] (26.0 K)

VQAv2 [50] (83.0 K)

OrcaMath [99] (200.0 K)

TQA [5] (1.0 K)

TabMWP [88] (23.0 K)

WizardCoder [93] (143.0 K)

Science (2.9%)

TextCaps [119] (22.0 K)

GQA [57] (72.0 K)

OpenCodeInterpreter [152] (66.0 K)

Data Engine (161.0 K)

LLAVAR [149] (20.0 K)

A-OKVQA [116] (50.0 K)

Dolly [36] (11.0 K)

PathVQA [53] (32.0 K)

ST-VQA [17] (17.0 K)

AlfWorld [146] (45.0 K)

Counting (8.5%)

ScienceQA [91] (12.0 K)

AI2D [65] (15.0 K)

ShareGPT [27] (40.0 K)

Filtered CLEVR (350.0 K)

Average

General

Knowledge

OCR&Chart

Vision-Centric

150k

53.7

68.0

51.3

45.2

50.5

250k

54.3

68.1

51.5

45.3

52.2

350k

54.3

67.4

51.4

46.0

52.3

450k

54.2

68.0

52.2

45.5

50.7

0

20

40

60

80

100

Percentage

4.50%

15.00%

15.00%

4.50%

0.8 %

4.40%

5.00%

5.00%

4.40%

0.87%

10.00%

5.00%

5.00%

10.00%

7.20%

2.0 %

5.00%

5.00%

2.0 %

8.71%

39.10%

25.00%

30.00%

39.10%

27.22%

20.00%

25.00%

30.00%

35.00%

34.52%

20.00%

20.00%

10.00%

5.00%

21.00%

Language

General

OCR

Counting

Math

Code

Science

Average

46.2

46.4

46.6

46.8

47.0

47.2

47.4

47.6

47.8

Average Score

Average

General

Knowledge

OCR&Chart

Vision-Centric

LLaVA-665K

40.7

64.7

45.2

20.8

32.0

Cambrian-10M

54.8

68.7

51.6

47.3

51.4

Cambrian-7M

55.9

69.6

52.6

47.3

54.1

exp-1

exp-2

exp-3

exp-4

exp-5

a limited number of learnable queries via global cross-attention. We also compare SVA with other connectors in Appendix H and show clear advantages.

Finding 8: Spatial inductive bias and deep interaction between LLM and vision feature help to better aggregate and condense vision features.

4 Instruction Tuning Data for Training MLLMs

4.1 Data Collection

Collecting Instruction Tuning Data from existing data sources Unlike language data, multimodal (visual) instruction-tuning data is much rarer and harder to collect. To address this, we use existing multimodal benchmarks and datasets involving visual interaction data, such as Visual Question Answering (VQA) and OCR data. To help maintain conversational abilities [147], we also collect a small volume of high-quality language-only instruction-following data. We categorize data into General conversation, OCR, Counting, Code, Math, Science, and Language-only data. We list the data sources in Fig. 8, and the details of data preparation in Appendix G.

Targeted Internet Data Collection Engine As observed in Fig. 8, there is an unbalanced distribution of data. Some categories, such as science, have very few data sources, and each source has limited samples. Inspired by previous works [73], we introduce a data engine to create large-scale, reliable, high-quality knowledge-based instruction tuning data (see Fig. 15). Details are in Appendix G.3. Our data engine produces a large volume of reliable scientific data, increasing the diversity in the data pool. We generate 161k science-related data points-400% more than the previous combined data sources.

Cambrian-10M We create a large pool of instruction tuning data, which we refer to as Cambrian10M. This pool contains approximately 9784k data points, offering a diverse range of data for our work and future research. We visualize its composition in Fig. 8.

4.2 Data Curation

Cambrian-10M is a large pool of instruction tuning data sourced from a variety of data sources, with an unbalanced data ratio between categories. Here, we take a preliminary step to study data curation by improving data balancing and adjusting data ratios.

Data Balancing We follow previous work [109, 138] to set thresholds t for the number of data points from a single data source. We choose t = 150 k, 250 k, 350 k, and 450 k in this section and observe an elbow effect in Table 2-finding that a threshold between 250 k and 350 k work the best for Cambrian-10M. We also plot in Appendix G.4 the cumulative sum of counts for entries sorted by counts from tail to head and we see this intermediate threshold prevents explosive heavy tail.

Data Ratio Cambrian-10M is designed for visual instruction tuning. Given the various capabilities of different types of data, it is essential to balance the ratio of these data types. We conduct pilot experiments with a fixed dataset size of 1350k, examining the impact of different data ratios. We visualize the results in Fig. 9 and summarize our findings as follows: (i) Balancing General, OCR and Language data is crucial. (ii) Performance on knowledge-intensive tasks is influenced by multiple factors, often requiring a mix of OCR, chart, reasoning, and general perception.

Cambrian-7M By applying data filtering to Cambrian-10M with our identified data ratio, we create a smaller but higher-quality dataset called Cambrian-7M. Table 3 showcases the benefits of a wellbalanced and carefully curated dataset. Despite having fewer samples, Cambrian-7M demonstrates improved performance. We additionally apply system prompts in Cambrian-7M to avoid the "answer machine phenomenon", see more details in Appendix G.2.

5 State of the Art Performance

Finally, we leverage the insights from all our previous studies to train a family of MLLMs we call Cambrian-1. We train models using LLM backbones of various scales: LLaMA-3-Instruct-8B [4], Vicuna-1.5-13B [151], and Hermes-2-Yi-34B [139]. Our vision component combines four modelsOpenAI CLIP ViT-L/14@336, SigLIP ViT-SO400M/14@384, OpenCLIP ConvNeXt-XXL@1024, and DINOv2 ViT-L/14@518 (Section 2.5)-via the Spatial Vision Aggregator (Section 3). We pre-train the connector using 2.5M adapter data and instruction tune using our Cambrian-7M data mix (Section 4.2). Our models are evaluated on the benchmarks categorized in Section 2.1, with results presented in Table 4. †† .

Cambrian-1 surpasses open-source models like LLaVA-NeXT and Mini-Gemini. Thanks to the SVA, Cambrian-1 excels in tasks requiring high-resolution image processing, even with only 576 image tokens-about 1/5 of the tokens used by LLaVA-NeXT and Mini-Gemini. Cambrian-1 also achieves comparable performance to the best proprietary models, such as GPT-4V, Gemini-Pro, and MM-1, on several benchmarks. We provide model weights, open-source code, datasets, and detailed recipes for model training and evaluation. We hope our work will strengthen the open research community and accelerate research in both visual representation learning and multimodal systems.

References

M. Acharya, K. Kafle, and C. Kanan. 'TallyQA: Answering complex counting questions'. In: AAAI . 2019.

A. Agrawal et al. 'Don't just assume; look and answer: Overcoming priors for visual question answering'. In: CVPR . 2018.

A. Ahmadyan et al. 'Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations'. In: CVPR (2021).

AI@Meta. 'Llama 3 Model Card'. In: (2024).

H. A. Alawwad et al. 'Enhancing Textbook Question Answering Task with Large Language Models and Retrieval Augmented Generation'. In: arXiv preprint arXiv:2402.05128 (2024).

J.-B. Alayrac et al. 'Flamingo: a visual language model for few-shot learning'. In: NeurIPS . 2022.

T. Aquinas. Quaestiones Disputatae de Veritate . q.2 a.3 arg.19, 1259.

Aristotle. Metaphysics . Ed. by T. by W. D. Ross. The Internet Classics Archive, 350BCE.

M. Assran et al. 'Self-supervised learning from images with a joint-embedding predictive architecture'. In: CVPR . 2023.

J. Bai et al. 'Qwen Technical Report'. In: arXiv preprint arXiv:2309.16609 (2023).

J. Bai et al. 'Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond'. In: (2023).

M. E. Banani et al. 'Probing the 3D Awareness of Visual Foundation Models'. In: arXiv preprint arXiv:2404.08636 (2024).

G. Baruch et al. 'ARKitScenes - A Diverse Real-World Dataset for 3D Indoor Scene Understanding Using Mobile RGB-D Data'. In: NeurIPS Datasets and Benchmarks Track (Round 1) . 2021.

J. Belouadi, A. Lauscher, and S. Eger. 'Automatikz: Text-guided synthesis of scientific vector graphics with tikz'. In: ICLR . 2024.

R. Birkl, D. Wofk, and M. Müller. 'Midas v3. 1-a model zoo for robust monocular relative depth estimation'. In: arXiv preprint arXiv:2307.14460 (2023).

A. F. Biten et al. 'Latr: Layout-aware transformer for scene-text vqa'. In: CVPR . 2022.

A. F. Biten et al. 'Scene text visual question answering'. In: ICCV . 2019.

G. Brazil et al. 'Omni3d: A large benchmark and model for 3d object detection in the wild'. In: CVPR . 2023.

J. Buchner. imagehash (fork) . https://github.com/JohannesBuchner/imagehash . 2021.

H. Caesar et al. 'nuscenes: A multimodal dataset for autonomous driving'. In: CVPR . 2020.

J. Cha et al. 'Honeybee: Locality-enhanced projector for multimodal llm'. In: CVPR . 2024.

S. Cha et al. 'Visually Dehallucinative Instruction Generation: Know What You Don't Know'. In: arXiv preprint arXiv:2402.09717 (2024).

D. J. Chalmers. 'Does Thought Require Sensory Grounding? From Pure Thinkers to Large Language Models'. In: Proceedings and Addresses of the American Philosophical Association 97 (2023), pp. 22-45.

Y. Chang et al. 'A survey on evaluation of large language models'. In: ACM Transactions on Intelligent Systems and Technology 15.3 (2024), pp. 1-45.

G. H. Chen et al. 'ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision-Language Model'. In: arXiv preprint arXiv:2402.11684 (2024).

L. Chen et al. 'Are We on the Right Way for Evaluating Large Vision-Language Models?' In: arXiv preprint arXiv:2403.20330 (2024).

L. Chen et al. 'Sharegpt4v: Improving large multi-modal models with better captions'. In: arXiv preprint arXiv:2311.12793 (2023).

X. Chen et al. 'Pali: A jointly-scaled multilingual language-image model'. In: ICLR . 2023.

X. Chen, S. Xie, and K. He. 'An empirical study of training self-supervised vision transformers'. In: ICCV . 2021.

Z. Chen et al. 'How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites'. In: arXiv preprint arXiv:2404.16821 (2024).

Z. Chen et al. 'Finqa: A dataset of numerical reasoning over financial data'. In: EMNLP . 2021.

Z. Cheng et al. 'HiTab: A hierarchical table dataset for question answering and natural language generation'. In: ACL . 2022.

M. Cherti et al. 'Reproducible scaling laws for contrastive language-image learning'. In: CVPR . 2023.

W.-L. Chiang et al. 'Chatbot arena: An open platform for evaluating llms by human preference'. In: arXiv preprint arXiv:2403.04132 (2024).

X. Chu et al. 'Mobilevlm v2: Faster and stronger baseline for vision language model'. In: arXiv preprint arXiv:2402.03766 (2024).

M. Conover et al. Free Dolly: Introducing the World's First Truly Open Instruction-Tuned LLM . 2023. URL: https://www.databricks.com/blog/2023/04/12/dolly-firstopen-commercially-viable-instruction-tuned-llm (visited on 06/30/2023).

W. Dai et al. 'Instructblip: Towards general-purpose vision-language models with instruction tuning'. In: NeurIPS . 2024.

H. Dong et al. 'Rlhf workflow: From reward modeling to online rlhf'. In: arXiv preprint arXiv:2405.07863 (2024).

A. Dosovitskiy et al. 'An image is worth 16x16 words: Transformers for image recognition at scale'. In: ICLR . 2021.

A. Fang et al. 'Data filtering networks'. In: ICLR . 2024.

X. Fu et al. 'BLINK: Multimodal Large Language Models Can See but Not Perceive'. In: arXiv preprint arXiv:2404.12390 (2024).

J. Gao et al. 'G-llava: Solving geometric problem with multi-modal large language model'. In: arXiv preprint arXiv:2312.11370 (2023).

P. Gao et al. 'Llama-adapter v2: Parameter-efficient visual instruction model'. In: arXiv preprint arXiv:2304.15010 (2023).

P. Gao et al. 'SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models'. In: arXiv preprint arXiv:2402.05935 (2024).

Y. Ge et al. 'Planting a seed of vision in large language model'. In: arXiv preprint arXiv:2307.08041 (2023).

A. Geiger, P. Lenz, and R. Urtasun. 'Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite'. In: CVPR . 2012.

R. Geirhos et al. 'Shortcut learning in deep neural networks'. In: Nature Machine Intelligence (2020).

R. Girshick et al. 'Rich feature hierarchies for accurate object detection and semantic segmentation'. In: CVPR . 2014.

Google. Gemini . 2023.

Y. Goyal et al. 'Making the v in vqa matter: Elevating the role of image understanding in visual question answering'. In: CVPR . 2017.

D. Gurari et al. 'Vizwiz grand challenge: Answering visual questions from blind people'. In: CVPR . 2018.

K. He et al. 'Masked autoencoders are scalable vision learners'. In: CVPR . 2022.

X. He et al. 'PathVQA: 30000+ Questions for Medical Visual Question Answering'. In: CoRR abs/2003.10286 (2020).

T. Hiippala et al. 'AI2D-RST: A multimodal corpus of 1000 primary school science diagrams'. In: Language Resources and Evaluation 55 (2021), pp. 661-688.

J. Hoffmann et al. 'Training compute-optimal large language models'. In: NeurIPS (2023).

Y.-C. Hsiao, F. Zubach, M. Wang, et al. 'Screenqa: Large-scale question-answer pairs over mobile app screenshots'. In: arXiv preprint arXiv:2209.08199 (2022).

D. A. Hudson and C. D. Manning. 'GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering'. In: CVPR . 2019.

A. Jaegle et al. 'Perceiver: General perception with iterative attention'. In: ICML . 2021.

J. Johnson et al. 'Clevr: A diagnostic dataset for compositional language and elementary visual reasoning'. In: CVPR . 2017.

N. Jouppi et al. 'Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings'. In: Proceedings of the 50th Annual International Symposium on Computer Architecture . 2023.

K. Kafle et al. 'Dvqa: Understanding data visualizations via question answering'. In: CVPR . 2018.

S. Kantharaj et al. 'Chart-to-text: A large-scale benchmark for chart summarization'. In: ACL . 2022.

S. Karamcheti et al. 'Prismatic vlms: Investigating the design space of visually-conditioned language models'. In: arXiv preprint arXiv:2402.07865 (2024).

M. Kazemi et al. 'Geomverse: A systematic evaluation of large models for geometric reasoning'. In: 2023.

A. Kembhavi et al. 'A diagram is worth a dozen images'. In: ECCV . 2016.

D. Kiela et al. 'The hateful memes challenge: Detecting hate speech in multimodal memes'. In: NeurIPS . 2020.

G. Kim et al. 'Donut: Document understanding transformer without ocr'. In: ECCV . 2022.

A. Kirillov et al. 'Segment anything'. In: ICCV . 2023.

R. Krishna et al. 'Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations'. In: IJCV (2016).

LAION. laion/gpt4v-dataset . 2023.

H. Laurençon, L. Tronchon, and V. Sanh. 'Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset'. In: arXiv preprint arXiv:2403.09029 (2024).

H. Laurençon et al. 'What matters when building vision-language models?' In: arXiv preprint arXiv:2405.02246 (2024).

A. C. Li et al. 'Internet Explorer: Targeted Representation Learning on the Open Web'. In: ICML . 2023.

A. C. Li et al. 'Your diffusion model is secretly a zero-shot classifier'. In: ICCV . 2023.

B. Li et al. LLaVA-NeXT: Stronger LLMs Supercharge Multimodal Capabilities in the Wild . 2024.

L. Li et al. 'Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models'. In: arXiv preprint arXiv:2403.00231 (2024).

Y. Li et al. 'Mini-gemini: Mining the potential of multi-modality vision language models'. In: arXiv preprint arXiv:2403.18814 (2024).

W. Lian et al. OpenOrca: An Open Dataset of GPT Augmented FLAN Reasoning Traces . https://https://huggingface.co/Open-Orca/OpenOrca . 2023.

T.-Y. Lin et al. 'Microsoft coco: Common objects in context'. In: ECCV . 2014.

H. Liu et al. 'Improved baselines with visual instruction tuning'. In: arXiv preprint arXiv:2310.03744 (2023).

H. Liu et al. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge . 2024.

H. Liu et al. 'Visual Instruction Tuning'. In: NeurIPS . 2023.

Y. Liu et al. 'Mmbench: Is your multi-modal model an all-around player?' In: arXiv preprint arXiv:2307.06281 (2023).

Y. Liu et al. 'On the hidden mystery of ocr in large multimodal models'. In: arXiv preprint arXiv:2305.07895 (2023).

Z. Liu and K. He. 'A Decade's Battle on Dataset Bias: Are We There Yet?' In: arXiv preprint arXiv:2403.08632 (2024).

Z. Liu et al. 'A convnet for the 2020s'. In: CVPR . 2022.

H. Lu et al. 'DeepSeek-VL: towards real-world vision-language understanding'. In: arXiv preprint arXiv:2403.05525 (2024).

P. Lu et al. 'Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning'. In: ICLR . 2023.

P. Lu et al. 'Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning'. In: NeurIPS . 2021.

P. Lu et al. 'Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning'. In: ACL . 2021.

P. Lu et al. 'Learn to explain: Multimodal reasoning via thought chains for science question answering'. In: NeurIPS . 2022.

P. Lu et al. 'Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts'. In: ICLR (2023).

Z. Luo et al. 'Wizardcoder: Empowering code large language models with evol-instruct'. In: ICLR . 2024.

A. Majumdar et al. 'OpenEQA: Embodied Question Answering in the Era of Foundation Models'. In: 2nd Workshop on Mobile Manipulation and Embodied Intelligence at ICRA 2024 . 2024.

K. Marino et al. 'OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge'. In: CVPR . 2019.

A. Masry et al. 'Chartqa: A benchmark for question answering about charts with visual and logical reasoning'. In: ACL . 2022.

M. Mathew, D. Karatzas, and C. Jawahar. 'Docvqa: A dataset for vqa on document images'. In: WACV . 2021.

B. McKinzie et al. 'Mm1: Methods, analysis & insights from multimodal llm pre-training'. In: arXiv preprint arXiv:2403.09611 (2024).

A. Mitra et al. Orca-Math: Unlocking the potential of SLMs in Grade School Math . 2024. arXiv: 2402.14830 [cs.CL] .

'OCR-VQA: Visual Question Answering by Reading Text in Images'. In: 2019.

OpenAI. ChatGPT . 2022.

OpenAI. gpt4o . 2024.

M. Oquab et al. 'Dinov2: Learning robust visual features without supervision'. In: TMLR (2023).

L. Ouyang et al. 'Training language models to follow instructions with human feedback'. In: NeurIPS . 2022.

A. Parker. In the blink of an eye: how vision sparked the big bang of evolution . 2003.

P. Pasupat and P. Liang. 'Compositional semantic parsing on semi-structured tables'. In: ACL . 2015.

J. Piaget, M. Cook, et al. The origins of intelligence in children . Vol. 8. 5. International Universities Press New York, 1952.

J. Pont-Tuset et al. 'Connecting Vision and Language with Localized Narratives'. In: ECCV . 2020.

A. Radford et al. 'Learning transferable visual models from natural language supervision'. In: ICML . 2021.

R. Rafailov et al. 'Direct preference optimization: Your language model is secretly a reward model'. In: NeurIPS . 2024.

M. Roberts et al. 'Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding'. In: ICCV . 2021.

R. Rombach et al. 'High-Resolution Image Synthesis With Latent Diffusion Models'. In: CVPR . 2022.

O. Russakovsky et al. 'Imagenet large scale visual recognition challenge'. In: IJCV (2015).

O. Sanseviero. LLM Evals and Benchmarking . 2022.

C. Schuhmann et al. 'Laion-5b: An open large-scale dataset for training next generation image-text models'. In: NeurIPS . 2022.

D. Schwenk et al. 'A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge'. In: ECCV . 2022.

M. Shridhar et al. 'ALFWorld: Aligning Text and Embodied Environments for Interactive Learning'. In: ICLR . 2021.

C. Si et al. 'Design2Code: How Far Are We From Automating Front-End Engineering?' In: arXiv preprint arXiv:2403.03163 (2024).

O. Sidorov et al. TextCaps: a Dataset for Image Captioning with Reading Comprehension . 2020. arXiv: 2003.12462 [cs.CV] .

A. Singh et al. 'Towards vqa models that can read'. In: CVPR . 2019.

S. Song, S. P. Lichtenberg, and J. Xiao. 'Sun rgb-d: A rgb-d scene understanding benchmark suite'. In: CVPR . 2015.

Q. Sun et al. 'Eva-clip: Improved training techniques for clip at scale'. In: arXiv preprint arXiv:2303.15389 (2023).

R. Tanaka, K. Nishida, and S. Yoshida. 'VisualMRC: Machine Reading Comprehension on Document Images'. In: AAAI . 2021.

B. J. Tang, A. Boggust, and A. Satyanarayan. 'Vistext: A benchmark for semantically rich chart captioning'. In: arXiv preprint arXiv:2307.05356 (2023).

S. Tong, E. Jones, and J. Steinhardt. 'Mass-producing failures of multimodal systems with language models'. In: NeurIPS . 2024.

S. Tong et al. 'Eyes wide shut? exploring the visual shortcomings of multimodal llms'. In: CVPR . 2024.

H. Touvron et al. 'LLaMA 2: Open foundation and fine-tuned chat models'. In: (2023).

H. Touvron et al. 'LLaMA: Open and efficient foundation language models'. In: arXiv preprint arXiv:2302.13971 (2023).

H. Tu et al. 'How many unicorns are in this image? a safety evaluation benchmark for vision llms'. In: arXiv preprint arXiv:2311.16101 (2023).

K. Vishniakov, Z. Shen, and Z. Liu. 'ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy'. In: ICML . 2024.

J. Wang et al. 'To see is to believe: Prompting gpt-4v for better visual instruction tuning'. In: arXiv preprint arXiv:2311.07574 (2023).

K. Wang et al. 'Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset'. In: arXiv preprint arXiv:2402.14804 (2024).

J. Wei et al. 'Chain-of-thought prompting elicits reasoning in large language models'. In: NeurIPS . 2022.

C. Wendler. wendlerc/RenderedText . 2023.

H. Wu et al. 'Q-instruct: Improving low-level visual abilities for multi-modality foundation models'. In: arXiv preprint arXiv:2311.06783 (2023).

P. Wu and S. Xie. 'V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs'. In: CVPR . 2024.

xAI. grok . 2024.

H. Xu et al. 'Demystifying clip data'. In: ICLR . 2024.

A. Young et al. 'Yi: Open foundation models by 01. ai'. In: arXiv preprint arXiv:2403.04652 (2024).

L. Yu et al. Modeling Context in Referring Expressions . 2016. arXiv: 1608.00272 [cs.CV] .

T. Yu et al. 'Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback'. In: arXiv preprint arXiv:2312.00849 (2023).

X. Yue et al. 'Mammoth: Building math generalist models through hybrid instruction tuning'. In: ICLR . 2024.

X. Yue et al. 'Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi'. In: CVPR . 2024.

M. Yuksekgonul et al. 'When and why vision-language models behave like bags-of-words, and what to do about it?' In: ICLR . 2022.

X. Zhai et al. 'Sigmoid loss for language image pre-training'. In: ICCV . 2023.

Y. Zhai et al. 'Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning'. In: arXiv preprint arXiv:2405.10292 (2024).

Y. Zhai et al. 'Investigating the catastrophic forgetting in multimodal large language models'. In: CPAL . 2024.

C. Zhang et al. 'Raven: A dataset for relational and analogical visual reasoning'. In: CVPR . 2019.

Y. Zhang et al. 'Llavar: Enhanced visual instruction tuning for text-rich image understanding'. In: arXiv preprint arXiv:2306.17107 (2023).

Y. Zhao et al. 'Pytorch fsdp: experiences on scaling fully sharded data parallel'. In: arXiv preprint arXiv:2304.11277 (2023).

L. Zheng et al. 'Judging llm-as-a-judge with mt-bench and chatbot arena'. In: NeurIPS . 2024.

T. Zheng et al. 'OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement'. In: arXiv preprint arXiv:2402.14658 (2024).

V. Zhong, C. Xiong, and R. Socher. 'Seq2sql: Generating structured queries from natural language using reinforcement learning'. In: 2017.

B. Zhou et al. 'Semantic understanding of scenes through the ade20k dataset'. In: IJCV (2019).

K. Zhou et al. 'Don't Make Your LLM an Evaluation Benchmark Cheater'. In: arXiv preprint arXiv:2311.01964 (2023).

B. Zhu et al. Starling-7b: Improving llm helpfulness & harmlessness with rlaif . 2023.

D. Zhu et al. 'Minigpt-4: Enhancing vision-language understanding with advanced large language models'. In: arXiv preprint arXiv:2304.10592 (2023).

F. Zhu et al. 'TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance'. In: ACL . 2021.

Y. Zhu et al. 'Visual7w: Grounded question answering in images'. In: CVPR . 2016.

A Broader Discussion

We advocate for using MLLMs as an interface to evaluate visual representations, as previous benchmarks are becoming saturated and do not adequately reflect the diverse and complex perception challenges of the real world. Our work highlights the current gap between language-supervised models and self-supervised learning models and demonstrates the potential of bridging this gap. However, it is known that features of language-supervised models behave like a bag-of-words [125, 144], underscoring the need for advancements in vision-only models to ensure better visual understanding. We hope to inspire future research into developing better vision-only models intended to be adapted into the MLLM setting, that more effectively leverage large-scale datasets [85] and preserve the advantages in visual grounding [126].

As we observe in Table 4, a well-trained open-source model such as Cambrian-1 can match or even outperform proprietary models on many existing benchmarks. However, the use and evaluation of MLLMs extend far beyond the current scope of benchmarks-to conversational ability, creativity, reliability, and overall user experience. Developing models solely based on benchmark results can result in an 'answer machine', over-optimized for benchmarks but lacking in practical interaction capabilities. Therefore, the development of MLLMs that better align with human and societal needs is a continuously evolving process, both in terms of evaluation and model development.

Our current Cambrian-1 model uses a moderate number of visual tokens and does not adopt the anyresolution strategy [30, 77, 81] to handle ultra high-resolution images or those with extreme aspect ratios, which require a larger number of visual tokens. For specialized tasks like V*Bench [136], which require processing ultra high-resolution images, increasing the resolution and number of visual tokens could lead to an HD version of the Cambrian-1 model.

One promising direction for post-training alignment is through reinforcement learning rather than supervised fine-tuning. Many MLLM studies, including Cambrian, primarily focus on supervised fine-tuning. Yet, recent advancements in LLMs [38, 104, 110, 156] and some in MLLMs [141, 146] suggest that reinforcement learning from human or environmental feedback can further improve models, potentially surpassing the limits of supervised fine-tuning, especially in decision-making abilities.

Cambrian-10M (Fig. 8 and Section 4) provides a rich pool of data for studying data curation in fine-tuning MLLMs. Our work takes an initial step in curating higher-quality data to enable more efficient and effective instruction tuning. We believe there is room for further improvement in the data curation pipeline, and we hope this work can serve as a foundation for future research.

Additionally, training large-scale models requires careful design of model sharding, data sharding, and infrastructure adaptations. In this work, we train our model on TPU-V4 [60] with FSDP [150] using TorchXLA. We share our experiences, technical challenges, and solutions in Appendix C. We also open-source our implementation and provide tutorials to help the community undertake large-scale training more efficiently.

To conclude, Cambrian-1 introduces a family of state-of-the-art MLLM models that achieve top performance across diverse benchmarks and excel in visual-centric tasks. We provide model weights, open-source code, datasets, and detailed recipes for model training and evaluation. We hope our work will strengthen the open research community and accelerate future advancements in both visual representation learning and multimodal systems.

B Multimodal LLMs: Preliminaries and Related Work

The key components of MLLM research include the Large Language Model , Visual Encoder , Multimodal Connector , Data Curation Pipeline , Instruction Tuning Strategy , and Evaluation & Benchmarking . Each component has its intricacies, and understanding their interactions presents significant challenges. Our study investigates these aspects from a vision-centric perspective.

Large Language Model Advanced LLMs [4, 101, 127, 128] are the foundation of an MLLM. After instruction-tuning on multimodal data, these models can be prompted to solve a variety of complex tasks and generate free-form responses leveraging input from a visual encoder. Recent MLLM research focuses on enhancing the LLM backbone [10, 75, 81], resulting in improved performance on benchmarks like MMMU [143] and AI2D [54]. However, this improvement raises the concern

that our current multimodal evaluation is biased by the development of LLMs, neglecting a true assessment of visual perception. For example, some benchmarks such as MMMU [143] are dominated by LLM capabilities, underscoring the need for evaluations that genuinely assess multimodality (see Section 2.1).

Visual Encoder Most MLLMs utilize language-supervised models like CLIP [109, 122, 145], which benefit from the massive scale of noisy web image-text data. However, there is a much broader pool of visual models that learn representations using only visual signals-such as self-supervised models [9, 103], segmentation [68], depth-supervised [15], and diffusion models [74, 112] (see Fig. 2). Recent work [87, 126] advocates for incorporating these diverse vision models into MLLMs. In this study, we systematically examine the impact of various vision backbones on MLLM performance (Section 1) and explore the benefits of model ensembles (Section 2.5).

Multimodal Connector Representations from a visual encoder cannot be natively processed by an LLM-they must be mapped into the LLM token space by a connector . There are three primary approaches to connector design: Resamplers [6], Q-Formers [11, 37], and MLP Projectors [43, 80, 82, 157]. We begin our exploration using an MLP projector, which is highly effective but presents challenges: the visual token count grows quadratically with image resolution, inhibiting scaling context length input resolution. For example, LLaVA-Next [81] requires 2880 visual tokens to process one 672px image. To address this, we explore new vision connector designs that process high-resolution images while maintaining a smaller number of visual tokens (Section 3).

Instruction Tuning Data Visual instruction tuning data is crucial but hard to collect, as it rarely naturally exists on the internet. Previous work [37, 80, 98] transforms existing VQA benchmarks [50, 69] into instruction tuning data, showing marked MLLM performance improvements. With this inspiration, we collect all VQA benchmarks and visual interaction data that we can find (Fig. 8), study data balancing and category mixtures (Section 4.2), and develop an internet data collection engine to fill in the gaps (Section 4.1).

Instruction Tuning Most current MLLMs leverage pre-trained LLMs and visual encoders, finetuning the LLM and connector using visual instruction tuning data. Some aspects of the tuning recipe are up for debate, including whether to pre-train the connector before joint fine-tuning with the LLM , and whether to freeze or unfreeze the vision encoder during fine-tuning [63, 98]. Additionally, some recent proprietary models explore end-to-end training from scratch [49, 102]. In this work, we use pre-trained models and revisit the debated recipe aspects with extensive studies, providing more insights for future MLLM research (Section 2.3).

Evaluation & Benchmarking There is an extensive set of benchmarks that evaluate various aspects of MLLMs, such as perception [45, 83], knowledge [91, 92], chart interpretation [84, 96], and visual capabilities [126, 136]. Instead of over-optimizing for specific benchmarks, we advocate for examining aggregates of benchmarks that focus on specific capabilities. To achieve this, we analyze existing benchmarks, categorize them, and assess the extent to which they measure multimodality (Section 2.1). Additionally, we find there are currently few benchmarks focused on vision-centric evaluation, and those that do exist contain relatively few images, leading to higher variance during evaluation. To address this issue, we propose a new vision-centric benchmark by reformulating classic vision tasks (Section 2.2).

C Training, Infrastructure, and Implementation

All models in this paper were trained using TPU-V4 pods [60]; we evaluate using NVIDIA A6000, A100, and H100 cards. The experiments in Section 2.4 require less than 24 hours on a TPU-V4-128, while our final Cambrian-1 models are trained in less than 4 days on a TPU-V4-512.

To enable and facilitate large-scale parallel training on TPUs, we employ TorchXLA with FSDP [150] to handle training sharding and parallelism. Training a large-scale multimodal model with TorchXLA on TPU is a challenging journey, as there are no open-source codebases and many critical features are not supported in the TorchXLA or TorchXLA FSDP libraries. To provide a brief taste of the difficulties: TPUs require a static graph throughout the program, which requires ground-up rewrites of dynamically-written open-source PyTorch codebases; model resuming is not implemented in TorchXLA, which is especially crucial when training on preemptable TPUs; existing TorchXLA FSDP tutorials fail to compile due to version changes in TorchXLA, updates in Hugging Face

Transformers & Accelerate, or simply inherent issues with the tutorial; loading very large models (over 30 billion parameters) with the TorchXLA FSDP library is natively impossible due to the 100GB memory constraints of TPU-V4s, and requires extensive workarounds.

To this end, we have rewritten or developed many new functions to make this research possible. For instance, we rewrote the TorchXLA FSDP Sharding API to load very large models; we implemented model resuming on TorchXLA; we rewrote parts of the Hugging Face Transformers FSDP and gradient checkpointing implementations to enable large-scale FSDP training. We are committed to open-sourcing our codebase and publishing a comprehensive tutorial to share our insights, with the hope of inspiring and supporting future research and open-source contributions to the TPU and TorchXLA ecosystem.

D Analyzing the Benchmarks

MLLMBenchmark Performance Confusion Matrix

We evaluate the benchmark scores for our one-stage, two-stage finetune-only and hybrid models, and then plot the correlation matrix for the pool of MLLM benchmarks. The correlation plot displays in Fig. 10. The result demonstrates that MMMU is less correlated in measuring model performance to other benchmarks. Nonetheless, we acknowledge it is widely used and therefore cluster it into knowledge-based QAs based on the nature of their questions.

MME

MMB

SEED-I

GQA

SQA-I

MMMU

MathVista

AI2D

ChartQA

OCRBench

TextVQA

DocVQA

MMVP

RW QA

CV-CB

MME

MMB

SEED-I

GQA

SQA-I

MMMU

MathVista

AI2D

ChartQA

OCRBench

TextVQA

DocVQA

MMVP

RW QA

CV-CB

1.00

0.97

0.96

0.96

0.82

0.65

0.75

0.82

0.78

0.82

0.84

0.79

0.87

0.84

0.78

0.97

1.00

0.97

0.95

0.85

0.73

0.75

0.84

0.80

0.87

0.87

0.81

0.87

0.88

0.79

0.96

0.97

1.00

0.97

0.79

0.70

0.77

0.85

0.78

0.83

0.84

0.79

0.91

0.91

0.83

0.96

0.95

0.97

1.00

0.70

0.69

0.71

0.76

0.68

0.73

0.74

0.69

0.87

0.83

0.78

0.82

0.85

0.79

0.70

1.00

0.62

0.75

0.78

0.89

0.94

0.95

0.93

0.71

0.83

0.72

0.65

0.73

0.70

0.69

0.62

1.00

0.57

0.54

0.53

0.65

0.65

0.58

0.51

0.60

0.62

0.75

0.75

0.77

0.71

0.75

0.57

1.00

0.65

0.78

0.78

0.80

0.78

0.74

0.74

0.52

0.82

0.84

0.85

0.76

0.78

0.54

0.65

1.00

0.83

0.86

0.86

0.83

0.81

0.85

0.76

0.78

0.80

0.78

0.68

0.89

0.53

0.78

0.83

1.00

0.95

0.96

0.98

0.75

0.85

0.66

0.82

0.87

0.83

0.73

0.94

0.65

0.78

0.86

0.95

1.00

0.98

0.95

0.76

0.89

0.72

0.84

0.87

0.84

0.74

0.95

0.65

0.80

0.86

0.96

0.98

1.00

0.98

0.79

0.89

0.74

0.79

0.81

0.79

0.69

0.93

0.58

0.78

0.83

0.98

0.95

0.98

1.00

0.73

0.85

0.70

0.87

0.87

0.91

0.87

0.71

0.51

0.74

0.81

0.75

0.76

0.79

0.73

1.00

0.90

0.79

0.84

0.88

0.91

0.83

0.83

0.60

0.74

0.85

0.85

0.89

0.89

0.85

0.90

1.00

0.84

0.78

0.79

0.83

0.78

0.72

0.62

0.52

0.76

0.66

0.72

0.74

0.70

0.79

0.84

1.00

0.6

0.7

0.8

0.9

1.0

E Cambrian Vision-Centric Benchmark (CV-Bench)

CV-Bench Curation Below we describe the procedure for programmatically constructing questions for each task. To ensure reliability, we also manually inspect each question , removing those that are unclear, ambiguous, or erroneous.

Spatial Relationship (2D). We consider images with two distinct ground-truth object categories and use visual prompts (bounding boxes) to avoid ambiguity when multiple instances are present. In these questions, we designate an anchor object, and the question asks for the direction of the other object relative to this anchor.

Object Counting (2D). This tests the model's ability to count objects. When generating options for these questions, we construct multiple-choice options that are similar to the correct answer. For

Spatial Relationship

Where is the cave located with

respect to the trees?

How many cars

are in the image?

Object Count

Which is closer to the camera,

sink

or

pillow

?

Relative Distance

Which is closer to the

chair

,

refrigerator

or

door

?

Source benchmark: ADE20K [154] and COCO [79]

Source benchmark: Omni3D [18]

example, if the correct answer is 4, the options might be 2, 3, 4, 5, & 6. We also include existence check examples where the correct count is 0.

Depth Order (3D). We consider images with two distinct categories (i.e., object A and object B) and use visual prompts (e.g., bounding boxes with two different colors) to avoid ambiguity. We define 'closer' as follows: object A is closer to the camera than object B only if the farthest vertex of object A is closer ‡‡ to the camera than the nearest vertex of object B by a specified offset.

Relative Distance (3D). We consider images with three distinct categories (i.e., anchor, object A, and object B), and use visual prompts (e.g., bounding boxes with three different colors) to avoid ambiguity. Object A is closer than object B only if the farthest distance from A's vertices is shorter than the shortest distance from B's vertices to the anchor object by a certain offset.

Curation Procedure We provide an overview of the data curation process in Fig. 12, which is conducted in a semi-automatic manner. The procedure consists of two main steps:

First, using the original benchmarks and their associated ground truth annotations, we generate query and answer pairs. These pairs are tailored to specific tasks: 2D-related tasks with COCO and ADE20K datasets, and 3D-related tasks with Omni3D.

Second, after generating the query and answer pairs, we engage human experts to manually filter out any incorrect or ambiguous queries to enhance the quality of benchmark. Each query is assigned one of three statuses: accepted (used as is), modified (where the incorrect answer is modified), and rejected (queries that are ambiguous, such as those too small or difficult to discern, even for human experts).

Following this two-stage process, we finalize the benchmark, which results in a total of 2638 image queries with improved accuracy and reliability. Subsequently, we will discuss the methods of human verification and the evaluation metrics used in this process.

Human verification

Spatial

2D Data

Relations

3DData

Accepted

ADE20k

Count

Manual

COCO

Generate

Filter,

AIIQ&A

Filter

Modified

Benchmark

Dataset

Q&A

Relative

Distance

OMNI3D

Rejected

Relative

Depth

There are multiple reasons for the above generated data to be inaccurate. One of the main reasons is sparse annotations, but occasionally there could be wrong annotations as well.

Thus, we need manual inspection to change/remove these examples generated. Here are a few criteria we followed while manually filtering COCO and ADE20k data.

For Counting question types, if all instances of a category are not annotated, the ground truth would have lower count than the actual number of instances appearing in the image. In a few cases where the image distinctly has different countable instances of the object, we change the options/answer. In case the count is ambiguous, we reject the data sample altogether.

For Relative Distance question types without annotation, if the question is asked about two objects A and B and if there are two instances of a specific category (say A), the relative location of A w.r.t B can be have multiple correct answers. We reject the sample in this case. We also reject cases with clear incorrect annotations.

Benchmark Evaluation To ensure that equal importance is given to both 2D and 3D tasks, we use an evaluation metric that is the average of the accuracies obtained from these tasks. Specifically, the overall performance is calculated as follows:

F Vision Models in MLLMs

As mentioned in Section 2.4, we use MLLM as an interface to evaluate vision model's different capabilities. Here, we list details in terms of the model selection, full results, and data split.

F.1 Details of Vision Models

In our exploration of versatile vision models, we select thirteen models and group them into four categories: language-supervised models (i.e., OpenAI CLIP [109], SigLIP [145], DFN-CLIP [40], EVA-CLIP [122] and OpenCLIP [33]), self-supervised models (i.e., DINOv2 [103], I-JEPA [9], MAE [52], MoCo v3 [29]), class-supervised models (ImagetNet22K ViT [39]) and other models such as stable diffusion [112], segmentation models like SAM [68], and depth estimation models like MiDaS [15]. To provide a clear understanding of the specific variant evaluated, we meticulously detail their backbone architectures, resolution, number of tokens, and hidden dimension sizes in Table 6. For models that output a large number of patches in the last layer (e.g., SAM and ConvNeXt) we interpolate to the number of tokens specified in Table 6, and denote interpolation with I .

F.2 Full Results of Different Vision Backbones

Here, we also show a ranking version of Fig. 5. We observe a clear advantage of CLIP models over non-CLIP models. We also observe that within the family of CLIP models, each model perform differently in different domains. This provide insight into both vision model development and data curation in training large vision models.

For the above-listed vision models in Table 6, they are integrated as the vision encoder of the MLLMs. These MLLMs are trained on various adapter adapter data splits (i.e., 0, 0.5 and 1.2 million), and subsequently fine-tuned on a 737K instruction tuning dataset provided in LLaVA-1.5[80]. For the adapter data splits, the 0M split indicates that no initial adapter pertaining phase is employed for the

Language Supervised

Self-Supervised & Other

MLLM. The 0.5M data split utilizes the 558K adapter data from LLaVA-1.5[80], while the 1.2M variant uses ShareGPT4V-PT dataset [27].

0M Adapter Data + 737K Instruction Tuning Data As shown in Table 9, we provide 20 results for different variants of the above-mentioned thirteen vision backbones. Among them, languagesupervised models show superior performance. Especially, OpenCLIP ConvNeXT-XXL@1024 model surpasses all other models on DocVQA with over 12%, indicating its potential to handle OCR-related benchmarks.

Vision Backbone

General

Knowledge

OCR & Chart

Vision-Centric

Model

Architecture

Average

MME P

MMB

SEED I

GQA

SQA I

MMMU V

MathVista M

AI2D

ChartQA

OCRBench

TextVQA

DocVQA

MMVP

RealWorldQA

CV-Bench 2D

CV-Bench 3D

Language Supervised

OpenAI CLIP

ViT-L/14@336

48.37

1,419.43 61.45

59.85 62.26

69.87 34.50 27.80 59.82

33.73 31.70 55.39 28.00

11.33 53.46 56.44 57.40

DFN-CLIP

ViT-L/14@224

38.78

1,172.50 49.53

49.74 52.94

67.74 34.00 27.30 56.99

16.75

4.87 44.81 11.19

6.67 46.97 40.29 52.08

DFN-CLIP

ViT-H/14@378

36.79

1,091.76 41.28

44.32 50.48

65.54 33.65 26.50 56.76

15.56

2.70 43.41 10.15

4.67 47.06 39.07 52.91

EVA-CLIP-02 ViT-L/14@336

45.84

1,325.17 58.21

62.99 62.03

68.67 35.00 27.50 58.26

19.40 22.50 51.08 16.36

24.67 52.68 52.98 54.83

SigLIP

ViT-L/16@384

48.80

1,383.42 61.02

63.56 61.85

68.91 35.29 29.70 57.87

34.96 29.60 56.73 28.31

23.33 52.68 52.95 54.83

SigLIP

ViT-SO400M/14@384

47.57

1,376.75 58.76

60.59 60.92

69.01 34.40 26.50 58.35

30.72 28.60 55.10 28.31

19.33 50.71 52.33 58.67

OpenCLIP

ConvNeXt-L@512

47.38

1,404.01 57.62

61.90 60.34

69.06 33.90 29.10 58.39

28.04 25.20 55.45 28.41

24.00 54.12 53.46 48.91

OpenCLIP

ConvNeXt-L@1024

39.02

1,139.60 14.64

49.59 37.91

65.71 34.30 27.30 54.13

32.97 12.05 52.61 38.36

9.67 47.45 52.68 38.04

OpenCLIP

ConvNeXt-XXL@1024 41.83 1,219.47 48.00

49.88 55.09

66.14 35.69 27.60 56.67

16.92

5.00 46.90 40.98

16.00 47.32 43.40 52.75

Self Supervised

DINOv2

ViT-L/14@336

41.18

1,262.66 49.62

56.80 60.30

65.10 35.00 26.40 56.41

16.48

3.10 44.04 11.90

18.67 50.20 49.43 52.25

DINOv2

ViT-L/14@518

40.60

1,242.48 51.00

53.39 60.38

64.55 34.50 26.20 57.53

15.11

2.90 44.28 10.95

14.00 48.63 46.13 57.90

MoCo v3

ViT-B/16@224

34.94

966.45 36.77

33.00 47.35

62.96 32.80 26.20 55.05

16.04

2.60 43.81 10.31

6.67 45.36 39.03 52.83

MoCo v3

ViT-L/16@224

34.70

1010.18 34.64

41.71 47.46

64.70 33.70 26.30 55.05

16.24

2.70 42.60 10.39

4.00 45.36 44.67 35.16

MAE

ViT-L/16@224

37.69

1,114.07 42.30

35.93 55.20

63.51 34.60 26.00 56.10

16.11

2.70 43.63 10.83

14.00 44.80 45.81 55.75

MAE

ViT-H/14@224

38.58

1,083.35 41.15

50.99 55.30

64.90 34.10 26.00 56.49

15.63

3.20 43.98 11.00

12.00 46.30 47.18 54.90

I-JEPA

ViT-H/14@224

38.88

1,132.07 44.68

51.74 55.37

66.04 34.20 26.40 56.09

15.84

3.00 43.66 11.48

10.67 46.01 46.74 53.50

Other

SAM

ViT-L/16@1024

31.74

585.78 20.34

36.34 39.85

65.49 34.50 25.10 53.92

16.16

2.70 42.37

9.25

2.00 44.44 35.65 50.50

SAM

ViT-H/16@1024

32.37

648.96 22.30

36.31 40.52

65.20 34.10 26.00 54.44

15.56

2.40 42.39

8.75

2.00 45.36 34.83 55.25

MiDaS 3.0

ViT-L/16@384

35.65

981.36 38.57

40.93 49.04

63.41 31.80 25.70 54.72

16.36

2.60 43.19 11.24

6.67 44.97 38.78 53.40

MiDaS 3.1

ViT-L/16@518

35.44

983.34 34.79

40.20 48.53

64.60 33.90 25.00 55.18

15.64

2.60 42.76 12.08

6.66 43.66 39.63 52.58

Diffusion

SD2.1/16@512

36.59

1,044.28 37.71

42.00 48.38

64.55 33.40 25.70 56.99

15.56

3.10 43.14 10.40

9.33 45.88 44.68 52.40

SupViT

ViT-L/16@224

40.13

1,197.39 46.55

54.72 57.27

65.94 34.00 28.00 56.22

16.44

3.10 43.52 11.82

16.67 46.67 48.49 52.75

SupViT

ViT-H/14@224

37.45

1,082.43 42.61

48.45 52.98

63.51 35.29 26.50 55.78

15.16

3.30 44.16 11.49

4.66 43.79 44.55 52.91

0.5M Adapter Data + 737K Instruction Finetune As shown in Table 10 and Table 9, the inclusion of an alignment stage with 0.5M data split results in a notable increase in performance for DFN-CLIP ViT-H/14@378, from 36.21 to 49.94. This substantial improvement highlights the value of the alignment stage for enhancing certain vision backbones, suggesting its importance in harnessing the full potential of vision models.

1.2M Adapter Data + 737K Instruction Finetune As we increase the amount of data in the alignment phase, we observe a consistent performance improvement for SigLIP ViT-SO400M/14@384

Vision Backbone

General

Knowledge

OCR & Chart

Vision-Centric

Model

Architecture

Average

MME P

MMB

SEED I

GQA

SQA I

MMMU V

MathVista M

AI2D

ChartQA

OCRBench

TextVQA

DocVQA

MMVP

RealWorldQA

CV-Bench 2D

CV-Bench 3D

Language-Supervised

OpenAI CLIP

ViT-L/14@336

49.03

1,413.51 60.34 62.17 60.81

69.76 36.49 29.90 58.48

36.80 30.20 57.63 30.98

21.33 51.63 52.57 54.75

DFN-CLIP

ViT-L/14@224

45.59

1,382.75 57.36 63.26 60.54

66.81 35.09 29.20 57.71

22.88 23.45 52.27 18.82

21.33 51.31 49.59 50.63

DFN-CLIP

ViT-H/14@378

50.62

1,500.45 62.64 66.44 62.53

70.75 35.69 30.30 58.78

39.20 29.80 56.98 31.39

29.33 53.59 54.96 52.58

EVA-CLIP-02 ViT-L/14@336

47.13

1,362.07 62.64 63.96 61.66

69.46 35.89 27.90 56.96

20.96 26.10 53.93 19.07

20.00 53.07 56.28 58.16

SigLIP

ViT-L/16@384

48.11

1,381.48 61.79 61.87 59.45

70.25 35.99 28.80 57.58

28.76 28.20 54.90 25.60

26.00 52.29 52.89 56.33

SigLIP

ViT-SO400M/14@384

50.41

1,327.79 62.13 63.92 61.31

70.38 36.99 30.00 59.52

40.08 33.20 60.37 36.58

22.00 53.99 55.59 54.08

OpenCLIP

ConvNeXt-L@512

48.01

1,366.85 59.66 62.89 61.31

68.77 36.99 28.50 59.29

27.88 28.50 57.57 29.48

16.00 53.20 53.81 55.91

OpenCLIP

ConvNeXt-L@1024

40.29

1,084.62 12.94 51.02 49.78

65.47 34.20 27.60 56.36

29.92 13.25 50.37 43.67

13.33 49.08 51.85 41.58

OpenCLIP

ConvNeXt-XXL@1024 50.45 1,405.65 57.96 63.58 62.41

68.02 34.30 29.40 59.62

42.96 26.20 61.82 42.67

28.67 55.16 49.92 54.16

Self-Supervised

DINOv2

ViT-L/14@336

42.64

1,283.95 54.64 59.03 60.19

66.39 35.29 25.70 58.03

16.00

3.20 45.39 11.79

20.00 50.59 53.51 58.33

MoCo v3

ViT-B/16@224

38.50

1,159.10 40.00 51.37 54.97

65.25 33.70 27.20 55.51

16.36

3.30 44.42 11.42

10.67 46.14 45.80 52.00

MoCo v3

ViT-L/16@224

37.71

1,074.13 41.19 49.46 53.61

63.66 33.70 27.40 55.83

17.04

3.40 43.84 11.98

8.00 46.27 48.69 45.59

MAE

ViT-L/16@224

39.99

1,138.35 44.60 54.91 56.69

65.64 36.19 27.90 56.48

17.20

3.20 44.45 12.42

14.00 47.32 48.83 53.08

I-JEPA

ViT-L/14@224

39.91

1,180.12 44.26 52.86 55.32

65.94 34.40 27.00 57.16

15.88

3.20 44.36 11.61

13.33 46.27 52.19 55.83

Other

SAM

ViT-H/16@1024

32.18

649.99 22.47 36.37 40.46

64.60 32.50 25.80 54.66

15.80

2.70 42.40

8.89

0.00 45.62 37.02 53.08

MiDaS 3.0

ViT-L/16@384

40.07

1,183.95 47.40 53.00 56.15

66.19 32.90 27.60 56.61

17.00

3.00 44.34 11.55

19.33 47.32 45.01 54.58

Diffusion

SD2.1/16@512

38.26

1,123.46 42.04 50.66 53.63

65.74 33.30 24.50 57.48

14.52

3.30 43.95 10.62

10.00 43.53 48.24 54.50

SupViT

ViT-L/16@224

39.66

1,186.88 48.43 54.28 56.35

65.49 33.00 28.10 57.16

17.56

2.80 44.92 12.23

12.67 47.06 43.59 51.67

from 46.79 to 49.72 to 53.09 across 0M, 0.5M to 1.2M data splits as shown in Table 9, Table 10 and Table 11.

Vision Backbone

General

Knowledge

OCR & Chart

Vision-Centric

Model

Architecture

Average

MME P

MMB

SEED I

GQA

SQA I

MMMU V

MathVista M

AI2D

ChartQA

OCRBench

TextVQA

DocVQA

MMVP

RealWorldQA

CV-Bench 2D

CV-Bench 3D

Language-Supervised

OpenAI CLIP

ViT-L/14@336

50.49

1,476.65

61.96 65.45 62.78

69.06 35.00 29.50

58.94

37.84 30.90 58.21 32.11

28.66 54.90 54.14 54.60

DFN-CLIP

ViT-L/14@224

46.01

1,341.14

56.68 63.74 60.75

66.96 33.80 28.65

57.04

23.32 23.20 52.85 18.97

26.67 51.44 52.91 52.08

DFN-CLIP

ViT-H/14@378

51.17

1,426.32

62.38 67.29 62.89

69.01 35.89 30.00

60.01

41.08 30.60 57.53 31.69

32.67 55.95 55.46 55.00

EVA-CLIP-02 ViT-L/14@336

49.71

1,449.78

64.00 67.53 63.60

69.91 35.49 28.40

59.16

24.76 27.10 55.39 21.63

34.67 55.69 57.83 57.75

SigLIP

ViT-L/16@384

50.87

1,424.20

59.40 65.48 62.56

68.67 35.99 29.70

59.29

40.52 33.50 59.59 35.20

28.00 53.33 55.42 56.08

SigLIP

ViT-SO400M/14@384

53.91

1,455.64

63.66 67.62 63.70

72.10 36.09 29.30

61.59

43.76 37.20 61.82 40.19

36.60 56.99 59.61 59.58

OpenCLIP

ConvNeXt-L@512

49.16

1,416.87

60.60 63.87 61.87

69.92 35.79 29.50

59.36

34.40 28.00 58.36 28.41

27.33 51.90 54.64 51.80

OpenCLIP

ConvNeXt-L@1024

51.00

1,392.92

58.21 65.47 62.89

67.43 34.90 29.90

59.13

46.08 25.50 62.14 44.13

26.67 55.29 53.57 55.08

OpenCLIP

ConvNeXt-XXL@1024

52.18

1,402.94

59.40 65.21 62.73

68.27 33.10 29.30

59.84

48.00 28.00 63.27 48.11

34.67 55.95 53.83 55.08

Self-Supervsied

DINOv2

ViT-L/14@336

41.85

1,190.81

51.83 56.90 60.38

66.04 34.20 27.40

56.41

16.44

3.30 45.12 11.79

21.33 49.67 53.91 55.33

MoCo v3

ViT-B/16@224

38.88

1,129.32

41.62 52.19 55.03

65.89 33.30 28.30

56.44

16.48

3.00 44.09 11.47

12.00 47.58 45.17 53.00

MoCo v3

ViT-L/16@224

37.07

1015.20

37.28 48.31 52.63

65.49 34.50 27.60

55.41

16.92

3.10 43.57 11.50

14.67 45.49 45.17 40.75

MAE

ViT-L/16@224

40.39

1,132.80

43.40 55.67 57.42

66.04 35.59 27.80

56.48

17.36

3.30 44.53 12.30

16.00 47.71 49.24 56.75

I-JEPA

ViT-H/14@224

40.27

1,207.88

45.79 54.51 56.15

65.29 34.40 27.10

56.19

16.20

3.20 43.45 11.58

18.00 45.88 49.57 56.58

Other

SAM

ViT-H/16@1024

32.54

682.81

23.32 36.16 40.32

65.20 33.20 26.50

54.21

15.68

2.50 41.76

8.98

1.33 46.80 37.66 52.90

MiDaS 3.0

ViT-L/16@384

39.15

1,132.18

46.21 51.75 55.57

66.30 33.70 26.70

56.06

17.08

3.10 43.65 11.66

15.30 45.75 44.44 52.58

Diffusion

SD2.1/16@512

39.51

1,168.52

40.00 53.80 55.33

64.60 35.00 26.10

57.16

15.36

3.10 44.23 11.06

18.67 47.32 48.04 53.90

Supervised

ViT-L/16@224

39.12

1,216.11

45.28 51.46 55.88

64.15 34.70 26.80

55.76

16.80

2.80 44.42 11.61

11.33 47.97 44.62 51.60

1.2M Adapter Data + 737K Instruction Finetune with Unfrozen Vision Model Here, we present the results of different vision models trained with 1.2m adapter data and 737K instruction tuning data in Appendix F.2. Comparing to Appendix I.2, we observe nearly all the models see improvement on most of the benchmarks, especially on the OCR & Chart and Vision-Centric benchmarks.

1.2M Adapter Data + 5M Instruction Finetune We present the results of 5M instruction tuning experiments in Fig. 6 here. In Table 12, we observe that after 5m instruction tuning, the gap between DINOv2 and CLIP models continue to bridge on general, knowledge and vision-centric benchmarks.

F.3 Model Ensemble

Model Ensemble Details We introduce the implementation details of the model ensemble in Section 2.5. For a given image, the image passes through each vision encoder to obtain the features

Vision Backbone

General

Knowledge

OCR & Chart

Vision-Centric

Model

Architecture

Average

MME P

MMB

SEED I

GQA

SQA I

MMMU V

MathVista M

AI2D

ChartQA

OCRBench

TextVQA

DocVQA

MMVP

RealWorldQA

CV-Bench 2D

CV-Bench 3D

Other

OpenAI CLIP

ViT-L/14@336

52.90

1,477.15 63.15 68.49 64.24

69.21 34.90 28.70 61.24

47.68 40.20 58.92 35.22

26.00 58.82 59.65 56.16

DFN-CLIP

ViT-L/14@224

45.80

1,364.91 55.15 63.06 59.94

67.28 35.59 29.80 58.65

20.48 23.40 52.23 18.29

22.67 50.85 53.59 53.50

EVA-CLIP-02 ViT-L/14@336

51.30

1,492.35 65.53 69.75 65.18

68.62 35.00 29.50 60.78

29.32 29.90 56.87 21.59

44.67 58.95 54.66 55.91

SigLIP

ViT-L/16@384

52.47

1,429.11 63.57 67.34 63.44

68.02 36.09 29.70 61.56

46.68 35.70 59.86 35.93

32.67 55.29 55.24 57.00

SigLIP

ViT-SO400M/14@384 55.27 1,489.05 66.55 69.59 64.58

70.45 35.69 29.20 62.34

51.28 40.80 63.28 43.02

38.00 59.61 61.58 53.91

OpenCLIP

ConvNeXt-L@512

52.76

1,467.38 63.40 66.92 63.17

69.16 34.90 29.70 58.45

52.04 35.40 61.87 38.79

30.67 56.21 54.24 55.91

Other

DINOv2

ViT-L/14@336

43.26

1,261.43 53.96 63.22 62.61

65.49 34.50 27.70 56.90

15.40

3.40 44.87 11.22

26.00 54.38 53.06 56.40

MoCo v3

ViT-B/16@224

39.51

1,175.34 41.70 53.31 56.15

65.25 33.40 28.30 55.76

15.48

3.20 44.42 11.10

18.00 46.67 45.38 55.25

MoCo v3

ViT-L/16@224

37.59

1075.39 39.15 50.14 53.65

65.49 34.60 27.30 55.44

17.28

3.00 44.21 11.70

14.67 44.58 45.38 41.12

MAE

ViT-L/16@224

41.43

1,181.51 45.53 58.92 58.75

64.65 35.00 29.20 57.12

16.88

3.10 44.67 11.74

18.67 49.67 53.04 56.83

I-JEPA

ViT-H/14@224

41.90

1,175.70 48.00 59.60 59.35

64.45 35.09 27.60 57.32

16.20

3.00 45.50 11.40

22.67 49.93 52.38 59.08

Other

MiDaS 3.0

ViT-L/16@384

38.28

1,065.26 42.64 50.95 56.10

65.39 35.00 27.20 52.98

15.96

2.80 43.49 11.23

12.00 46.14 43.41 53.90

Supervised

ViT-L/16@224

40.01

1,222.41 47.40 54.15 57.26

64.35 34.40 26.40 56.09

16.20

3.20 44.73 11.74

14.00 46.41 49.33 53.40

Vision Backbone

General

Knowledge

OCR & Chart

Vision-Centric

Method

Backbone

Unfreeze

Average

MME P

MMB

SEED I

GQA

SQA I

MMMU V

MathVista M

AI2D

ChartQA

OCRBench

TextVQA

DocVQA

MMVP

RealWorldQA

CV-Bench 2D

CV-Bench 3D

OpenAI CLIP ViT-L/14@336

×

55.85

1,577.33 69.70 70.22 63.33

73.67 36.19 36.60 64.80

49.12 36.90 60.33 39.79

32.67 55.56 66.11 59.75

DINOv2

ViT-L/14@336

×

45.36

1,373.14 57.02 64.58 61.67

67.13 36.19 30.70 60.62

19.04

3.40 46.39 13.27

26.67 52.68 59.81 57.91

OpenAI CLIP ViT-L/14@336

✓

57.44

1,585.34 68.68 71.47 63.96

77.39 36.09 37.30 65.12

59.36 48.00 62.39 45.24

31.33 56.21 61.09 56.16

DINOv2

ViT-L/14@336

✓

47.40

1,366.65 61.62 69.72 63.68

68.72 36.29 35.50 60.88

18.64

4.40 47.92 14.66

34.67 54.64 60.98 57.83

from the last layer. The shape of each model's output differs depending on the resolution and patch size of each vision model. To resolve these differences, we interpolate the output of each model to a fixed number of tokens, using 576 tokens in our implementation, as described in Section 2.5. Our example code for interpolation can be seen below.

We then concatenate the model outputs along the feature dimension and use a larger MLP to project the concatenated visual tokens into the LLM token space.

Full results on Model Ensemble We present all the benchmarks from the model ensemble experiment in Section 2.5 in Table 14. As discussed in Section 1 and Section 2.4, this comprehensive view of benchmarks provides a better understanding of the model's performance compared to simply averaging across benchmarks. Adding a vision-only SSL model enhances the MLLM's performance in vision-centric benchmarks while maintaining strong capabilities in other categories.

G Data

G.1 Catalog of Visual Instruction Data

Here, we provide a comprehensive catalog of visual instruction datasets utilized in our study. The datasets are categorized based on their primary focus, including general conversation and VQA data, OCR-related data, counting data, knowledge-based data, and language-only data. Table 15 summarizes these datasets and their respective references.

Average

General

GQA

MME-P

MMB

SEED-I

Knowledge

SQA-I

MMMU

MathVista

AI2D

OCR & Chart

ChartQA

OCRBench

TextVQA

DocVQA

Vision-Centric

MMVP

V*Star

RealWorldQA

CV-Bench

All Model Average

Language-Supervised Average

OpenAI CLIP ViT-L/14@336

SigLIP SO400M/14@384

SigLIP ViT-L/16@384

OpenCLIP ConvNeXt-L@512

EVA-CLIP-02 ViT-L/14@336

DFN-CLIP ViT-L/14@224

Self-Supervised Average

DINOv2 ViT-L/14@336

MAE ViT-L/16@224

MoCo v3 ViT-B/16@224

I-JEPA ViT-H/16@224

Other Average

MiDaS 3.0 ViT-L/16@384

SupViT ViT-L/16@224

2.79

2.50

2.13

1.43

2.40

4.13

0.17

-0.98

0.40

1.16

0.91

4.42

7.55

7.16

1.43

2.90

4.18

12.82

3.58

3.88

2.32

3.47

2.25

1.39

1.82

2.97

2.90

0.48

-0.91

0.05

0.92

2.16

9.67

19.32 13.99

1.92

8.59

3.05

6.21

0.63

4.74

1.56

4.62

2.16

2.33

0.03

1.92

4.64

0.81

0.22

-0.29

-2.71

3.90

14.43 26.00 30.10

1.22

9.69

3.68

-9.28

4.25

7.14

6.50

2.89

2.77

1.38

2.30

4.54

2.91

-0.70

-2.29

-1.11

-0.34

1.22

8.42

17.18

9.68

2.36

7.04

1.60

3.83

2.18

4.60

-3.10

3.28

2.77

1.41

0.34

7.02

2.84

0.89

-0.95

0.27

0.00

3.82

5.54

15.20

6.57

0.45

2.07

4.44

16.68

2.39

3.67

0.66

7.51

3.76

2.10

3.56

4.62

4.78

-1.21

-1.09

-2.49

0.68

-1.53 26.10 51.28 26.43

6.01

36.54

6.87

12.22

5.96

8.30

3.49

3.44

2.79

2.48

2.94

2.39

3.29

0.49

-1.85

-1.38

3.87

2.74

6.83

18.42 10.33

2.67

-0.18

4.98

28.84 -6.02

5.85

-0.20

-0.90

-0.74

-1.33

1.77

-2.70

-1.07

2.61

0.48

5.30

4.01

2.82

-3.33 -12.18 0.86

-1.17

-3.58

-3.26 -15.00 -4.99

-1.15

2.00

3.37

4.21

3.44

2.91

3.51

7.11

0.12

-1.30

0.38

1.99

0.70

-0.65

-3.79

-0.65

1.31

-3.54

7.97

28.63 10.82

5.13

2.84

4.88

6.22

3.69

5.93

4.11

11.11

0.29

-0.83

0.88

1.09

0.87

-2.30

-6.33

3.03

-0.55

-4.83

11.47 21.89 23.99

9.48

0.20

3.25

4.29

2.32

4.30

4.91

5.84

0.03

-2.10

-1.66

5.04

1.13

-1.42

-2.76

-6.06

0.31

-4.55

8.12

16.69 16.42

4.11

3.66

1.36

2.25

2.04

4.08

0.19

2.15

-0.66

-0.97

0.30

0.00

-1.20

-1.12

-6.07

6.67

0.75

-3.23

3.91

50.00 -1.31

-1.91

2.51

3.97

4.10

5.70

-2.66

4.83

9.34

0.81

-1.29

2.01

1.85

2.01

2.24

0.00

-6.25

4.72

-1.55

8.37

25.94

4.16

8.83

5.00

-0.39

-0.19

1.71

-2.70

-1.52

1.84

-0.65

-0.53

1.49

0.19

-2.45

-1.17

-5.06

2.30

0.16

-1.28

0.01

1.00

-2.05

-1.20

3.53

-2.39

-3.42

0.95

-5.91

-7.73

-1.55

-1.20

-1.37

3.86

1.87

-5.49

-2.66

-6.56

-9.68

-0.37

-3.69

-2.25 -21.57 -1.43

0.85

0.30

1.60

3.05

2.47

0.52

4.69

5.23

-0.09

0.32

-0.87

-1.49

0.59

0.31

-3.57 14.29

0.69

1.12

2.26

23.57 -2.67

-3.26

6.77

20

10

0

10

20

30

40

50

Heatmap depicting the percentage change in performance across multiple benchmarks when visual encoders are unfrozen compared to when they are kept frozen during fine-tuning. The color gradient indicates the magnitude of the performance change after unfreezing visual encoders-white indicates no change, red is a positive change, and blue is a negative change. Notably, unfreezing leads to significant gains in OCR Chart tasks for most Language-Supervised Models, as reflected by the deep red cells. ConvNeXt, in particular, shows substantial improvements, demonstrating the benefits of updating this visual encoder during fine-tuning.

Vision Backbone

General

Knowledge

OCR & Chart

Vision-Centric

Method

Average

MME P

MMB

SEED I

GQA

SQA I

MMMU V

MathVista M

AI2D

ChartQA

OCRBench

TextVQA

DocVQA

MMVP

RealWorldQA

CV-Bench 2D

CV-Bench 3D

SigLIP+DINOv2

51.61

1,432.02

61.28 65.99 63.30

68.82 35.69 29.40 60.01

43.00 35.70 60.40 37.54

30.00 53.99 55.52 53.58

SigLIP+DINOv2+ConvNext

54.52

1,503.51

63.83 67.97 63.95

70.40 35.99 29.30 60.69

48.20 36.90 64.97 45.53

34.67 58.69 55.74 60.33

SigLIP+DINOv2+ConvNext+CLIP

54.74

1,479.46

63.32 67.63 64.04

71.39 35.49 29.10 59.88

50.24 39.60 64.55 46.12

32.67 58.95 58.54 60.42

SigLIP+ConvNext

54.53

1,494.97

64.60 67.98 63.58

71.05 34.90 29.80 60.85

50.64 38.00 64.53 46.52

32.00 57.91 58.83 56.58

CLIP+ConvNext

54.45

1,511.08

63.83 67.41 63.63

70.80 35.09 30.40 59.91

51.32 35.00 64.45 47.88

33.33 57.25 56.32 59.08

SigLIP+DINOv2+ConvNext-L

53.78

1,450.64

63.57 67.79 63.63

71.34 34.80 30.20 61.04

49.32 37.70 64.05 45.83

30.00 56.21 58.08 54.33

SigLIP+CLIP+ConvNext-L

54.53

1,507.28

63.23 68.64 63.63

71.10 35.89 30.90 59.97

52.36 38.50 65.40 47.92

28.67 57.25 57.66 55.92

G.2 Additional System Prompts used in Cambrian Data

Here, we investigate a phenomenon we term the 'answer machine phenomenon'. We observe that a well-trained MLLM may excel at VQA benchmarks, but lack basic conversational abilities and default to outputting short, curt responses (see examples in Fig. 14). This discrepancy arises because benchmark questions typically require responses that are limited to a single option, choice, or worddiverging from the more broad and realistic use cases of MLLMs. Similar phenomena have been discussed in other LLM studies [114, 151, 155].

We suspect that this issue stems from instruction tuning data containing an excessive number of short-response VQA tasks, leading to catastrophic forgetting in LLMs. To address this, we incorporate additional system prompts during training. We append prompts such as ' Answer the question using a single word or phrase. ' before questions that generate a single word or phrase in the response. Full details of the system prompts used are provided in Appendix G.2. After integrating these system prompts, we observe that while the model's benchmark performance remains unchanged, its conversational ability improves dramatically. For example, in Fig. 14, models with system prompts produce longer and more engaging responses while answering questions correctly. The system prompts also enhance the model's performance on reasoning-related tasks, such as math problems, by encouraging a chain of thoughts [133] followed by the answer.

This underscores the necessity of developing evaluation protocols like the Chatbot Arena [34] for MLLMs, despite the challenges in collecting large-scale, real-world interaction data. While performing well on benchmarks is important, it is equally crucial to ensure the model can engage in meaningful and natural interactions. The overall user experience and the model's conversational abilities are paramount, as a model that excels in benchmarks but fails to converse effectively cannot meet the needs of practical applications.

As our Cambrian data includes instructions/questions and responses of different types and formats (e.g., Short response with a single word or regular response as a complete sentence), it is important to specify the required response format in the instruction prompt to avoid ambiguity and possible conflicts. Some of the datasets already include such prompts and we add proper prompts for the remaining datasets. The detailed response formatting prompts we additionally add are listed in Table 16.

G.3 Data Engine

Comprehensive Implementation Details of the Data Engine

The data engine is designed to generate instruction tuning data for knowledge-based fields, where previous works rarely covers and MLLMs are not reliable to distill for from. The data engine takes in a given field, such as 'Physics', utilizing reliable web sources like Wikipedia. Below are the various stages involved in the process. We also visualize this process in Fig. 15:

Stage 1 - Topic Generation: We start by compiling a list of fields and subfields and subsequently generate topics for each field using a Large Language Model (LLM), such as GPT-4. In this stage,

What city is this? If I want to tour around the landmark in this picture, where should I go?

Please solve this question step by step.

How many people are in this image?

Based on this chart, what is the most popular tea?

What is unusual about this image? Please answer in details.

Describe this image please.

There are six people in the image.

There are six people in the image.

Based on the chart, black tea appears to be the most popular

type of tea in the U.S. Bubble Tea Market, as it consistently

has the highest sales volume across all years displayed.

Black Tea

This is New York City, and the prominent landmark in the image

is the Washington Square Arch, which is located in the Greenwich

Village neighborhood. To tour around this landmark, you should

visit the nearby Washington Square Park, which is a popular spot

for locals and tourists alike. The park offers a variety of

activities, including walking paths, a fountain, and a statue of

George Washington. Additionally, the surrounding area is rich

with historical buildings, art galleries, and restaurants,

making it a vibrant and culturally diverse place to explore.

New York City

The unusual aspect of this image is the anthropomorphic design of

the church's bell tower, which is given human-like features.

Specifically, the two windows on the tower are designed to

resemble eyes, and the cross at the top of the tower is stylized

to look like a nose. This creative design choice is not typical

for church architecture, which usually favors more traditional

and less whimsical features. The intention behind such a design

could be to add a playful or artistic element to the building,

possibly to attract attention or to reflect a unique character of

the church or the community it serves.

The image is unusual because it features a church with a design

that incorporates elements typically associated with a face. The

two circular windows on the tower serve as eyes, and the cross

on top of the tower is positioned to resemble a nose. This

anthropomorphic design is not commonly found in church

architecture, which often favors more traditional and

conventional styles. The presence of the palm trees adds a

tropical or subtropical feel to the scene, which may not be the

usual setting for such a building. The overcast sky suggests a

gloomy or somber mood, which contrasts with the whimsical nature

of the church's design.

The image captures a serene moment in a bathroom

setting. A fluffy, gray cat is comfortably seated

on the edge of a white bathtub. The cat's gaze is

directed upwards, possibly at something outside the

frame, giving it a contemplative expression. The

backdrop is a simple white tiled wall, which

contrasts with the cat's fur, making it the focal

point of the image. The overall scene suggests a

quiet, peaceful moment in a domestic setting.

Cat in bathtub

To solve the equation (\sqrt{k+2} -

x = 0), we need

to isolate (k). Let's start by adding (x) to both

sides of the equation to get rid of the subtraction:

(\sqrt{k+2} = x)

Now, to isolate (k), we square both sides of the

equation:

((\sqrt{k+2})^2 = x^2)

This simplifies to:

(k+2 = x^2)

Now, we substitute (x = 9) into the equation:

(k+2 = 9^2)

(k+2 = 81)

Finally, we subtract 2 from both sides to solve for

(k):

(k = 81 -

2)

(k = 79)

Therefore, the value of (k) when (x = 9) is 79. The

correct answer to the question is D.

Answer is C.

w/o system prompt

w/ system prompt

U.S,Bubble Tea Market

Sipk, ty 1om, 2190 2000-(0/50 M05or

8.6%

4/

In the equation above, k is a constant. If x = 9, what

0=x-+

is the value of k ?

A)

B)

C)

16

D)

6

Fields

Topics

Filtered

Web Data

Relevant

Text

Q&A

Web

Knowledge

LLM

Search

Engine

API

Parser

Image

VQA Data

LLM

we processed 30 fields, resulting in 3660 topics. We then post-process the output of LLMs into json formats. For example, the topic data for Physics looks like below.

Stage 3 - Parsing: In this stage, we parse each web page to extract image-caption-text tuples. We aim to identify the blocks containing an image, the image's caption, and relevant textual content. Below is an example of the parsed data for the same topic, "Electric Field and Electric Potential":

Stage 4 - Data Generation: We generate dataset in this stage, ensuring high quality. We first filter out data samples with fewer than 50 words in the text. Then, instead of downloading images directly from the links retrieved during web parsing, we download high-resolution images from the original sources. We then convert formats like SVG or GIF into a common standardized format, PNG.

Question-Answer pairs are generated by using LLM such as GPT-3.5 from the image metadata, caption, and contextual text. These Q&A pairs and the image form our VQA dataset. We generated 165k data samples. Here is an example of the generated data:

"caption": "Representing the Coulomb gauge magnetic vector potential A, flux density B, and current density J fields around a toroidal inductor of circular cross section. Thicker lines indicate field lines of higher average intensity. Circles in the cross section of the core represent the B-field coming out of the picture, plus signs represent B-field going into the picture. \u2207 \u22c5 A = 0

B=0

B=0

B=0

B=0

B=0

B=0

Axisof symmetry

General

Knowledge

OCR & Chart

Vision-Centric

# of Data

Average

MME P

MMB

SEED I

GQA

SQA I

MMMU V

MathVista M

AI2D

ChartQA

OCRBench

TextVQA

DocVQA

MMVP

RealWorldQA

CV-Bench 2D

CV-Bench 3D

t

= 150

k

4015k

53.74

1,512.3

67.0

68.3

61.2

73.4

35.1

34.3

62.4

44.6

39.1

58.5

38.5

30.0

55.2

61.98

54.7

t

= 250

k

5218k

54.31

1,475.9

67.3

69.2

61.6

73.4

35.9

34.5

62.4

46.5

36.9

59.2

38.6

32.0

56.6

63.68

57.5

t

= 350

k

5883k

54.27

1,461.9

66.2

68.9

61.6

73.8

36.4

32.8

62.5

46.8

38.3

59.3

39.3

31.3

54.9

62.68

60.4

t

= 450

k

6383k

54.15

1,534.1

67.6

66.3

61.9

72.9

35.1

36.9

63.8

45.8

38.6

58.4

39.4

28.0

53.6

64.60

56.8

This data engine is designed for scalability and efficiency and is capable of handling extensive data generation tasks using multithreading techniques.

G.4 Full results on data curation experiment

Data Balance via Fitlering t As discussed in Section 4.2, if left unfiltered, the data pool is dominated by noisy, unbalanced data sources such as CLEVR and DVQA, leading to pronounced exponential tails. However, as we apply different t values to filter data from each source, the exponential tails become less pronounced, resulting in a more balanced dataset. We also present all the results in Table 17. t value 250k has the highest average across all benchmarks; 250k and 350k also have the highest performance across many individual benchmarks.

Here, we plot the cumulative sum of counts for entries sorted by counts from tail to head. From Fig. 17, we see this intermediate threshold prevents explosive heavy tail.

Data Ratio Studies We present the full results of our data ratio study in Table 18. The table highlights the importance of finding an optimal data ratio that balances different aspects of MLLM. Experiment 5 achieves well-rounded performance with its selected data ratio.

0

10

20

30

40

50

60

70

0m

2m

4m

6m

8m

10m

Cumulative Sum of Counts for Entries

Data Mix 1 (unfiltered)

Data Mix 2 (t=450k)

Data Mix 3 (t=350k)

Data Mix 4 (t=250k)

Data Mix 5 (t=150k)

Sorted Entries

General

Knowledge

OCR & Chart

Vision-Centric

Average

MME P

MMB

SEED I

GQA

SQA I

MMMU V

MathVista M

AI2D

ChartQA

OCRBench

TextVQA

DocVQA

MMVP

RealworldQA

CV-Bench 2D

CV-Bench 3D

exp1

47.49

1,309.10 58.00 60.10 54.00

72.40 34.80 31.20 59.10

34.20 34.50 54.20 33.00

13.30 47.60 57.40 50.58

exp2

47.78

1,351.70 60.30 61.20 55.40

72.80 35.20 29.50 59.10

31.20 33.40 54.20 30.50

15.30 48.20 58.40 52.25

exp3

48.28

1,299.53 60.56 61.79 55.74

72.04 34.90 32.10 59.40

33.20 33.90 54.15 31.90

21.30 48.60 58.52 49.41

exp4

47.47

1,288.98 58.16 61.47 55.00

71.05 37.10 28.20 58.50

33.72 34.50 55.07 31.69

20.66 47.06 56.30 46.58

exp5

48.96

1,363.26 60.48 63.18 55.92

70.35 35.70 31.40 57.19

32.88 34.60 54.74 32.10

22.70 47.30 58.83 57.75

G.5 737K and 5M Mixes

0.7M For the 0.7M data we used in Section 2.4, We add a small number of OCR and chart data to LLaVA 665K, specifically 15,501 AI2D, 14,999 DocVQA, and 13,000 DVQA data points. This results in a 737K mix, which covers all categories in training MLLMs. This data mix allows us to study visual representations efficiently.

5.0M For the 5M data mixes we use in Section 2.4, we apply data filtering discussed in Section 4.2 and apply t =150k on all multimodal instruction data in Cambrian-10M.

G.6 Test Image Leakage in Visual Instruction Training Data

One potential concern with our targeted data engine (Section 4.1) is that instruction-tuning data collected from the open web could introduce data leakage. To address this, we systematically analyze the extent of direct image matches between our training data and our test sets. Using difference hashing (dHash) [19], we compute hashes for all images in the training data and test sets. We then compare these hash sets to determine how many test images overlap with our training data, reporting the number of collisions in Table 19.

Across all fifteen datasets, our targeted data engine finds only 32 test images in total, amounting to just 0.06% of the test data. This low overlap percentage dispels concerns that our data engine inadvertently targets specific test sets. When analyzing the full Cambrian10M dataset-which is 15x larger than LLaVA-665k-we observe only 6x more matching test images (7,244 compared to 1,034 in LLaVA-665k). This discrepancy suggests that Cambrian10M's scale does not inherently result in excessive overlap with test sets. Instead, any overlap likely arises from the natural reuse of training images across benchmark datasets rather than targeted duplication.

It is important to emphasize that while some exact image matches are found, this does not imply that the exact image-question pairs have been encountered during training. Unlike in traditional unimodal computer vision research, where an image alone constitutes a data point, the multimodal paradigm treats each image-text (question-answer) pair as unique. Consequently, seeing a test image during training is not equivalent to 'training on the test set' as long as the associated text (question-answer) pairs differ. This distinction ensures that Cambrian10M respects the integrity of test evaluations, even in cases where images might appear in both training and test sets.

We encourage future research exploring the impact of image-only leakage on the performance of MLLMs. Understanding this influence may yield insights into the boundaries of model generalization and guide future best practices for dataset construction in multimodal learning.

G.7 Broader Impacts

We conducted a preliminary analysis of the Cambrian dataset, focusing on the distribution of male, female, and neutral pronouns. Our findings show the following distribution: 38.35% male pronouns, 17.99% female pronouns, and 43.66% neutral pronouns.

We recognize that training models on biased data can perpetuate these biases. Addressing bias by artificially modifying data distributions-such as through rebalancing or applying fairness constraints-can help mitigate this issue, but it also presents challenges. These include the potential loss of generalization and the risk of introducing new biases. Additionally, identifying and mitigating bias in Multimodal Large Language Models (MLLMs) is particularly complex, given the interaction

between different data modalities. We believe that openness in model development and data curation will accelerate research aimed at understanding and mitigating these potential harms.

H Implementation Details

Cambrian Models For our final Cambrian models, we use 2.5M adapter data which is comprised of 1.2M captioning data from shareGPT4V [27] and 1.2M captioning data used in MiniGemini [77].

SVA We provide here ablation studies of SVA module.

Wefurther conduct ablation experiments using OpenAI CLIP ViT-L/14@336 + OpenCLIP ConvNeXtL@1024 as our base model combination. We focus on the OCR & chart categories to assess the impact on high-resolution visual understanding. The results show that increasing capacity via D or G improves performance and that allowing vision aggregation across multiple layers by adding cross-attention layers within the LLM also enhances performance.

Compared with other spatial-based connectors like C/D-Abstractor [21] which are designed for single vision feature maps, our SVA module can dynamically combine visual features from multiple vision models with varying resolutions. Besides, our spatial inductive bias in SV A can better compress spatial information compared with such methods. To isolate the effect of spatial inductive bias, we consider the case of token reduction using a single vision encoder. Specifically, we use OpenAI CLIP ViT-L as the vision model and compress its original 576 tokens to 36 tokens using our SV A module and other connectors. We compare our SVA module with three baselines: 1) Direct interpolation + MLP, 2) C-Abstractor [21], and 3)LDPv2 Projector [35] (similar to C-Abstractor but more lightweight). For fair comparisons, we do not include multi-layer aggregation inside the LLM for our SVA baseline, and the results are shown in Table 21. Compared with the simple MLP baseline, C-Abstractor performs better on General and Vision-Centric tasks but inferior on Knowledge and OCR & Chart tasks. LDPv2 performs similarly to the MLP baseline. Our SVA consistently demonstrates superior performance across all categories, especially in OCR & Chart and Vision-Centric tasks, demonstrating its effectiveness in information compression.

We introduce learnable k m × k m positional encodings in the vision features when k m > 1 . Besides, during cross-attention, the query is augmented with a global feature obtained by global pooling over the vision features, which is concatenated with q i,j to better guide the aggregation process. In our experiments, the feature maps of all vision encoders except for ConvNext are interpolated to 576 × 576 ( m k = 1 for L = 24 ). For ConvNext, we first interpolate the feature maps from its 4 stages to 96 × 96 ( m k = 4 for L = 24 ) and then channel-wise concatenate them to form its final vision feature map similar to [77].

For experiments in Section 3, we set D = 3 , G = 1 and add cross-attention layers between the layers of LLM with a stride equal to 3. For our final Cambrian models, we set D = 3 , G = 1 and insert multiple cross-attention layers in LLM considering the tradeoff between performance and efficiency. For Cambrian-8B, Cambrian-13B, and Cambrian-34B, the strides of cross-attention layers inside the LLM are 3, 4, and 9 respectively.

To study the importance of visual features from different vision models to different image categories, we further investigate the attention score distribution in our SV A module. We evaluate our Cambrian8b model on GQA, DocVQA, and ScienceQA (representing three different benchmark categories), and the attention distribution results are shown in Table 22. We can see that on real-world images (GQA), the contribution of different vision models is relatively uniform, in part due to the similar characteristics of SigLIP and CLIP. On document-type images (DocVQA) which are text-heavy and often high-resolution, the influence of SigLIP increases and that of ConvNext greatly increases to aid in high-resolution information processing. For scientific images (ScienceQA) composed of illustrations and diagrams about different science categories, the contribution of SigLIP is further increased while the portion of DINOv2 decreases compared to GQA.

Table 22: Attention distribution studies. The attention distribution among different vision encoders varies with different image categories.

Unfreezing While unfreezing is largely beneficial (Section 2.3 and Fig. 13), it has a significant speed drawback. Given fixed computational resources, unfreezing visual encoders slows down the fine-tuning process by approximately 50-55%. For initial explorations or when computational overhead is a concern, leaving the visual encoders frozen can be a practical strategy. This allows for quicker iterations and tuning, especially during early research phases, while still providing valuable insights. Ultimately, unfreezing is recommended for achieving the best performance once the setup has been optimized.

I Evaluation Details

I.1 System Prompts Used in Evaluation

To ensure the reproduction of our results, we also include the system prompts we used in this work. The system prompts for our models can be found in Table 25. Additionally, we release the prompts we used while evaluating our models on the various benchmarks in Table 26. We hope this sets a precedent for future research to improve the reproducibility of benchmark results.

I.2 Ablation Study on Fuzzy Matching Vs LLM Judgement

We use fuzzy matching to evaluate responses in some benchmarks, since MLLMs can answer questions with auxillary phrases. To study the effectiveness of our fuzzy matching, we compare our model accuracy through fuzzy matching with the model accuracy obtained when we use LLM as a grader.

The LLM grader is sensitive to the prompt given to it while grading, and we prompt the LLM (we use OpenAI GPT-3.5-turbo and GPT-4-turbo as our graders) with few shot grading examples, which we notice significantly improves grading accuracy. An example of such a prompt is given below.

LLM Grader Prompt

You are a reliable grader. Reply with only either of the following 2 words: CORRECT or INCORRECT.

Backbone

Data

Adapter

Instruction Tuning

Experiment

LLM

Vision

Adapter

Instruction Tuning

lr

wd

bs

lr

wd

bs

vision lr

0M Adapter+737K IT

Vicuna-1.5-7B

OpenAI CLIP ViT-L/14@336

0

737k

-

-

-

2e-5

0

512

-

0.5M Adapter+737K IT

Vicuna-1.5-7B

OpenAI CLIP ViT-L/14@336

0.5M

737k

1e-3

0

512 2e-5

0

512

-

1.2M Adapter+737K IT

Vicuna-1.5-7B

OpenAI CLIP ViT-L/14@336

1.2M

737k

1e-3

0

512 2e-5

0

512

-

Unfreeze Vision

Vicuna-1.5-7B

OpenAI CLIP ViT-L/14@336

1.2M

737k

1e-3

0

512 2e-5

0

512

1e-5

Model Ensemble

Vicuna-1.5-7B

Chosen Combination

1.2M

737k

1e-3

0

512 2e-5

0

512

Data Balance

Vicuna-1.5-7B

OpenAI CLIP ViT-L/14@336

0

Mix Based on three

t

-

-

-

2e-5

0

512

-

Data Ratio

Vicuna-1.5-7B

OpenAI CLIP ViT-L/14@336

0

1350k Based on Ratio

-

-

-

2e-5

0

512

-

LLaVA 665K

Vicuna-1.5-7B

OpenAI CLIP ViT-L/14@336

0

LLaVA 665K

-

-

-

2e-5

0

512

-

Cambrian-10M (Data)

Vicuna-1.5-7B

OpenAI CLIP ViT-L/14@336

0

Cambrian-10M

-

-

-

2e-5

0

512

-

Cambrian-7M (Data)

Vicuna-1.5-7B

OpenAI CLIP ViT-L/14@336

0

Cambrian-7M

-

-

-

2e-5

0

512

-

Cambrian-1-8B

Llama-3-Ins-8B

SVA with 4 encoders

∗

2.5M

Cambrian-7M

1e-4

0

512 2e-5

0

512

-

Cambrian-1-13B

Vicuna-1.5-13B

SVA with 4 encoders

∗

2.5M

Cambrian-7M

1e-4

0

512 2e-5

0

512

-

Cambrian-1-34B

Hermes-2-Yi-34B SVA with 4 encoders

∗

2.5M

Cambrian-7M

1e-4

0

512 2e-5

0

1024

-

answer:

25

gt_answer:

29

evaluation:

INCORRECT

answer:

Yes

gt_answer:

Yes

evaluation:

CORRECT

answer:

80

gt_answer:

80

evaluation:

CORRECT

answer:

Ireland

gt_answer:

Italy

evaluation:

INCORRECT

answer:

UK

gt_answer:

UK

evaluation:

CORRECT

answer:

2019

gt_answer:

2011

evaluation:

INCORRECT

answer:

{answer}

gt_answer:

{gt_answer}

You will be given an 'answer' and a 'gt_answer' (ground truth answer) ,and you must reply with either CORRECT or INCORRECT based on the response. Tolerate a 0.05 relative error for numerical answers.

answer:

25

gt_answer:

29

evaluation:

INCORRECT

answer:

Yes

gt_answer:

Yes

evaluation:

CORRECT

answer:

80

gt_answer:

80

evaluation:

CORRECT

answer:

Ireland

gt_answer:

Italy

evaluation:

INCORRECT

answer:

UK

gt_answer:

UK

evaluation:

CORRECT

answer:

2019

gt_answer:

2011

evaluation:

INCORRECT

answer:

{answer}

gt_answer:

{gt_answer}

evaluation:

We conduct an ablation study on the benchmarks that require fuzzy matching and present the results in Table 24. We discover that fuzzy matching provides reliable results compared to an LLM grader. We recommend using a more capable model (such as GPT-4-turbo) for grading benchmarks that have more subjective responses (such as numbers and words).

Method

Eval

SEED I

GQA

SQA I

MMMU

AI2D

ChartQA

OCRBench

TextVQA

MMVP

RealWorldQA

Cambrian-1 8B

74.7

64.6

80.4

42.7

73.0

73.3

62.4

71.7

51.3

64.2

Cambrian-1 8B

GPT3.5 Grading

78.4

65.8

82.0

38.9

78.1

71.2

67.0

69.2

49.3

63.5

∆

+3.7 +1.2 +1.6

-3.8 +5.1

-2.1 +4.6

-2.5

-2.0

-0.7

Cambrian-1 13B Fuzzy Matching

74.7

64.6

80.4

40.4

73.0

73.3

62.4

71.7

51.3

64.2

Cambrian-1 13B GPT3.5 Grading

77.3

64.7

81.3

37.2

78.2

71.4

67.1

75.6

46.0

64.3

∆

+2.9 +0.4 +2.0

-3.2 +4.6































































































J More results of Cambrian-1 Model

In Fig. 18, we also plot our Cambrian-1 performance as well as GPT-4 performances. In the plot, it is clear that Cambrian-1 offers competitive performance compared to proprietary models in most categories. We also showcase some examples in Fig. 19, demonstrating that the model effectively attends to details in images despite using only 576 tokens.

K Potential Misuse & Mitigation Strategies

We recognize that there are ethical concerns regarding the potential misuse of multimodal large language models like Cambrian-1, particularly in generating misleading content or spreading misinformation. Below, we outline the main risks and provide strategies to address them:

Misinformation Cambrian-1 could be used to create misleading text descriptions of images, leading to false narratives or misrepresentations. For instance, such models might be leveraged by social media bots to manipulate public opinion during elections or other critical events.

Hallucination Similar to any large language model, Cambrian-1 may produce information that is not based on facts or actual input data. This phenomenon, often called "hallucination," can be dangerous if users assume the model's output is entirely accurate without verification.

To mitigate these risks, users should exercise caution and critical thinking when interpreting outputs generated by Cambrian-1. It is important to verify the information produced by the model, particularly if the results are intended for sensitive or high-stakes applications. Users must be aware of the potential for hallucinations, where the model produces information not grounded in facts, and take steps to cross-check and validate any critical outputs. Additionally, implementing content filtering as a safeguard can help flag potentially harmful or misleading content before it is disseminated.

Why is this image funny?

The humor in this image comes from the

unexpected twist on the classic poem

'Roses are red, violets are blue.' Instead

of a romantic or sweet sentiment, the poem

is humorously altered to a more mundane

and slightly unsettling scenario, which is

a bug in the bed. This juxtaposition of a

traditional romantic poem with a common,

albeit slightly unpleasant, household

occurrence creates a comedic effect.

Which character is this?

The character in the image is Homer Simpson,

a central character from the animated

television series 'The Simpsons.' Homer is

known for his distinctive yellow skin,

casual attire, and often humorous or

oblivious behavior.

What does this image remind you of?

The image might remind you of a playful

and creative culinary presentation,

possibly inspired by a world map, where

each piece of food represents a different

continent. It could also evoke thoughts of

a fun cooking challenge or a themed meal

where the food is arranged to mimic the

shape of the Earth.

Please solve the problem step by step:

To solve this problem, we need to

determine which option correctly completes

the sentence in a way that adheres to the

conventions of Standard English. The

sentence is:

"In 1937, Chinese American screen actor

villains and secondary characters but

never a heroine, finally got a starring

role in Paramount Pictures' Daughter of

Shanghai, a film that _____ "expanded the

range of possibilities for Asian images on

screen."

The correct option should be the one that

maintains the subject-verb agreement and

the appropriate punctuation.

Option A: "critic, Stina Chyn, claims" -

This option is incorrect because it uses a

comma after the name, which is not

standard in English.

Option B: "critic, Stina Chyn, claims," -

This option is also incorrect because it

uses a comma after the name and an extra

comma at the end of the sentence, which is

not standard in English.

Option C: "critic Stina Chyn claims" -

This option is correct because it

maintains the subject-verb agreement and

does not use any unnecessary commas.

Option D: "critic Stina Chyn, claims," -

This option is incorrect because it uses a

comma after the name and an extra comma at

the end of the sentence, which is not

standard in English.

Therefore, the correct answer is Option C:

List all the countries in this image in

the format of a json {1: 2:, etc}. Why

are the flags here?

{

"2": "Belgium",

"3": "France",

"4": "Germany",

"5": "Portugal",

"6": "Scotland",

"7": "Spain",

"8": "Turkey"

}

The flags are here to represent the

participating countries in the UEFA EURO

2024 tournament, which is hosted by

Germany. Each flag corresponds to a

country that is either a host or a

participant in the event, as indicated by

the tickets below the flags. The tickets

suggest that these countries are either

hosting matches or have qualified to play

in the tournament.

ROSES ARE RED

BUG IN YO BED

In 1937, Chinese American screen actor Anna May

Wong, who had portrayed numerous villains and

secondary characters but never a heroine, finally got

a starring role in Paramount Pictures' Daughter of

Shanghai, a film that

“expanded the range

of possibilities for Asian images on screen."

Which choice completes the text so that it conforms

to the conventions of Standard English?

A)

B)

C)

D)

EURO2024

CKET

L Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have detailed all claims mentioned in the abstract and introduction in the remaining part of the paper and Appendix.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [NA]

Justification: We have discussed the limitations of our work in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not include any theoretical assumptions and proofs in our paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We will publicly release all code, hyperparameters, model checkpoints, and datasets for reproducing our experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Yes, as highlighted in the abstract, we will release our model weights, code, tools, dataset, and all recipes.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes. We include all training and evaluation recipes in the main paper and Appendix, and will additionally include them in the code release.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We do report error bars and other statistical results for our experiments. Part of these results are included in Fig. 3, Fig. 4 and Fig. 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: We will include these in our code release.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conform NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work has no societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We used pre-trained LLM, which already has safeguards. As for datasets, we have conducted filtering to avoid safety risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have credited and mentioned the assets we used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We have explained the data engine and new benchmark in our paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents extensive studies on existing MLLM benchmarks addressing the difficulties involved in consolidating and interpreting results from various tasks for MLLM designs. Moreover, the authors also propose Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector to fuse vision features with LLMs while reducing vision tokens. In addition, authors also collect high quality visual instruction-tuning data. The proposed model, Cambrian-1, achieves the state-of-the-art performance on multiple MLLM benchmarks.\n\nSTRENGTHS:\n1, Overall writing is good and easy to follow. The motivation is clear. \n\n2, The goal of this work is interesting. A good study for the effects on Large Language Model, Visual Encoder, Multimodal Connector, Data Curation Pipeline, Instruction in the MLLM system.\n\n3, Experiments are extensive, and several findings are useful when designing MLLM models.\n\n4, A new dataset Cambrian-7M is proposed, which may benefit the MLLM fields for further research. \n\n5, The performance looks good compared with LLaVA-Next. If open-sourced, this model can definitely benefit for the community.\n\n6, Open-sourced code and model.\n\nWEAKNESSES:\n1, The technical novelty is limited, combining multiple vision experts into MLLM is not new. Moreover, fusing visual tokens dynamically is also not new in dynamic network design. \n\n[-] Sphinx: the joint mixing of weights, tasks, and visual embeddings for multi-modal large language models, arixv-2023.\n\n[-] Mova: Adapting mixture of vision experts to multimodal context, arxiv-2024.\n\nI think there are more close related works than [1][2]. The authors should cite all these works as respect.\n\n\n2, I also have one question, What if you do not use the dynamic token but only using proposed tuning datasets, compared with LLaVA-1.6.\n\n3, Several findings are well known and also verified from previous works. For example, Finding-6 is common knowledge using high resolution visual encoder. \n\n[-] Vary: scaling up the vision vocabulary for large vision-language models\n\n\n4, Several figures are useless and common knowledge for MLLM community. For example, Fig.1 and Fig.2 can be merged into one figure. \n\n5, The ablation studies for SVA are not enough. For example, which tokens are more important in which datasets? This needs further analysis. Moreover, the effect of increasing instruction tuning data size is not well explored.\n\nGiven these, I rate this work as weak accept.\n\nQUESTIONS:\nSee the weakness part.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a multimodal large language models (MLLMs), named Cambrian-1, designed with a vision-centric approach. In current MLLM researches, the choices of visual encoder are not sufficiently explored. This study utilizes MLLM performance as a visual representation evaluator, showing different characteristics over differently trained vision encoders and revealing that various widely-used MLLM benchmarks are disconnected from visual understanding capability but connected to language capability. Furthermore, this study proposes spatial vision aggregator (SVA) to effectively connect vision and language models with spatial inductive bias. Additionally, curation of high-quality visual instruction-tuning dataset and its distribution balancing are discussed. As a result, Cambrian-1 achieves state-of-the-art performances and provides an open cookbook for MLLMs.\n\nSTRENGTHS:\n- This paper is notably well-written and easy to follow.\n- Section 3.1 shows the limitations of MLLM benchmarks. The finding that several existing benchmarks like MMMU, which were considered important benchmarks in the MLLM field, do not properly evaluate multimodal capabilities is very interesting.\n- This study releases model weights, code, tools, datasets, and detailed recipes, which is a great contribution to this field.\n\nWEAKNESSES:\n- There exists a previous work about vision-language connectors with spatial inductive bias [1]. The comparison or at least discussion between the proposed SVA and C/D-Abstractor [1] is essential but lacks.\n- There are many overlapping findings with existing studies. For example, language-supervised models are effective [2], high-res encoders are beneficial [3], increasing data size and spatial inductive bias are advantageous for connectors [1], and so on. I believe that re-examining these aspects and analyzing them in different settings has its own contribution due to the empirical nature of this field. Nevertheless, it is difficult to attribute high value to the overlapping findings.\n- Findings 7 (the second Findings 6 in the paper, seems to be a typo) is not consistent with the results. The finding claims that performance improves with the vision encoder ensemble, but Table 11 does not seem to support this. For example, SigLIP+DINOv2 performs worse than sole SigLIP.\n\n[1] Cha, Junbum, et al. \"Honeybee: Locality-enhanced projector for multimodal llm.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.  \n[2] Chen, Xi, et al. \"Pali-3 vision language models: Smaller, faster, stronger.\" arXiv preprint arXiv:2310.09199 (2023).  \n[3] Liu, Haotian, et al. \"Improved baselines with visual instruction tuning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nQUESTIONS:\nIn Table 11, why are there two entries for \"SigLIP+DINOv2+ConvNext\" with different numbers?\n\nLIMITATIONS:\nThis study construct a new dataset based on web search, but it does not appear to address any privacy issues. It would be better to address this issue.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper conducts a comprehensive study of multimodal LLMs from a vision-centric perspective. Different from the lines of previous literature which aim to propose new architectures/algorithms for multimodal LLMs, this paper carefully splits the design space of visual parts of multimodal LLMs into several individual parts and diagnoses each with controlled experiments. This leads to several innovative conclusions about the visual aspects of multimodal LLMs, including the validity of standard benchmarks, the choices of visual encoders, etc.\n\nSTRENGTHS:\nGenerally, the paper is a pleasure to read.\n\n- The paper is well-motivated. MLLMs are differentiated from pure LLMs by the visual components. Thus it is quite natural to study the visual aspects of MLLMs.\n- The paper draws rich connections from the visual representation learning literature, which I see as an original perspective, putting the paper into the appropriate position and delivering more valuable information to a broader community.\n- The controlled analysis is precise and rigorous with carefully designed experiments. Particularly, the experiments start with an examination of existing benchmarks, which is a prerequisite of all following experiments.\n- The conclusions are insightful and can be valuable for the future development of multimodal LLMs.\n\nWEAKNESSES:\n- The experiments only consider one particular LLaVa-like formulation of MLLMs built upon a pretrained LLMs, while Sota MLLMs like GPT-4o and Reka are more likely to have completely different training diagrams and architectures, e.g., treating images and texts equally and training a native multimodal LLMs from scratch, or training with interleaved visual and text contents instead of fixed image-first formulations. The value of the paper is thus limited.\n\n-  A minor point: There is no analysis of why the findings hold.\n\nQUESTIONS:\n- What's your opinion of \"native\" multimodal LLMs? Do you think the findings in the paper will in a way transfer to more advanced models?\n\nLIMITATIONS:\nThe authors have discussed the limitations quite adequately.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores Multimodal Large Language Models (MLLMs) and constructs the Cambrian-1 series models. This approach builds a series of advanced MLLMs through five key pillars, achieving exceptional performance in vision-centric tasks and diverse benchmark tests. By exploring different visual encoders, the method designs a novel spatial-aware connector, SVA, to reduce the number of annotations and enhance the integration of visual and language models. The method also curates high-quality visual instruction fine-tuning data from public sources, emphasizing the importance of balanced data distribution and discussing various instruction fine-tuning strategies. Additionally, this paper critically analyzes and supplements existing MLLM benchmarks, introducing the vision-centric benchmark CV-Bench to more accurately evaluate the models' visual-centric capabilities. This approach achieves top performance across diverse benchmarks and excels in visual-centric tasks.\n\nSTRENGTHS:\nThe paper aims to bridge the gap in visual understanding by exploring Multimodal Large Language Models (MLLMs) from a vision-centric perspective. By investigating various visual encoders, this method introduces an innovative spatial-aware connector, SVA, which minimizes the need for annotations and improves the synergy between visual and language models.  Additionally, the paper offers a critical assessment and enhancement of current MLLM benchmarks, presenting a new vision-centric benchmark, CV-Bench, to more precisely measure the models' visual-centric abilities. Cambrian-1 achieves top performance across diverse benchmarks and excels in visual-centric tasks. The paper is well written and the experiment is well solid.\n\nWEAKNESSES:\nNo Weaknesses. See questions\n\nQUESTIONS:\n(1) As the number of cross-attention layers (D) and distinct groups of learnable queries (G) increases, the performance does not show continuous improvement (lines 256-258). It is worth exploring whether performance saturation occurs with the increase of (D) and (G).\n\n(2) Instruction tuning data collected from open web maybe raise the potential data leakage. And provide a statistical analysis of the data categories.\n\n(3) Table 2 suggests that increasing the value t does not continuously enhance performance. The proportion of data varies across different tasks. Additionally explore the data scaling law.\n\n(4) From Cambrian 10M to 7M, whether higher data quality results in better model performance.\n\n(5) Statistical analysis of the response length and the difficulty, diversity distribution of the instruction data.\n\n(6) Compare the performence of current MLLMs like LLaVA and BLIP-2 using the same data and other models (i.e., Minicpm v2.5) with Cambrian-1.\n\n(7) Evaluate model performance on high-resolution images or tasks with extreme aspect ratios, (i.e., V*Bench).\n\n(8) Determine whether training the visual encoder in all tasks outperforms freezing it, and compare the convergence speed of end-to-end training versus two-stage training.\n\n(9) Table 11 indicates that integrating more vision encoders does not necessarily lead to higher performance improvements, as seen with models like MMB, VStar, and MMEP.\n\n(10) Provide detailed information of the parameter counts, training duration, and  training hyperparameters for different model backbones.\n\n(11) The paper improves performance across various tasks by integrating most of the current vision encoders. Could a unified visual encoder be used instead?\n\n(12) Some related work needs to be included and discussed.\n\nZhu D, Chen J, Shen X, et al. Minigpt-4: Enhancing vision-language understanding with advanced large language models[J]. arXiv preprint arXiv:2304.10592, 2023.\n\nWang W, Chen Z, Chen X, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks[J]. Advances in Neural Information Processing Systems, 2024, 36.\n\nChen G, Liu X, Wang G, et al. Tem-adapter: Adapting image-text pretraining for video question answer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 13945-13955.\n\nHuang X, Wang J, et al. Segment and Caption Anything[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024\n\nLiu Y, Zhang C, et al. Universal Segmentation at Arbitrary Granularity with Language Instruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nGao P, Han J, Zhang R, et al. Llama-adapter v2: Parameter-efficient visual instruction model[J]. arXiv preprint arXiv:2304.15010, 2023.\n\nLIMITATIONS:\nThe authors have discussed the limitations of this paper, and this paper has no direct negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-30"}