Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Authors: shengbang  tong
ellis  brown
penghao  wu
sanghyun  woo
manoj  middepogu
sai  charitha
akula  jihan
yang  shusheng
yang  adithya
iyer  xichen
pan austin wang
rob  fergus
yann  lecun
saining  xie
vision model pretrained
vision model connector
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 19:32:35

Abstract: We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. While stronger language models can enhance multimodal capabilities, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning research. This gap hinders accurate sensory grounding in real-world scenarios. Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations, offering new insights into different models and architectures—self-supervised, strongly supervised, or combinations thereof—based on experiments with over 15 vision models. We critically examine existing MLLM benchmarks, addressing the difficulties involved in consolidating and interpreting results from various tasks. To further improve visual grounding, we propose spatial vision aggregator (SVA), a dynamic and spatially-aware connector that integrates vision features with LLMs while reducing the number of tokens. Additionally, we discuss the curation of high-quality visual instruction-tuning data from publicly available sources, emphasizing the importance of distribution balancing. Collectively, Cambrian-1 not only achieves state-of-the-art performances but also serves as a comprehensive, open cookbook for instruction-tuned MLLMs. We provide model weights, code, supporting tools, datasets, and detailed instruction-tuning and evaluation recipes. We hope our release will inspire and accelerate advancements in multimodal systems and visual representation learning.

Main paper: 
 introduction: 
 There is a long-standing debate in philosophy about whether understanding and meaning in language require sensory grounding. Aristotle's emphasis on acquiring knowledge through sensory experience and empirical observation was central to his ancient Peripatetic school and remains influential to this day [8] ; Aquinas famously formalized these ideas in the 13th century with the Peripatetic axiom: "Nihil est in intellectu quod non sit prius in sensu" (Nothing is in the intellect that was not first in the senses) [7] . Though many philosophers disagree [23] , it is evident that having robust and highly capable sensory grounding is at least beneficial. Consider the Cambrian explosion, during which the emergence of vision is believed [105] to have been crucial for early animals to not only find food and avoid predators but also to evolve and improve. In fact, most human knowledge (and nearly Neg.ImageNet-1K [113] Language Supervised CLIP [109] SSL-Contrastive DINOv2 [103] SSL-Masking MAE [52] Diffusion Stable Diffusion [112] Depth SupervisedMiDaS [15] Segmentation Supervised SAM [68] Figure 2 : Examples of various vision models, objectives, and architectures studied. Image from [48] .• Connector Design: We design a new dynamic and spatially-aware connector that integrates vision features with LLMs while reducing the number of tokens. §3 • Instruction Tuning Data: We curate high-quality visual instruction-tuning data from public sources, emphasizing the importance of distribution balancing. §4 • Instruction Tuning Recipes: We discuss instruction tuning strategies and practices. §2.3 • Benchmarking: We analyze existing MLLM benchmarks, cluster them into 4 intuitive groups, and introduce a new vision-centric benchmark "CV-Bench". §2.1, §2.2We defer a detailed review of the fundamental components and methodologies that underpin MLLM research to Appendix B 
evaluating visual representations through mllms: 
 Current MLLMs predominantly rely on CLIP [109] as the visual encoder due to its pre-alignment with language and ease of adaptation to the LLM token space. However, strong language priors can be a double-edged sword-they compensate for deficiencies in learning effective visual representations [126] and diminish insights gained from extensive visual representation learning research.In this section, we systematically evaluate how various visual encoder choices (see Fig. 2 ) impact the multimodal capabilities of MLLMs. We also advocate for using MLLM evaluation as a robust framework for assessing visual representation methods, moving beyond traditional protocols like linear probing and end-to-end fine-tuning to more faithfully reflect the diverse perception challenges in real-world scenarios and to better guide the development of improved visual representations.To effectively evaluate visual representations and MLLMs, we first need to select benchmarks that accurately assess the multimodal capabilities of these models. We use a suite of commonly used benchmarks [24, 45, 54, 57, 83, 84, 91, 92, 96, 97, 120, 126, 137, 143] , which is the intersection of those used in recent MLLM research [75, 77, 137] . To help interpret our results, we begin by analyzing the benchmarks themselves. Here, we train MLLMs with 23 different vision backbones (see Table 6 ) from a variety of model families (see Fig. 2 ) using a 2-stage instruction tuning process initially proposed in [82] : first training connector on 1.2M adapter data from ShareGPT-4V [27] followed by fine-tuning both the connector and LLM on 737K instruction tuning data (see more details in Appendices G.5 and H). Full benchmark results in Table 9 .Who's answering the question: the LLM or MLLM? Determining whether a benchmark truly needs visual input to be solved has been a persistent challenge in vision-language research [2, 26, 50, 94] . In this study, we compare the performance of MLLMs with and without visual input § , and also calculate the expected score via randomly guessing. These three conditions are visualized in Fig. 3 -left, with benchmarks sorted by the difference between the average score with vision enabled and disabled. SQA-I ¶ , MMMU, MathVista, and AI2D display less than a 5% gap between vision enabled and disabled, suggesting that these benchmarks may not significantly depend on visual input and rather heavily rely on the base LLM. TextVQA and GQA both demonstrate a nearly 40% positive gap between random guessing and vision-disabled scores, implying a strong language bias in these benchmarks. On the other hand, the vision-disabled performance on benchmarks like MMVP is notably worse than random guessing, suggesting that strong visual grounding is particularly crucial.Clustering the Benchmarks To better understand the different aspects of MLLM performance, we analyze the correlations between the performance of our 23 MLLMs on each benchmark. A § We note that our instruction-tuning data includes text-only data, so text-only questions are not OOD. ¶ The subset of SQA [91] with images. with bubble size corresponding to benchmark size. We label the clusters as "General" in green, "Knowledge" in yellow, "Chart & OCR" in red, and "Vision-Centric" in blue.confusion matrix (Fig. 10 ) reveals that certain benchmarks, such as MMMU, are largely uncorrelated with the others. We perform principal component analysis on the benchmark scores and observe the formation of clusters corresponding to "General," "Knowledge," "Chart & OCR," and "Vision-Centric" categories (Fig. 3 -right). We assign MMMU to the knowledge category based on the types of questions it includes (see Appendix D). We also find that existing vision-centric benchmarks [126, 137] are of insufficient size (see Fig. 3 -right), challenging the robustness of evaluating such capabilities. These benchmarks do not cover crucial visual elements such as depth and spatial awareness.Finding 1: Most benchmarks do not properly measure vision-centric capabilities, and the ones that do have very few samples.To address the limitations of existing vision-centric benchmarks, we introduce the Cambrian Vision-Centric Benchmark (CV-Bench). With 2638 manually-inspected examples, CV-Bench provides significantly more examples than other vision-centric MLLM benchmarks-3.5× more than Real-WorldQA [137] and 8.8× more than MMVP [126] . By repurposing standard vision benchmarks [18, 79, 154] || , we can assess models at classic vision tasks within a multimodal context. Leveraging the rich ground truth annotations from the benchmarks, we formulate natural language questions that probe the fundamental 2D and 3D understanding of the models.As visualized in Fig. 11 , CV-Bench evaluates 2D understanding via spatial relationships & object counting, and 3D understanding via depth order & relative distance. We refer details to Appendix E.Finding 2: Existing vision benchmarks can be effectively repurposed into VQA questions, enabling the assessment of vision-centric MLLM capabilities.MLLMs start with pre-trained LLM and vision backbones, connecting these modules with a connector such as a projector (MLP). The original LLaVA [80, 82] proposes a 2-stage frozen training process: first, pre-training a connector between frozen LLM and vision backbones using adapter data, and then fine-tuning both the connector and LLM with instruction tuning data while leaving the vision encoder frozen. Various studies [27, 63, 81, 98] have drawn different conclusions regarding the optimal training methodology for MLLMs. Here, we revisit this topic with extensive experiments.For our experiments, we tune a set of MLLMs using Vicuna-1.5-7B as the LLM backbone and each of our 23 vision models (Table 6 ) as the visual encoder. We use a 737K instruction tuning data mix for all experiments here (see Appendix H). All hyperparameters are matched across each experimental setting-highlighting the impact of different tuning strategies with each visual encoder. All experimental settings and results are tabulated in Appendix F. Recent work [63] advocates for skipping connector pre-training, claiming this "reduces compute cost without harming downstream performance." To explore whether this claim holds-especially when using non-language-supervised visual encoders-we conduct experiments using 0, 0.5M, and 1.2M adapter data. Following LLaVA's recipe [82] , we tune only the connector on the adapter data during this first phase, before unfreezing the LLM and connector during instruction tuning on the 737K mix. Fig. 4 shows that pre-training the connector first enhances model performance and that more adapter data further improves performance across all domains. Thus, we subsequently adopt 2-stage training with 1.2M adapter data as our standard setup.Finding 3: Two-stage training is beneficial; more adapter data further improves results.There are also mixed practices in freezing [63, 80, 82] or unfreezing [44, 81] vision backbones during fine-tuning. Some argue that unfreezing the vision backbone significantly degrades performance [63] . Our experiments demonstrate that unfreezing benefits performance across all benchmarks except for a marginal change in knowledge benchmarks (Fig. 4 ). We suspect this is due to the composition of the 737K instruction tuning data and the LLMheavy focus of these benchmarks (see Section 2.1). We note that unfreezing the vision backbone introduces additional computational overhead, which prohibits testing on some larger vision models under current sharding strategies (see more details in Appendix H).Finding 4: Unfreezing the vision encoder is widely beneficial. Language-supervised models always benefit; SSL models particularly benefit on vision-centric benchmarks.As discussed in earlier sections, MLLMs provide a new interface to explore aspects of vision models beyond traditional benchmarks like ImageNet-1k linear probing. We study the 2-stage instruction tuning setting using 1.2M adapter data, 737K fine-tuning data, and frozen visual encoders to allow comparison of the widest range of models. We evaluate on benchmarks detailed in Section 2.1, calculating the average performance ** for each category and visualize the results in Fig. 5 (full results in Appendix F). Our findings highlight the advantages of language-supervised models over non-CLIP models across all benchmark categories, with significantly better performance on chart and OCR-related benchmarks. We hypothesize that this is due to CLIP's training data, such as LAION [115] , containing abundant OCR and text-heavy data, whereas SSL and other vision models primarily train on natural images with significantly less text content. It is also noteworthy that language-supervised models are typically trained with a very large pool of data, ranging from 400 million [109] to 10 billion [28] samples, whereas the largest vision self-supervised training dataset, like DINOv2, consists of only 142 million samples [103] .Additionally, we observe that higher-resolution models particularly enhance performance on chart and vision-centric benchmarks while remaining neutral on general VQA and knowledge-based VQAs. While the majority of the backbones we examine are ViT-based [39] , ConvNet-based architectures (such as OpenCLIP ConvNeXt [86] ) are inherently well-suited for high-resolution image processing [130] and can produce superior results on OCR & Chart and Vision-Centric benchmarks. In vision-centric benchmarks, the gap between language-supervised and other types of vision models is smaller, with a well-trained self-supervised DINOv2 model even outperforming some language-supervised models. Narrowing the gap between Language-and Self-Supervised models Above, we observe that DINOv2 stands midway between SSL models and language-supervised models on general and knowledge benchmarks, even outperforming some language-supervised models on vision-centric benchmarks. Here, we study whether the continued finetuning of an MLLM based on a SSL model can achieve performance similar to that of a language-supervised model. Specifically, we scale up the instruction tuning data from 737K to 5M (see more details in Appendix G.5), and instruction tune MLLMs with DINOv2 ViT-L/14@336 and OpenAI CLIP ViT-L/14@336 encoders in both frozen and unfrozen settings. In Fig. 6 , we observe that by unfreezing the vision backbone, the DINOv2-based MLLM fine-tuned with 5M data surpasses the MLLM trained with a CLIP model on 0.7M data. Additionally, the gap between DINOv2 and the CLIP models is reduced under the 5M setting.Finding 6: Language supervision offers strong advantages, but the performance gap can be narrowed with SSL methods given enough data and proper tuning.As observed in Fig. 5 , different vision encoders excel in different aspects of MLLM performance. In this study, we explore the potential of combining multiple vision encoders to leverage their distinctive representations, aiming to build a more capable MLLM. Given that different vision encoders use varying architectures and image resolutions, we interpolate to a fixed number of visual tokens (576) in this subsection (see details in Appendix F.3). We then concatenate these tokens along the feature dimension, following a method similar to A-MoF proposed in [126] .Our study indicates that adding a non-language-supervised model (DINOv2) can improve benchmark performance, especially in vision-centric tasks. Notably, even OCR benchmarks benefit from incorporating DINOv2. This highlights the importance of self-supervised learning models in complementing language-supervised models to achieve robust multimodal understanding. Detailed results and configurations are available in Appendix F.3.However, this naive strategy has two limitations: 1) it employs interpolation, which can lead to information loss, especially with vision encoders with high-resolution feature maps, and 2) it treats each model equally via simple concatenation. Therefore, we seek a more effective strategy that can more flexibly leverage model combinations with less information loss.Finding 7: Combining multiple vision encoders, including SSL models, can enhance MLLM performance across various benchmarks, particularly in vision-centric tasks. 
spatial vision aggregator (sva): a new connector design: 
 To effectively aggregate features from multiple vision encoders and prevent the information loss introduced by interpolation, we use a set of learnable latent queries that interact with multiple vision features via cross-attention layers [37] . In particular, our approach incorporates two new vision-centric design principles:1. We introduce spatial inductive bias by explicitly defining the aggregation space for each token in the query.2. We aggregate vision features multiple times across the LLM layers, enabling the model to repeatedly access and integrate necessary visual information.To facilitate information aggregation via cross-attention, we create a C-dimension learnable latent token x ∈ R C that is repeated L × L times to form a 2D grid, serving as the query X ∈ R L 2 ×C . The set of visual features F from N vision encoders serve as the context (i.e., key and value). We ensure the output resolution of every vision encoder is a multiple of L. Formally, the feature map of the k-th vision encoder (F k ) has a resolution of m k L × m k L × C, where m k is a positive integer multiplier, and L is the height/width of the learnable 2D grid with hidden dimension C.To maintain the spatial structure during cross-attention, we align each token in the query with a specific sub-region of the feature maps in all vision encoders. Formally, a token at row i and column j in the query x i,j corresponds to the sub-regionF k [m k • i : m k • (i + 1), m k • j : m k • (j + 1)] ∈ R m 2 k ×Cof the k-th vision feature map. As a result, a token x i,j aggregates a total of k m 2 k features from N vision encoders through cross-attention (see Fig. 7-left ).Specifically, the updated query vector q * i,j ∈ R 1×C at position (i, j) is computed asTransformer Block 1 Text Tokens Latent Tokens ! L ! × C SVA Transformer Block k SVA • • • • • • G × D Enc 1 • • • • • • Enc N Enc k Cross Attention ! V i s u a l T o k e n s% % % & " ',)Figure 7 : Spatial Vision Aggregator (SVA). We propose SVA, a dynamic and spatially-aware connector that integrates multiple vision features with LLMs while reducing the number of tokens.9EQUATIONwhereqi,j = W Q x i,j ∈ R 1×C , k i,j,k = W K k F k [m k • i : m k • (i + 1), m k • j : m k • (j + 1)] ∈ R m 2 k ×C , v i,j,k = W V k F k [m k • i : m k • (i + 1), m k • j : m k • (j + 1)] ∈ R m 2 k ×C .Here, q i,j is the query vector at position (i, j), calculated using the query projection matrixW Q ∈ R C×C .The key vectors k i,j,k and value vectors v i,j,k are computed for each vision encoder k using their respective key and value projection matricesW K k ∈ R C×C and W V k ∈ R C×C . Since k m 2 kfeatures are aggregated into a single token, we effectively reduce the number of tokens.Multi-layer vision aggregation Although our proposal effectively aggregates features from multiple vision encoders, there is still potential information loss with high-resolution input (large m k ) or multiple vision encoders (large N ). Here, a single token would have to handle a larger amount of context information during aggregation. To prevent this, we allow cross-attention to occur multiple times by inserting our proposal throughout the LLM layers-allowing consistent access to the uncompressed visual information (see Fig. 7 -right).Hyperparameters To flexibly modulate capacity, we introduce two hyperparameters D and G, which indicate the number of cross-attention layers and distinct groups of learnable queries used between the vision models and the LLM, respectively. D and G are always set to 1 for cross-attention layers within LLM layers. We provide ablation studies on the selection of D and G in Appendix H. We demonstrate the efficacy of SVA module using the best vision model combination results from the previous section and a Vicuna-1.5-7B base LLM. Specifically, we employ a combination of four vision encoders: OpenAI CLIP ViT-L/14@336, SigLIP ViT-SO400M/14@384, OpenCLIP ConvNeXt-XXL@1024, and DINOv2 ViT-L/14@518. We compare our method with two strong baselines: 1) concatenation-based [126] and 2) Re-sampler [11, 72] . Here, we include two variants of our SVA module. The standard one, "SVA", uses D = 3, G = 1, and inserts cross-attention blocks inside the LLM with a layer stride of 3. To isolate the advantages of spatial inductive biases, we include another SVA variant, "SVA-no-multi-agg", that does not add cross-attention blocks inside the LLM and sets D = 3 and G = 3. Table 1 shows that SVA outperforms both baselines, with a significant improvement in the OCR & chart category. In contrast, the Resampler-which lacks spatial inductive biases-struggles to condense concatenated tokens from various vision towers into a limited number of learnable queries via global cross-attention. We also compare SVA with other connectors in Appendix H and show clear advantages.Finding 8: Spatial inductive bias and deep interaction between LLM and vision feature help to better aggregate and condense vision features. 
instruction tuning data for training mllms 4.1 data collection: 
 Collecting Instruction Tuning Data from existing data sources Unlike language data, multimodal (visual) instruction-tuning data is much rarer and harder to collect. To address this, we use existing multimodal benchmarks and datasets involving visual interaction data, such as Visual Question Answering (VQA) and OCR data. To help maintain conversational abilities [147] , we also collect a small volume of high-quality language-only instruction-following data. We categorize data into General conversation, OCR, Counting, Code, Math, Science, and Language-only data. We list the data sources in Fig. 8 , and the details of data preparation in Appendix G.Targeted Internet Data Collection Engine As observed in Fig. 8 , there is an unbalanced distribution of data. Some categories, such as science, have very few data sources, and each source has limited samples. Inspired by previous works [73] , we introduce a data engine to create large-scale, reliable, high-quality knowledge-based instruction tuning data (see Fig. 15 ). Details are in Appendix G.3. Our data engine produces a large volume of reliable scientific data, increasing the diversity in the data pool. We generate 161k science-related data points-400% more than the previous combined data sources.We create a large pool of instruction tuning data, which we refer to as Cambrian-10M. This pool contains approximately 9784k data points, offering a diverse range of data for our work and future research. We visualize its composition in Fig. 8 .Cambrian-10M is a large pool of instruction tuning data sourced from a variety of data sources, with an unbalanced data ratio between categories. Here, we take a preliminary step to study data curation by improving data balancing and adjusting data ratios. Data Balancing We follow previous work [109, 138] to set thresholds t for the number of data points from a single data source. We choose t = 150k, 250k, 350k, and 450k in this section and observe an elbow effect in Table 2 -finding that a threshold between 250k and 350k work the best for Cambrian-10M. We also plot in Appendix G.4 the cumulative sum of counts for entries sorted by counts from tail to head and we see this intermediate threshold prevents explosive heavy tail.Data Ratio Cambrian-10M is designed for visual instruction tuning. Given the various capabilities of different types of data, it is essential to balance the ratio of these data types. We conduct pilot experiments with a fixed dataset size of 1350k, examining the impact of different data ratios. We visualize the results in Fig. 9 and summarize our findings as follows: (i) Balancing General, OCR and Language data is crucial. (ii) Performance on knowledge-intensive tasks is influenced by multiple factors, often requiring a mix of OCR, chart, reasoning, and general perception.Cambrian-7M By applying data filtering to Cambrian-10M with our identified data ratio, we create a smaller but higher-quality dataset called Cambrian-7M. Table 3 showcases the benefits of a wellbalanced and carefully curated dataset. Despite having fewer samples, Cambrian-7M demonstrates improved performance. We additionally apply system prompts in Cambrian-7M to avoid the "answer machine phenomenon", see more details in Appendix G.2. 
state of the art performance: 
 Finally, we leverage the insights from all our previous studies to train a family of MLLMs we call Cambrian-1. We train models using LLM backbones of various scales: LLaMA-3-Instruct-8B [4] , Vicuna-1.5-13B [151] , and Hermes-2-Yi-34B [139] . Our vision component combines four models-OpenAI CLIP ViT-L/14@336, SigLIP ViT-SO400M/14@384, OpenCLIP ConvNeXt-XXL@1024, and DINOv2 ViT-L/14@518 (Section 2.5)-via the Spatial Vision Aggregator (Section 3). We pre-train the connector using 2.5M adapter data and instruction tune using our Cambrian-7M data mix (Section 4.2). Our models are evaluated on the benchmarks categorized in Section 2.1, with results presented in Table 4 . † † .Cambrian-1 surpasses open-source models like LLaVA-NeXT and Mini-Gemini. Thanks to the SVA, Cambrian-1 excels in tasks requiring high-resolution image processing, even with only 576 image tokens-about 1/5 of the tokens used by LLaVA-NeXT and Mini-Gemini. Cambrian-1 also achieves comparable performance to the best proprietary models, such as GPT-4V, Gemini-Pro, and MM-1, on several benchmarks. We provide model weights, open-source code, datasets, and detailed recipes for model training and evaluation. We hope our work will strengthen the open research community and accelerate research in both visual representation learning and multimodal systems.We advocate for using MLLMs as an interface to evaluate visual representations, as previous benchmarks are becoming saturated and do not adequately reflect the diverse and complex perception challenges of the real world. Our work highlights the current gap between language-supervised models and self-supervised learning models and demonstrates the potential of bridging this gap. However, it is known that features of language-supervised models behave like a bag-of-words [125, 144] , underscoring the need for advancements in vision-only models to ensure better visual understanding. We hope to inspire future research into developing better vision-only models intended to be adapted into the MLLM setting, that more effectively leverage large-scale datasets [85] and preserve the advantages in visual grounding [126] .As we observe in Table 4 , a well-trained open-source model such as Cambrian-1 can match or even outperform proprietary models on many existing benchmarks. However, the use and evaluation of MLLMs extend far beyond the current scope of benchmarks-to conversational ability, creativity, reliability, and overall user experience. Developing models solely based on benchmark results can result in an "answer machine", over-optimized for benchmarks but lacking in practical interaction capabilities. Therefore, the development of MLLMs that better align with human and societal needs is a continuously evolving process, both in terms of evaluation and model development.Our current Cambrian-1 model uses a moderate number of visual tokens and does not adopt the anyresolution strategy [30, 77, 81] to handle ultra high-resolution images or those with extreme aspect ratios, which require a larger number of visual tokens. For specialized tasks like V*Bench [136] , which require processing ultra high-resolution images, increasing the resolution and number of visual tokens could lead to an HD version of the Cambrian-1 model.One promising direction for post-training alignment is through reinforcement learning rather than supervised fine-tuning. Many MLLM studies, including Cambrian, primarily focus on supervised fine-tuning. Yet, recent advancements in LLMs [38, 104, 110, 156] and some in MLLMs [141, 146] suggest that reinforcement learning from human or environmental feedback can further improve models, potentially surpassing the limits of supervised fine-tuning, especially in decision-making abilities.Cambrian-10M (Fig. 8 and Section 4) provides a rich pool of data for studying data curation in fine-tuning MLLMs. Our work takes an initial step in curating higher-quality data to enable more efficient and effective instruction tuning. We believe there is room for further improvement in the data curation pipeline, and we hope this work can serve as a foundation for future research.Additionally, training large-scale models requires careful design of model sharding, data sharding, and infrastructure adaptations. In this work, we train our model on TPU-V4 [60] with FSDP [150] using TorchXLA. We share our experiences, technical challenges, and solutions in Appendix C. We also open-source our implementation and provide tutorials to help the community undertake large-scale training more efficiently.To conclude, Cambrian-1 introduces a family of state-of-the-art MLLM models that achieve top performance across diverse benchmarks and excel in visual-centric tasks. We provide model weights, open-source code, datasets, and detailed recipes for model training and evaluation. We hope our work will strengthen the open research community and accelerate future advancements in both visual representation learning and multimodal systems.The key components of MLLM research include the Large Language Model, Visual Encoder, Multimodal Connector, Data Curation Pipeline, Instruction Tuning Strategy, and Evaluation & Benchmarking. Each component has its intricacies, and understanding their interactions presents significant challenges. Our study investigates these aspects from a vision-centric perspective.Large Language Model Advanced LLMs [4, 101, 127, 128] are the foundation of an MLLM. After instruction-tuning on multimodal data, these models can be prompted to solve a variety of complex tasks and generate free-form responses leveraging input from a visual encoder. Recent MLLM research focuses on enhancing the LLM backbone [10, 75, 81] , resulting in improved performance on benchmarks like MMMU [143] and AI2D [54] . However, this improvement raises the concern that our current multimodal evaluation is biased by the development of LLMs, neglecting a true assessment of visual perception. For example, some benchmarks such as MMMU [143] are dominated by LLM capabilities, underscoring the need for evaluations that genuinely assess multimodality (see Section 2.1).Visual Encoder Most MLLMs utilize language-supervised models like CLIP [109, 122, 145] , which benefit from the massive scale of noisy web image-text data. However, there is a much broader pool of visual models that learn representations using only visual signals-such as self-supervised models [9, 103] , segmentation [68] , depth-supervised [15] , and diffusion models [74, 112] (see Fig. 2 ).Recent work [87, 126] advocates for incorporating these diverse vision models into MLLMs. In this study, we systematically examine the impact of various vision backbones on MLLM performance (Section 1) and explore the benefits of model ensembles (Section 2.5).Multimodal Connector Representations from a visual encoder cannot be natively processed by an LLM-they must be mapped into the LLM token space by a connector. There are three primary approaches to connector design: Resamplers [6] , Q-Formers [11, 37] , and MLP Projectors [43, 80, 82, 157] . We begin our exploration using an MLP projector, which is highly effective but presents challenges: the visual token count grows quadratically with image resolution, inhibiting scaling context length input resolution. For example, LLaVA-Next [81] requires 2880 visual tokens to process one 672px image. To address this, we explore new vision connector designs that process high-resolution images while maintaining a smaller number of visual tokens (Section 3).Instruction Tuning Data Visual instruction tuning data is crucial but hard to collect, as it rarely naturally exists on the internet. Previous work [37, 80, 98] transforms existing VQA benchmarks [50, 69] into instruction tuning data, showing marked MLLM performance improvements. With this inspiration, we collect all VQA benchmarks and visual interaction data that we can find (Fig. 8 ), study data balancing and category mixtures (Section 4.2), and develop an internet data collection engine to fill in the gaps (Section 4.1).Instruction Tuning Most current MLLMs leverage pre-trained LLMs and visual encoders, finetuning the LLM and connector using visual instruction tuning data. Some aspects of the tuning recipe are up for debate, including whether to pre-train the connector before joint fine-tuning with the LLM , and whether to freeze or unfreeze the vision encoder during fine-tuning [63, 98] . Additionally, some recent proprietary models explore end-to-end training from scratch [49, 102] . In this work, we use pre-trained models and revisit the debated recipe aspects with extensive studies, providing more insights for future MLLM research (Section 2.Evaluation & Benchmarking There is an extensive set of benchmarks that evaluate various aspects of MLLMs, such as perception [45, 83] , knowledge [91, 92] , chart interpretation [84, 96] , and visual capabilities [126, 136] . Instead of over-optimizing for specific benchmarks, we advocate for examining aggregates of benchmarks that focus on specific capabilities. To achieve this, we analyze existing benchmarks, categorize them, and assess the extent to which they measure multimodality (Section 2.1). Additionally, we find there are currently few benchmarks focused on vision-centric evaluation, and those that do exist contain relatively few images, leading to higher variance during evaluation. To address this issue, we propose a new vision-centric benchmark by reformulating classic vision tasks (Section 2.2).All models in this paper were trained using TPU-V4 pods [60] ; we evaluate using NVIDIA A6000, A100, and H100 cards. The experiments in Section 2.4 require less than 24 hours on a TPU-V4-128, while our final Cambrian-1 models are trained in less than 4 days on a TPU-V4-512. We evaluate the benchmark scores for our one-stage, two-stage finetune-only and hybrid models, and then plot the correlation matrix for the pool of MLLM benchmarks. The correlation plot displays in Fig. 10 . The result demonstrates that MMMU is less correlated in measuring model performance to other benchmarks. Nonetheless, we acknowledge it is widely used and therefore cluster it into knowledge-based QAs based on the nature of their questions. CV-Bench Curation Below we describe the procedure for programmatically constructing questions for each task. To ensure reliability, we also manually inspect each question, removing those that are unclear, ambiguous, or erroneous.Spatial Relationship (2D). We consider images with two distinct ground-truth object categories and use visual prompts (bounding boxes) to avoid ambiguity when multiple instances are present. In these questions, we designate an anchor object, and the question asks for the direction of the other object relative to this anchor.Object Counting (2D). This tests the model's ability to count objects. When generating options for these questions, we construct multiple-choice options that are similar to the correct answer. ForWhere is the cave located with respect to the trees?How many cars are in the image?Which is closer to the camera, sink or pillow?Which is closer to the chair, refrigerator or door?Source benchmark: ADE20K [154] and COCO [79] Source benchmark: Omni3D [18] Figure 11 : Cambrian Vision-Centric Benchmark (CV-Bench). We repurpose standard vision benchmarks to evaluate the fundamental 2D and 3D visual understanding of MLLMs. See Section 2.2 for more details.Type Determine which of the two distinct objects is closer to the camera.Omni3D 600Determine which of the two distinct objects is closer to the anchor object.Omni3D 600 The examples are sourced from ADE20K [154] , COCO [79] , and Omni3D [18] .example, if the correct answer is 4, the options might be 2, 3, 4, 5, & 6. We also include existence check examples where the correct count is 0.Depth Order (3D). We consider images with two distinct categories (i.e., object A and object B) and use visual prompts (e.g., bounding boxes with two different colors) to avoid ambiguity. We define "closer" as follows: object A is closer to the camera than object B only if the farthest vertex of object A is closer ‡ ‡ to the camera than the nearest vertex of object B by a specified offset.Relative Distance (3D). We consider images with three distinct categories (i.e., anchor, object A, and object B), and use visual prompts (e.g., bounding boxes with three different colors) to avoid ambiguity. Object A is closer than object B only if the farthest distance from A's vertices is shorter than the shortest distance from B's vertices to the anchor object by a certain offset.Curation Procedure We provide an overview of the data curation process in Fig. 12 , which is conducted in a semi-automatic manner. The procedure consists of two main steps:First, using the original benchmarks and their associated ground truth annotations, we generate query and answer pairs. These pairs are tailored to specific tasks: 2D-related tasks with COCO and ADE20K datasets, and 3D-related tasks with Omni3D.Second, after generating the query and answer pairs, we engage human experts to manually filter out any incorrect or ambiguous queries to enhance the quality of benchmark. Each query is assigned one of three statuses: accepted (used as is), modified (where the incorrect answer is modified), and rejected (queries that are ambiguous, such as those too small or difficult to discern, even for human experts).Following this two-stage process, we finalize the benchmark, which results in a total of 2638 image queries with improved accuracy and reliability. Subsequently, we will discuss the methods of human verification and the evaluation metrics used in this process.Human verification ‡ ‡ We use the Euclidean distance. There are multiple reasons for the above generated data to be inaccurate. One of the main reasons is sparse annotations, but occasionally there could be wrong annotations as well.Thus, we need manual inspection to change/remove these examples generated. Here are a few criteria we followed while manually filtering COCO and ADE20k data.For Counting question types, if all instances of a category are not annotated, the ground truth would have lower count than the actual number of instances appearing in the image. In a few cases where the image distinctly has different countable instances of the object, we change the options/answer. In case the count is ambiguous, we reject the data sample altogether.For Relative Distance question types without annotation, if the question is asked about two objects A and B and if there are two instances of a specific category (say A), the relative location of A w.r.t B can be have multiple correct answers. We reject the sample in this case. We also reject cases with clear incorrect annotations.Benchmark Evaluation To ensure that equal importance is given to both 2D and 3D tasks, we use an evaluation metric that is the average of the accuracies obtained from these tasks. Specifically, the overall performance is calculated as follows:EQUATIONOverall Accuracy = Accuracy 2D + Accuracy 3D 2As mentioned in Section 2.4, we use MLLM as an interface to evaluate vision model's different capabilities. Here, we list details in terms of the model selection, full results, and data split.In our exploration of versatile vision models, we select thirteen models and group them into four categories: language-supervised models (i.e., OpenAI CLIP [109] , SigLIP [145] , DFN-CLIP [40] , EVA-CLIP [122] and OpenCLIP [33] ), self-supervised models (i.e., DINOv2 [103] , I-JEPA [9] , MAE [52] , MoCo v3 [29] ), class-supervised models (ImagetNet22K ViT [39] ) and other models such as stable diffusion [112] , segmentation models like SAM [68] , and depth estimation models like MiDaS [15] . To provide a clear understanding of the specific variant evaluated, we meticulously detail their backbone architectures, resolution, number of tokens, and hidden dimension sizes in Table 6 . For models that output a large number of patches in the last layer (e.g., SAM and ConvNeXt) we interpolate to the number of tokens specified in Table 6 , and denote interpolation with I .We extract features following the practice in [12] Supervision Type Here, we also show a ranking version of Fig. 5 . We observe a clear advantage of CLIP models over non-CLIP models. We also observe that within the family of CLIP models, each model perform differently in different domains. This provide insight into both vision model development and data curation in training large vision models.For the above-listed vision models in Table 6 , they are integrated as the vision encoder of the MLLMs. These MLLMs are trained on various adapter adapter data splits (i.e., 0, 0.5 and 1.2 million), and subsequently fine-tuned on a 737K instruction tuning dataset provided in LLaVA-1.5 [80] . For the adapter data splits, the 0M split indicates that no initial adapter pertaining phase is employed for the Here, "SigLIP" = ViT-SO400M/14@384, "DINOv2" = ViT-L/14@518, "ConvNext" = OpenCLIP ConvNeXt-XXL@1024, and "CLIP" = OpenAI CLIP ViT-L/14@336.Here, we investigate a phenomenon we term the "answer machine phenomenon". We observe that a well-trained MLLM may excel at VQA benchmarks, but lack basic conversational abilities and default to outputting short, curt responses (see examples in Fig. 14 ). This discrepancy arises because benchmark questions typically require responses that are limited to a single option, choice, or worddiverging from the more broad and realistic use cases of MLLMs. Similar phenomena have been discussed in other LLM studies [114, 151, 155] .LVIS-Instruct4V [131] , SketchyVQA [129] , OODVQA [129] , VizWiz [51] , ALLaVA [25] , IDK [22] , Q-Instruct [135] , LAION GPT-4V [70] , Hateful-Memes [66] , Visual7W [159] , Visualmrc [123] , AlfWorld [117] , LNQA [108] , LLaVA150K [82] , ShareGPT [27] , VQAv2 [50] , GQA [57] , OKVQA [95] , A-OKVQA [116] , RefCOCO [140] , VisualGenome [69] , GPT-4V recorded chat OCR Related Data LLAVAR [149] , ChartQA [96] , DocVQA [97] , DVQA [61] , ArxivQA [76] , AI2D [65] , ScreenQA [56] , SynthDog [67] , IconQA [89] , WTQ [106] , WikiSQL [153] , FinQA [31] , HiTab [32] , TAT-QA [158] , TabMWP [88] , Chart2Text [62] , VisText [124] , InfoVQA [16] , ST-VQA [17] , Rendered-Text [134] , OCRVQA [100] , TextCaps [119] , ShareGPTOCRData [27] Counting Data TallyQA [1] , CLEVR [59] Knowledge-Based Data Code: Design2Code [118] , WebSight [71] , Datikz [14] Math: MathVision [132] , Geo170K [42] , TQA [5] , Inter-GPS [90] , RAVEN [148] , GeomVerse [64] Science: ScienceQA [91] , PathVQA [53] Language Only Data Dolly [36] , MathInstruct [142] , WizardCoder [93] , OrcaMath [99] , OpenCodeInterpreter [152] , OpenOrca [78] Table 15 : Visual Instruction-Tuning Data CatalogWe suspect that this issue stems from instruction tuning data containing an excessive number of short-response VQA tasks, leading to catastrophic forgetting in LLMs. To address this, we incorporate additional system prompts during training. We append prompts such as "Answer the question using a single word or phrase." before questions that generate a single word or phrase in the response. Full details of the system prompts used are provided in Appendix G.2. After integrating these system prompts, we observe that while the model's benchmark performance remains unchanged, its conversational ability improves dramatically. For example, in Fig. 14 , models with system prompts produce longer and more engaging responses while answering questions correctly. The system prompts also enhance the model's performance on reasoning-related tasks, such as math problems, by encouraging a chain of thoughts [133] followed by the answer.This underscores the necessity of developing evaluation protocols like the Chatbot Arena [34] for MLLMs, despite the challenges in collecting large-scale, real-world interaction data. While performing well on benchmarks is important, it is equally crucial to ensure the model can engage in meaningful and natural interactions. The overall user experience and the model's conversational abilities are paramount, as a model that excels in benchmarks but fails to converse effectively cannot meet the needs of practical applications.As our Cambrian data includes instructions/questions and responses of different types and formats (e.g., Short response with a single word or regular response as a complete sentence), it is important to specify the required response format in the instruction prompt to avoid ambiguity and possible conflicts. Some of the datasets already include such prompts and we add proper prompts for the remaining datasets. The detailed response formatting prompts we additionally add are listed in Table 16 .The data engine is designed to generate instruction tuning data for knowledge-based fields, where previous works rarely covers and MLLMs are not reliable to distill for from. The data engine takes in a given field, such as "Physics", utilizing reliable web sources like Wikipedia. Below are the various stages involved in the process. We also visualize this process in Fig. 15 :Stage 1 -Topic Generation: We start by compiling a list of fields and subfields and subsequently generate topics for each field using a Large Language Model (LLM), such as GPT-4. In this stage, What is unusual about this image? Please answer in details.Describe this image please.There are six people in the image.There are six people in the image.Based on the chart, black tea appears to be the most popular type of tea in the U.S. Bubble Tea Market, as it consistently has the highest sales volume across all years displayed.This is New York City, and the prominent landmark in the image is the Washington Square Arch, which is located in the Greenwich Village neighborhood. To tour around this landmark, you should visit the nearby Washington Square Park, which is a popular spot for locals and tourists alike. The park offers a variety of activities, including walking paths, a fountain, and a statue of George Washington. Additionally, the surrounding area is rich with historical buildings, art galleries, and restaurants, making it a vibrant and culturally diverse place to explore.The unusual aspect of this image is the anthropomorphic design of the church's bell tower, which is given human-like features. Specifically, the two windows on the tower are designed to resemble eyes, and the cross at the top of the tower is stylized to look like a nose. This creative design choice is not typical for church architecture, which usually favors more traditional and less whimsical features. The intention behind such a design could be to add a playful or artistic element to the building, possibly to attract attention or to reflect a unique character of the church or the community it serves.The image is unusual because it features a church with a design that incorporates elements typically associated with a face. The two circular windows on the tower serve as eyes, and the cross on top of the tower is positioned to resemble a nose. This anthropomorphic design is not commonly found in church architecture, which often favors more traditional and conventional styles. The presence of the palm trees adds a tropical or subtropical feel to the scene, which may not be the usual setting for such a building. The overcast sky suggests a gloomy or somber mood, which contrasts with the whimsical nature of the church's design.The image captures a serene moment in a bathroom setting. A fluffy, gray cat is comfortably seated on the edge of a white bathtub. The cat's gaze is directed upwards, possibly at something outside the frame, giving it a contemplative expression. The backdrop is a simple white tiled wall, which contrasts with the cat's fur, making it the focal point of the image. The overall scene suggests a quiet, peaceful moment in a domestic setting.To solve the equation (\sqrt{k+2} -x = 0), we need to isolate (k). we processed 30 fields, resulting in 3660 topics. We then post-process the output of LLMs into json formats. For example, the topic data for Physics looks like below.Physics { "Classical Mechanics": [ "Newton's Laws of Motion", "Conservation of Energy", "Conservation of Momentum", "Harmonic Motion", "Rotational Dynamics", "Gravitation and Orbits", "Fluid Dynamics", "Elasticity and Plasticity", "Friction", "Waves and Sound", "Velocity and Acceleration", "Angular Momentum", "Statics and Equilibrium", "Kinematics of Particles", "Dynamics of Systems of Particles", "Collisions", "Centripetal Force and Acceleration", "Lagrangian and Hamiltonian Mechanics", "Chaos Theory", "Equations of Motion" ], "Electromagnetism": [ "Coulomb's Law", "Electric Field and Electric Potential", "Gauss's Law", "Capacitance and Dielectrics", "Current and Resistance", "https://en.wikipedia.org/wiki/Voltage", "https://en.wikipedia.org/wiki/Electricity", "https://en.wikipedia.org/wiki/Electrostatics", "https://en.wikipedia.org/wiki/Electric_dipole_moment", "https://en.wikipedia.org/wiki/Magnetic_vector_potential", "https://en.wikipedia.org/wiki/Electric-field_screening", "https://en.wikipedia.org/wiki/Electric_flux" ],Stage 3 -Parsing: In this stage, we parse each web page to extract image-caption-text tuples. We aim to identify the blocks containing an image, the image's caption, and relevant textual content. Below is an example of the parsed data for the same topic, "Electric Field and Electric Potential": { "Electric Field and Electric Potential", [ { "section": "Electrostatics", "text": "An electric potential at a point r in a static electric field E is given by the line integral where C is an arbitrary path from some fixed reference point to r; it is uniquely determined up to a constant... The generalization of electric potential to this case is described in the section Generalization to electrodynamics.", "images": [ { "caption": "Electric potential of separate positive and negative point charges shown as color range from magenta (+), through yellow (0), to cyan (-). Circular contours are equipotential lines. Electric field lines leave the positive charge and enter the negative charge." }, { "caption": "Electric potential in the vicinity of two opposite point charges." } ],"link": https://en.wikipedia.org/wiki/Electric_potential, "title": "Electric potential", "field": "Physics", "subfield": "Electromagnetism", "topic": "Electric Field and Electric Potential" }, ... ] } Stage 4 -Data Generation: We generate dataset in this stage, ensuring high quality. We first filter out data samples with fewer than 50 words in the text. Then, instead of downloading images directly from the links retrieved during web parsing, we download high-resolution images from the original sources. We then convert formats like SVG or GIF into a common standardized format, PNG.Question-Answer pairs are generated by using LLM such as GPT-3.5 from the image metadata, caption, and contextual text. These Q&A pairs and the image form our VQA dataset. We generated 165k data samples. Here is an example of the generated data: has been assumed", "section": "Depiction of the A-field", "...", "Question": "What do the thicker lines signify?", "Answer": "The thicker lines in the image represent paths of higher average intensity in the magnetic vector potential A field. Shorter paths have higher intensity, ensuring that the path integral remains constant. These lines are drawn to visually convey the general behavior and intensity distribution of the A-field around the toroidal inductor" } This data engine is designed for scalability and efficiency and is capable of handling extensive data generation tasks using multithreading techniques.Data Balance via Fitlering t As discussed in Section 4.2, if left unfiltered, the data pool is dominated by noisy, unbalanced data sources such as CLEVR and DVQA, leading to pronounced exponential tails. However, as we apply different t values to filter data from each source, the exponential tails become less pronounced, resulting in a more balanced dataset. We also present all the results in Table 17 . t value 250k has the highest average across all benchmarks; 250k and 350k also have the highest performance across many individual benchmarks.Here, we plot the cumulative sum of counts for entries sorted by counts from tail to head. From Fig. 17 , we see this intermediate threshold prevents explosive heavy tail.We present the full results of our data ratio study in Table 18 . The table highlights the importance of finding an optimal data ratio that balances different aspects of MLLM. Experiment 5 achieves well-rounded performance with its selected data ratio. For the 5M data mixes we use in Section 2.4, we apply data filtering discussed in Section 4.2 and apply t=150k on all multimodal instruction data in Cambrian-10M.One potential concern with our targeted data engine (Section 4.1) is that instruction-tuning data collected from the open web could introduce data leakage. To address this, we systematically analyze the extent of direct image matches between our training data and our test sets. Using difference hashing (dHash) [19] , we compute hashes for all images in the training data and test sets. We then compare these hash sets to determine how many test images overlap with our training data, reporting the number of collisions in Table 19 .Across all fifteen datasets, our targeted data engine finds only 32 test images in total, amounting to just 0.06% of the test data. This low overlap percentage dispels concerns that our data engine inadvertently targets specific test sets. When analyzing the full Cambrian10M dataset-which is 15x larger than LLaVA-665k-we observe only 6x more matching test images (7, 244 We encourage future research exploring the impact of image-only leakage on the performance of MLLMs. Understanding this influence may yield insights into the boundaries of model generalization and guide future best practices for dataset construction in multimodal learning.We conducted a preliminary analysis of the Cambrian dataset, focusing on the distribution of male, female, and neutral pronouns. Our findings show the following distribution: 38.35% male pronouns, 17.99% female pronouns, and 43.66% neutral pronouns.We recognize that training models on biased data can perpetuate these biases. Addressing bias by artificially modifying data distributions-such as through rebalancing or applying fairness constraints-can help mitigate this issue, but it also presents challenges. These include the potential loss of generalization and the risk of introducing new biases. Additionally, identifying and mitigating bias in Multimodal Large Language Models (MLLMs) is particularly complex, given the interaction between different data modalities. We believe that openness in model development and data curation will accelerate research aimed at understanding and mitigating these potential harms.Cambrian Models For our final Cambrian models, we use 2.5M adapter data which is comprised of 1.2M captioning data from shareGPT4V [27] and 1.2M captioning data used in MiniGemini [77] .SVA We provide here ablation studies of SVA module. We further conduct ablation experiments using OpenAI CLIP ViT-L/14@336 + OpenCLIP ConvNeXt-L@1024 as our base model combination. We focus on the OCR & chart categories to assess the impact on high-resolution visual understanding. The results show that increasing capacity via D or G improves performance and that allowing vision aggregation across multiple layers by adding cross-attention layers within the LLM also enhances performance.Compared with other spatial-based connectors like C/D-Abstractor [21] which are designed for single vision feature maps, our SVA module can dynamically combine visual features from multiple vision models with varying resolutions. Besides, our spatial inductive bias in SVA can better compress spatial information compared with such methods. To isolate the effect of spatial inductive bias, we consider the case of token reduction using a single vision encoder. Specifically, we use OpenAI CLIP ViT-L as the vision model and compress its original 576 tokens to 36 tokens using our SVA module and other connectors. We compare our SVA module with three baselines: 1) Direct interpolation + MLP, 2) C-Abstractor [21] , and 3)LDPv2 Projector [35] (similar to C-Abstractor but more lightweight). For fair comparisons, we do not include multi-layer aggregation inside the LLM for our SVA baseline, and the results are shown in We introduce learnable k m × k m positional encodings in the vision features when k m > 1. Besides, during cross-attention, the query is augmented with a global feature obtained by global pooling over the vision features, which is concatenated with q i,j to better guide the aggregation process. In our experiments, the feature maps of all vision encoders except for ConvNext are interpolated to 576×576 (m k = 1 for L = 24). For ConvNext, we first interpolate the feature maps from its 4 stages to 96 × 96 (m k = 4 for L = 24) and then channel-wise concatenate them to form its final vision feature map similar to [77] .For experiments in Section 3, we set D = 3, G = 1 and add cross-attention layers between the layers of LLM with a stride equal to 3. For our final Cambrian models, we set D = 3, G = 1 and insert multiple cross-attention layers in LLM considering the tradeoff between performance and efficiency. For Cambrian-8B, Cambrian-13B, and Cambrian-34B, the strides of cross-attention layers inside the LLM are 3, 4, and 9 respectively.To study the importance of visual features from different vision models to different image categories, we further investigate the attention score distribution in our SVA module. We evaluate our Cambrian-8b model on GQA, DocVQA, and ScienceQA (representing three different benchmark categories), and the attention distribution results are shown in Table 22 . We can see that on real-world images (GQA), the contribution of different vision models is relatively uniform, in part due to the similar characteristics of SigLIP and CLIP. On document-type images (DocVQA) which are text-heavy and often high-resolution, the influence of SigLIP increases and that of ConvNext greatly increases to aid in high-resolution information processing. For scientific images (ScienceQA) composed of illustrations and diagrams about different science categories, the contribution of SigLIP is further increased while the portion of DINOv2 decreases compared to GQA. Unfreezing While unfreezing is largely beneficial (Section 2.3 and Fig. 13 ), it has a significant speed drawback. Given fixed computational resources, unfreezing visual encoders slows down the fine-tuning process by approximately 50-55%. For initial explorations or when computational overhead is a concern, leaving the visual encoders frozen can be a practical strategy. This allows for quicker iterations and tuning, especially during early research phases, while still providing valuable insights. Ultimately, unfreezing is recommended for achieving the best performance once the setup has been optimized.To ensure the reproduction of our results, we also include the system prompts we used in this work.The system prompts for our models can be found in Table 25 . Additionally, we release the prompts we used while evaluating our models on the various benchmarks in Table 26 . We hope this sets a precedent for future research to improve the reproducibility of benchmark results.We use fuzzy matching to evaluate responses in some benchmarks, since MLLMs can answer questions with auxillary phrases. To study the effectiveness of our fuzzy matching, we compare our model accuracy through fuzzy matching with the model accuracy obtained when we use LLM as a grader.The LLM grader is sensitive to the prompt given to it while grading, and we prompt the LLM (we use OpenAI GPT-3.5-turbo and GPT-4-turbo as our graders) with few shot grading examples, which we notice significantly improves grading accuracy. An example of such a prompt is given below.You are a reliable grader. Reply with only either of the following 2 words: CORRECT or INCORRECT. J More results of Cambrian-1 ModelIn Fig. 18 , we also plot our Cambrian-1 performance as well as GPT-4 performances. In the plot, it is clear that Cambrian-1 offers competitive performance compared to proprietary models in most categories. We also showcase some examples in Fig. 19 , demonstrating that the model effectively attends to details in images despite using only 576 tokens.We recognize that there are ethical concerns regarding the potential misuse of multimodal large language models like Cambrian-1, particularly in generating misleading content or spreading misinformation. Below, we outline the main risks and provide strategies to address them:1. Misinformation Cambrian-1 could be used to create misleading text descriptions of images, leading to false narratives or misrepresentations. For instance, such models might be leveraged by social media bots to manipulate public opinion during elections or other critical events.2. Hallucination Similar to any large language model, Cambrian-1 may produce information that is not based on facts or actual input data. This phenomenon, often called "hallucination," can be dangerous if users assume the model's output is entirely accurate without verification.To mitigate these risks, users should exercise caution and critical thinking when interpreting outputs generated by Cambrian-1. It is important to verify the information produced by the model, particularly if the results are intended for sensitive or high-stakes applications. Users must be aware of the potential for hallucinations, where the model produces information not grounded in facts, and take steps to cross-check and validate any critical outputs. Additionally, implementing content filtering as a safeguard can help flag potentially harmful or misleading content before it is disseminated.Why is this image funny?The humor in this image comes from the unexpected twist on the classic poem 'Roses are red, violets are blue.' Instead of a romantic or sweet sentiment, the poem is humorously altered to a more mundane and slightly unsettling scenario, which is a bug in the bed. This juxtaposition of a traditional romantic poem with a common, albeit slightly unpleasant, household occurrence creates a comedic effect.The character in the image is Homer Simpson, a central character from the animated television series 'The Simpsons.' Homer is known for his distinctive yellow skin, casual attire, and often humorous or oblivious behavior.What does this image remind you of?The image might remind you of a playful and creative culinary presentation, possibly inspired by a world map, where each piece of food represents a different continent. It could also evoke thoughts of a fun cooking challenge or a themed meal where the food is arranged to mimic the shape of the Earth.To solve this problem, we need to determine which option correctly completes the sentence in a way that adheres to the conventions of Standard English. The sentence is: "In 1937, Chinese American screen actor Anna May Wong, who had portrayed numerous villains and secondary characters but never a heroine, finally got a starring role in Paramount Pictures' Daughter of Shanghai, a film that _____ "expanded the range of possibilities for Asian images on screen."The correct option should be the one that maintains the subject-verb agreement and the appropriate punctuation. Option A: "critic, Stina Chyn, claims" -This option is incorrect because it uses a comma after the name, which is not standard in English. Option B: "critic, Stina Chyn, claims," -This option is also incorrect because it uses a comma after the name and an extra comma at the end of the sentence, which is not standard in English. Option C: "critic Stina Chyn claims" -This option is correct because it maintains the subject-verb agreement and does not use any unnecessary commas. Option D: "critic Stina Chyn, claims," -This option is incorrect because it uses a comma after the name and an extra comma at the end of the sentence, which is not standard in English. Therefore, the correct answer is Option C: "critic Stina Chyn claims."• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: We used pre-trained LLM, which already has safeguards. As for datasets, we have conducted filtering to avoid safety risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We have credited and mentioned the assets we used.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.|| Omni3D assets are sourced from[3,13,20,46,111,121].† † For the General Average, we note that GPT-4's performance on the GQA test set is low, possibly because other models are trained on the GQA training set, whereas the training set used for GPT-4 is unclear. 
 
 Appendix:  


Decision: The paper presents Cambrian-1, a family of multimodal large language models with a vision-centric approach. The authors explore the use of various visual encoders and propose a novel spatial vision aggregator (SVA) to enhance the integration of vision and language models. The work critically evaluates existing benchmarks and curates a large dataset for visual instruction tuning. Cambrian-1 achieves state-of-the-art performance across multiple benchmarks and provides an open-source toolkit for further research in multimodal systems.

The contributions of the paper are solid. There are comprehensive evaluation of multimodal benchmarks, identifying gaps in visual understanding. The introduction of the SVA module, shows notable performance improvements in specific tasks like OCR and chart understanding. The open-source contributions including model weights, datasets, and code, will likely to benefit the research community considerably. Extensive experiments with various visual encoders offers valuable insights into their performance. Despite some limitations in novelty, the paper makes a valuable asset for the research community. The paper's performance on multiple tasks and its rigorous experimentation further support its acceptance.

Reviews: review: paper_summary: paper_summary: The paper explores Multimodal Large Language Models (MLLMs) and constructs the Cambrian-1 series models. This approach builds a series of advanced MLLMs through five key pillars, achieving exceptional performance in vision-centric tasks and diverse benchmark tests. By exploring different visual encoders, the method designs a novel spatial-aware connector, SVA, to reduce the number of annotations and enhance the integration of visual and language models. The method also curates high-quality visual instruction fine-tuning data from public sources, emphasizing the importance of balanced data distribution and discussing various instruction fine-tuning strategies. Additionally, this paper critically analyzes and supplements existing MLLM benchmarks, introducing the vision-centric benchmark CV-Bench to more accurately evaluate the models' visual-centric capabilities. This approach achieves top performance across diverse benchmarks and excels in visual-centric tasks.
strength_weakness: strength_weakness: No Weaknesses. See questions
questions: questions: (1) As the number of cross-attention layers (D) and distinct groups of learnable queries (G) increases, the performance does not show continuous improvement (lines 256-258). It is worth exploring whether performance saturation occurs with the increase of (D) and (G).

(2) Instruction tuning data collected from open web maybe raise the potential data leakage. And provide a statistical analysis of the data categories.

(3) Table 2 suggests that increasing the value t does not continuously enhance performance. The proportion of data varies across different tasks. Additionally explore the data scaling law.

(4) From Cambrian 10M to 7M, whether higher data quality results in better model performance.

(5) Statistical analysis of the response length and the difficulty, diversity distribution of the instruction data.

(6) Compare the performence of current MLLMs like LLaVA and BLIP-2 using the same data and other models (i.e., Minicpm v2.5) with Cambrian-1.

(7) Evaluate model performance on high-resolution images or tasks with extreme aspect ratios, (i.e., V*Bench).

(8) Determine whether training the visual encoder in all tasks outperforms freezing it, and compare the convergence speed of end-to-end training versus two-stage training.

(9) Table 11 indicates that integrating more vision encoders does not necessarily lead to higher performance improvements, as seen with models like MMB, VStar, and MMEP.

(10) Provide detailed information of the parameter counts, training duration, and  training hyperparameters for different model backbones.

(11) The paper improves performance across various tasks by integrating most of the current vision encoders. Could a unified visual encoder be used instead?

(12) Some related work needs to be included and discussed.

Zhu D, Chen J, Shen X, et al. Minigpt-4: Enhancing vision-language understanding with advanced large language models[J]. arXiv preprint arXiv:2304.10592, 2023.

Wang W, Chen Z, Chen X, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks[J]. Advances in Neural Information Processing Systems, 2024, 36.

Chen G, Liu X, Wang G, et al. Tem-adapter: Adapting image-text pretraining for video question answer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 13945-13955.

Huang X, Wang J, et al. Segment and Caption Anything[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024

Liu Y, Zhang C, et al. Universal Segmentation at Arbitrary Granularity with Language Instruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Gao P, Han J, Zhang R, et al. Llama-adapter v2: Parameter-efficient visual instruction model[J]. arXiv preprint arXiv:2304.15010, 2023.
limitations: limitations: The authors have discussed the limitations of this paper, and this paper has no direct negative societal impact.
paper_summary: paper_summary: The paper conducts a comprehensive study of multimodal LLMs from a vision-centric perspective. Different from the lines of previous literature which aim to propose new architectures/algorithms for multimodal LLMs, this paper carefully splits the design space of visual parts of multimodal LLMs into several individual parts and diagnoses each with controlled experiments. This leads to several innovative conclusions about the visual aspects of multimodal LLMs, including the validity of standard benchmarks, the choices of visual encoders, etc.
strength_weakness: strength_weakness: - The experiments only consider one particular LLaVa-like formulation of MLLMs built upon a pretrained LLMs, while Sota MLLMs like GPT-4o and Reka are more likely to have completely different training diagrams and architectures, e.g., treating images and texts equally and training a native multimodal LLMs from scratch, or training with interleaved visual and text contents instead of fixed image-first formulations. The value of the paper is thus limited.

-  A minor point: There is no analysis of why the findings hold.
questions: questions: - What's your opinion of "native" multimodal LLMs? Do you think the findings in the paper will in a way transfer to more advanced models?
limitations: limitations: The authors have discussed the limitations quite adequately.
paper_summary: paper_summary: This paper introduces a multimodal large language models (MLLMs), named Cambrian-1, designed with a vision-centric approach. In current MLLM researches, the choices of visual encoder are not sufficiently explored. This study utilizes MLLM performance as a visual representation evaluator, showing different characteristics over differently trained vision encoders and revealing that various widely-used MLLM benchmarks are disconnected from visual understanding capability but connected to language capability. Furthermore, this study proposes spatial vision aggregator (SVA) to effectively connect vision and language models with spatial inductive bias. Additionally, curation of high-quality visual instruction-tuning dataset and its distribution balancing are discussed. As a result, Cambrian-1 achieves state-of-the-art performances and provides an open cookbook for MLLMs.
strength_weakness: strength_weakness: - There exists a previous work about vision-language connectors with spatial inductive bias [1]. The comparison or at least discussion between the proposed SVA and C/D-Abstractor [1] is essential but lacks.
- There are many overlapping findings with existing studies. For example, language-supervised models are effective [2], high-res encoders are beneficial [3], increasing data size and spatial inductive bias are advantageous for connectors [1], and so on. I believe that re-examining these aspects and analyzing them in different settings has its own contribution due to the empirical nature of this field. Nevertheless, it is difficult to attribute high value to the overlapping findings.
- Findings 7 (the second Findings 6 in the paper, seems to be a typo) is not consistent with the results. The finding claims that performance improves with the vision encoder ensemble, but Table 11 does not seem to support this. For example, SigLIP+DINOv2 performs worse than sole SigLIP.

[1] Cha, Junbum, et al. "Honeybee: Locality-enhanced projector for multimodal llm." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.  
[2] Chen, Xi, et al. "Pali-3 vision language models: Smaller, faster, stronger." arXiv preprint arXiv:2310.09199 (2023).  
[3] Liu, Haotian, et al. "Improved baselines with visual instruction tuning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
questions: questions: In Table 11, why are there two entries for "SigLIP+DINOv2+ConvNext" with different numbers?
limitations: limitations: This study construct a new dataset based on web search, but it does not appear to address any privacy issues. It would be better to address this issue.
paper_summary: paper_summary: This paper presents extensive studies on existing MLLM benchmarks addressing the difficulties involved in consolidating and interpreting results from various tasks for MLLM designs. Moreover, the authors also propose Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector to fuse vision features with LLMs while reducing vision tokens. In addition, authors also collect high quality visual instruction-tuning data. The proposed model, Cambrian-1, achieves the state-of-the-art performance on multiple MLLM benchmarks.
strength_weakness: strength_weakness: 1, The technical novelty is limited, combining multiple vision experts into MLLM is not new. Moreover, fusing visual tokens dynamically is also not new in dynamic network design. 

[-] Sphinx: the joint mixing of weights, tasks, and visual embeddings for multi-modal large language models, arixv-2023.

[-] Mova: Adapting mixture of vision experts to multimodal context, arxiv-2024.

I think there are more close related works than [1][2]. The authors should cite all these works as respect.


2, I also have one question, What if you do not use the dynamic token but only using proposed tuning datasets, compared with LLaVA-1.6.

3, Several findings are well known and also verified from previous works. For example, Finding-6 is common knowledge using high resolution visual encoder. 

[-] Vary: scaling up the vision vocabulary for large vision-language models


4, Several figures are useless and common knowledge for MLLM community. For example, Fig.1 and Fig.2 can be merged into one figure. 

5, The ablation studies for SVA are not enough. For example, which tokens are more important in which datasets? This needs further analysis. Moreover, the effect of increasing instruction tuning data size is not well explored.

Given these, I rate this work as weak accept.
questions: questions: See the weakness part.
limitations: limitations: None

Comments: title: Author Response to Ethics Reviewer VuFm
Rebuttal comments
comment: Thanks for your reply! I appreciate your perspective on the native multimodal LLMs.
Thank you for the rebuttal. My concerns are addressed by the authors. Therefore, I will keep my score.
We thank Reviewer K2Un for requesting an ethics review regarding "data privacy, copyright, and consent" for our submission, and further thank Ethics Reviewer VuFm for their valuable perspective and recommendations regarding such ethical considerations. We appreciate the importance of the points you raise and will address them in detail in our revised manuscript.

---

> **Q1: Data Privacy, Security, Copyright, and Consent**

We will expand our section explaining our data collection process, focusing on how we ensure compliance with data protection regulations, maintain privacy, respect copyright, and address consent issues.

Our datasets come from two sources: existing datasets that we modify and/or use, and new data scraped from the web via our data engine.

1. **Existing datasets:** We only use datasets with licenses that explicitly permit usage and repackaging for research purposes. We currently list all datasets used in Appx. E1, but will add information on their licenses, copyright information, and how we maintain compliance with each dataset’s terms.
2. **Data engine:** Our web agent collects data solely from Wikipedia, which is licensed under CC BY-SA (https://en.wikipedia.org/wiki/Wikipedia:Copyrights). We will add details on how we adhere to this license.
3. **Consent considerations:**
   - Existing datasets: We will review and report on any consent procedures used by the datasets we incorporate, acknowledging any limitations or concerns.
   - Wikipedia data: We will mention the implicit consent given by Wikipedia contributors under the site's terms of service.


> **Q2: Bias and Fairness**

We appreciate the importance of addressing potential biases related to demographic variables. While comprehensive mitigation of bias in MLLMs is not the objective of our work—and, in our opinion, is challenging enough to warrant a standalone work in itself—we recognize its significance. We will include a section including:

1. An analysis of demographic variables in our Cambrian-10M dataset, highlighting any observed patterns or imbalances.
2. A discussion of how training a model on data with inherent biases perpetuates them, as well as challenges and potential unintended consequences of artificially modifying data distributions to address said biases.
3. Acknowledgment that the thorough mitigation of bias in large-scale MLLMs is a complex issue worthy of dedicated research, and we hope our analysis will spur future work focused on this critical issue.


> **Q3: Transparency and Interpretability of the Model’s decision-making processes**

We appreciate the reviewer's suggestion around model interpretability. However, it's important to note that **understanding the decision-making processes of LLMs / MLLMs is a challenging research problem in itself, with only very preliminary work in this direction.** Recent work [1][2][3] has made initial progress toward the mechanistic interpretability of LLMs, but such methods are not yet applicable to open-source models or MLLMs.

While full interpretability of a MLLM's decision-making process is beyond the current state-of-the-art, we can provide some insights into the workings of our SVA module. Specifically, as described in our rebuttal to Reviewer ED7N, we will include an analysis of the attention scores across different vision encoders in our SVA module for various types of images and tasks. This analysis will shed light on how different vision models contribute to the overall visual understanding in different scenarios.

> **Q4: Potential Misuse and Mitigation Strategies**

We appreciate the reviewer's concern about potential misuse of our technology.

However, we must clarify a fundamental misunderstanding: **MLLMs like Cambrian-1 are not capable of generating deepfakes.** They take text + images as input and produce text as output. They cannot generate, manipulate, or synthesize images or videos, as deepfakes entail.

That said, we acknowledge that MLLMs can potentially be misused in other ways, and we take these concerns seriously. We will add discussion of potential risks and mitigation strategies, including:

1. **Misinformation:** MLLMs could be used to generate misleading text descriptions of images or to craft false narratives about visual content. To mitigate this, we will provide guidelines for responsible use of our model.

2. **Hallucination:** Like LLMs, MLLMs might generate outputs not grounded in facts or input data. We will clearly communicate this limitation to users.

---

Thank you again for your valuable feedback on the ethical considerations of our work. Please let us know if you have any other questions or suggestions.


### References
- [1] Bricken et al. Towards monosemanticity: Decomposing language models with dictionary learning. 2023.
- [2] Templeton et al. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. 2024.
- [3] Lieberum et al. Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2. 2024.
Thank you for your clear responses. As most my concerns are addressed, I will maintain the original rating of accept.
Thanks for your reply. The rebuttal solves my concern. I keep original rating as weak accept.