CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts
Authors: Jiachen Li
Xinyao Wang
Sijie Zhu
Chia-Wen Kuo
Lu XU
Fan Chen
Jitesh Jain
Humphrey Shi
Longyin Wen
Venue: NeurIPS 2024 poster (2024)
Keywords: Mixture-of-Experts
Multimodal LLM
Primary area: machine_vision

Abstract: Recent advancements in Multimodal Large Language Models (LLMs) have focused primarily on scaling by increasing text-image pair data and enhancing LLMs to improve performance on multimodal tasks. However, these scaling approaches are computationally expensive and overlook the significance of efficiently improving model capabilities from the vision side. 
Inspired by the successful applications of Mixture-of-Experts (MoE) in LLMs, which improves model scalability during training while keeping inference costs similar to those of smaller models, we propose CuMo, which incorporates Co-upcycled Top-K sparsely-gated Mixture-of-experts blocks into both the vision encoder and the MLP connector, thereby enhancing the multimodal LLMs with neglectable additional activated parameters during inference.
CuMo first pre-trains the MLP blocks and then initializes each expert in the MoE block from the pre-trained MLP block during the visual instruction tuning stage, with auxiliary losses to ensure a balanced loading of experts.
CuMo outperforms state-of-the-art multimodal LLMs across various VQA and visual-instruction-following benchmarks within each model size group, all while training exclusively on open-sourced datasets.

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts
Jiachen Li
Xinyao Wang
Sijie Zhu
Chia-Wen Kuo²
Lu Xu²
Fan Chen?
Jitesh Jain'
Humphrey Shi Longyin Wen
SHI Labs @ Georgia Tech & UICC
BanteDacce Inc., San Jose
https://githun Com//SI--abaaCuti
Abstract
Recent advancements in Multimodal Large Language Models (LLMs) have focused primarily on scaling by increasing text-image pair data and enhancing LLMs improve performance on multimodal tasks. However, these scaling approaches are computationally expensive and overlook the significance efficiently improving model capabilities from the vision side. Inspired by the successful applications of Mixture-of- of-exes (MoE) in LLMs, which improves model scalability during training while keeping inference costs similar to those of smaller models, we propose CuMo, which incorporates Co-upcycled Top-K sparsely-gated Mixture- of-experts blocks into both the vision encoder and the MLP connector, thereby enhancing the multimodal LLMs with neglectable additional activated parameters during inference. CuMo first pre-trains the MLP blocks and then initializes each expert in the MoE block from the pre-trained MLP block during the visual instruc- tion tuning stage, with auxiliary losses ensure balanced loading of experts. CuMo outperforms state-of-the-ar multimodal LLMs across various VQA and benchmarks within each model size group, all while training exclusively on open-sourced datasets.
1
Introduction
The advent of GPT-4V [53] has sparked excitement within open-source communities to transform large language models (LLM) into multimodal LLMs. Recent multimodal LLMs [11, 46, 2] typically integrate pre-trained vision encoders with MLP connectors to LLMs, with visual instruction tuning data to fine-tune the pre-trained LLMs, enhancing their visual understanding capabilities. To further scale multimodal LLMs, previous efforts [44, 45, 39, 51, 7, 42] primarily focus on training the model with a more extensive collection of text-image paired data and employing stronger LLMs, significantly increasing training efforts. On the vision side, recent work concentrates on leveraging multiple vision encoders [43, 18] to enrich visual content, employing larger vision encoders [9], and using advanced vision-languaal
connectors [5] to improve performance on multimodal tasks. However, these techniques result in an increased number of additional parameters and generate extra visual tokens for LLMs to process, making it inefficient to scale.
In terms of efficiently scaling up models, Mfxture-of-Expeets (MoE) has become the de-facto framework in modern large-scale neural networks, particularly in natural language processing (NLP). Most large language models (LLM) are built upon the transformer [64] architecture, wherein sparse
Work done during internship at ByteDance San Jose, CA.
Work at ByteDance. Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

SQA-IM6
LLoVA-Wild
Layer
MLP
Layer Norm
Norm
make N capies
100 MME
MLP
MLP
Top Router
Weighted Sum
MMBBecc
SEED-IMS
MLP
MMMU
Co-Upcycled MoE block
Mini-Gemini Vicuno-78
MM1 78 (Privcte)
LLXVA-NeXT Mistral-78
Initidization of Co Upeycced McE blocks in CuMo
CuMo Mistrol-78 (Ours)
Figure I: Left: Each MLP expert within the MoE block during the visual instruction tuning stage is initialized from the corresponding pre-trained MLP in CuMo. Right: CuMo outperforms strong open-sourced models such as Mini-Gemini and LLaVA-NeXT, as well as the private MMI model.
MoE is used to replace the dense MLP block with the Top-K K sparsely-gated MoE block [57]. Recent sate-offtheeart open-sourced [28, 61] and private [55, 51] LLMs have predominantly adopted the sparse MoE architecture. These models are scaled up using the MoE design during training while maintaining relatively lower inference costs as only selected MLP experts are activated during the feed-forward process. Nevertheless, the development and optimization of MoE-based models have been largely tailored to LLMs, and the exploration of scaling multimodal LLMs with MoE, especially on the vision side, remains largely unexplored.
Motivated by these observations, we introduce CuMo, which integrates Top-K sparsely-gated MoE blocks into the vision encoder and the MLP connector of multimodal LLMs. We also explore the associated training recipe and methodology for CuMo. Firstly, we pre-train the MLP connector and perform pre-finetuning warm up the whole model without introducing the MoE architecture, which stabilizes the following visual instruction tuming stage with newly incorporated sparse MoE blocks. Then, replace each MLP block with the sparse MoE block in the MLP connector and the vision encoder through co-upcycling. Each expert within the sparse MoE block is initialized from the corresponding MLP block after the pre-training and the pre-fine pre-finetuning stages, shown in Figure 1. Additionally, each MoE block contains Top-K K router trained from scratch to select experts during the visual instruction tuning stage with auxiliary losses the router maintain balanced loading of experts. We conduct further comparisons between co-upcycled LLMs and pre-trained MoF-based L.MMs. The results show that the pre-trained MoB--based I.LMS significantly outperform the co-upeycled LLMs. As result, the co-upeycling of LLMs is not included in CuMo. Our models are trained fully on open-sourced datasets that converted to visual instruction following formats. Experimental results demonstrate that CuMo outperforms other state-of-the-a multimodal LLMs on various VQA and multimodal instruction-followwe benchmarks within the same model size group, as illustrated in Figure 1.
Our contributions can be summarized as follows:
We introduce CuMo, which integrates co-upeycled pearsely-gateds MoE layers into both the MI. connector and the vision encoder, enhancing the multimodal L.M with slightly additional activated parameters from the vision side.
We outline the training methodology for CuMo, including three-stage training process with auxiliary losses stabilize training and ensure balanced loading of experts.
We train CuMo exclusively open-sourced datasets and pre-trained models. It outperforms state-of-the-a open-sourced and private multimodal L.LMs across multiple competitive benchmarks within each model size group.

---

2 Related Works
2.1 Multimodal LLM
While the ultimate goal for mulitmodal models may be generative across various modalities [66, 3, 60), modern multimodal LLMs primarily focus on integrating additional modalities, such as vision, into LLMs. InstructBLIP [11] adopts Q-Former [37] to sample from visual tokens for LLM to feed- forward and follow the instructions. Flamingo [1] and IDEFICS [23, 32] use shared decoder for visual-language understanding. Qwen-VL [2] uses three-stage training to convert QwenL.M to Qwen- VI. L.L VA series [46, 44, 45] adopt visual instruction tuning that uses instruction-follont
data to convert L.M into multimodal I.I.M. ShareGPT4V [7] collects detailed image caption data from GPT4V to augment the LLaVA models. HoneyBee [5] investigates different designs of the MLP connector for better alignment. VILA [42] unfreezes the LLM during pre-training with interleaved image-text data. MoE-LLaVA [41] adopts the MoE design in small LLMs and reaches comparable performance to LLaVA with large LLMs. VCoder [26] adopts various vision adapters to enhance visual perception abilities. SPHINX [43, 18] adopts multiple visual encoders to enrich the visual features with scaled data and models. InternLM- M-comppose [69, 12] is trained with interleaved text-image composition data and achieves state-of-the-al performance. Intern [9] scales up the vision encoder to a 6B ViT model. MM1 [51] summarizes the essential steps towards building a strong multimodal L.M from pre-trained LLM. Mini-Gemini [39] further collects guided generation into the pipeline.
2.2 Mixture-of-
Mixture-offExperts [24] is proposed to utilize a set of expert networks to address specific tasks by employing gating network determine the selection of these experts. Recently, it has gained popularity in the design of large language models [15]. The mainstream practice [57] is to replace the dense MLP layers with Top-K sparsely-gated mixture-of-experts (MoE) layers in the transformer [64]. MoE in Language Subsequent works [33, 16] have further scaled up MoE-based large language models with improved stability and load balancing of experts. The design of gating networks often involves selecting the top-k experts for each token [57, 33]. Various routing strategies have been explored, such as choosing top-k tokens by experts [71], one-to-one matching between experts and tokens [34]. Besides routing strategies, maintaining the load balance of experts is crucial for training MoE models. ST-MoE [73] adopts loading balancing loss and router-z loss to ensure a balanced distribution of the experts. Upeycling [31] proposes training sparse experts from dense checkpoints to stabilize training and lower the cost. Recent large language models like Gemini-Pre [55] and DBRX [61] are also based on the MoE design.
MoE in Vision The success of MoE extends to the vision community, particularly following the popularity of vision transformers [13, 4, 72, 21, 20, 25, 36]. V-MoE [56] reaches comparable performance to dense ViT while only requiring half of the compute. LIMoE [52] replaces dense MI.P layers with MoE layers in CL.IP and observes improvements in zero-shot image classification. Residual MoE [65] corporates residual design into MoE transformer and saves over 30% training cost. A-MEE [8] proposes adaptive MoE framework for multi-task learning.
3 Method
In this section, we first review the sparse MoE block structure and the upcycling strategy utilized in previous studies. Subsequently, we describe how these sparsely-gated MoE blocks are integrated into each module of multimodal LLMs using co-upcycling strategies. Then, we introduce the three-stage training process and auxiliary loss functions employed to stabilize training and balance the loads of experts.
3.1 Revisit Sparse MoE
Sparse MoE Structure Previous mainstream practice [57] is to replace the dense MLP blocks with sparsely- mixture-of-experts blocks. Given input X E NNC. and MI.P block,
Xout ML)(X) E IR NCC.
(I)
3

---

The dog is engoging activity of surfing.
Weighted-Soe Sum
(dense/MoE)
MLP 2
MLP4
MLP-MoE
Word Embedding
CLIP-MoE
What is the dog doing
Top-K MoE
Figure 2: Architecture of CuMo. CuMo incorporates sparse Top-K MoE blocks into the CLIP vision encoder and vision-language MLP connector, thereby improving the multimodal LLM capabilities from the vision side. Skip connections are omitted for simplicity. Further implementation details are provided in Section 3.2.
To scale up the model with multiple MLP blocks in parallel, a sparse MoE block includes a router network to select Top-K experts out of S total experts. This router network has a linear layer to compute the normalized weight matrix based on the inputs X for voting, resulting in
W = Softmax(Lincar(X) E NxS
The Top-K experts selected for each token based on W, and the re-normalized weights WK R x are computed using
Wx K (oftmax(TopK(WW) NxK
(3)
Each selected expert is represented by an MLP block, and the final output is obtained through re-weighted sum
X > W MLP,(X) NNC.s.
the output Xout maintains the same dimension as the output of a single dense MLP block.
Sparse Upeycling Training MoE-based designs from scratch can be unstable and costly. Sparse Upcycling [31] addresses this challenge by initializing the experts in each MoE block from the corresponding MLP block in pre-trained dense checkpoints. This initialization approach provides better starting point for training MoE-basee models and reduces training costs compared to training from scratch.
3.2 CuMo Architecture
Sparse MoE in MLP Connector The MLP connector converts visual tokens into word embedding space, aligning dimensions between visual and text tokens. An effective architecture for the vision- language connector is an MLP block [44] that contains two linear layers. We start from single MLP block and replace it with a Top-K sparse MoE block, incorporating a Top-K router and a set of experts for projecting visual tokens into word embedding space.
Sparse MoE in Vision Encoder Vision encoders extract image features as sequences of visual tokens for reasoning in LLMs. CLIP [54] is one the most popular pre- treitrained vision encoders for multimodal LLM since pre-trained on large-scale image-text pairs, which makes it suitable for processing images for multimodal usage. The visual encoding part of CLIP is a ViT [13] model, which has consecutive MI. blocks in the transformer encoder. We substitute each MI. block with Top-K sparse MoE block, retaining skip connections alongside MoE block outputs.

---

LLM
LLM
LLM
MLP
MLP
MLP-MoE
Co-Upcycle
CLIP
CLIP
CLIP-MoE
Pre- Training
See-FineTTunngg
Visual Instruction Tuning
Figure 3: Training Stages of CuMo. The first stage involves pre-training the MLP for better alignment. Subsequently, the pre-finetuning stage trains all parameters as warm-up before the next stage. Finally, the MLP experts within each MoE block are initialized from the weights of the corresponding MLP block, followed by training all parameters in the visual instruction tuning stage.
Sparse MoE in LLM In terms of using MoE in LLM, we compare the co-upcycled LLM with pre-trained MoE-based LLM. We start from Mistral-7B and the upcycled Mistral-7B-MoE slightly outperforms Mistral-7B on certain benchmarks. However, considering the constrained knowledge base of upcycled experts from Mistral-7B, we compare it with the pre-trained Mixtral 8x7B with pre-trained experts of diverse knowledge base. Experimental results reveal that pre-traie pre-trained Mixtral 8x7B significantly outperforms -B-Mral-7B-MoE. As result, LLM is not co-upeyeled with CLIP and MLP connectors since it brings marginal improvements with great additional parameters.
3.3 Training Recipe
Co-Upeycling MoE blocks We start with training the added MoE blocks from scratch while the model is struggling to converge. Attempts to address this issue with lower learning rates perform worse compared to the baseline. As result, we adopt a co-upcycling approach, initializing each module that integrates sparsely-gated MoE blocks with pre-trained MLPs to replace corresponding MLP blocks, as shown in Figure This strategy consistently improves training stability and model performance.
Three-Stage Training To further enhance training stability, we adopt three-stage training strategy for CuMo models, as illustrated in Figure 3. In the first stage, we only pre-train the ML. connector, given that the vision encoder and LLM have already undergone pre-training on large-scale data. During the second pre-finetuning stage, we train all parameters using high-quality caption data to warm up the entire model before introducing MoE blocks in the subsequent stage. The third stage involves visual instruction finetuning, where the multimodal LLM is scaled up with upcycled MoE blocks and trained visual instruction tuning data.
Loss Function To maintain a load balance between experts in each MoE block, we adopt auxiliary losses based on the language modeling cross-entropy loss. The auxiliary losses comprise loading balance loss and router z-loss [73]. Hence, the total loss is
L Lee + LgLs + L.L.
(5)
Here, Loe represents the language modeling loss, which computes the cross-entropy of next-token predictions. ag and ax denote coefficients for loading balance loss Ls and router z-loss Ly, set to 0.1 and 0.01, respectively, across all experiments. These auxiliary losses, abbreviated as bzloss in Section 4, are individually applied to the MLP connector, vision encoder, and LLM for simplicity.
4 Experiments
We train the CuMo models on mixture of open- sourced datasets, which are converted into the visual instruction tuning format. Then, we conduct comprehensive evaluations of the performance of CuMo models across various competitive VQA-based and dedruucion-following--oo Additionally, we perform ablation studies on each module with upcycled MoE. blocks with qualitative analysis of the results.
benchmarks.
5

---

Act.
SQA
Text VQA
MMB
MM
VQA
LLaVA WiM
SEED IMG
MMMU
Method
LLM
(R)
IMG
GQA
POPE
MME
EN
CN
Ver
V2
val
78 to 13B Models
InstrcsRI [1]]
Vicuna-7R
7.9
60.55
49.2
36.0
23.7
26.2
60.5
Qwes-VI. VL.-Chot
Qwen-7B
61.5
57.5
1487.5
LLaVA.vl.5 [46]
Vicuna-7B
7.1
66.8
58.2
62.0
85,9
1510.7
64.3
58.3
30.5
78.5
63.4
66.1
VILA [42]
Vicuna-7B
7.1
68.2
64.4
62.3
85.5
1533.0
68.9
61.7
34.9
79.9
69..7
61.1
StareGPPTV [7]
Vicuna-BB
7.1
68.4
1567.4
68.8
62.2
37.6
80.6
72.6
69.7
LLD [3S]
Vicuna-7B
86.0
1521.4
79.3
SPHIINX-Iniem2 [18]
Mn-emLLM2-77
70.4
58.1
56.2
86.9
1260,4
57.9
36.5
75.5
57.6
68.8
LLaVA-NeXT [+15]
Mistral-70 7B
7.6
72.8
65.7
61.8
86.7
1/98
68.7
61.2
17.3
82.2
K3.2
72.2
35.5
LLaVA-NeXT[S [-5]
Vicuma-BB
7.1
70.1
64.9
64.2
86.5
1519
67.4
60.6
43.9
81.8
81.6
70.2
35.8
Mini-Cimmini [59]
Vicuna-7R
7.3
1929
40.8
[51]
MM1-7B
72.8
86.6
1529.3
79.0
69.9
37.0
Instm:tBL1PP [1]]
Vicuna-138
14.2
63.1
50.7
49.5
78.9
1212.8
25.6
58.2
63.1
LL.VA,-1..5 (44)
Vicuna-13B
13.4
71.6
61.3
63.3
85.9
1531.3
67.7
63.6
35.4
80.0
70.7
68.2
36.4
VILA [42]
Vicuna-138
13.4
73.7
66.6
63.3
84.2
1570.1
70.3
64.3
38.8
80.8
73.0
62.8
Intern VL.Caa
Vicuma1138
61.5
66.6
87.6
1586.4
LLAMA-VID [88]
Vicuma-1388
70.0
65.0
86.0
1542.3
66.6
80.0
62.3
SPHINX--Plus [18]
LLMMA2113
74.2
65.7
89.1
1457.7
71.0
47.9
71.7
74.8
Mini-Gieminif 5o]
Vicuna-138 13B
13.6
65.9
1565
68.5
46.0
38.
LLaVA-NeXT (45]
Vicuna-133 13B
13.4
73.6
67.1
65.4
86.2
1575
64.4
48.4
82.8
87.3
71.9
36.2
CaMo
Mistral-10 7B
7.8
73.9
67.0
64.9
86.7
1548.6
73.0
66.5
51.0"
82.2
85.71
72.1
39.1
7B MoE Models
OPHINX.MoB
Mixtral-8x TB
74.5
68.0
63.8
89.6
1485.3
71.3
40.9
81.1
70.2
73.0
31.1
[5
MBM
72.8
87.6
1629.0
47.0
83.4
40.9
Mini Gemiii [39]
Mixtral-8
13.5
69.2
1639
75.6
45.8
41.8
CaMo
Mixtral-8x7B
13.5
77.9
66.0
63.8
85.7
1639.5
75.3
68.0
48.7'
81.8
84.71
73.2
45.0
Table 1: Comparisons between CuMo and other offte-of-the-arr art multimodal LLMs competitive benchmarks. These models are grouped by the size of the base LLM and bold indicates the best performance on certain benchmark. Act.: activated parameters during inference. Numbers with are averaged by three inference runs of querying GPT API.
4.1 Implementation Details
Training Datasets During pre-training, we only utilize LLaVA-558K [46] to train the MLP con- nector for better alignment. In the subsequent pre-finetuning stage, detailed image caption data from ALLaVA [6] is employed to warm up all parameters of the multimodal LLM. For the final
visual instruction tuning stage, mixture of datasets including LLaVA-665K [44], ShareGPT4V [7],
LAION-GPT [14], DOc [62], ChartQA [49], A12D [29], Info [50], SynDog-EN [30], VLLaVA [6], and LIMA [70] is utilized to train the CuMo models with upeycled MoE blocks. The total data size for visual instruction tuning is approximately 1.65 million, and all training data are publicly accessible. The detailed breakdown of the training dataset is listed in Appendix A.
Evaluation Benchmarks Evaluation of CuMo models primarily focuses on academic VQA-based datasets such as VQAv2 [19], GQA [22]. Science-QA [48], and TextVQA [59]. well as instruction- following- LMM benchmarks including POPE [40], MME [17], MMBench [47], SEED- Bench [35], LLaVA-Wiin [46], and MM- Vet [67]. Additionally, the challenging MMMU [68] is evaluated to assess the visual reasoning abilities of the multimodal LLMs.
Training Settings We employ the pre-trained CLIP ViT-L [54] as the vision encoder, two-layer MLP as the vision-language connector, and Mistral-7B [27] as the LLM to establish the baseline model following V. v1.5 [44]. We only use LLaVA-5588 [44] as pre-training data and LLaVA- 665K [44] as visual instruction tuning data to train the bascline model and make ablation studies for comparisons. The learning rate is set to le-3 for pre- pre-training the MLP connector and reduced to 2c-5 for visual instruction tuning of both the MLP connector and CLIP. To further stabilize the visual instruction tuning process after scaling up with additional data, the learning rate is lowered to 2e-6 for all parameters of the CuMo models in the final results. More hyperparameters of the training process is listed in Appendix B.
Evaluation Settings During evaluation, we adhere to the settings outlined the LLaVA series [44], employing a greedy decoding strategy for all benchmarks. The data and questions are converted into visual instructions to prompt the multimodal LLMss. For benchmarks that utilize GPT API for evaluation, we adopt gpt-4-0613 for LLa [46].

---

SQQA
Text VQA
MMBeacc
MM
VQA v2
LLAVA Wild
SEED
Metbod
LLM
PT
IT
IMG
GQA
POPE
MME
EN
CN
Vet
IMG
InstructBLPP
Vicuna-7B
1.2M
60.5
49.2
36.0
23.7
60.9
InstructBLPP [11]
Vicuna-1.3 13B
129M
1.2M
63.1
50.7
49.5
78.9
1212.8
25.6
58.2
63.1
IDEFICS-9B [23]
LJaMA-78
353M
1M
25.9
38.4
48.2
25.2
50.9
IDEFICS-S08 [25]
LLaMA-658
45.2
38.1
60.0
Qwen-VL. [2]
Qwen-7B
1.4
50M
67.1
63.8
59.3
38.2
7.4
78.8
56.3
Qwen-VI.-Che [3]
Qwen-7B
148
50M
68.2
61.5
57.5
1487.5
60.6
56.7
78.2
58.2
LLaVA-v1.5
Vicuna-78
665K
66.8
58.2
62.0
85.9
78.5
63.4
I.aVA-vi..'
Mistral-7B
558K
663K
72.8
57.6
60.0
86.1
1414,9
66.5
60.1
32.1
78.2
69.4
56.4
CuMo
Mistral-BB B
558K
663K
71.7
59.3
63.2
87.1
1428.6
69.6
62.6
34.3
80.6
68.8
69.6
Table Comparisons between CuMo Mistral-7B and other multimodal LMM models with limited training data. The best performance are highlighted in bold. LLaVA-v1.5' with Mistral-7B is reproduced by us as baseline model.
4.2 Main Results
Comparison with SO Multimodal LLMss In Table 1, we present comparison of CuMo models with other state-of-the-art instruction-follovinng-- based on the size of the base LLMs. including 7B models, 13B models, and 7B MoE models. CuMo Mistral-7B outperforms other 7B-based state-of-the-at multimodal LLMs across multiple benchmarks. Moreover, the performance of the CuMo Mistral-7B model is comparable to many 13B-based multimodal LLMs. In the case of Mixtral-8x 7B models, CuMo achieves results on par with SPHINX-MOE, MMI, and Mini-Gemini. LLaMA-based LLMs [10, 63] are not utilized in our experiments due to license constraints.
multimodal L.LMs. We categorize the models
Comparison under limited training data To further evaluate the effectiveness of the co- co-upcycled MoE. blocks, we train the vanilla CuMo mistral-7B under limited training data in Table 2. shows that CuMo outperforms other 7B models and reaches comparable performance to LLaVA-v1.5 Vicuna-13B under the same training data.
4.3 Ablation Study
Upeycle MLP connector to MLP-MoE We initiate the ablation study by replacing the MLP connector with upcycled MLP-MoF, as depicted in Table 3(a). We start with Top 2-in-4 router and train the MoE blocks from scratch, which leads to a clear performance drop on all benchmarks. Then, we adopt the upcycling strategy to initialize the MLP experts. We observe marginal improvements over the baseline, considering each expert comprises only two linear layers. Subsequently, the incorporation of bzloss to ensure a balanced loading of experts in the MLP-MoE, yields noticeable enhancements on MMVet. However, employing Top 2-in-8 router with upcycling and bzloss results in slight performance decline, possibly due to the limited visual instruction tuning data to train robust and well-balanced eight experts.
Empower CLIP with CLIP-MoE In Table 3(b), initially unfreezing CLIP based on MLP-MoE leads to noticeable improvements on TextVQA and MMVet benchmarks. However, training with the added Top2-in- -n- MoE blocks in CLIP from scratch proves unsuccessful, as the model fails to converge even with largely reduced learning rates. Consequently, adopting upcycled MoE blocks during the visual instruction tuning stage yields further enhancements on the TextVQA, MMVet, and SEED benchmarks, as well as more stable training process.
Upcycle LLM vs Pre-trained LLM-MoE Upon replacing all MI.P blocks with sparsely-gated MoE blocks in the visual part, we further investigate the utilization of the MoE architecture in the LLM. Starting from the Mistral-7B model, we first lower the learning rate to 2c-6 to set the baseline and the following experiments since learning rate of 2c-5 induces training instabilities. Then, we upcycle each MLP block with sparsely-gated MoE block, initializing the weight of each expert from the pre-trained MLP block. As demonstrated in Table 3(c), the upcycled Mistral-4xxl 7B and 8 8x77 7B outperform the Mistral-7B model slightly except for TextVQA. However, considering that the upcycled experts significantly increase parameters without introducing new knowledge, we replace the upcycled Mistral Sx7B with Mixtral 8x [28]. In Mixtral 8x7B, all expert layers are pre-trained on large-scale language data, providing superior initialization and similar training stability compared to upeycling. The results indicate that CuMo Mixtral-8xx 8xBB outperforms its upcycled
7

---

Method
SQA
VQA"
Me
SEED
Method
SQA
VQA"
MMVer
SEED
Baseline Mistral-TB
72.8
57.6
32.1
66.4
MLP-MoE
73.3
57.4
33.1
67.4
TTp 2-in-4 & Scratch
68.1
65.1
+ Unffrezze CLIP
72.0
58.9
34.7
69.0
Top 2-in-4 4 & Upcycle
73.7
57.2
32.3
67.1
"Top 2-tn-4 & beloss
72.8
59.7
35.4
69.8
+ beloss
73.5
57.4
33.1
67.4
Top 2-in-8 & Upcycle
73.4
57.6
32.4
67.2
Top 2-in-8 & beloss
71.0
59.0
33.6
69.2
(a) MLP-MoE
(b) CLIP-MoE
Method
SQA
VQAT
MMVer
SEED
1x
2x
3x
SQA
VQA
MMVet
SEED
MLP-MME & CLI
72.8
59.7
35.4
69.8
71.7
59.3
34.3
69.6
lower Ir to 2e-6
Mistral 7x78 Upcycle
71.7
72.8
57.0
59.3
34.3
35.2
69.9
69.6
4
71.7
60.6
35.0
69.7
Mistral 8x7B & Upcycle
73.2
56.4
35.7
70.5
72.9
61.0
37.0
69.7
Mistral 8x78
74.2
60.6
40.0
72.6
V
72.2
60.5
36.9
70.1
(c) LLM-MoE
(d) Multi-resolution Feature
Method
CL.IP
ML.
L.M
Total
Method
SQA
VQA
MM
SEED
Mistral-7B
0.30
0.025
7.25
7.58
No PFT
71.7
59.3
34.3
69.6
MLP-MMoF
0.30
0.05
7.25
7.60
+ShareGPT4V
72.4
61.7
36.5
70.0
CLIP-MoE
0.50
0.05
7.25
7.80
ALLaVA
73.0
62.8
37.2
70.9
= Mixtral-x77B
0.50
0.05
12,90
13.45
(e) Tuniing Stage
(f) Activated billions parameters
Table 3: Ablation Studies during building CuMo. Each row represents different configuration, with changes or additions marked using = and + symbols, respectively. Settings highlighted with a light blue background are those adapted for final model in Table 1. For (b): all MoE blocks in CLIP are initialized with upcycling.
counterparts significantly and is employed in the final models with bzloss to maintain a balanced loading of experts.
Multi-Resolution Visual Features Incorporating multi-resolution imputs is crucial for enhancing the understanding of image content in multimodal L.MMs. Following the approach outlined in S2[58), introduce multi-resolution inputs to CLIP and concatenate the feature maps channel-wise to maintain the total number of visual tokens consistent with low-resolution inputs. As illustrated in Table 3()), an empirical combination of 3 and x reaches the best performance and we adopt this configuration for the final CuMo models.
Pre-Fine Pre-FineTuning Stage Previous ablation studies were conducted directly after the pre-training of the MLP connector, leading observed training instabilities during visual instruction tuning. To address this, we introduce a pre- feeetinetuinng stage using high-quality image caption data, wherein all parameters unfrozen. In Table 3(e), we leverage caption data from ALLaVA for this stage. Results indicate that AIL data proves to be superior option, providing fewer but higher-quality captions for training, ultimately leading improved performance.
Added Parameters In Table 3(f), we keep track of the added activated parameters during infer- ence of CuMo. It shows that adding MoE blocks in the vision side upon MLP connector and CLIP only brings 0.22B extra parameters compared to the baseline model on Mistral-7BB More details can be found in Appendix C.
Uopan
Loyer
Laye-10 10
Leye11 16
4.4 More Analysis
Laye
Expert Distribution As shown in Figure we visualize the expert distributions in the MoE block from selected layers in CLIP during infer- ence. The dataset used for analyzation is the test set of the MME benchmark. The distribution indicates that the selected experts during infer-
0%
85%
90%
75%
100%
Figure 4: Expert distributions of MoE blocks in CLIP. We select layers from CLIP and summarize the activated experts during inference.
8

---

LLaVA-Yi-34B
the is well stroked of bunch
of greoonss Hone's what see
contoner stronberro, and for blue platts cen contorery that caue used foud shonge randig the radiy sat fow buttee eight cortuby wil or pict, and cormo egits precison There
mies inclding what spperss
shalf
oplae (rud,
heod buttce the drsver ot the bettee camaing mene fros predece rekidnng shet lest like bunch of grenn avene and peartly zone healtly
Ioety greess enriggnone oegasped appeand
with
MiniGemini-Vi-g4g
The refeigendor filed with variety of procenes inolding fruits, noptiddes danry products and benerags, There are soand batriks, Iidy cenhoring Addiiinally there
darry products, and corten cuas. The Iridge alm canta adicsion fresh produce, is brocest presence these aightt that
lncladig whe was the Fridee prrfors bolozoed der with wis heatily badde heveniges
CuMto-Matral-7
The grzceria pr preserted the fridge include von cnty frosh produce such be peppers, lattooe, brecosk, and cacurberss Thene are des cooking os cor disent The presince these neee sagestt thun the fridge is wel shockd who bobonc of Frest vegetclles dery, and condinionts indeating a haaehal that valus healthy and varied diet.
cor you describe what greceries are presented in this fridge
LL-VA-VV-34B
The imgge preaeee it from the unimatted movi "Gees"" In festures voncty of chemectens, bonk suror and farmestical wlick one net fagire is young buy Wayel chancctery, bor mateed chere members and other characters fie. The charsctene depicted stylladd cximched and
e gerter Surmecd
the raage i snd from the vree.
MiriGemini-yye -Yi-48
the Irage, Therr one totel people lests dus bey the foreg roud the two costuttee and riis choractane eacted oround chaletene
people included the
CuMo- Mistrol-78
how many people one in this image
hotdl is The includa chinas chrrachers seoted cround the hible and 2 chanc stording the tible. The choma ters are homee and suoett Fig ane with the deeletss ganes being port the boy of the beer celetiictios
the
LLaVA- 34B
Age litrorr withh is 2015 Aepnion aqareers the basdd the Menal Creice apperkeno fwon the Asergery The ii i seaul Thee Anereer (2002) and the 2ith fil in the Marid Chamahic Univers (MOU) 21 vea di dinched by The potter from ny crver the maln chonctwes from the file ieckriin Tran Men Thas conse nession the Hill Black Welkow and The tits Avangers agperere catures actting appear be cttyaape with seus dustruction mappertng nq
This
corflict
Ulrrat leaetly depleyed
cester, and Mervel vita
bothon.
The file' plot realeed aund the Auagert sthopping og to stop viloon Uffec from enccting his plass for kuman ear actionn The movie is ixperteres
contniuting to the onga papdarity of the Mariel Chs notic universe
MiniGemini 31.348
The mene paster for "iee Averpera" 2002 Amanion repeherr th produced by Mlared brukkss and detmibtted by Wist braney Pottess Morr buttils chile semeovmm Coorent ort Zchanson Jermny Rennar Ten Hiltleeter end Senusl coction The pist rewive aword Nick
AVENGERS
dimsttr the pepeshees organsaioon in snottgrna ther
desigeed to icent coctueas the chytoop the todkgw el eageeting the deae of the fet The nenase dots in auilikitty
ohe
The
recra ting heam sqpererne ner
chanacles
ane the we
Las,
nuture of
filn, with the monoche intioting
eydinil tnclccigg 1b. which was a sigiffiontt acling partt the tine of its ndeau.
CyMe- Mistrol---
The paster is for the eane "hhe An Ausaeerc of whice is part of the Morvll Geonoiic Us weree It features teoe of oaprr mene, each with that the fin laader the costumess my tigether Ther movic's pet incker
cestrol fyyre, codle
fictie fastapy skmerrz and cinurn gratera noat The paman cadipad to camvey the apl: sale and the righ makes at the file de woll or the divrrs hoon of heroes that
preairex citystope and explution. bocky aand thet
con you introduce movie based poster
inpects
velsss
Figure 5: Dialogues between the user and multimodal LLMs on challenging images. We highlight the correct answers and hallucinations from the responses of the multimodal LLMs.
ence are evenly spread across layers, providing further evidence of the effectiveness of the auxiliary losses in maintaining load balance.
Dialogue Comparisons Presented in Figure 5, we contrast the responses from CuMo-Mistral- 7B, LLaVA-Yi-34B. and MiniGemini- Yi-34B under challenging content understanding cases. It demonstrates that CuMo-Mistral--B can effectively follow instructions and provide mostly correct answers to challenging questions derived from complex scenes. However, CuMo also exhibits instances of hallucinations, such as responding with "2 characters standing on the table", highlighting the need for further investigation to mitigate hallucinations and improve reliability of CuMo.
The main limitation of CuMo is that, similarly to other large language models, it can generate hallucinated responses. This may constrain its potentials in real-world multimodal applications like used as a chatbot. Future works, such as Reinforcement Learning with Human Feedback (RLHF) and Retrieval Augmented Generation (RAG), can be undertaken to mitigate these hallucinations and improve the model's ' reliability.
Limitations
9

---

5 Conclusion
In this study, we introduce the sparse mixture-of-exper design into multimodal LLMs from the vision side. Specifically, we replace each MLP block with Top-K sparse MoE block in the MLP connector and the vision encoder. To enhance training stability, we employ a three-stage training approach, incorporating upcycled MoE blocks during the visual instruction tuning stage, along with auxiliary bzloss to maintain balanced loading of experts. All CuMo models are trained and evaluated on fully open-sourced datasets and benchmarks. Through extensive experiments and ablation studies, we validate the effectiveness of the upcycled MoE blocks each module. CuMo outperforms ofte-of-theeart models across multiple competitive benchmarks within the same group of model sizes.
Acknowledgments We extend our gratitude to Chunyuan Li, Lei Chen, and Haibin Lin for their insightful and valuable discussions throughout this project. Li, Jain, Shi are in part supported by National Science Foundation CAREER Award #2427478, and by National Science Foundation and the Institute of Education Sciences, U.S. Department Education under Award #2229873- National AI Institute for Exceptional Education, Beckman Institute and ECE Department at UIUC, and Georgia Institute of Technology.
References
[1] Jean-Bape Sann-Bapistte Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-237366, 2022. 3
[2] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023. 1, 3, 6,7
[3] Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaolc Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion scale. In International Conference on Machine Learning (ICML), pages 1692-1717. PMLR, 2023. 3
[4] Nicolas Carion, Francisco Massa, Gabricl Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. European conference on computer vision. Springer, 2020.
[5] Junbum Cha. Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal Ilm. arXiv preprint arXiv:2312.06772,, 2023. 1,3
[6] Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for lite vision-language model, arXiv:2402. 1684, 2024. 6
[7] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4y: Improving large multi-modal models with better captions, 2023. 3,
[8] Tianlong Chen, Xuxi Chen, Xianzhi Du Abdullah Rashwan, Fan Yang, Huizhong Chen, Zhangyang Wang, and Yeqing Li. Adamy-moc: Adaptive multi-task vision mixture-of-expert In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 17357, October 2023, 3
[9] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Intervvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint ar 14224 2023. 1,
[10] WL Chiang, Z Li, Z Lin, Y Sheng, Z Wu, H Zhang, L Zheng, S Zhuang. Y Zhuang, JE Gonzalez, et al. Vieuna: An spen-source chatbot impressing 4 with 90%* chatgpt quality, mar. 2023.
[11] Wenliang Dai, Junnan Li, Dongxu Li, Authony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li. Pascale Fung, and Steven Hoi. Instructblip: Towards seneral-purposs. vision- language models with instruction tuning. arXiv preprint arXiv:2305.066 066000 2023. 1, 6.
10

---

[12] Xiaoyi Dong. Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan. Wenwei Zhang, Yining Li, et al. Interlnnxccompoorr24hhh A pioneering large vision- language model handling resolutions from 336 pixels 4k hd. arXiv preprint urXiv:2404.06333 06512, 2024.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai. Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXiv:2010. /9929, 2020. 3,
[14] LAION eV. Laion/gpt4v-dattee -attssttt datasets at hugging face. 6
[15] William Fedus, Jeff Dean, and Barret Zoph. review of sparse expert models in deep learning. arXiv preprint or X1:::220900006 2022. 3
[16] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022. 3
[17] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang. Xiawu Zheng. et al. Mme: comprehensive evaluation benchmark for multimodal large language models. 2306. 13394, 2023. 6
[18] Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijic Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang. Wenqi Shao, Chao Xu, Conghui He, Junjun He, Hao Shao, Pan Lu, Hongsheng Li, and Yu Qiao. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. ArXiv, abs/2402.05935, 2024. 1,3,6 6
[19] Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017. 6
[20] Ali Hassani and Humphrey Shi. Dilated neighborhood attention transformer. arXiv preprint
vrXiv:2209.15001, 2022. 3
[21] Ali Hassani, Steven Walton. Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2023. 3
[22] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 6
[23] IDEFICS. Introducing idefies: An open reproduction of State-of-the-eee visual language model. https:
//hugiigfface wo//blog/ddfics,..//o/ooooo.../e 2023. 7,7
[24] Robert A Jacobs, Michael I Jordan. Steven Nowlan, and Geoffrre E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-8) 87, 1991. 3
[25] Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer to rule universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2989-2998, 2023.
[26] Jitesh Jain, Jianwei Yang, and Humphrey Shi. Veoder: Versatile vision encoders for multimodal large language models. arXiv preprint arXiv:2312.1422 2023.
[27] Alhert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint XXXi:3331000625, 2023.
[28] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv:2401.04088. 2024. 2.7
[29] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. diagram is worth a dozen images. In ECCV, 2016. 6
[30] Geewook Kim. Teakgyu Hong. Moonbin Yim, Jeong Yeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang. Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In European Conference Computer Vision, pages 517. Springer, 2022. 6
[31] Aran Komatsuzaki, Joan Puigcerver, James Lee- Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay. Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture- of experss from dense checkpoints, 2023, 3,

---

[32] Hugo Laurengon. Léo Tronchon. Matthieu Cord, and Victor Sanh. What matters when building vision- language models? arXiv preprint 0tt::24005002246. 2024. 3
[33] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam M. Shazeer, and Z. Chen. Gshard: Sealing giant models with conditional computation and automatic sharding. ArXiv, abs/2006. 2020. 3
[34] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models, 2021. 3
[35] Bohao Li, Rui Wang. Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal Ilms with generative comprchension. 2.X.::2307.161255 2023. 6
[36] Jiachen Li. Vidit Goel, Marianna Ohanyan, Shant Navasardyan, Yunchao Wei, and Humphrey Shi. Vm- former: End-to-end video matting with transformer. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages
2024. 3
[37] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image -mage pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 2023.
[38] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image worth 2 tokens in large language models, 1rX1v:2311.17043, 2023. 6
[39] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modalln vision language models, 2024. 1, 3,
[40] Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv:2305. 10355, 2023.
[41] Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-Ilava: Mixture of experts for large vision- models. arXiv preprint arXiv:2401. 15947, 2024. 3
[42] Ji Lin, Hongxu Yin, Wei Ping. Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. arXiv preprint 2r 2023. 1, 6
[43] Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming Hc, Hongsheng Li. and Yu Qiao. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models, 2023. 1,
[44] Haotian Liu, Chumyuan Li, Yuheng Li, and Yong Jac Lec, Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 2023. 1, 3, 4, 6,7
[45] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. 1,3,6 6
[46] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023. 1, 1,3,6 6
[47] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv:2307.06258 2023. 6
[48] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal rcasoning via thought chains for science question answering. In NewrlPS, 2022.
[49] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv:2203. 102444 2022. 6
[50] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographievga. In Proceedings the IEEE/CVF Winter Conference Applications of Computer Vision, pages 1697-1706, 2022. 6
[51] Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang. Ruoming Pang. Peter Grasch, Alexander Toshev, and Yinfei Yang. Mml: Methods, analysis, insights from multimodal Ilm pre-training, 2024. 2,
12

---

[52]
Basil Mustafa, Carlos Riquelme, Joan Puigcerver. Rodolphe Jenatton, and Neil Houlsby. Multimodal contrastive learning with limoe: the language-image mixture of experts, 2022. 3
[53] OpenAI. Gpt-ov(ision) system card. 2023.
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamola Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In ICML, 2021. 4,
[55] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jeffrey Dean, and et al. Oriol Vinyals. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,
2024. 2,
[56] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton. André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts, 2021.
[57] Noam Shazeer, Azalia Mirhoseini. Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017, 2,3
[58] Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do we not need larger vision models? arXiv preprint arX3x:2433. 13043, 2024. 8
[59] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. CVPR, 2019.
[60] Zineng Tang, Ziyi Yang. Chenguang Zhu, Michael Zeng, and Mohit Bansal, Any-to-any generation via composable diffusion. Advances in Neural Information Processing Systems, 36, 2024. 3
[61] The Mosaic Research Team. Introducing dbrx: A new state-of-the open Ilm, March 2024. 2,
[62] Ruben Tito, Dimosthenis Karatzas, and Emnsst Valveny. Document collection visual question answering. In ICDAR 2021, 2021. 6
[63] Hugo Touvron and et al. Louis Martin, Llama 2: Open foundation and fine-tuned chat models, 2023. 7
[64] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 2017. 1,3
[65] Lemeng Wu, Mengchen Liu, Yinpeng Chen, Dongdong Chen, Xiyang Dai, and Lu Yuan. Residual mixture of experts, 2022. 3
[66] Xingqian Xu, Zhangyang Wang. Gong Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all one diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7754-7765, 2023.
[67] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv:2308.022 024999 2023. 6
[68] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang. Weiming Ren, Yuxuan Sun. Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin. Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang. Huan Sun, Yu Su, and Wenhu Chen, Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024. 6
[69] Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. Internim- xocomi--.coooorr. vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15:: 2023.
3
[70] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024. 6
[71] Yanqi Zhou, Tao Lei,, Hanxiao Liu, Nan Du Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoe Lee, and James Laudon. Mixture-of-experts with expert choice routing, 2022. 3
[72] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations. 2020. 3
[73] Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, 2022. 3.5
13

---

Appendix
The supplementary material elaborates on further aspects of our work concerning the experimental setups and dataset usage. In Appendix A, provide details the datasets used for the visual instruction tuning stage and how we converted the mixture of datasets into the visual instruction following formats. In Appendix B, we present the hyperparameters used for the three-stage trainings. In Appendix E, we include additional examples of dialogues between the user and our CuMo models.
Dataset Details
As outlined Table 4, we provide detailed information on the datasets utilized for the three-stage training process mentioned in Section 3.3. All data converted into the Astruction-followiiie format for training. For the Syndog-EN and DVQA datasets, we didn't 't use the entire training set as observed that large portion of synthetic data negatively impacts the zero-shot performance of the multimodal LLMs.
Dataset
Size
Pre-Training
LCS-5588
558K
Pre-Finetuning
ALLaVA-Caption
708K
Visual Instruction Tuning
LLaVA-665K
665K
ShareGPT4V
102K
LAION-GPT-V
11K
Doc
10K
SynDog-EN EN
50K
ChartQA
4K
DVQA
50K
AI2D
2K
InfoVQA
4K
ALLaVA
708K
LIMA
1K
ALLaVA-Text
143K
Table 4: List of datasets used for three training stages.
Model
Vision Encoder
ImageNet Acc.
Res.
Params.
TexiVQA
MM
SEED
LLaVA-v1.5 CuMo
CIIP-ViTAL CLIP-VVT-L
76.6
336
0.30B
57.6
32.1
66.4
76.6
336
0.50B
59.3
34.3
69.6
LLaVA-v1.5 CuMo
SigLIP-
83.2
384
0.43B
58.1
32.5
67.5
SigLIP-SO400M
83.2
384
0.72B
59.4
34.1
69.8
Table 5: CuMo under different vision encoders.
B Experimental Setup Details
Table provides an overview the main hyperparameters used during the three-stage training process. For the final results presented in Table 1, the model was trained using 32 x A100 GPUs with total batch size 256 and learning rate of 4e-6. All ablation studies were conducted with total batch size of 128 and learning rates of 2c-5 and 2c-6, detailed in Section 4.3.
14

---

Hyperparameter
PT
PFT
VIT
Learning rate
le-3
2e-6
4e-6
LR schedule
Cosine
Cosine
Cosine
Batchsize per GPU
32
8
GPUs
8xA100
16xA100
32xA100
Zero
Zero2
Zero3
Zero3-offload
Optimizer
AdamW W
AdamW W
AdamW
MLPP
Open
Open
Open
CLIP
Freeze
Open
Open
LLM
Freeze
Open
Open
MoE blocks
Max token
2048
4096
4096
Table 6: Hyperparameters used in three-stage training on Mistral-7B. PT: Pre-Training stage. PFT:
Tre-FineTuning stage. VIT: Visual Instruction tuning stage.
C More Vision Encoders
Table 5 shows CuMo with different vision encoders. Here we use the pre-trained SigLIP-S040MM the vision encoder and add MoE to the vision encoder. SigLIP-S0400M has much better performance on ImageNet zero-shot classification than CLIP-ViT-L (83.2 vs 76.6). The added MoE can still make improvements this stronger vision encoder but the average improvement shrinks compared to CLIP-Vin L. However, the training data here is limited to LLaVA-66 for quick verification, which may not show the full potential of the model if training with more data.
CuMo
CLIP
MLP
LLM
Total
Mistral-7B
0.30B
0.025B
7.25B
7.58B
Activation Params
0.30B
0.025B
7.25B
7.58B
Top 2-in-4 MLP-MoE
0.30B
0.10B
7.25B
7.65B
Activation Params
0.30B
0.05B
7.25B
7.60B
Top 2-in-4 CLIP-MoE
0.91B
0.10B
7,25B
8.26B
Activation Params
0.50B
0.05B
7.25B
7,80B
= Mixtral-8x7B
0.91B
0.10B
46.70B
47.71B
Activation Params
0.50B
0.05B
12.90B
13.45B
Table 7: Change of model parameters of CuMo. corresponds to Act. of CuMo in Table
The 7.80B and 13.45B activation parameters
D Model Parameters
We include Table to illustrate the evolution of parameters in the CuMo model throughout its con- struction process. The LLM constitutes a significant proportion of the total parameters, underscoring the potential for further scaling up the vision encoders to bolster the strength of multimodal LLMs.
E
More Dialogues
We add more dialogues between the questions from the users and the response from CuMo-7B in Figure 6.
F Border Impact
The potential border impact of CuMo, as an open-sourced multimodal I.MM, relies on two aspects: the development of artificial general intelligence (AGI) and the open-source community.
15

---

What
The the Ponsche Tayeon, which electric vebicle produced by the German automebile nanufacturer Porsche. Toycas meludd the sleping reoflins, the shope, of the headlights, ond the specific style of the whesls. The model nome "Taycon" is visible the front of the car, confirning its identity. This particular Heely
distintive design elements that
hugher-and versions. of whst appears to be performance oniented features.
sporty design and the presence
What onimals ore this image.
The animal image are oot deg, ident identifiable by its tabby pattiens and the shape of itss eors and eyes. The dog is Border Collie. recegnizable
domestic shart-haized faline,
medun-longhh eyes. Both oninals are common domestic pets and are humons.
shope
componiinchip
you fell what happored image
The image captures noment where three Individuals are standing baside a. lorge piace of equpnentt which oppoors
hardwore, presence yellow warning lobel suggests that the coupment is either new, being serviced, or requies coution when sparating moving The setting seens to be e profiees ona anvironmntt possisly a date technical faclityy given the The individaols are dressed cosuel enortscouull oftire, which night indioate work-rslated context The exect event on eccaston not cearr from the image alone, out It could be a delivery. installtion,, inspaction of the equimentt The image does provide explicit information. specific actions
the
orgarized space.
that took place prion
moment.
many
image
There seven charocters this
How mony colories astimated this
Eatimating the exect rumber of calories in a didh from a photograph is not possible without specific memmrnmens and ingnedient quantities. Howeer based the vieble ingredients and typical counts for wach, are could make educnted guess boxt neadles vegetables and proticn chicken typicall ranges from pooruts and sesome seeds would cdd few dozen colories each. The sauce, which appears dressing would rarge
colories. edkition
Therefore, rsugh 620 calories, but this is very reugh imae and vary signfiicantly based on the achuol ingredients and partion sizet.
night
Figure 6: More dialogues of CuMo-7B. We highlight the correct answers and hallucinations from the responses of CuMo.
AGI:
The development of AGI is core topic, with the central component being the LLM. A multimodal LLM enhances its abilities to understand content such as images and videos, making it more versatile agent and positive contributor towards AGI.
Open-sourced MLLM: Open-sourced weights and code can accelerate the development of MLLMs. However, this also has potential negative impacts on society. Therefore, we have added non-socicty license to constrain the usage of our model.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We summarize contributions and scope in the abstract and introduction with the main experimental results to support our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We add section 4.4 to discuss the limitations of our CuMo model.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
This paper is application oriented and we conduct extensive experiments to validate our assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all experimental details including the datasets, hyperparameters, and training devices that we used.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide all the open-sourced datasets and codes that used for training and evaluation.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr (ogllc/guides/Coeeeee
ooooonnioiil icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wttpp.ccpppuiiiieeeeeeeepppppppp cctpublic//aeccccccccccccccc /gwii/gg///////eee /Ca///eoeubbiiic Sestsuumissiooos ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the training and inference details for reproducing and under- standing our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For results with statistical variance like querying GPT API, we averaged the results by three times reduce the variance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide the details of the compute resources settings to reproduce our results.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https http///eeerppp.eeeee wwtprrippp.cpppppcccccce cc/publicc/thiessuudeee
Answer: [Yes]
Justification:
We strictly follow the code of ethics during the project.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We add the discussions of border impact in Appendix F.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We provide usage guidelines for users to adhere when accessing to our model.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We strictly follow the original licenses of existing assets including datasets and codes.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our models and codes well documented for reproduction and protected by licenses.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not include crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not include crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "S0iWaNeU6S", "reviewer": "Reviewer_NGUV", "summary": "The paper presents upcycling for large multimodal models (LMMs). It specifically looks at how to enable upcycling for the different components of an auto-regressive based multimodal model (e.g., LLaVA). It shows how the MLP and vision-encoder (in this case, CLIP) are the two modules that should be upcycled and instead of relying on upcycled LLMs, it is better to use pretrained MoE models (for their specific example of upcycled-Mistral vs Mixtral). The paper outlines the training recipe, which is based on 3 stages, with the first focusing on enabling a stable multi-modal model, and then incorporating their CuMo recipe in two stages. To enable stability and balance across the introduced experts, the authors enabled both load balancing loss for the experts and $z_{loss}$ (as suggested in the ST-MoE paper) as auxiliary losses. They apply these auxiliary losses to the two different upcycled MoEs separately. This is followed by a detailed training recipe and evaluation (both qualitative and quantitative) and ablations that explain their design choices.", "questions": "__Suggestions__:\n\n1. In Table 1, please highlight all the best numbers, not just where the CuMo model is the best (both for the 7B dense & MoE sections)\n2. In Table 2, TextVQA is highlighted as the best with the CuMo model, but the QwenVL model has a higher score. How is the highlighting done in this case - if it is still the \"best\" result (despite # data samples used), then it will be good to highlight the appropriate model. \n3. For both table captions, it would be good to mention \"best results are highlighted in bold\" so that it is clear to the user.\n4. For Figure 6, please mention which model generates the responses - Cumo with Mistral or Mixtral.\n5. In the Appendix intro paragraph, the authors mention the \"M3\" model in the last line, but no other reference has been made to this model before. It would be good to clarify this. \n\n__Questions__:\n1. The authors mention a non-society license in the impacts section - is there a specific license being applied that the authors cannot speak about as it will break double-blind? Or is it a general enough license so it is well understood how the release is regulated?\n2. To clarify, the authors mention only the datasets in the checklist (question 5), but in the impacts section, they mention releasing both code and model. It would be good to be consistent on this front. \n3. For the usage guidelines (question 11), there is nothing directly mentioned in the paper. Is the idea that the license, when released, will have clear guidelines for this?", "limitations": "1. The authors list hallucination as the main limitation of the model, which they propose can be corrected using RLHF and help improve reliability. While RLHF can help to some extent with the hallucination problem in the context of multi-modal LLMs, it firstly aligns models with human preferences on tasks of interest (e.g., improving captioning capabilities). Additional systems such as RAG also play a role in this case. I'd recommend potentially mentioning this.\n\n2. In a similar vein to RLHF, since the model has not been aligned, I'd also recommend that the authors mention the potential for biased / non-helpful outputs from their model.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper outlines a clear recipe for upcycling in the context of multimodal models, which has not been explored in literature before.\n    1. The authors support their design choices through well-designed ablations -  particularly for the MoE blocks (which form the core of their method) - for the MLP connector and CLIP model. They show the benefits of using pretrained MoE LLM models over upcycling a dense LLM.\n    2. They present the benefits of using different auxiliary losses (balancing + $z_{loss}$, which they term as bzloss)  to train the upcycled model.\n2. The paper relies on fully open datasets, and it presents all training settings and hyper-parameters, enabling the reproduction of its results.\n    1. The authors show quantitative results, which show for models of similar # active parameters (during inference), the models are competitive with other popular LMMs of similar sizes and outperform some benchmarks. They show this with two different LLMs (Mistral + Mixtral), showing their method is composable with different LLMs. For results that use the GPT4 API, they present the statistical average of 3 API calls to calibrate their results. They give a full breakdown of how the # active parameters are computed for their models based on the upcycled components.\n    2. They follow this up with some qualitative analysis for the balance in experts (relying on the bzloss), seeing an approximately equal distribution of tokens through the experts. They also show some dialogue examples based on sample images for 3 different LMMs, highlighting the benefits of using their method.\n3. For some of their training recipes, they show studies of using high-fidelity data and how relying on the latest training methods, such as multi-resolution features, helps boost their overall model quality.", "weaknesses": "1. One main criticism of the upcycling setup is the limit on the gains from upcycling. The original setting [1] compares the upcycled model to the original MoE model in a 100% dense compute setting and shows that it takes ~20% additional capacity to catch up to the upcycled model (on the LLM setting). While it does take more computing on the ViT models, do the authors have any intuition on when training MoE-based ViTs (such as V-MoE [2]) and MoE connectors from scratch will be much better / potentially start outperforming the dense-only case? \n\n2. Another criticism of the setup is the diminishing gains as the base model size increases (see Figures 2 & 3 in [1]). Do the authors have an intuition of how this will apply to the CuMo setup? If we scale the CLIP-ViT model or use a much stronger model like SigLIP-ViT, will the gains from CuMo still hold? Please note that I'm not expecting new experiments or results here — just a sense of the authors' intuition.\n\n\n[1] Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints, https://arxiv.org/abs/2212.05055", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Limit on the gains from upcycling and when training MoE-based ViTs or MoE connectors from scratch will be much better / potentially start outperforming the dense-only case?**\n\nA1: The conclusion of using 20% additional capacity to catch up the upcycled model in original sparse upcycling and V-MoE is based on a much more intensive computation and data budget for pre-training LLM. However, in the CuMo setup, our motivation for using upcycling is to stabilize the training of the MoE module under a small data and compute budget, because training from scratch of connectors or ViTs is not comparable to the dense models due to the training instabilities as shown in Table 3(a). To estimate the gains of upcycling compared to training from scratch, we may need to train a CLIP-MoE from scratch, which is out of our training budget and beyond the scope of our work.\n\n**Q2: Diminishing gains as the base model size increases and how this will apply to the CuMo setup? If we scale the CLIP-ViT model or use a much stronger model like SigLIP-ViT, will the gains from CuMo still hold?**\n\n|                  | ImageNet | Res. | Params. | TextVQA | MMVet | SEED |\n|------------------|----------|------|---------|---------|-------|------|\n| CLIP-ViT-L       | 76.6     | 336  | 0.30B   | 57.6    | 32.1  | 66.4 |\n| + MoE            | -        | 336  | 0.50B   | 59.3    | 34.3  | 69.6 |\n| SigLIP-SO400M    | 83.2     | 384  | 0.43B   | 58.1    | 32.5  | 67.5 |\n| + MoE            | -        | 384  | 0.72B   | 59.4    | 34.1  | 69.8 |\n\nA2: We think the diminishing gains also exist in the CuMo setup if we use larger and stronger pre-trained CLIP with MoE while keeping training data unchanged. Here we use the pre-trained SigLIP-SO400M as the vision encoder and add MoE to it as shown in the table above. SigLIP-SO400M has a much better performance on ImageNet zero-shot classification than CLIP-ViT-L (83.2 vs 76.6). The added MoE can still make improvements to this stronger vision encoder but the average improvement shrinks compared to CLIP-ViT-L. However, the training data here is limited to LLaVA-665K for quick verification, which may not show the full potential of the model if training with more data.\n\n**Q3: Suggestions regarding Table 1, 2, captions, Figure 6 and Appendix.**\n\nA3: Thanks for the suggestions. We'll update Table 1 by highlighting the best performance numbers in each section and Table 2 by highlighting QwenVL's TextVQA number, as well as the table captions to make them clear to the audience. For Figure 6, the responses are generated by CuMo-Mistral-7B. For the Appendix, the 'M3 model' refers to the CuMo-Mistral-7B, we'll revise it in the updated version as well.\n\n**Q4: Non-society license, release, and usage guidelines.**\n\nA4: We plan to release the code under Apache 2.0 and the weights of all CuMo checkpoints under CC BY-NC 4.0 for non-commercial use. All the datasets and pre-trained weights we used for training and evaluation are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses.\n\n**Q5: Limitations.**\n\nA5: Thanks for the suggestions. We'll add discussions of RAG and the potential for biased outputs in the limitation section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7UjaRXwSWQ", "reviewer": "Reviewer_rTFY", "summary": "The paper introduces CuMo, a novel approach to enhancing multimodal large language models (LLMs) by integrating sparsely-gated Mixture-of-Experts (MoE) blocks into both the MLP connector and vision encoder. CuMo addresses the challenge of scaling multimodal LLMs effectively by leveraging MoE's efficiency in parameter usage during training and inference. Key contributions include a detailed exploration of MoE integration strategies across different components of LLMs, a three-stage training methodology to stabilize model training, and the introduction of auxiliary losses for expert load balancing. Experimental results demonstrate that CuMo outperforms existing state-of-the-art multimodal LLMs on various benchmarks, showcasing its efficacy in enhancing model performance while managing computational efficiency", "questions": "see weaknesses", "limitations": "see weaknesses", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Innovative Integration of MoE: Integrating sparsely-gated Mixture-of-Experts (MoE) blocks into both the MLP connector and vision encoder of multimodal large language models (LLMs) represents a novel approach. This approach is not only innovative in terms of architecture but also in its application to enhance multimodal understanding.\n\nExperimental Rigor: The authors provide a thorough experimental evaluation, comparing CuMo against state-of-the-art models on multiple benchmarks. This comprehensive evaluation includes ablation studies, which validate the effectiveness of their proposed approach.", "weaknesses": "Issue: While the paper touches upon the scalability benefits of MoE, there's limited discussion on the computational efficiency and training time required for integrating MoE blocks into multimodal LLMs. This is crucial as MoE designs can potentially introduce additional computational costs during training.\n\nWhile the paper focuses on performance metrics, there's a lack of discussion on the interpretability of MoE decisions within CuMo. Understanding how MoE blocks make decisions and whether they exhibit consistent behavior across different inputs is crucial for deploying models in real-world applications.\n\nThe paper shows that Mixtral-8×7B's CuMo does not consistently outperform Mini-Gemini on several datasets, particularly on TextVQA and MME. This discrepancy could be due to architectural differences, especially Mini-Gemini's specialization in high-resolution data. The author should conduct a comparative study with non-MoE architectures like Vicuna or Llama3, focusing solely on CuMo's projection MoE and Vision Encoder MoE. This would provide insights into how CuMo's MoE-based approach performs against architectures not leveraging MoE.\n\n Table 1 highlights CuMo's results in datasets like VQAv2, SeedImg, and MMbench, even though it does not consistently achieve the best performance. This can mislead readers about CuMo's comparative performance against other models.\n\nThe experiments primarily focus on Mistral or Mixtral-8×7B architectures with MoE integration. There's a lack of exploration into how CuMo's MoE-based enhancements compare against non-MoE architectures like Vicuna or Llama3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Computational efficiency and training time of MoE.**\n\n| CuMo                     | CLIP  | MLP   | LLM   | Total  | Time       |\n|--------------------------|-------|-------|-------|--------|------------|\n| Mistral-7B               | 0.30B | 0.025B| 7.25B | 7.58B  | ~16h       |\n| + Top 2-in-4 MLP-MoE     | 0.30B | 0.10B | 7.25B | 7.65B  | ~16h       |\n| + Top 2-in-4 CLIP-MoE    | 0.91B | 0.10B | 7.25B | 8.26B  | ~20h       |\n\nA1: We provided the breakdown of additional parameters during training in Table 6 and here we further include the training time of CuMo-Mistral-7B with MLP-MoE and CLIP-MoE. We used a single 8xA100 machine and LLaVA-665K data for reference here. Note that we only used the model parallelism in deepspeed for implementation while the training can be faster if expert parallelism is further added. \n\n**Q2: Lack of discussion on the interpretability of MoE decisions within CuMo.**\n\n| Subset    | Layer ID | Top 1 Expert Ratio |\n|-----------|----------|--------------------|\n| OCR       | 8        | 31.54%             |\n| Color     | 7        | 33.97%             |\n| Code      | 18       | 34.49%             |\n| Reasoning | 1        | 35.01%             |\n\nA2: Following Section 5 of Mixtral-8x7B[1], we did the expert distribution analysis in Figure 4, which shows that the experts are loaded pretty balanced overall across the layers. We further used the subsets of images in MME by topic and found that they show a preference towards certain experts in some layers, which may imply some hidden patterns of the assignment of experts based on the applications or topics. We may update these results in Section 4.4 as parts of the analysis.\n\n[1] Mixtral of Experts, https://arxiv.org/abs/2401.04088\n\n**Q3: The paper shows that Mixtral-8×7B's CuMo does not consistently outperform Mini-Gemini on several datasets, particularly on TextVQA and MME.**\n\nA3: CuMo-Mixtral-8x7B is better than Mini-Gemini-Mixtral-8x7B on MME (+0.5), MMMU(+3.2), and MM-Vet (+2.9), while worse on TextVQA (-3.2) and MMBench (-0.3), as shown in Table 1. One main reason behind this is that Mini-Gemini takes high-res inputs and benchmarks like TextVQA are sensitive to input resolution as shown in the Table 2 \\& 3 in Mini-Gemini. \n\n**Q4: The experiments primarily focus on Mistral or Mixtral-8×7B architectures with MoE integration. The author should conduct a comparative study with non-MoE architectures like Vicuna or Llama3, focusing solely on CuMo's projection MoE and Vision Encoder MoE.**\n\n|                  | TextVQA | MMVet | SEED |\n|------------------|---------|-------|------|\n| Mistral-7B       | 57.6    | 32.1  | 66.4 |\n| + MLP-MOE & CLIP-MoE | 59.3 | 34.3  | 69.6 |\n| Vicuna-7B-v1.5   | 58.2    | 30.5  | 66.1 |\n| + MLP-MOE & CLIP-MoE | 59.4 | 32.6  | 68.5 |\n\nA4: Our ablation studies are mainly on the Mistral-7B, which is a non-MoE LLM to verify the effectiveness of MLP-MoE and CLIP-MoE. We further evaluate CuMo on Vicuna-7B-v1.5 to make comparisons as shown in the table above under the same LLaVA-665K training data. The CLIP-MoE and MLP-MoE also make improvements over the Vicuna-7B-v1.5.\n\n**Q5: Table 1 highlights are misleading.**\n\nA5: Thanks for the suggestions. We'll revise that and highlight the best performance across models in each session of Table 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NrOUkJ0JHL", "reviewer": "Reviewer_5mox", "summary": "This paper is motivated by the fact that traditional scaling approaches are computationally expensive and overlook the significance of efficiently improving model capabilities from the vision side. The authors introduce CuMo, a method to enhance MLLMs by incorporating sparsely-gated Mixture-of-Experts (MoE) blocks into the vision encoder and MLP connector. CuMo employs a three-stage training process and has achieved impressive results across multiple benchmarks.", "questions": "See weakness.", "limitations": "The authors mention that the main limitation of this work is the hallucination problem and indicate that it will be improved in future work.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written, and the proposed method is simple and easy to follow.\n2. This paper has addressed an important issue: traditional MoE-based MLLMs are computationally expensive and overlook the vision encoder.\n3. CuMo achieves good performance across various benchmarks.", "weaknesses": "1. The author employ the MoE method in the vision encoder and connector, but don't explain why using this approach enhances the model's capabilities.\n2. The use of the MoE method inherently expands the trainable parameters to some extent. The author should consider whether merely increasing the parameters of the vision encoder alone will also enhance the model's capabilities. This will provide readers with valuable insights.\n3. CuMo's LLM is mainly limited to Mistral-8$times$7B. Since this paper primarily explores the expansion of the vision encoder and connector, I think the author should include comparisons with other LLM backbones, such as Qwen[1] and LLaMA[2], to demonstrate that the model's improvements are not due to differences in the inherent capabilities of the LLM itself.\n4. Since CuMo is three-stage training, the increased amount of training data may lead to an unfair comparison with other baselines, such as LLaVA1.5.\n\n[1] Qwen Technical Report.\n\n[2] LLaMA: Open and Efficient Foundation Language Models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Why MoE in the vision encoder and connector enhances the model's capabilities**\n\nA1: MoE has been widely used in LLMs to improve the capacity [1] of text generation as it improves the model size during training while keeping inference costs lower at inference. In our work, we apply MoE in the vision encoder and connectors of multimodal LLMs, which has the potential to generate versatile visual tokens and further improve the model capabilities on many vision-language instruction-following tasks. We verify our assumptions with detailed ablation studies on the effectiveness of the proposed MLP-MoE and CLIP-MoE.\n\n[1] Shard: Scaling Giant Models with Conditional Computation and Automatic Sharding\n\n**Q2: Whether merely increasing the parameters of the vision encoder alone will also enhance the model's capabilities for comparisons**\n\n| Model          | ImageNet | Res. | Params | TextVQA | MMVet | SEED |\n|----------------|----------|------|--------|---------|-------|------|\n| CLIP-ViT-L     | 76.2     | 336  | 0.30B  | 57.6    | 32.1  | 66.4 |\n| SigLIP-SO400M  | 83.1     | 384  | 0.43B  | 58.1    | 32.5  | 67.5 |\n| CLIP-ViT-H     | 78.0     | 224  | 2.52B  | 49.2    | 29.5  | 58.2 |\n\nA2: It depends. We compare the CLIP-ViT-L encoder with the other two larger vision encoders: CLIP-ViT-H and SigLIP-SO400M. CLIP-ViT-H has 8x more parameters than CLIP-ViT-L while performing much worse due to low-resolution inputs. SigLIP-SO400M has 0.13B more parameters and performs consistently better than CLIP-ViT-L. These findings have been also explored in recent multimodal LLMs [2,3] that the model size is not the top factor that affects the overall performance of multimodal LLMs.\n\n[2] MM1: Methods, Analysis, and Insights from Multimodal LLM Pre-training\n\n[3] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs\n\n| Model                    | TextVQA | MMVet | SEED |\n|--------------------------|---------|-------|------|\n| SigLIP-SO400M            | 58.1    | 32.5  | 67.5 |\n| + CLIP-MoE & MLP-MoE     | 59.4    | 34.1  | 69.8 |\n\nWe further apply CLIP-MoE and MLP-MoE to the stronger SigLIP-SO400M vision encoder and they can still make improvements upon a larger and more powerful vision encoder.\n\n**Q3: CuMo's LLM is mainly limited to Mixtral-8x7B, and the author should include comparisons with other LLM backbones.**\n\n| Model                    | TextVQA | MMVet | SEED |\n|--------------------------|---------|-------|------|\n| Mistral-7B               | 57.6    | 32.1  | 66.4 |\n| + MLP-MOE & CLIP-MoE     | 59.3    | 34.3  | 69.6 |\n| Vicuna-7B-v1.5           | 58.2    | 30.5  | 66.1 |\n| + MLP-MOE & CLIP-MoE     | 59.4    | 32.6  | 68.5 |\n\nA3: CuMo is mainly focusing on Mistral-7B and Mixtral-8x7B. Our ablations are based on Mistral-7B and verified the effectiveness of CLIP-MoE and MLP-MoE separately. We further evaluate CuMo on Vicuna-7B-v1.5 as shown in the table above with added CLIP-MoE and MLP-MoE to show their effectiveness.\n\n**Q4: Since CuMo is three-stage training, the increased amount of training data may lead to an unfair comparison with other baselines, such as LLaVA1.5.**\n\nA4: In Table 2, we maintain a fair comparison to LLaVA1.5 under the same training data and CuMo-7B consistently performs better than LLaVA1.5, either under Vicuna7B or Mistral-7B. In Table 1, we compare CuMo with models that use various datasets even with private datasets like LLaVA-NeXT and MM1, while CuMo is trained under fully open-sourced datasets. We may remove LLaVA1.5 from Table 1 and keep LLaVA-NeXT for comparison in the updated version to avoid confusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their thoughtful comments. We feel encouraged that the reviewers find that:

- The innovation of our method integrates MoE design into the vision side of current multimodal LLMs (NGUV, rTFY, 5mox), and the implementation is simple and easy to follow (5mox).
- We provide detailed ablations to validate the effectiveness of the proposed method (NGUV, rTFY) and achieve good performance across various benchmarks (rTFY, 5mox).
- Our work is based on fully open datasets and we present all training settings with hyper-parameters for reproduction of the results (NGUV).

We also appreciate the suggestions from all reviewers that help us continue to improve the draft. We attempted our best to address the questions using the individual responses below.

Meta-review: This paper proposes to use a sparse MoE block in the MLP connector and the vision encoder for multimodal LLMs. In particular, for stable and fast MoE training, the proposed method exploits a three-stage training with a upcycling strategy and auxiliary losses. Experimental results on various benchmark multimodal understanding tasks show that the proposed MoE-based MLLM outperforms existing MLLMs having similar model sizes and rigorous ablation studies validate the effectiveness of the proposed architecture and training strategy.

Overall, the paper is well-written and easy to understand, and the authors sufficiently address most of concerns and issues raised by the reviewers.

In terms of the technical novelty, actually, the proposed MLLM is somewhat incremental from the combination of existing methods for its MoE blocks, initialization, and even training losses, even though it is first to apply the MoE-based vision encoder to MLLMs. However, rigorous empirical validation from extensive experiments on multiple benchmark tasks with in-depth analysis and comparison with recent state-of-the-art MLLMs sufficiently supports its feasibility and powerfulness. 

Based on that all reviewers scored in a positive direction and the above contribution, I recommend the paper to be accepted in NeurIPS.
Decision: Accept (poster)