MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models
Authors: Leyang Shen
Gongwei Chen
Rui Shao
Weili Guan
Liqiang Nie
Venue: NeurIPS 2024 poster (2024)
Keywords: Multimodal Large Language Model
Mixture of Experts
Primary area: generative_models

Abstract: Multimodal large language models (MLLMs) have demonstrated impressive capabilities across various vision-language tasks. However, a generalist MLLM typically underperforms compared with a specialist MLLM on most VL tasks, which can be attributed to task interference. In this paper, we propose a mixture of multimodal experts (MoME) to mitigate task interference and obtain a generalist MLLM. Our MoME is composed of two key components, a mixture of vision experts (MoVE) and a mixture of language experts (MoLE). MoVE can adaptively modulate the features transformed from various vision encoders, and has a strong compatibility in transformation architecture. MoLE incorporates sparsely gated experts into LLMs to achieve painless improvements with roughly unchanged inference costs. In response to task interference, our MoME specializes in both vision and language modality to adapt to task discrepancies. Extensive experiments show that MoME significantly improves the performance of generalist MLLMs across various VL tasks.

MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models
Leyang Shen" Gongwei Chen' Rui Shao Weili Guan Liqiang Nie
School of Computer Science and Technology, Harbin Institute of Technology. Shenzhen {chengongwei, shaorui, guanweili, nieliqiang) @hit edu.cn
Abstract
Multimodal large language models (MLLMs) have demonstrated impressive ca- pabilities across various vision-language tasks. However, a generalist MLLM typically underperforms compared with specialist MLLM most VL tasks, which can be attributed to task interference. In this paper, we propose a mixture multimodal experts (MoME) mitigate task interference and obtain generalist MILMM. Our MoME composed of two key components, a mixture of vision experts (MoVE) and mixture of language experts (MoLE). MoVE can adaptively modulate the features transformed from various vision encoders, and has a strong compatibility in transformation architecture. MoLE incorporates sparsely gated experts into LLMs to achieve painless improvements with roughly unchanged inference costs. In response to task interference, our MoME specializes in both vision and language modality to adapt to task discrepancies. Extensive experiments show that MoME significantly improves the performance of generalist MLLMs across various VL tasks.
Introduction
Recently, Multimodal Large Language Models (MILMs) 39 35
have witnessed re-
63 and Modality 52 70], MLLMs demonstrate excellent multimodal comprehensive abilities, especially in solving wide range of vision-language (VL) tasks 3 42
markable progress. With the help of Large Language Models (LLMs)
13
Encoders (50 30 17
57 56], such Image Cpation, Visual Question Answering, Referring Expression Comprehension, and Optical Character Recognition. However, it is increasingly acknowledged that generalist MLLM tends have lower performance compared to a specialist MLLM on most VL tasks This phenomenon can be attributed to task interference, fundamental and crucial issue in multi-takk learning [16 [4] (64).
37
55
[12 g as depicted Fig. (a).
There are some preliminary attempts 15 6 12
5
8
58
to address this issue in instruction-tuning
58 among these attempts to use a mixture of experts (MoE) in MLLMs, aiming for each expert specialize in several tasks. However, these works only investigate MoE LLMs and primarily concentrate on textual differences between tasks, overlooking the equally important visual information. As shown in Fig. (b-c), we analyze distribution of various VL tasks across both vision and language modalities. It is evident that images from different groups of VL tasks exhibit distinct feature distributions, do texts. Inspired by this observation, we argue that handling task interference needs to comprehensively exploit task differences in both vision and language modalities.
MLLMs. The most promising direction 12
'Equal contribution Corresponding authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Comperison of specialist and generalist models
(61 Sentence Semantic Analysis
[cl image Embedding Analysic
0.
Specialitt
Generali
Document
Document Racared
04
100%
REC
500%
0.0
0.1
-02
General
REC
BBG
Decumeee
-0.5
-0.4
-0.3
-02
0.0 0.2
D.1
-40 -3
-10
Groups
Figure 1: VL data distribution visualization and model performance comparisons. Experimental results in (a) show that a generalist model trained on mixed dataset underperforms most specialist models trained on separate task groups. The feature distributions visualized in (b) and (c) show significant discrepancies across VI. tasks in both images and instructions.
To mitigate task interference, we devise Mixture of Multimodal Experts (MoME) and integrate it into MLLMs. Our MoME consists of a Mixture Vision Experts (MoVE) for adaptively aggregating features from various vision encoders [52 17 [0], and a Mixture of Language Experts (MoLE) for leveraging multiple sparsely-activated parameter-efficient adapters. To avoid feature mismatch in different vision encoders, we propose adaptive deformable transformation (ADT) module in MoVE and use to transfer features of vision encoders into a unified-length sequence of feature vectors. Our ADT module combines adaptive average pooling and deformable attention [71] to obtain compressed and self ceffeenhanced visual features. After feature transformation, our VE uses an instance-level soft router modulate and aggregate transformed visual features according to the instructions. Our MoL.E introduces several parameter-effiiien adapters 99] as experts and integrates them by using an instance-level sparsely-ace acttivvted router. Due to the utilization of adapters, MoLE be integrated into each feed-forward network layer of an LLM and only incurs a few computational costs with consistent performance gains.
To comprehensively evaluate the multimodal understanding ability of MoME, we collect an amount of VL tasks to form an instruction-tume -tttinn dataset and split them into four groups. Extensive experiments show that both Mo and MoLE can consistently improve performance across all groups tasks. Notably, MoVE
achieve average performance gain of 12.87 points across all VL tasks, and improve by over 20 points on the "Document" group. Furthermore, we visualize the expert load distributions of MoVE and MoL. across various tasks. The experts in both MoVE and MoLE exhibit a relatively clear specialization in different groups of VL tasks. For example, the "Document" group of tasks has strong preference for the "Pix2Struct" vision expert. The expert specialization is strong evidence to demonstrate that our MoME dynamically selects experts to adapt to task differences and mitigate task interference.
Our main contributions are summarized as follows:
We propose MoME by simultaneously designing mixtures of experts tailored for both the vision encoder and the LLM, resulting in generalist MLLMs with the ability combat task interference.
Through statistical analysis, we demonstrate that our MoME specializes in both vision and language modality, effectively adapting to the varying requirements of different VL tasks.
Extensive experiments show that our MoME possesses excellent multimodal understanding abilities and achieves superior performances across various groups of VL tasks.
Related Work
2.1 Vision Encoders in MILMs
Vision encoders play important roles in the perception ability of recent MLLMs by encoding visual information into visual tokens, enabling LLMs to understand information on visual modalities. Most Multimodal Large Language Models (MLLM) use CLIP-VIT (52) as their vision encoder to provide the basic image-level perception of an image for LLMs, which is useful for tasks such as image caption and general VQA. However, Tong et al. (62) have found that CLIP-VIT struggles to encode some visual patterns, severely limiting perception and preventing MLLMs from becoming generalist.

---

To alleviate this, recent works 33 62 27 integrated different vision encoders [52] 17 50 70 29 into a single MLLM, which enhanced the perception of MLLM. However, none have effectively mitigated the interference among different visual features, resulting in sub-optimal utilization of each encoder. Differently, we explore the task differences and propose MoVE to perform self-enhanced transformation and adaptive routing among features from different encoders, achieving consistently high performances across vast VL tasks.
2.2 Mixture of Experts in Large Models
Mixture of Experts (MoE) (24] is type of structure with multiple expert networks working together, where each expert is responsible for part of the knowledge space. By only activating specific experts adaptively during inference, MoE can reduce resource consumption and enhance reasoning speed, which is useful for LLM. Existing MoE LLM [18 26 14 usually replace the feed-forwar forward network (FFN) with the MoE layer. Each MoE layer consists of a router and multiple expert networks and each token is assigned several expert networks by the router. MoE LLMs tend to outperform dense models with the same inference activate parameters.
In addition to its effectiveness in foundation models, some works 12 g 58 MoE in the visual instruction tuning (36] phrase of MLLMs to mitigate task interference, aiming for each expert to specialize in several tasks. Some of them replicate the original FFN in LLMs, while others insert multiple low-rank adaptation [22] modules parallel with the original FFN, converting LLM into multi- culti-expert architecture. However, they primarily concentrate LLM but overlook task interference within the visual perceiving process of MLLMs. In contrast, we comprehensively exploit task interference in both vision and language modalities and propose MoME to mitigate them with specialized vision and language experts.
20
32
have utilized
3 Methods
To design a generalist MLLM with powerful multimodal understanding capabilities, we begin by analyzing task interference, a common challenge in current MLLMS (Section (.1). Then, we propose our Mixture of Multimodal Experts and introduce its main components in Section 3.2
3.1 Analysis of Task Interference
Task interference is a fundamental and crucial issue in multi-task learning. As current generalist MLLMs are trained with a number of tasks, they naturally suffer from this issue especially when the number of tasks increases. To investigate this issue in a scenario of MILMs, we will analyze the feature distribution and the performance change of MLLMS on a tailored instruction-tuning dataset.
To demonstrate the external manifestations of task interference, we first construct mixed instruction- tuning dataset with various VL tasks and split all tasks into four groups. The performance comparisons between MLLMs trained on the mixed dataset and each task group are illustrated in Fig, (a). It is shown that a generalist model trained on the mixed dataset underperforms specialist models on three task groups We conclude that the generalist model suffers from notable task interference problem.
In the era of Large models, there are some attempts to handle task interference from various perspec- tives, such as instruction, architecture, and dataset configuration. Here, we focus on the mainstream direction, architecture design, and try to explore robust architecture to combat task interference. In terms of architecture, most existing works prefer mixture of experts in LLMss. The experts can be forward networks or parameter-efficien modules. However, we argue that this paradigm of architectural design is sub-optimal as visual and textual information should be given equal importance.
In Fig. (b-c), we investigate the feature distribution of various task groups on vision and language modalities, respectively. All samples of each task group are fed into vision and text encoders to produce modality features. These features are transformed by using PCA and then visualized to show the distribution. We observe that the feature distributions of different task groups exhibit significant differences in both the vision and language modalities. As mentioned above, the textual differences can be addressed by multiple experts in L.MMs, but visual differences lack effective handling. In the following section, we will introduce our Mixture of Multimodal Experts, which simultaneously handles visual and textural differences between tasks to mitigate task interference.
3

---

Avoid Injury during home isclation
LLM
CLIP
Original
Adaptive Deformable
Dynamic Routing
Add & Norm
DINO
Original
Feat.
FFN
MoLE
Pix2Struct
Original
Add & Norm
Feat.
Self-Attention
Instruction
(a) Adaptive Deformable Transformation
(b) Dynamic Routing
(c) MoLE
Pooling
Coarse Features
Fused Feat
Hidden State
Original Features
Deformable Layers
Adapter
Self-Alln
SoftMax
Detomnable X-Alin
MLP
Adapter
Sentence Embedding
CLIP
DIO
Sentence
Sparse
Selt-Enhanced Features
Embedding
Router
Figure 2: The overall architecture of the proposed MoME. The model obtains compressed and self- enhanced visual features from distinct vision encoders through adaptive deformable transformation (a) and aggregates them by dynamic routing (b). The MoLE blocks (c) are integrated into each FFN layer of LLM to improve multitasking capability with little cost.
3.2 Architecture
As illustrated in Figure 2 we present our novel MoME architecture that dynamically mixes vision and language experts. The proposed architecture aims to adaptively aggregate visual information (32.1 and select LLM pathways (3.2.2) based on the given instructions.
3.2.1 Mixture of Vision Experts
Before introducing our MoVE architecture, we will present pilot study that inspires design three MLLMs (consists of vision encoder, projection, and LLM), which share the same architecture except vision en- coders. These three MLLMs use three distinct vision encoders, CLIP, DINOv2, Pix2Struct, respectively. After training them using the same data and strategies, we found that MILMs with different vision encoders excelled in specific tasks, as presented in Fig. 3 the MILMM with CL.IPVij is good at general tasks and regional caption tasks. the MLLM with DINOv2 excels in REC, visual grounding task. the MLLM with Pix2Struct is outstanding in text-intensive document understanding tasks. However, each model had weaknesses and none could achieve uniformly excellent performance across all tasks.
design MoVE. Specifically,
Rerormacc
20
Re2frrutt
Figure 3: Comparison of MLLMs with different vision encoders.
Inspired by the above study, we propose MoVE to combine various off off-the-shell vision encoders and effectively align and aggregate their visual features. The key components in MoVE are an adaptive deformable transformation module and an instruction-based soft router. The former aims to align the features from various vision encoders, and the latter seeks modulate and aggregate transformed features based on the instructions.

---

Adaptive Deformable Transformation. Given the diversity in architecture and training methods of different vision encoders, the issue of mismatched visual representations in terms of sequence often focus on models like CLIP-ViT [2 and DINO [50], which share similar data processing pipeline, the mismatch problems are less important and frequently overlooked. However, the aspect ratios of Pix2Struct [30] feature shapes vary depending on the input image, Simply combining them through padding and addition will lead to the misalignment among visual tokens and the damage of visual information. To tackle this challenge, we innovatively design an adaptive deformable transformation (ADT) module that effectively transforms features from diverse vision encoders into unified-length feature sequence f. ensuring more coherent and informative visual representations for subsequent aggregations.
length and feature space becomes significant. While current researches [62 27 X
As illustrated in Fig. 2 (a), the ADT module consists of 2D adaptive pooling layer and M deformable attention layers (I)). It attempts to automatically select the corresponding information from original features to refine the coarse features obtained by 2D adaptive pooling D(J),
= (D())
(1)
Inspired by Deformable DERT 71 we choose deformable cross-attention for its 2D sampling mechanism, which is ideal for interactions among visual feature maps of varying shapes. Meanwhile, it converges fast and has computational and memory efficiency. Specifically, each deformable layer consists of a self-attention layer, a deformable cross-attention,, and a forwafd layer. The crucial select operation occurs in the deformable poos-attenttoo layer, which takes the output of the upper self-attention as query E R RLXC, samples the original feature map f E RH xxxW XC and outputs result O E R RLxC" In this module, the first step is to generate attention weights E IR L N,,, and L sets of 2D sampling points, denoted as p, from the input queries q. For each set, there are N₁, x Np points, where N₁, signifies the number of attention heads and Np represents the number of points sampled by each head. The sample points and attention weights generation process is as follows,
Piyk = (Po(a)s + R E {1, (L....,L):. E {l..... Nn), (L....,,,, Npp w - Pp)(()
where P denotes the linear projector and R E Lx2 is learnable vector called reference point. Then, the corresponding information is sampled by attention heads from the value feature maps P(f); projected and split on the last dimension for each head. The sampling mechanism of each attention head as follows,
Ojj Wifk Samplo(P.())) Pijk,, (t, 4J.j. (L,...,,NNk)
The results of multiple attention heads are concatenated and projected to obtain the output feature O, which is the input of subsequent feed-forward layer.
= Polo)
Instance-level Soft Router. Since images from different groups of VL tasks exhibit distinct feature distributions, there no one-fits-lll strategy to aggregate them. To address this, we propose to generate a customized fusion ratio for each sample based on its instruction.
Specifically, devise instance-level soft router G., depicted in Fig (b). The router generates corresponding ratios for the visual representations from different vision encoders, followed by a weighted sum of these visual representation features f. which can be formulated as,
G1(I) -SoftMax(Wa(((WWW b,)) + bz)
(6)
N F Σ G.(Dxxff.
where N the number of vision experts and a denotes GeLU 21). The router is a multilayer perceptron (MLP) followed by SoftMax layer to ensure that the sum of the weights equals 1. The instruction first passed through Sentence-BERT 53 to extract sentence embedding 7, which then fed into the router.
5

---

Table Comparison of various MoVE strategies. "Gen." and "Doc." respectively denote average performances on General and Document. "Avg." means the average of the four group scores. The best performances are marked as bold.
Strategy
Performance REG
#
Encoder
Transformation
Aggregation
Gen.
REC
Doc.
Avg.
1
CLIP
AvgPool
75.04
61.42
58.79
30.84
56.52
2
DINO
AvgPool
66.09
71.52
55.58
22.10
53.82
3
Pix
AvgPool
40.75
39.26
32.11
47.05
39.79
4
MoVE
AvgPool
Addition
70.36
74.89
57.55
32.83
58.91
5
MoVE ADT
Addition
74.35
76.93
61.01
39.23
62.88
6
MoVE
ADT
Router
79.05
81.92
63.82
52.77
69.39
3.2.2 Mixture of Language Experts
We introduce MoE architecture LLM, aiming for each expert specialize several tasks. However, conventional MoE methods typically incorporate multiple parallel FFN layers in one block, significantly increasing training costs and memory consumption. To meet the multi-task learning needs in the instruction tuning stage, we incorporate several teffeette-effiiintt adapters [9] parallel FFN. Each adapter enhances the original FFN with task sseciipcciiic understanding capabilities, thus effectively enhancing the multitasking abilities with few computation costs.
We insert MoE block parallel each FFN layer in LLM. As depicted in Fig. (c), The MoE block consists of several low-rank adapters and an instance-level sparsely-activated router G₁, The adapter designed
bottleneck structure for computational efficiency, featuring a down-projection layer Pdown. a ReLU layer a, and an up-projection layer Pup. Moreover, a learnable scalar is multiplied the output weigh the importance adaptively. The whole low-rank adaptation process is as follows,
= (oppo((Poon)) down(x)))
(8)
The router is MLP network followed by top-1 gate function to ensure the output is one-hot vector G(T) E 1) The router generates the selection based on the sentence embeddings / used in MoVE. Each sample is routed to the corresponding adapter calculate the adapted value o, which can be further added to the output the original FFN. The whole process the MoLE block is as follows,
Gn(T)R Uk
(9)
where K denotes the number of experts.
4
Experimental Results and Analysis
We collect 24 datasets and categorize them into four groups for instruction-tuning and evaluation, the details of which can found Appendix B
4.1 Analysis on MoVE
We conduct experiments on MLLMs with different vision encoders under the same training strategy to verify the effectiveness of the two key components in MoVE: ADT and router. Experimental results are summarized in Table We take the multitasking performances of MLLMs with a single vision encoder as our baselines. The adaptive average pooling is applied to the visual representation from DINO and Pix2Struct, ensuring that the lengths of visual tokens fed into LLM are consistent. Experiments #1 show that MLLMs using CLIP. DINO, and Pix2Struct as vision encoders exhibit distinct strengths in General, REC, and Document tasks, respectively. Moreover, in the REG task, which requires both captioning and visual grounding abilities, the performance of MILMs with CLIP and DINO significantly surpasses that of those using Pix2Struct. We can conclude that a single vision encoder cannot meet the visual perception needs of all tasks.

---

85%-
10%-
58%
486
33%
Bdl Nona
Figure 4: Distribution of vision experts routing results. In each bar, the lengths of different colors represent the frequency with which each expert is selected.
To make different vision encoders work together in perception, we first aggregate different visual representations by addition (44). The average performance does improve compared with models with single vision encoders (#1-3). However, such a straightforward method does bring much improvement, and even some sub-items have declined. This is due to the mismatch and interference among different visual features, which severely compromise their respective visual information. To make visual features aligned well and reduce information loss, we first replace the pooling with the proposed ADT network. As demonstrated in Experiment #5, ADT consistently enhances performance across four task groups, yielding an average improvement of points. Based on the transformed visual features, we further replace the naive addition mixture process with the router to modulate and aggregate them adaptively according to instructions (#6). This achieves an impressive performance that significantly outperforms the addition method and the methods with a single vision encoder across all sub-tasks. Additional experiments on the deformable mechanism can be found in Appendix C These experimental results prove that ADT and router are crucial and effective components of MoVE to mitigate interference and optimally utilize the capabilities of each visual expert.
Visualization of Routing Results. To provide a deeper understanding of the MoVE's adaptive routing process, we visualize the routing outcomes across various tasks. Since the feature scale varies across different vision encoders, we cannot directly consider the output of the router as expert importance. Instead, integrate the output features from vision encoders, taking the magnitude of weighted feature vector as the importance metric. Because the final aggregated result will lean towards the side with the larger magnitude.
As displayed in Fig. for tasks that involve text recognition and graph understanding. such as ChartQA and Doc VQA, the features from the Pix2Struct encoder are dominant. In contrast, the model utilizes more CLIP features image-level VQA tasks like COCOCapion [11]. NoCaps (2]. and Flickr30K [69]. Notably, for TextCaps [59]. task that requires two kinds of visual information, the router shows preference for balancing the CLIP and Pix2Struct branches. For tasks that focus on visual grounding ability like REC [42), and REG (37, the model uses more DINO features to perceive region-level visual information. These observations indicate that MoVE can adaptively modulate the features transformed from various vision encoders.
4.2 Analysis on MoLE
We conduct ablation experiments to explore the best practice of MoLE, which are summarized in Table We take the model with a single adapter in each FFN (#1) ) as bascline, which suffers severe task interference. Then, replace the plain adapter with MoLE. As summarized in Table #2-4, we test three kinds of MoLE routers with different inputs: token hidden states (MoLE-T), sentence embedding (MoLE-I), and a mixture of both (MoLE-IT). The token hidden states and sentence embedding are concatenated on the last dimension as the input for the MoLE-IT router. The experimental results indicate that all MoLE variations outperform the plain adapter, with the centence-embedleee based router achieving the highest average performance.
We also explore two strategies for expert load balance in MoLE, which are tabulated in Table 2 #5-6.
MoL.E-IGGS introduces variability to the router by adding distrilddst noise to the logits 25
aiming to prevent certain experts from never being selected. MoLE-I+LB uses auxiliary loss [18 to

---

Table 2: Comparison different MoLE configurations "-T", "-I", and "-IT" respectively represent MoLE with routers based on token, instance, and both. "GS" and "LB" represent the implementation
of Gumble Softmax [5] and Load Balance [8] based on the MoL.EI,, respectively.
#
VE
LE
Gen.
REC
REG
Doc.
Avg.
CLIP
Adapter
74.50
63.80
59.24
31.73
57.32
CLIP
MoLE-IT
75.61
65.63
59.17
32.46
58.22
CLIP
MoLE-T
75.35
66.09
58.09
32.12
57.91
CLIP
MoLE-I
75.62
66.95
60.90
32.27
58.94
5
CLIP
MoLE-I GS
74.87
63.97
59.00
31.69
57.38
CLIP
MoLE-I+ LB
75.42
64.74
59.48
32.05
57.92
MoVE
Adapter
79.05
81.92
63.82
52.77
69.39
8
MoVE
MoLE-I
79.65
81.58
64.83
53.69
69.94
30%
25%
20%
LLM Liger
LLM
Figure 5: Distribution of language experts routing results. The figures depict the expert load conditions of four selected datasets. In each bar, the lengths of different colors represent the frequency with which each expert is selected.
impose load balancing. However, find that these methods not suitable for MoL.F as they perform worse than the plain MoLE-I configuration.
Through comprehensive comparative experiments, choose MoLE-I as the default configuration for MoME. By training based on the MoVE model, MoME further enhances the multitasking capability of MLLM, shown in Experiments #7 and #8.
Visualization of routing results. To provide a deeper understanding of our MoLE module, we sample several instances from each dataset and calculate their average routing outcomes. In Fig. []] we present the expert load conditions of four selected datasets, each representing kind of VL task. At the beginning, the router assigns equal probabilities each expert it randomly initialized. However, after training, the routing distributions of MoE blocks vary significantly across tasks, as shown Fig-5] This means language experts in our MoLE module gradually specialize in distinct task domains during training. In the inference process, the router adaptively chooses the optimal expert achieve strong multitasking capabilities. Meanwhile, we can observe strong resemblances in the expert routing results of similar tasks, which are further analyzed in Appendix D
4.3 Comparison with state-of-the-art MLLMS
We summarize the evaluation results of MoME and other MLLMs with similar resource consumption on popular VI. tasks in Table 3 The results show that our MoME method achieves promising outcomes on most datasets compared with other generalist and MoE MILMss, especially on TextCaps, Flicker30K, and IconQA.
4.4 Qualitative Analisys
We present several visualized examples of our MoME model from distinct dataset groups along with their MoVE and MoL. routing results in Fig. 6 the REC case, DINOv2 accounts for nearly 50% among vision experts, providing fine-graines visual information. So the model can recognize the blue

---

Table Comparison with state-of-the-art MLLMs with similar resource consumption. MoME
achieves superior performances most datasets and is capable of a broader range of VL tasks.
Doc
Chart
Text
Text
Flickr 30K
Icon
Ref
Model
VSR
GQA
VQA
QA
Caps
VQA
QA
COCO
Shikra-7B
80.2
Ferret-7B 68
82.5
IBLIP 15
50.7
82.4
43.1
54.3
49.2
LLaVA-v1.5 35
58.2
62.0
LION 5
108.8
87.4
54.89
73.8
51.6
DocPedia 19
47.1
46.9
60.2
MoE-LLaVA 32
50.2
61.1
MixLoRA [58
40.0
51.2
MoCLE 20
57.1
81.9
46.3
64.7
49.3
LLaVA-
30.0
42.4
MoME
50.8
57.2
130.8
53.2
94.6
61.4
61.9
59.7
83.2
REC
Document Question
Question the given image, can you find the blue car right and lel me its cocrdinates?
Which the micro blogging social site that limits each past 140 characiers?
Answer:
Twitter
383, 180.1.0,0..0..........000... 1.0.0.899)
General
Question
words, what the gnated
Please describe this image in short.
j0.516,0.80,,0 w08.8.0227.8.... 0.864 image apart?
Answer
man a hoodle and a child in grean jacket standing sidewalk the
laying on the grass yellow tag around neck.
Figure 6: Visualization of samples along with their routing result distributions. The MoVE distributions on the left represent Pix2Struct, DINOv2, and CLIP-ViT from top to bottom. MoLE is on the bottom, with different colors indicating different experts.
in the background and provide its precise bounding box. The Pix2Struct branch accounts for 70% in the Document case for structured text understanding. The REG case utilizes information from both the CLIP-ViT VIT and DINOv2 locate objects and generate captions. In contrast, the conventional caption task in the General group only requires an image-level perception, so the CLIP-VIT is dominant. Remarkably, we can observe significant differences among the MoLE routing results. These examples show how MoME selects vision and language experts to adapt to various tasks.
Conclusion
This work investigates task interference when training a generalist MLLM across various VL tasks. To mitigate it, we propose MoME, which specializes both vision and language modality to adapt to task differences. Extensive experiments validate the efficiency of MoME generalist MLLM.
However, due to the limitations of computing resources, We have not yet expanded our approach to more data and more modalities for experiments. Nonetheless, we believe that the proposed MoME is versatile and can applied to constructing generalist models in wider range of multimodal domains. We hope MoME will inspire research in general- -urppoe multimodal AI and its applications.

---

6 Acknowledgement
This study is supported by National Natural Science Foundation of China (Grant No. 62306090, No. 62476071, I, No. 62236003), Natural Science Foundation of Guangdong Province of China (Grant No. 1024A1515..0147), and Shenzhen College Stability Support Plan (Grant No.
GXWD0020081144422000
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmirt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[2] Harsh Agrawal, Karan Desai, Yufei Wang. Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning scale. In Proceedings of the IEEF/CVF international conference on computer vision, pages 9948-8957, 2019.
[3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick. and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425 2433, 2015.
[4] Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291 4301, 2019.
[5] Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nie. Lion: Empowering multimodal large language model with dual-Level visual knowledge. In CVPR, 2024.
[6] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krish- namoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning. arXiv preprint 13X1v:2310.09478, 2023.
[7] Keqin Chen, Zhao Zhang, Weili Zeng. Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's referential dialogue magic. arXiv preprint (.V::2366.11995 2023.
[8] Shaoxiang Chen, Zequn Jie, and Lin Ma. Llava-mole: Sparse mixture of lora experts for mitigating data confficts in instruction finetuning mllms. arXiv preprint arXiv:2401.1616, 2024.
19] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664-16678 2022.
[10] Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv: 1909.02164, 2019.
[11] Xinlei Chen, Hao Fang. Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 1504.003355 2015.
[12] Zeren Chen, Ziqin Wang, Zhen Wang, Huayang Liu, Zhenfei Yin, Si Liu, Lu Sheng, Wanli Ouyang, Yu Qiao, and Jing Shao. Octavius: Mitigating task interference in milms via lora-moe. In ICLR, 2024.
[13] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%6* chatgpt quality, March 2023.
[14] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseckmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprini 24l.:2400000666 2024.
[15] Wenliang Dai, Junnan Li, Dongxu Li. Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructhlip: Towards general-purpose vision-language models with instruction tuning, 2023.
[16] Chuntao Ding, Zhichao Lu, Shangguang Wang, Ran Cheng, and Vishnu Naresh Boddeti. Mitigating task interference in multi task learning via explicit task routing with non- learnable primitives. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7765, 2023.
10

---

[17] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang- and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, pages 19358-19369 2023.
[18] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
[19] Hao Feng, Qi Liu, Hao Liu, Wengang Zhou, Houqiang Li, and Can Huang. Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding. arXiv preprint
2023.
[20] Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Dit-Yan Young. James T Kwok, and Yu Zhang. Mixture of cluster-conditional lora experts for vision-language instruction tuning. arXiv preprint arXi::2312.12379 2023.
[21] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), arXiv preprint arXiv: 1etti16060881111 2016.
[22] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models, arXiv preprint arXiv:2106.08 09685, 2021.
[23] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real visual reasoning and compositional question answering. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[24] Robert A Jacobs, Michael I Jordan. Steven Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
[25] Fric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint ar 1.01 144, 2016.
[26] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint arXiv:2401.000 240,08800,04888, 2024.
[27] Dongsheng Jiang, Yuchen Liu, Songlin Liu, Xiaopeng Zhang, Jin Li, Hongkai Xiong, and Qi Tian. From clip to dino: Visual encoders shout in multi-modal large language models. arXiv preprint 2310022310.00885,, 2023.
[28] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg, Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNIP), pages 787-798, 2014.
[29] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Yen Lo. et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[30] Kenton Lee, Mandar Joshi, Tulia Raluca Ture, Hexiang Hu, Fangyu I Liu, Julian Martin Fisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Sei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference Machine Learning, pages 18893-18912. PMLR, 2023.
[31] Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-1: Hybrid multimodal memory empowered agents excel morizon tasks. In NeurIPS, 2024.
[32] Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-Ilava: Mixture of experts for large languaee models, arXiv preprint arXiv:2401. 15947, 2024.
[33] Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for modal large language models. arXiv preprint MxXi::2311.0777. 2023.
[34] Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635-651 2023.
[35] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, arXiv preprint arXiv:2310.03744.200 2023.

---

[36] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[37] Jingyu Liu, Liang Wang, and Ming- Hsuan Yang. Referring expression generation and comprehension via attributes. In Proceedings of the IEEE International Conference on Computer Vision, pages 4856-4864, 2017.
[38] Ilya Loshchilov and Frank Hutter. arXiv:1711.05101 2017.
Decoupled weight decay regularization. arXiv preprint
[39] Haoyu Lu, Wen Liu, Bo Zhang. Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseck-vl: Towards real-world vision-language understanding. arXiv preprint 20Xv:22033005523 2024.
[40] Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:21 132144 2021.
[41] Kevis-Kokitsi Maninis, Lija Radosavovic, and Iasonas Kokkinos. Attentive single-tasking multiple tasks. In Proceedings of the JEEE/CVF Conference on Compuder Vision and Pattern Recognition, pages 1851-1866 2019.
[42] Junhua Mao. Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy, Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference computer vision and paltern recognition, pages 1 11-20, 2016.
[43] Junhua Mao. Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy, Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1 10-20, 2016.
[44] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok- A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/evf conference on computer vision and pattern recognition, pages 3195-3204- 2019.
[45] Ahmed Masry, Do Xuan Long. Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.1023 10244, 2022.
[46] Minesh Mathew, Viraj Bagal, Rubèn Tito. Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pages 1706, 2022.
[47] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on upplications of computer vision, pages 2200-2209, 2021.
[48] Brandon McKinzie, Zhe Gan, Jean- Philipp Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng. Floris Weers. al. Mml: Methods, analysis insights from multimodal Ilm pre-training. arXiv preprint X40:::40330096/11 2024.
[49] Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, and Pratyush Kumar. Plotga: Reasoning over scientific plots. The IEEE Winter Conference on Applications of Computer Vision (WACV), March 2020.
[50] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacddin El-Nouby, et al. Dinov2: Learnnng robust visual features without supervision. arXiv preprint WxXiv:230.007193 2023.
[51] Panupong Pasupat and Percy Liang. Compositional semantic parsing semi-structured tables. arXiv preprint arXiv: 500.00000000 001555000000000 2015.
[52] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8766 8763. PMLR, 2021.
[53] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084, 2019.
[54] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pages 146-162 Springer. 2022.
12

---

[55] Rui Shao, Xiangyuan Lan, Jiawei Li, and Pong C Yuen. Multi-adversarial discriminative deep domain generalization for face presentation attack detection. In Proceedings the IEEE/CVF conference on computer vision and patter recognition, pages 10023-10031, 2019.
[56] Rui Shao, Tianxing Wu, and Ziwei Liu. Detecting and grounding multi-modal media manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6904-6913. 2023.
[57] Rui Shao, Tianxing Wu, Jianlong Wu, Liqiang Nie, and Ziwei Liu. Detecting and grounding media manipulation and beyond. JEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[58] Ying Shen. Zhiyang Xu, Qifan Wang. Yu Cheng, Wenpeng Yin. Lifu Huang. Multimodal instruction tuning with conditional mixture of lora. arXiv preprint 1.XXi::2402118966 2024.
[59] Oleksii Sidorov, Ronghang Hu, Mareus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part II 16, pages 742-758 Springer, 2020.
[60] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages $317-8326, 2019.
[61] Stacey Svetlichnaya. Deepform: Understand structured documents at scale, 2019.
[62] Shengbang Tong. Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. arXiv preprint arXiv:2401.06200 06229, 2024.
[63] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. J 13971, 2023.
[64] Simon Vandenhende. Stamatios Georgoulis, Wouter Van Gansbeke, Mare Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE transactions pattern analysis and machine intelligence, 44(7):3614 0113331146666 4-33,,, 2021.
[65] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus based image description evaluation. Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4566 4575, 2015.
[66] Jiabo Ye, Aawen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:2310.05126 2023.
[67] Qilang Ye, Zitong Yu, Rui Shao, Xinyu Xie, Philip Torr, and Xiaochun Cao. Cat: Enhancing multimodal large language model to answer questions in dynamic audio-visual scenarios. In ECCV, 2024.
[68] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du. Bowen Zhang. Zirui Wang, Liangliang Cao, Shih-Fu Chang. and Yinfei Yang. Ferret: Refer and ground anything any anyhhere at any granularity. arXiv preprint arXiv:2310.07704, 2023.
[69] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[70] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEF/VV International Conference on Computer Vision, pages 11975-11986, 2023.
[71] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXivv2010.04159, 2020.
13

---

A Implementation Details
A.1 Architecture Details
The MoME includes three off-the-shelf vision encoders: CiTP-ViT, DINOv2. and Pix2Struct. For CLIP-VIT, we use ViT-G/14 from EVA [17] without the last layer. The DINOv2 [50] is the official pre- trained version ViT-L/14 without registers. For Pix2Struct, we use the vision branch of pre-trained Pix2Struct-base model [3) The ADT consists of 2D adaptive average pooling layer and a six-layer deformable attention network. The hidden size of ADT the same as its corresponding vision encoder. We use 8 attention heads and each head samples 4 points in the deformable cross-attention process. MoLE, hidden dimension each adapter set to 64, use Vicuna-v1.5(7B) as our pre-trained LLM.
A.2 Training Details
Our training process comprises two stages, stage we train the model with MoVE and a single adapter in LLM for 30k steps with batch size of 64. The learning rate is warmed up linearly from 0 to 5e-4 across 1000 steps and then reduces to minimum optimizer is employed with 31 0.9, $2= 0.999, and weight decay of 0.05, In stage 2, we load the checkpoint of stage and replicate the weights of adapters to initialize MoLE, while keeping everything else unchanged.
using cosine decay. The Adam (38)
We use a single node with A800 80GB for all experiments, the entire training is done in one day including stage and stage 2.
B Details of Multitasking Benchmark
We collected 24 datasets and categorized them into four groups for instruction-tune and evaluation, as shown in Fig. For most vision-language (VL) tasks, we used the datasets in both the training and evaluation phases. However, set, and we exclude the VSR training data due to its simplicity. During the training process, we mix the datasets within the same group into one large dataset, the probability of each sub-dataset being sampled equals their size as a proportion of the total. However, we ensure the sample ratio of each group dataset is the same. For evaluation,
only use NoCaps for evaluation because only has evaluation
compute the overall score for each category by
averaging its subitem evaluation results. We follow InstructBLIP (15), Shikra 77 and UReader [66] for evaluation metrics, which are tabulated in Table 5 Notably, the model only takes images as visual information without introducing OCR tokens like InstructBL.P.
General
REC
COCO
Flickr30K
NoCaps
RefCOCO
RefCOCO+
ReiCOCOg
OKVQA
VQAV2
AOKVQA
REG
VSR
IconQA
RefCOCO
RefCOCO+
RefCOCOg
Document
Train
ChartQA
TabFact
TextCaps
DeepForm
VisualMRC
Evaluation
DocVQA
InfographicsVOA
WikiTableQuestions
TextVQA
Evaluation
Figure 7: Multitask learning and evaluation datasets and their corresponding categories.
14

---

Table 4: Ablation studies deformable mechanism. "Gen." " and "Doc." " respectively denote average performances on General and Document. "Avg." means the average of the four group scores. The best performances are marked as bold.
Deformable Mechanism
Aggregation
Gen.
REC
REG
Doc.
Avg.
Router
75.27
78.17
61.64
42.46
64.39
Router
79.05
81.92
63.82
52.77
69.39
C Additional Ablation Experiments
To evaluate the effectiveness of the deformable mechanism of ADT, conduct ablation ex- periment by replacing the deformable cross-attention in MoME with standard cross-attention. As shown in Table 4 the model without deformable mechanism (#1) presents much worse performances than the original MoME (#2) consistently. We attribute this decline to the strong inductive bias of deformable attention in processing 2D feature maps.
D Additional Visualization and Analysis
We present the routing distributions across different datasets of all MoLE routers in Fig. 8 In general, the routing results differ significantly among different task groups, while the routing preferences are similar within the same task group. From the perspective of router preferences, the datasets can be classified text-rich (ChartQA Tex(Caps), caption (COCOCap Flickr30K), VQA (IconQA GQA), REC, and REG. It can prove that the MoLE captures the modularity of the tasks and mitigates task interferences through differential expert routing.
E Societal Impacts
MoME utilizes pre-trained large language models (LLMs), which inherently carry limitations from LLMs. These limitations include the potential for generating inaccurate information or biased outputs. To address these issues, we enhance the model's visual perception ability with MoVA and conduct vision- vision-language instruction tuning on high-quality datasets. Despite these improvements, we advise caution and recommend thorough safety and fairness assessments before deploying MoME models in any downstream applications.
15

---

Table 5: Summary of the evaluation datasets.
Task
Dataset
Split
Metric
COCOCap
karpathy-test karpathy-test
CIDEr 65 (t)
Flickr30K
CIDEr 65
NoCaps 2
val
CIDEr 65 (1)
OKVQA
val val
Accuracy(1)
General
AOKVQA [54]
Accuracy(t)
GQA 23
test-dev
Accuracy(f)
Visual Spatial Reasoning (VSR)
34
val
Accuracy(1)
IconQA 40
test
Accuracy(1)
RefCOCO
28
val & testA & testB Accuracy(
REC
RefCOCO+
28
val testA & testB
Accuracy (f)
RefCOCOg
43
val & test
Accuracy(
RefCOCO 28
val & testA & testB
CIDEr 65 (1)
REG
RefCOCO+
28
val testA & testB
CIDEr 65 651(1)
RefCOCOg
43
val & test
CIDEr (1)
ChartQA 45
test
Relax Accuracy [49 (1)
TabFact
[10
test
Accuracy(1)
DeepForm 61
test
Score
DocVQA 47
test
ANLS 4 (1)
Document
InfographicsVQA [46
test
ANLS 4 (1)
WikiTableeQeetiions (WTQ) 51
test
Accuracy(1)
TextCaps (59)
val
CIDEr (65
TextVQA [60
val
Accuracy(1)
16

---

Expert 0
Expert
Exper1
Expert
Layer
Expert
Expert
Experi
Expert
Layer
100%
73%
29%
28
1009 1
Layer
Layer
30%
2354
1000
Layer
Layer
255
5004
237
Layer
Layer
2044
Layer
Layer 10
layer 11
Layer 12
Layer 13
Layer 14
230
Layer 15
Layer 16
%0
Layer Layer 17
Layer 18
%0
Layer 1ayer11
L 20
Layer
Layes
1000
Leyer 23
Layer 24
Layer 25
Layer 26
2006
Layer 27
Layer 28
Layer 29
Layer 30
10001
Layer 31
Layer 32
Councol
abssact
Wna
Reecaao
coccccap
Recopp
BiK
DRVO4
ADRYZA
REC
Chanti
Blstect
Crstform
Infage
WnG Rereiii
TentCant
rococap
orvoz
ADKVIDA
0034
RBC
Name
Figure 8: Distribution of all language experts routing results across all tasks.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of work are discussed in the Conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
The formulas are numbered and refer-eeferenced correctly in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We fully and clearly describe the model architecture in the main body, and provide detailed training process and model configuration in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [No]
Justification: We use publicly available datasets for training and evaluation, which enables easy reproducibility. Upon acceptance, we will release the code and data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
public/guidese https//ooooouaasssssooooooooo
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips //nips.c//pil wccppublicgguuuiecceeeeeee.oo.ee
http://ooSSuuiiss..eoeee Stt.SSbbbiiiioooiiiiii.io.. for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide detailed training and test configuration details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: MLLM researches typically do not test and report error bars of experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http::
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss societal impacts in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We describe safeguards about the pre-trained language model in the Appendix.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification: The creators and original owners the assets used in the paper are properly credited and explicitly mentioned with respect.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We will release our code with detailed documentation the future.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: we do not conduct any crowdsourcing experiments this item is not applica- ble.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: we do not conduct any experiments with human subjects so this item is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "Ilib2fDjMS", "reviewer": "Reviewer_1g6K", "summary": "In this paper, the authors introduce a mixture of multimodal experts (MoME) to reduce task interference and develop a generalist MLLM. MoME consists of two main components: a mixture of vision experts (MoVE) and a mixture of language experts (MoLE). MoVE can adaptively adjust features transformed from different vision encoders and boasts strong compatibility with various transformation architectures. MoLE integrates sparsely gated experts into LLMs, achieving seamless improvements while keeping inference costs nearly unchanged.", "questions": "One advantage of MoE is its fast inference speed. Could you conduct an experiment to verify the model's inference speed?", "limitations": "yes, the author explains the limitations of their study and potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The analysis of task interference and mixture of vision experts in this paper is clear, highlighting the necessity of a vision mixture of experts\n2. The experiments are well-conducted and quite comprehensive\n3. The study demonstrates strong performance on most datasets compared with other generalist and MoE MLLMs", "weaknesses": "1. From Table 1, we can see that both ADT and Router have achieved notable improvements. Could you explain the internal mechanisms behind this? I am quite curious as to why the improvements are so significant.\n2. In Table 3, many values are missing. Could you add some results from more general multimodal benchmarks to make the experiments more comprehensive, such as MME, MMbench, MM-Vet, LLaVA$^W$, Science$^{QA}$, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Reasons for notable improvements in Table 1.\n\n**A1:** When mixing visual representations from different vision experts, they are first transformed into a unified-length sequence of feature vectors and then aggregated, each step of which will severely damage the visual information. \n\n- Transforming visual representations of different sequence lengths into unified lengths using downsample pooling will cause inevitable information loss since it’s rule-based and static.\n- Visual representations are in different feature spaces due to the diversity in data domains and training methods. They will interfere with each other if simply adding them together and cause information loss.\n\nThe proposed ADT and router are dynamic and learnable and can effectively mitigate information loss in both steps:\n\n- Transformation: The ADT module uses deformable attention to compensate for the information loss caused by downsample pooling.\n- Aggregation:  The dynamic router can adaptively aggregate features according to task demands, maximizing the retention of visual information appropriate to each task.\n\nConsequently, the proposed MoVE achieved notable improvement compared to the baseline in Table 1.\n\n---\n\n**Q2:** General multimodal benchmark results.\n\n**A2:** **Our MoME focuses on multitasking ability and has advantages in benchmarks that contain diverse types of tasks.** However, recent multimodal benchmarks (e.g. MMBench, MME) are primarily organized in a VQA style with multiple choice formats, having a rather limited scope of tasks and instruction templates. For example, all the instructions in MMBench are single-choice with fewer than four options and exhibit high similarity, which does not align with forms of human expression. \n\nIn contrast, we evaluate MoME on a multitasking benchmark that contains many types of tasks with diverse instructions and more closely resembles human instructions in a real-world environment. In the Table below, we compare MoME with LLaVA v1.5 on the Multitasking Benchmark. The original LLaVA model performs poorly on the Multitasking Benchmark since it is trained with a limited variety of tasks. To ensure a fair comparison, we retrain it using the same data and settings as MoME. While some improvements are observed, its performance remained significantly lower than ours. MoME exhibits a clear advantage in multitasking due to its effectiveness in mitigating conflicts.\n\n|  | Gen. | REC | REG | Doc. | Avg. |\n| --- | --- | --- | --- | --- | --- |\n| LLaVA-v1.5-7B (Original) | 69.78 | 46.40 | 71.85 | 22.06 | **52.52** |\n| LLaVA-v1.5-7B (Retrained) | 75.04 | 61.42 | 58.79 | 30.84 | **56.52** |\n| LLaVA-v1.5-7B (w/ DINO&PixStruct) | 70.36 | 74.89 | 57.55 | 32.83 | **58.91** |\n| MoME-7B | 79.65 | 81.58 | 64.83 | 53.69 | **69.94** |\n\n---\n\n**Q3:** Inference Speed.\n\n**A3:** The typical MoE in LLMs has a fast inference speed because of its sparse activation mechanism. However, there are other types of MoE that pursue performance improvements by mitigating the task conflict rather than efficiency [1,2]. Inspired by these works, we designed MoVE and MoLE for better performance with only a slight increase in parameters. Specifically, MoVE and MoLE result in just a 1.36% and 0.668% increase in parameters, and a 1.076% and 5.959% inference speed.\n\n[1] Ye, Qinyuan, Juan Zha, and Xiang Ren. \"Eliciting and understanding cross-task skills with task-level mixture-of-experts.\".\n\n[2] Zadouri, Ted, et al. \"Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning.\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d2q2X7tYV4", "reviewer": "Reviewer_cdtp", "summary": "The paper proposed a MoE design for MLLM, it utilize MoE in both visual encoding procedure and LLM decoding procedure. The paper utilized a dynamic routing module to mix visual features from different experts, and adopted a multi-adapter structure to combine the knowledge of differnent language experts. Widely conducted experiments showcase the effectiveness of the proposed method on various downstream tasks.", "questions": "1.\tWhat is the differnece between this work and the previous works discussing MoE in vision (the several papers I listed in the weaknesses section), how can this paper brough new knowledge or designs that is perviously unknown or not sufficiently expored by the community?\n2.\tHow does the MoE structure affects the inference speed, are there any quantitative results?", "limitations": "The authors already adequaately addressed the potential negative societal impact.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The paper widely conducted quantitative results on several multimodal downstream tasks to prove the effectness of the proposed MoE design. The paper also adds qualitative results to intuitively showcase the performance.", "weaknesses": "The design proposed in this paper seems to be not innovative enough. Its key design philosophy has already been brought out in many previous works. As for MoE in vision, the CVPR 2024 paper “Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs” combines CLIP and DINOv2 features to improve visual representations, the arxiv 2023 paper “SILC: Improving Vision Language Pretraining with Self-Distillation” combines the learning objectives of CLIP and DINOv2 for better pretraining outcome. Moreover, the references [12, 5, 8, 54] cited in this paper has already investigated MoE in LLMs, as stated in the introduction section. \nIt is already a consensus that MoE in MLLM, especially combining multimodal pretrained feature (such as CLIP) and pure visual pretrained feature (such as DINOv2), can improve model performance. Combining a third feature to improve performance (in this paper, the feature from Pix2Struct) besides CLIP and DINOv2 is a pure engineering design, which could not add up to the technical contribution of this paper. Moreover, the ‘Dynamic Router’ proposed in this paper is also a simple MLP network which is widely used for combining different feature. To sum up, it is hard to claim all these designs as a contribution of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Key design philosophy has already been brought out in many previous works.\n\n**A1:** The motivation and capabilities of MoVE are completely different from the previous work.\n\n1. “**Eyes Wide Shut**” introduced an additional DINO encoder and chose to interleave visual features, which ended up with redundant visual features of twice the length. In contrast, we utilized the MoE technique to dynamically aggregate features and keep the token length unchanged, which is more efficient and adaptive.\n2. “**SILC**” combined the learning objectives of CLIP and DINOv2 by self-distillation to improve image-text contrastive learning. It did not focus on multitasking nor did it use the MoE technique. In contrast, we concentrated on multitasking MLLMs, innovatively utilizing the MoE technique to leverage pre-trained vision encoders and effectively mitigate interference among them.\n\nTherefore, we believe that our MoVE is a new and novel design that is distinct from previous works.\n\n---\n\n**Q2:** It is already a consensus that MoE in MLLM, especially combining multimodal pre-trained features (such as CLIP) and pure visual pre-trained features (such as DINOv2), can improve model performance.\n\n**A2:** It is a consensus that combining different visual features can improve model performance. However, combining them in a MoE style has not been explored, which is not trivial. Existing MLLM works have demonstrated combining two or more vision encoders can significantly boost the visual perception ability of MLLM, but they just use simple addition or interleave [1]. In our MoVE, we aim to dynamically and adaptively aggregate these visual features according to task demands. Table 1 in the manuscript shows the significant improvement of our MoVE compared to simple addition.\n\n[1] Tong, Shengbang, et al. \"Eyes wide shut? exploring the visual shortcomings of multimodal llms.\"\n\n---\n\n**Q3:** Combining Pix2Struct features is a pure engineering design.\n\n**A3:** Introducing Pix2Struct is not an engineering design. In contrast, it reveals several representative problems including aggregating features with different shapes and sizes and encoders focusing on different data domains. As stated in the manuscript, the aspect ratios of Pix2Struct feature shapes vary depending on the input image and it focuses on text-rich images. Experiments in Table 1 have shown that simply pooling and aggregating these diverse features can lead to severe interference. Instead, we proposed MoVE to adaptively transform and aggregate different kinds of features from CLIP (224x224), DINOv2 (448x448), and Pix2Struct (arbitrary shape) and achieved significant improvement. The ability to handle features of arbitrary shape makes MoME more versatile and valuable.\n\n---\n\n**Q4:** ‘Dynamic Router’ is a simple MLP.\n\n**A4:** The ‘Dynamic Router’ is not just a simple MLP; its significance lies in its working mechanism rather than its internal structure. Its working mechanism allows MoVE to dynamically and adaptively aggregate visual features, greatly reducing interference and improving performance by over 6 points, as shown in Table 1 of the manuscript. Furthermore, existing work [1] has verified that the internal architecture has a negligible effect on performance.\n\n[1] Ye, Qinyuan, Juan Zha, and Xiang Ren. \"Eliciting and understanding cross-task skills with task-level mixture-of-experts.\"\n\n---\n\n**Q5:** Differences from previous works & New knowledge or designs.\n\n**A5:** Compared to multi-modal large language models that are equipped with more than one vision encoder, MoME revealed the severe conflicts among visual features and proposed an efficient and novel method to tackle the interferences within both the transformation and aggregation process. Compared to works that simply employ typical MoE design into vision encoders, we deeply explore the dilemma of multi-modal large language models and propose a framework that can benefit from a variety of pre-trained models while avoiding interferences among them.\n\nIn this work, we highlighted the task interference problem in both the textual and visual information, while previous works only investigated MoE in LLMs and primarily concentrated on textual differences between tasks, overlooking the equally important visual information.\n\n---\n\n**Q6:** Quantitative results of Inference speed changes.\n\n**A6:** For MoVE, the number of trainable parameters is 112.83M, accounting for 1.36% of the total parameters and increasing inference time by 1.076%.\n\nFor MoLE, we use lightweight adapters as our experts, which only increase the parameters by 0.668% and inference time by 5.959%.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q9Lr2VrlJX", "reviewer": "Reviewer_ZwrT", "summary": "The paper proposes to mitigate task interference during multimodal instruction tuning with a mixture of experts, in both the language and the encoder side. The paper is well written, contains insightful analysis and shows improvements over baselines.", "questions": "Please the weaknesses section (e.g. 4-5-6)", "limitations": "Some limitations and societal impacts are discussed in the paper or the appendix.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Mixture of experts is an important topic that is heavily studied in LLMs but little in multimodal models.\n- The problem of task interference is an important problem that is more present in a multimodal setting.\n- The paper is well written and easy to follow.\n- The proposed approach is well motivated with insightful analysis and show improvements over baselines", "weaknesses": "1. While the paper shows improvements over different baselines in Tab.3, the scores are still lagging behind, compared to methods without MoE, such as LLaVA (which the paper builds on top of). Knowing that the proposed approach have significantly more parameters, pretrained for more steps (stage 1) and use different visual encoders.\n2. The paper claims that “This means the language experts in our MoLE module gradually specialize in distinct task domains during training.” However the visualization does not support that. For instance, in Fig.5  E4 is used in most tasks types, while E1 is relatively less used. But it is not clear if we can map different experts to different task groups.\n3. The paper is not evaluated on common and recent multimodal benchmarks: SEED, MME, MM-Vet, POPE, VQAv2 … that are considered in other methods like LLaVA. \n4. The design of MoE are very different for the LLM and visual encoders. Each visual encoder is considered as an expet. Did the authors experiment with typical MoE (replicating FFN) in a single visual encoder?\n5. The papers used deformable cross-attention, but I did not find any experiment to support this design choice, compared to using simple cross-attention. Did the authors conduct this experiment?\n6. The paper states that load balancing did not help. Any insights on why this is the case? knowing that these losses are typically used in most MoE papers. Also, did the authors encounter any instabilities during training? It is important to discuss these in the paper, as instability is a major problem in MoEs training. \n7. The paper main novelty seems in applying MoE also in the vision encoders. This question is also explored in previous works with slightly different context such as [1] and [2]. I found the novelty limited in this regard.\n\n[1] Mustafa, Basil, et al. \"Multimodal contrastive learning with limoe: the language-image mixture of experts.\" Advances in Neural Information Processing Systems 35 (2022): 9564-9576.\n\n[2] Shen, Sheng, et al. \"Scaling Vision-Language Models with Sparse Mixture of Experts.\" Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Lag behind LLaVA with more parameters, vision encoders, and training time.\n\n**A1:** The original LLaVA includes much fewer tasks than our MoME for training and evaluation, we reported only the results on the shared tasks in Table 3 of the manuscript. **Thus, it is improper and inaccurate to conclude that our MoME lags behind LLaVA.**\n\nIn the following, we evaluate the original LLaVA model on our Multitasking Benchmark (which contains many types of VL tasks) for a fair and comprehensive comparison. Meanwhile, we retrain both the original LLaVA model and LLaVA with additional DINO and Pix2Struct encoders using the same datasets and settings as MoME. The results are summarized below:\n\n|  | Gen. | REC | REG | Doc. | Avg. |\n| --- | --- | --- | --- | --- | --- |\n| LLaVA-v1.5-7B (Original) | 69.78 | 46.40 | 71.85 | 22.06 | **52.52** |\n| LLaVA-v1.5-7B (Retrained) | 75.04 | 61.42 | 58.79 | 30.84 | **56.52** |\n| LLaVA-v1.5-7B (w/ DINO&PixStruct) | 70.36 | 74.89 | 57.55 | 32.83 | **58.91** |\n| MoME-7B | 79.65 | 81.58 | 64.83 | 53.69 | **69.94** |\n\nThe table shows that **MoME (avg. 69.94 points) has significant advantages over the LLaVA model (Original avg. 52.52 points & Retrained avg. 56.52 points) on the multitasking benchmark.** Additionally, simply adding encoders to LLaVA (avg. 58.91 points) results in limited improvement and lags significantly behind MoME with the same vision encoders, training time, and nearly the same number of parameters. The substantial performance gains of MoME are achieved through the proposed ADT and Dynamic Router, which adaptively mitigate interference.\n\n---\n\n**Q2:** Visualization of MoLE.\n\n**A2:** Since each LLM layer contains an independent MoLE block, there is no relationship between experts in different layers. Therefore, the statement that \"E4 is used in most task types, while E1 is relatively less used\" might be misunderstood, as E1 has different meanings across layers. Instead, from  Fig. 5 we can see that the utilization of experts varies significantly across different tasks, indicating the specialization of language experts. For example, REC and REG tasks primarily use E2 in layer 12, while TabFact uses E1, and AOKVQA uses both. \n\nMoreover, from Fig.8, we can observe that the routing results differ significantly among different task groups, while the routing preferences are similar within the same task group, which means different experts specialized in different task groups. We have included an excerpt of Fig. 8 in the rebuttal supplement material, from which we can see clear differences among text-rich (ChartQA - TextCaps), caption (COCOCap - Flickr30K), VQA (IconQA - GQA), REC, and REG tasks.\n\n---\n\n**Q3:** Multimodal benchmark results.\n\n**A3: Our MoME focuses on multitasking ability and has advantages in benchmarks that contain diverse types of tasks.** However, recent multimodal benchmarks (e.g. MMBench, MME) are primarily organized in a VQA style with multiple choice formats, having a rather limited scope of tasks and instruction templates. For example, all the instructions in MMBench are single-choice with fewer than four options and exhibit high similarity, which does not align with forms of human expression. \n\nIn contrast, we evaluate MoME on a multitasking benchmark that contains many types of tasks with diverse instructions and more closely resembles human instructions in a real-world environment. As seen in Table of **A1**, MoME shows significant performance improvements over LLaVA original, LLaVA retrained on Multitasking Benchmark, and LLaVA with more visual encoders.\n\n---\n\n**Q4:** Design of MoVE (typical MoE)\n\n**A4:** The MoVE module is designed to leverage powerful, off-the-shelf pre-trained vision encoders each specialized in a specific domain, which are more suitable for multi-modal large language models that are designed to be general and versatile.\n\nMoreover, fine-tuning vision encoders was proven by [1] to be resource-consuming and often led to bad results on usually small-scale VL instruction datasets. Consequently, we chose to freeze the pre-trained vision encoders and regard them as vision experts.\n\n[1] Wang, Guangzhi, et al. \"What Makes for Good Visual Tokenizers for Large Language Models?.\"\n\n---\n\n**Q5:** Experiments supporting deformable cross-attention.\n\n**A5:** We supplemented an ablation experiment by replacing the deformable cross-attention in ADT with cross-attention, which can be found in Table 1 of the rebuttal supplement material. \n\nThe table shows MoVE with cross-attention (Avg. 64.39 points) presents much worse performances than ADT (Avg. 69.39 points) consistently. We attribute it to the powerful inductive bias of deformable attention in processing 2D feature maps. We will add this experiment in the revised version.\n\n---\n\n**Q6:** Load balancing and instabilities of MoLE.\n\n**A6:** We did not encounter instabilities in the training process of MoLE so the load balancing loss did not help. We infer that this is because the lightweight design of adapters makes the MoE block stable. The same phenomenon was also recorded in [1].\n\n[1] Chen, Zeren, et al. \"Octavius: Mitigating task interference in mllms via moe.\"\n\n---\n\n**Q7:** Novelty limited.\n\n**A7:** The motivation, structure, and context of our method all have significant differences compared to previous works. “LIMoE” and “Scaling Vision-Language Models with Sparse Mixture of Experts.” both employ typical MoE into CLIP encoders to scale up and improve performances. In contrast, the experts of MoVE are different vision encoders each specialized in a specific domain. We proposed an effective way to make use of the powerful off-the-shelf pre-trained vision encoders and mitigate the interference among them. Moreover, we comprehensively explored task differences in both vision and language modalities and significantly enhanced the multitasking capability of multimodal large language models. Therefore, it can be argued that MoME is novel and provides many valuable insights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers (R#1g6k, R#cdtp, R#ZwrT) for their time and efforts in providing constructive feedback. We are very encouraged that reviewers found our work effective (R#1g6k, R#cdtp, R#ZwrT), with a clear analysis of task interference (R#1g6k), comprehensive experiments (R#1g6k, R#cdtp), and insightful analysis (R#ZwrT). We have built an official repository for providing well-structured open-source codes (released upon acceptance).

We have responded to all questions and comments in each review. Additionally, supplement material is included in the attached PDF to help clarify related concerns. We hope these responses provide a more comprehensive view of our paper. Please kindly consider increasing your rating if your concerns have been addressed.

Meta-review: This paper studies the problem of building a multimodal generalist by mixture of experts motivated by the observation of task interference. Existing MoE models tend to focus on the LLM side, but this paper investigates the task inference on the vision representation and proposes ADT and dynamic routing techniques to aggregate dis-similar visual representation. The ablations show that the proposed techniques are effective in improving a multimodal generalists across dis-similar task space. 

This paper receives divergent reviews with scores 6, 5, 4. All reviewers appreciate the comprehensive experiments which demonstrates the benefits of the proposed approach, and the analysis which are intuitive (cdtp), clear (1g6k) and insightful (ZwrT).  In addition, the importance of understanding multimodal task interference is noted by ZwrT. Considering the reviews and rebuttal, the main objection to acceptance is raised by reviewer cdtp who considers the novelty incremental relative to previous works such as “Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs”, "SILC: Improving Vision Language Pretraining with Self-Distillation" and other works using MoEs in LLMs. The author provided a detailed response and the reviewer adjusted the score accordingly, while maintaining that "the design philosophy of this work—mixture of vision expert—is pretty much similar to previous works" and leaning to reject.

Upon careful inspection of the works cited by cdtp and the author rebuttal, I notice that the two cited papers “Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs”, "SILC: Improving Vision Language Pretraining with Self-Distillation" are actually studying very different problems than this paper. The first paper focuses on probing the short comings of the existing VLMs and demonstrate effects of MoF (mixture of features) by simply adding or interleaving features on their benchmarks. The second paper proposes to add self-distillation objective to contrastive learning so that the model is capable of both global/local recognition. Thus, the similarity claimed by cdtp is not very convincing in my humble opinion. In addition, as a principle, I do not think the similarity in high-level design philosophy alone should be the ground for rejection. Having inspected the paper myself and other related multimodal MoE works, I do not think this justification of weak reject are well-supported by evidences.

Given that the other two reviews are positive, I'm leaning to recommend acceptance of this paper given the importance of the problem and the comprehensive analysis/experiments noted by all the reviewers. However, due to the lower number of positive reviews, I'd like us to have more discussion on this paper.
Decision: Accept (poster)