MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models

Authors: leyang  shen
gongwei  chen
rui  shao
weili  guan
liqiang  nie
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 13:13:59

Abstract: Multimodal large language models (MLLMs) have demonstrated impressive capabilities across various vision-language tasks. However, a generalist MLLM typically underperforms compared with a specialist MLLM on most VL tasks, which can be attributed to task interference. In this paper, we propose a mixture of multimodal experts (MoME) to mitigate task interference and obtain a generalist MLLM. Our MoME is composed of two key components, a mixture of vision experts (MoVE) and a mixture of language experts (MoLE). MoVE can adaptively modulate the features transformed from various vision encoders, and has a strong compatibility in transformation architecture. MoLE incorporates sparsely gated experts into LLMs to achieve painless improvements with roughly unchanged inference costs. In response to task interference, our MoME specializes in both vision and language modality to adapt to task discrepancies. Extensive experiments show that MoME significantly improves the performance of generalist MLLMs across various VL tasks.

Main paper: 
 introduction: 
 Recently, Multimodal Large Language Models (MLLMs) [39, 35, 48, 31, 67] have witnessed remarkable progress. With the help of Large Language Models (LLMs) [13, 63, 1] and Modality Encoders [50, 30, 17, 52, 70] , MLLMs demonstrate excellent multimodal comprehensive abilities, especially in solving a wide range of vision-language (VL) tasks [3, 42, 37, 55, 57, 56] , such as Image Cpation, Visual Question Answering, Referring Expression Comprehension, and Optical Character Recognition. However, it is increasingly acknowledged that a generalist MLLM tends to have lower performance compared to a specialist MLLM on most VL tasks [12, 8] , as depicted in Fig. 1 (a) . This phenomenon can be attributed to task interference, a fundamental and crucial issue in multi-task learning [16, 41, 64] .There are some preliminary attempts [15, 6, 12, 5, 8, 58] to address this issue in instruction-tuning MLLMs. The most promising direction [12, 5, 8, 58] among these attempts is to use a mixture of experts (MoE) in MLLMs, aiming for each expert to specialize in several tasks. However, these works only investigate MoE in LLMs and primarily concentrate on textual differences between tasks, overlooking the equally important visual information. As shown in Fig. 1 (b-c ), we analyze the distribution of various VL tasks across both vision and language modalities. It is evident that images from different groups of VL tasks exhibit distinct feature distributions, as do texts. Inspired by this observation, we argue that handling task interference needs to comprehensively exploit task differences in both vision and language modalities. To mitigate task interference, we devise a Mixture of Multimodal Experts (MoME) and integrate it into MLLMs. Our MoME consists of a Mixture of Vision Experts (MoVE) for adaptively aggregating features from various vision encoders [52, 17, 50] , and a Mixture of Language Experts (MoLE) for leveraging multiple sparsely-activated parameter-efficient adapters. To avoid feature mismatch in different vision encoders, we propose an adaptive deformable transformation (ADT) module in MoVE and use it to transfer features of vision encoders into a unified-length sequence of feature vectors. Our ADT module combines adaptive average pooling and deformable attention [71] to obtain compressed and self-enhanced visual features. After feature transformation, our MoVE uses an instance-level soft router to modulate and aggregate transformed visual features according to the instructions. Our MoLE introduces several parameter-efficient adapters [9] as experts and integrates them by using an instance-level sparsely-activated router. Due to the utilization of adapters, MoLE can be integrated into each feed-forward network layer of an LLM and only incurs a few computational costs with consistent performance gains.To comprehensively evaluate the multimodal understanding ability of MoME, we collect an amount of VL tasks to form an instruction-tuning dataset and split them into four groups. Extensive experiments show that both MoVE and MoLE can consistently improve performance across all groups of tasks. Notably, MoVE can achieve an average performance gain of 12.87 points across all VL tasks, and improve by over 20 points on the "Document" group. Furthermore, we visualize the expert load distributions of MoVE and MoLE across various tasks. The experts in both MoVE and MoLE exhibit a relatively clear specialization in different groups of VL tasks. For example, the "Document" group of tasks has a strong preference for the "Pix2Struct" vision expert. The expert specialization is strong evidence to demonstrate that our MoME dynamically selects experts to adapt to task differences and mitigate task interference.Our main contributions are summarized as follows:• We propose MoME by simultaneously designing mixtures of experts tailored for both the vision encoder and the LLM, resulting in generalist MLLMs with the ability to combat task interference. • Through statistical analysis, we demonstrate that our MoME specializes in both vision and language modality, effectively adapting to the varying requirements of different VL tasks. • Extensive experiments show that our MoME possesses excellent multimodal understanding abilities and achieves superior performances across various groups of VL tasks.2 Related WorkQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
vision encoders in mllms: 
 Vision encoders play important roles in the perception ability of recent MLLMs by encoding visual information into visual tokens, enabling LLMs to understand information on visual modalities. Most Multimodal Large Language Models (MLLM) use CLIP-ViT [52] as their vision encoder to provide the basic image-level perception of an image for LLMs, which is useful for tasks such as image caption and general VQA. However, Tong et al. [62] have found that CLIP-ViT struggles to encode some visual patterns, severely limiting perception and preventing MLLMs from becoming generalist.To alleviate this, recent works [33, 62, 27] integrated different vision encoders [52, 17, 50, 70, 29 ] into a single MLLM, which enhanced the perception of MLLM. However, none have effectively mitigated the interference among different visual features, resulting in sub-optimal utilization of each encoder. Differently, we explore the task differences and propose MoVE to perform self-enhanced transformation and adaptive routing among features from different encoders, achieving consistently high performances across vast VL tasks.Mixture of Experts (MoE) [24] is a type of structure with multiple expert networks working together, where each expert is responsible for a part of the knowledge space. By only activating specific experts adaptively during inference, MoE can reduce resource consumption and enhance reasoning speed, which is useful for LLM. Existing MoE LLM [18, 26, 14] usually replace the feed-forward network (FFN) with the MoE layer. Each MoE layer consists of a router and multiple expert networks and each token is assigned to several expert networks by the router. MoE LLMs tend to outperform dense models with the same inference activate parameters.In addition to its effectiveness in foundation models, some works [12, 8, 58, 20, 32] have utilized MoE in the visual instruction tuning [36] phrase of MLLMs to mitigate task interference, aiming for each expert to specialize in several tasks. Some of them replicate the original FFN in LLMs, while others insert multiple low-rank adaptation [22] modules in parallel with the original FFN, converting LLM into multi-expert architecture. However, they primarily concentrate on LLM but overlook task interference within the visual perceiving process of MLLMs. In contrast, we comprehensively exploit task interference in both vision and language modalities and propose MoME to mitigate them with specialized vision and language experts.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitations of our work are discussed in the Conclusion section. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methods: 
 To design a generalist MLLM with powerful multimodal understanding capabilities, we begin by analyzing task interference, a common challenge in current MLLMs (Section 3.1). Then, we propose our Mixture of Multimodal Experts and introduce its main components in Section 3.2.Task interference is a fundamental and crucial issue in multi-task learning. As current generalist MLLMs are trained with a number of tasks, they naturally suffer from this issue especially when the number of tasks increases. To investigate this issue in a scenario of MLLMs, we will analyze the feature distribution and the performance change of MLLMs on a tailored instruction-tuning dataset.To demonstrate the external manifestations of task interference, we first construct a mixed instructiontuning dataset with various VL tasks and split all tasks into four groups. The performance comparisons between MLLMs trained on the mixed dataset and each task group are illustrated in Fig. 1 (a). It is shown that a generalist model trained on the mixed dataset underperforms specialist models on three task groups. We conclude that the generalist model suffers from a notable task interference problem.In the era of Large models, there are some attempts to handle task interference from various perspectives, such as instruction, architecture, and dataset configuration. Here, we focus on the mainstream direction, architecture design, and try to explore a robust architecture to combat task interference. In terms of architecture, most existing works prefer a mixture of experts in LLMs. The experts can be feed-forward networks or parameter-efficient modules. However, we argue that this paradigm of architectural design is sub-optimal as visual and textual information should be given equal importance.In Fig. 1 (b-c ), we investigate the feature distribution of various task groups on vision and language modalities, respectively. All samples of each task group are fed into vision and text encoders to produce modality features. These features are transformed by using PCA and then visualized to show the distribution. We observe that the feature distributions of different task groups exhibit significant differences in both the vision and language modalities. As mentioned above, the textual differences can be addressed by multiple experts in LLMs, but visual differences lack effective handling. In the following section, we will introduce our Mixture of Multimodal Experts, which simultaneously handles visual and textural differences between tasks to mitigate task interference. Feat.Feat.Add & Norm As illustrated in Figure 2 , we present our novel MoME architecture that dynamically mixes vision and language experts. The proposed architecture aims to adaptively aggregate visual information (3.2.1) and select LLM pathways (3.2.2) based on the given instructions. Before introducing our MoVE architecture, we will present a pilot study that inspires us to design MoVE. Specifically, we design three MLLMs (consists of vision encoder, projection, and LLM), which share the same architecture except vision encoders. These three MLLMs use three distinct vision encoders, CLIP, DINOv2, Pix2Struct, respectively. After training them using the same data and strategies, we found that MLLMs with different vision encoders excelled in specific tasks, as presented in Fig. 3 . the MLLM with CLIP-ViT is good at general tasks and regional caption tasks. the MLLM with DINOv2 excels in REC, a visual grounding task. the MLLM with Pix2Struct is outstanding in text-intensive document understanding tasks. However, each model had weaknesses and none could achieve uniformly excellent performance across all tasks.Inspired by the above study, we propose MoVE to combine various off-the-shell vision encoders and effectively align and aggregate their visual features. The key components in MoVE are an adaptive deformable transformation module and an instruction-based soft router. The former aims to align the features from various vision encoders, and the latter seeks to modulate and aggregate transformed features based on the instructions.Adaptive Deformable Transformation. Given the diversity in architecture and training methods of different vision encoders, the issue of mismatched visual representations in terms of sequence length and feature space becomes significant. While current researches [62, 27, 33] often focus on models like CLIP-ViT [52] and DINO [50] , which share similar data processing pipeline, the mismatch problems are less important and frequently overlooked. However, the aspect ratios of Pix2Struct [30] feature shapes vary depending on the input image. Simply combining them through padding and addition will lead to the misalignment among visual tokens and the damage of visual information. To tackle this challenge, we innovatively design an adaptive deformable transformation (ADT) module that effectively transforms features from diverse vision encoders f into a unified-length feature sequence f , ensuring more coherent and informative visual representations for subsequent aggregations.As illustrated in Fig. 2 (a), the ADT module consists of a 2D adaptive pooling layer and M deformable attention layers (D). It attempts to automatically select the corresponding information from original features f to refine the coarse features obtained by 2D adaptive pooling D(f ),EQUATIONInspired by Deformable DERT [71] , we choose deformable cross-attention for its 2D sampling mechanism, which is ideal for interactions among visual feature maps of varying shapes. Meanwhile, it converges fast and has computational and memory efficiency. Specifically, each deformable layer consists of a self-attention layer, a deformable cross-attention, and a feed-forward layer. The crucial select operation occurs in the deformable cross-attention layer, which takes the output of the upper self-attention as query q ∈ R L×C , samples the original feature map f ∈ R H×W ×C , and outputs result O ∈ R L×C . In this module, the first step is to generate attention weights w ∈ R L×N h ×Np and L sets of 2D sampling points, denoted as p, from the input queries q. For each set, there are N h × N p points, where N h signifies the number of attention heads and N p represents the number of points sampled by each head. The sample points and attention weights generation process is as follows,EQUATIONwhere P denotes the linear projector and R ∈ R L×2 is a learnable vector called reference point. Then, the corresponding information is sampled by attention heads from the value feature maps P(f ) j projected and split on the last dimension for each head. The sampling mechanism of each attention head is as follows,EQUATIONThe results of multiple attention heads are concatenated and projected to obtain the output feature O, which is the input of subsequent feed-forward layer.EQUATIONInstance-level Soft Router. Since images from different groups of VL tasks exhibit distinct feature distributions, there is no one-fits-all strategy to aggregate them. To address this, we propose to generate a customized fusion ratio for each sample based on its instruction.Specifically, we devise an instance-level soft router G s , as depicted in Fig 2 (b ). The router generates corresponding ratios for the visual representations from different vision encoders, followed by a weighted sum of these visual representation features f , which can be formulated as,EQUATIONEQUATION)where N is the number of vision experts and σ denotes GeLU [21] . The router is a multilayer perceptron (MLP) followed by a SoftMax layer to ensure that the sum of the weights equals 1. The instruction is first passed through Sentence-BERT [53] to extract sentence embedding I, which is then fed into the router. We insert an MoE block parallel to each FFN layer in LLM. As depicted in Fig. 2 (c), The MoE block consists of several low-rank adapters and an instance-level sparsely-activated router G h . The adapter is designed as a bottleneck structure for computational efficiency, featuring a down-projection layer P down , a ReLU layer σ, and an up-projection layer P up . Moreover, a learnable scalar s is multiplied in the output to weigh the importance adaptively. The whole low-rank adaptation process is as follows, y = s • P up (σ(P down (x)))The router is an MLP network followed by a top-1 gate function to ensure the output is a one-hot vector G(I) ∈ {0, 1} K . The router generates the selection based on the sentence embeddings I used in MoVE. Each sample is routed to the corresponding adapter to calculate the adapted value o, which can be further added to the output of the original FFN. The whole process of the MoLE block is as follows,EQUATIONwhere K denotes the number of experts.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [No] Justification: We use publicly available datasets for training and evaluation, which enables easy reproducibility. Upon acceptance, we will release the code and data.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental results and analysis: 
 We collect 24 datasets and categorize them into four groups for instruction-tuning and evaluation, the details of which can be found in Appendix B.We conduct experiments on MLLMs with different vision encoders under the same training strategy to verify the effectiveness of the two key components in MoVE: ADT and router. Experimental results are summarized in Table 1 . We take the multitasking performances of MLLMs with a single vision encoder as our baselines. The adaptive average pooling is applied to the visual representation from DINO and Pix2Struct, ensuring that the lengths of visual tokens fed into LLM are consistent. Experiments #1-3 show that MLLMs using CLIP, DINO, and Pix2Struct as vision encoders exhibit distinct strengths in General, REC, and Document tasks, respectively. Moreover, in the REG task, which requires both captioning and visual grounding abilities, the performance of MLLMs with CLIP and DINO significantly surpasses that of those using Pix2Struct. We can conclude that a single vision encoder cannot meet the visual perception needs of all tasks. To make different vision encoders work together in perception, we first aggregate different visual representations by addition (#4). The average performance does improve compared with models with single vision encoders (#1-3). However, such a straightforward method does not bring much improvement, and even some sub-items have declined. This is due to the mismatch and interference among different visual features, which severely compromise their respective visual information. To make visual features aligned well and reduce information loss, we first replace the pooling with the proposed ADT network. As demonstrated in Experiment #5, ADT consistently enhances performance across four task groups, yielding an average improvement of 4 points. Based on the transformed visual features, we further replace the naive addition mixture process with the router to modulate and aggregate them adaptively according to instructions (#6). This achieves an impressive performance that significantly outperforms the addition method and the methods with a single vision encoder across all sub-tasks. Additional experiments on the deformable mechanism can be found in Appendix C. These experimental results prove that ADT and router are crucial and effective components of MoVE to mitigate interference and optimally utilize the capabilities of each visual expert.Visualization of Routing Results. To provide a deeper understanding of the MoVE's adaptive routing process, we visualize the routing outcomes across various tasks. Since the feature scale varies across different vision encoders, we cannot directly consider the output of the router as expert importance. Instead, we integrate the output features from vision encoders, taking the magnitude of the weighted feature vector as the importance metric. Because the final aggregated result will lean towards the side with the larger magnitude.As displayed in Fig. 4 , for tasks that involve text recognition and graph understanding, such as ChartQA and DocVQA, the features from the Pix2Struct encoder are dominant. In contrast, the model utilizes more CLIP features in image-level VQA tasks like COCOCapion [11] , NoCaps [2] , and Flickr30K [69] . Notably, for TextCaps [59] , a task that requires two kinds of visual information, the router shows a preference for balancing the CLIP and Pix2Struct branches. For tasks that focus on visual grounding ability like REC [42] , and REG [37] , the model uses more DINO features to perceive region-level visual information. These observations indicate that MoVE can adaptively modulate the features transformed from various vision encoders.We conduct ablation experiments to explore the best practice of MoLE, which are summarized in Table 2 . We take the model with a single adapter in each FFN (#1) as baseline, which suffers severe task interference. Then, we replace the plain adapter with MoLE. As summarized in Table 2 #2-4, we test three kinds of MoLE routers with different inputs: token hidden states (MoLE-T), sentence embedding (MoLE-I), and a mixture of both (MoLE-IT). The token hidden states and sentence embedding are concatenated on the last dimension as the input for the MoLE-IT router.The experimental results indicate that all MoLE variations outperform the plain adapter, with the sentence-embedding-based router achieving the highest average performance.We also explore two strategies for expert load balance in MoLE, which are tabulated in Table 2 #5-6. MoLE-I+GS introduces variability to the router by adding Gumbel-distributed noise to the logits [25] , aiming to prevent certain experts from never being selected. MoLE-I+LB uses auxiliary loss [18] to Visualization of routing results. To provide a deeper understanding of our MoLE module, we sample several instances from each dataset and calculate their average routing outcomes. In Fig. 5 , we present the expert load conditions of four selected datasets, each representing a kind of VL task. At the beginning, the router assigns equal probabilities to each expert as it is randomly initialized. However, after training, the routing distributions of MoE blocks vary significantly across tasks, as shown in Fig. 5 . This means the language experts in our MoLE module gradually specialize in distinct task domains during training. In the inference process, the router adaptively chooses the optimal expert to achieve strong multitasking capabilities. Meanwhile, we can observe strong resemblances in the expert routing results of similar tasks, which are further analyzed in Appendix D.We summarize the evaluation results of MoME and other MLLMs with similar resource consumption on popular VL tasks in Table 3 . The results show that our MoME method achieves promising outcomes on most datasets compared with other generalist and MoE MLLMs, especially on TextCaps, Flicker30K, and IconQA.We present several visualized examples of our MoME model from distinct dataset groups along with their MoVE and MoLE routing results in Fig. 6 . In the REC case, DINOv2 accounts for nearly 50% among vision experts, providing fine-grained visual information. So the model can recognize the blue Please describe this image in short.A man in a hoodie and a child in a green jacket are standing on a sidewalk in the rain.  
conclusion: 
 This work investigates task interference when training a generalist MLLM across various VL tasks.To mitigate it, we propose MoME, which specializes in both vision and language modality to adapt to task differences. Extensive experiments validate the efficiency of MoME as a generalist MLLM.However, due to the limitations of computing resources, We have not yet expanded our approach to more data and more modalities for experiments. Nonetheless, we believe that the proposed MoME is versatile and can be applied to constructing generalist models in a wider range of multimodal domains. We hope MoME will inspire new research in general-purpose multimodal AI and its applications.  
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We provide detailed training and test configuration details in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: MLLM researches typically do not test and report error bars of experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provide sufficient information on the computer resources in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss societal impacts in the Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We describe safeguards about the pre-trained language model in the Appendix.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] .Justification: The creators and original owners of the assets used in the paper are properly credited and explicitly mentioned with respect.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: We will release our code with detailed documentation in the future. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Architecture DetailsThe MoME includes three off-the-shelf vision encoders: CLIP-ViT, DINOv2, and Pix2Struct. For CLIP-VIT, we use ViT-G/14 from EVA [17] without the last layer. The DINOv2 [50] is the official pretrained version of ViT-L/14 without registers. For Pix2Struct, we use the vision branch of pre-trained Pix2Struct-base model [30] . The ADT consists of a 2D adaptive average pooling layer and a six-layer deformable attention network. The hidden size of ADT is the same as its corresponding vision encoder. We use 8 attention heads and each head samples 4 points in the deformable cross-attention process. In MoLE, the hidden dimension of each adapter is set to 64. We use Vicuna-v1.5(7B) [13] as our pre-trained LLM.Our training process comprises two stages. In stage 1, we train the model with MoVE and a single adapter in LLM for 30k steps with a batch size of 64. The learning rate is warmed up linearly from 0 to 5e-4 across 1000 steps and then reduces to a minimum of 0 using cosine decay. The AdamW [38] optimizer is employed with β 1 = 0.9, β 2 = 0.999, and a weight decay of 0.05. In stage 2, we load the checkpoint of stage 1 and replicate the weights of adapters to initialize MoLE, while keeping everything else unchanged.We use a single node with A800 80GB × 8 for all experiments, the entire training is done in one day including stage 1 and stage 2.We collected 24 datasets and categorized them into four groups for instruction-tuning and evaluation, as shown in Fig. 7 . For most vision-language (VL) tasks, we used the datasets in both the training and evaluation phases. However, we only use NoCaps for evaluation because it only has an evaluation set, and we exclude the VSR training data due to its simplicity. During the training process, we mix the datasets within the same group into one large dataset, so the probability of each sub-dataset being sampled equals their size as a proportion of the total. However, we ensure the sample ratio of each group dataset is the same. For evaluation, we compute the overall score for each category by averaging its subitem evaluation results. We follow InstructBLIP [15] , Shikra [7] , and UReader [66] for our evaluation metrics, which are tabulated in Table 5 . Notably, the model only takes images as visual information without introducing OCR tokens like InstructBLIP. To evaluate the effectiveness of the deformable mechanism of ADT, we conduct an ablation experiment by replacing the deformable cross-attention in MoME with standard cross-attention. As shown in Table 4 , the model without deformable mechanism (#1) presents much worse performances than the original MoME (#2) consistently. We attribute this decline to the strong inductive bias of deformable attention in processing 2D feature maps.We present the routing distributions across different datasets of all MoLE routers in Fig. 8 . In general, the routing results differ significantly among different task groups, while the routing preferences are similar within the same task group. From the perspective of router preferences, the datasets can be classified as text-rich (ChartQA -TextCaps), caption (COCOCap -Flickr30K), VQA (IconQA -GQA), REC, and REG. It can prove that the MoLE captures the modularity of the tasks and mitigates task interferences through differential expert routing.MoME utilizes pre-trained large language models (LLMs), which inherently carry limitations from LLMs. These limitations include the potential for generating inaccurate information or biased outputs.To address these issues, we enhance the model's visual perception ability with MoVA and conduct vision-language instruction tuning on high-quality datasets. Despite these improvements, we advise caution and recommend thorough safety and fairness assessments before deploying MoME models in any downstream applications. COCOCap [11] karpathy-test CIDEr [65] (↑) Flickr30K [69] karpathy-test CIDEr [65] (↑) NoCaps [2] val CIDEr [65] (↑) OKVQA [44] val Accuracy(↑) AOKVQA [54] val Accuracy(↑) GQA [23] test-dev Accuracy(↑) Visual Spatial Reasoning (VSR) [34] val Accuracy(↑) IconQA [40] test Accuracy(↑)REC RefCOCO [28] val & testA & testB Accuracy(↑) RefCOCO+ [28] val & testA & testB Accuracy(↑) RefCOCOg [43] val & test Accuracy(↑)REG RefCOCO [28] val & testA & testB CIDEr [65] (↑) RefCOCO+ [28] val & testA & testB CIDEr [65] (↑) RefCOCOg [43] val & test CIDEr [65] (↑)ChartQA [45] test Relax Accuracy [49] (↑) TabFact [10] test Accuracy(↑) DeepForm [61] test F1 Score(↑) DocVQA [47] test ANLS [4] (↑) InfographicsVQA [46] test ANLS [4] (↑) WikiTableQuestions (WTQ) [51] test Accuracy(↑) TextCaps [59] val CIDEr [65] (↑) TextVQA [60] val Accuracy(↑)Figure 8 : Distribution of all language experts routing results across all tasks.NeurIPS Paper ChecklistJustification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper studies the problem of building a multimodal generalist by mixture of experts motivated by the observation of task interference. Existing MoE models tend to focus on the LLM side, but this paper investigates the task inference on the vision representation and proposes ADT and dynamic routing techniques to aggregate dis-similar visual representation. The ablations show that the proposed techniques are effective in improving a multimodal generalists across dis-similar task space. 

This paper receives divergent reviews with scores 6, 5, 4. All reviewers appreciate the comprehensive experiments which demonstrates the benefits of the proposed approach, and the analysis which are intuitive (cdtp), clear (1g6k) and insightful (ZwrT).  In addition, the importance of understanding multimodal task interference is noted by ZwrT. Considering the reviews and rebuttal, the main objection to acceptance is raised by reviewer cdtp who considers the novelty incremental relative to previous works such as “Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs”, "SILC: Improving Vision Language Pretraining with Self-Distillation" and other works using MoEs in LLMs. The author provided a detailed response and the reviewer adjusted the score accordingly, while maintaining that "the design philosophy of this work—mixture of vision expert—is pretty much similar to previous works" and leaning to reject.

Upon careful inspection of the works cited by cdtp and the author rebuttal, I notice that the two cited papers “Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs”, "SILC: Improving Vision Language Pretraining with Self-Distillation" are actually studying very different problems than this paper. The first paper focuses on probing the short comings of the existing VLMs and demonstrate effects of MoF (mixture of features) by simply adding or interleaving features on their benchmarks. The second paper proposes to add self-distillation objective to contrastive learning so that the model is capable of both global/local recognition. Thus, the similarity claimed by cdtp is not very convincing in my humble opinion. In addition, as a principle, I do not think the similarity in high-level design philosophy alone should be the ground for rejection. Having inspected the paper myself and other related multimodal MoE works, I do not think this justification of weak reject are well-supported by evidences.

Given that the other two reviews are positive, I'm leaning to recommend acceptance of this paper given the importance of the problem and the comprehensive analysis/experiments noted by all the reviewers. However, due to the lower number of positive reviews, I'd like us to have more discussion on this paper.

Reviews: review: paper_summary: paper_summary: In this paper, the authors introduce a mixture of multimodal experts (MoME) to reduce task interference and develop a generalist MLLM. MoME consists of two main components: a mixture of vision experts (MoVE) and a mixture of language experts (MoLE). MoVE can adaptively adjust features transformed from different vision encoders and boasts strong compatibility with various transformation architectures. MoLE integrates sparsely gated experts into LLMs, achieving seamless improvements while keeping inference costs nearly unchanged.
strength_weakness: strength_weakness: 1. From Table 1, we can see that both ADT and Router have achieved notable improvements. Could you explain the internal mechanisms behind this? I am quite curious as to why the improvements are so significant.
2. In Table 3, many values are missing. Could you add some results from more general multimodal benchmarks to make the experiments more comprehensive, such as MME, MMbench, MM-Vet, LLaVA$^W$, Science$^{QA}$, etc.
questions: questions: One advantage of MoE is its fast inference speed. Could you conduct an experiment to verify the model's inference speed?
limitations: limitations: yes, the author explains the limitations of their study and potential negative societal impact.
paper_summary: paper_summary: The paper proposed a MoE design for MLLM, it utilize MoE in both visual encoding procedure and LLM decoding procedure. The paper utilized a dynamic routing module to mix visual features from different experts, and adopted a multi-adapter structure to combine the knowledge of differnent language experts. Widely conducted experiments showcase the effectiveness of the proposed method on various downstream tasks.
strength_weakness: strength_weakness: The design proposed in this paper seems to be not innovative enough. Its key design philosophy has already been brought out in many previous works. As for MoE in vision, the CVPR 2024 paper “Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs” combines CLIP and DINOv2 features to improve visual representations, the arxiv 2023 paper “SILC: Improving Vision Language Pretraining with Self-Distillation” combines the learning objectives of CLIP and DINOv2 for better pretraining outcome. Moreover, the references [12, 5, 8, 54] cited in this paper has already investigated MoE in LLMs, as stated in the introduction section. 
It is already a consensus that MoE in MLLM, especially combining multimodal pretrained feature (such as CLIP) and pure visual pretrained feature (such as DINOv2), can improve model performance. Combining a third feature to improve performance (in this paper, the feature from Pix2Struct) besides CLIP and DINOv2 is a pure engineering design, which could not add up to the technical contribution of this paper. Moreover, the ‘Dynamic Router’ proposed in this paper is also a simple MLP network which is widely used for combining different feature. To sum up, it is hard to claim all these designs as a contribution of this paper.
questions: questions: 1.	What is the differnece between this work and the previous works discussing MoE in vision (the several papers I listed in the weaknesses section), how can this paper brough new knowledge or designs that is perviously unknown or not sufficiently expored by the community?
2.	How does the MoE structure affects the inference speed, are there any quantitative results?
limitations: limitations: The authors already adequaately addressed the potential negative societal impact.
paper_summary: paper_summary: The paper proposes to mitigate task interference during multimodal instruction tuning with a mixture of experts, in both the language and the encoder side. The paper is well written, contains insightful analysis and shows improvements over baselines.
strength_weakness: strength_weakness: 1. While the paper shows improvements over different baselines in Tab.3, the scores are still lagging behind, compared to methods without MoE, such as LLaVA (which the paper builds on top of). Knowing that the proposed approach have significantly more parameters, pretrained for more steps (stage 1) and use different visual encoders.
2. The paper claims that “This means the language experts in our MoLE module gradually specialize in distinct task domains during training.” However the visualization does not support that. For instance, in Fig.5  E4 is used in most tasks types, while E1 is relatively less used. But it is not clear if we can map different experts to different task groups.
3. The paper is not evaluated on common and recent multimodal benchmarks: SEED, MME, MM-Vet, POPE, VQAv2 … that are considered in other methods like LLaVA. 
4. The design of MoE are very different for the LLM and visual encoders. Each visual encoder is considered as an expet. Did the authors experiment with typical MoE (replicating FFN) in a single visual encoder?
5. The papers used deformable cross-attention, but I did not find any experiment to support this design choice, compared to using simple cross-attention. Did the authors conduct this experiment?
6. The paper states that load balancing did not help. Any insights on why this is the case? knowing that these losses are typically used in most MoE papers. Also, did the authors encounter any instabilities during training? It is important to discuss these in the paper, as instability is a major problem in MoEs training. 
7. The paper main novelty seems in applying MoE also in the vision encoders. This question is also explored in previous works with slightly different context such as [1] and [2]. I found the novelty limited in this regard.

[1] Mustafa, Basil, et al. "Multimodal contrastive learning with limoe: the language-image mixture of experts." Advances in Neural Information Processing Systems 35 (2022): 9564-9576.

[2] Shen, Sheng, et al. "Scaling Vision-Language Models with Sparse Mixture of Experts." Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.
questions: questions: Please the weaknesses section (e.g. 4-5-6)
limitations: limitations: Some limitations and societal impacts are discussed in the paper or the appendix.

Comments: title: Reminder to review the rebuttal
comment: As the interactive discussion window is about to close, we sincerely invite the reviewer cdtp to read our follow-up response. We hope that our explanations effectively address your concerns, and we would appreciate it if you could consider revising your rating based on this information.
We are happy to hear that we've addressed your concerns, and we thank the reviewer again for the feedback!
We are happy to hear that we've addressed your concerns, and we thank the reviewer again for the feedback!
Thanks for your reply. We would like to clarify some concepts again and sincerely hope you will reassess the innovation and contributions of our work.

1. The innovative aspect of this paper, the “mixture of vision experts,” differs significantly from previous works [1,2,3]. In the context of the availability of numerous pre-trained models, we have innovatively proposed a dynamic and adaptive approach to fuse vision encoders each specialized in a specific domain. To the best of our knowledge, the adaptive mixture of various pre-trained vision encoders has not been explored in the works of MLLM, as these works only consider addition or concat [4,5]. Prior to the era of MLLM, existing works [1,2,3] focused on applying typical MoE methods (experts are several identical sub-networks) within Vision Encoders and training from scratch, which is a completely different technical solution. **We acknowledge that our MoVE shares a similar design philosophy with previous works at a broad conceptual level (effectively utilizing many visual branches in a vision-language model). However, as mentioned above, the differences in technical details (e.g., the combination framework of vision branches, the feature processing method, and the design purpose) are also significant.** 
2. We would like to argue against the contention that our ADT did not "bring new knowledge to the model." As stated in the manuscript and rebuttal, **one of the major issues in combining various pre-trained vision encoders is the misalignment among visual tokens**. This issue is caused by the differences in the pre-training setting and architecture of these vision encoders. Simply combining them (pooling and addition) will result in significant information loss. Thus, we propose Adaptive Deformable Transformation to mitigate the information loss by adaptively refining the pooled features. As shown in Table 1 of the manuscript and the table below, our ADT is very helpful and achieves **an average gain of almost 4 points**. To conclude, we believe that **our ADT is a promising method to resolve the discrepancies of various pre-trained vision encoders**, instead of “a simple adjust or compression in token length”.

| Strategy | Gen. | REC | REG | Doc. | Avg. |
| --- | --- | --- | --- | --- | --- |
| Pool + Add | 70.36 | 74.89 | 57.55 | 32.83 | **58.91** |
| ADT + Add | 74.35 | 76.93 | 61.01 | 39.23 | **62.88** |
| MoVE (ADT + Router) | 79.05 | 81.92 | 63.82 | 52.77 | **69.39** |

We hope that these explanations can address your concerns, and we would greatly appreciate it if you could consider giving us a higher rating.

---

**Reference**

[1] Naeem, Muhammad Ferjad, et al. "Silc: Improving vision language pretraining with self-distillation."

[2] Mustafa, Basil, et al. "Multimodal contrastive learning with limoe: the language-image mixture of experts."

[3] Shen, Sheng, et al. "Scaling vision-language models with sparse mixture of experts."

[4] Tong, Shengbang, et al. "Eyes wide shut? exploring the visual shortcomings of multimodal llms."

[5] Jiang, Dongsheng, et al. "From clip to dino: Visual encoders shout in multi-modal large language models."
Thank the authors for addressing most of my concerns, I will keep my positive rating.
I appreciate the authors' efforts to summarize their contributions. However, I still believe that the design philosophy of this work—mixture of vision expert—is pretty much similar to previous works. A simple adjust or compression in token length won't brought new knowledge to the model. Nevertheless, I appreciate the detailed explanation of the paper's design difference comparing to previous works, as well as the inference speed provided by the authors. Therefore, I would like to raise my score to 4.
Dear Reviewer cdtp,

Thank Reviewer cdtp again for the valuable comments. We have provided the response to each of the concerns raised in the review, and we are eager to continue the conversation. As the interactive discussion window will close soon, we kindly invite the reviewer to read our response to see if there are any further questions.

Thank you!

Best regards,

Authors
Thanks for the detailed response. The reviewers addressed some raised points and added additional experiments supporting their approach. Based on their feedback I will increase my score.