MoVA: Adapting Mixture of Vision Experts to Multimodal Context

Authors: zhuofan  zong
bingqi  ma
dazhong  shen
guanglu  song
hao  shao
dongzhi  jiang
hongsheng  li
yu  liu
cuhk  mmlab
sensetime  research
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 07:14:53

Abstract: As the key component in multimodal large language models (MLLMs), the ability of the visual encoder greatly affects MLLM's understanding on diverse image content. Although some large-scale pretrained vision encoders such as vision encoders in CLIP and DINOv2 have brought promising performance, we found that there is still no single vision encoder that can dominate various image content understanding, e.g., the CLIP vision encoder leads to outstanding results on general image understanding but poor performance on document or chart content. To alleviate the bias of CLIP vision encoder, we first delve into the inherent behavior of different pre-trained vision encoders and then propose the MoVA, a powerful and novel MLLM, adaptively routing and fusing task-specific vision experts with a coarse-to-fine mechanism. In the coarse-grained stage, we design a context-aware expert routing strategy to dynamically select the most suitable vision experts according to the user instruction, input image, and expertise of vision experts. This benefits from the powerful model function understanding ability of the large language model (LLM). In the fine-grained stage, we elaborately conduct the mixture-of-vision-expert adapter (MoV-Adapter) to extract and fuse task-specific knowledge from various experts. This coarse-to-fine paradigm effectively leverages representations from experts based on multimodal context and model expertise, further enhancing the generalization ability. We conduct extensive experiments to evaluate the effectiveness of the proposed approach. Without any bells and whistles, MoVA can achieve significant performance gains over current state-of-the-art methods in a wide range of challenging multimodal benchmarks.

Main paper: 
 introduction: 
 Significant achievements in multimodal large language models (MLLMs) [1, 2, 3, 4, 5, 6, 7] have been witnessed due to their remarkable proficiency in solving open-world tasks. MLLMs acquire visual perception capacity while inheriting sophisticated reasoning abilities and knowledge from large language models (LLMs) [8, 9, 10] . The core idea behind MLLMs is projecting the vision representation into an LLM through a projector, facilitating a general-purpose multimodal understanding. General multimodal understanding requires comprehending complex image contexts across various tasks and scenarios. The CLIP [11] vision encoder, pre-trained on large-scale image-text pairs with a contrastive loss, is widely considered as a flexible and popular choice among the latest leading MLLMs. However, training data and optimization target of the vision encoder determine its Table 1 : Comparison of CLIP vs. state-of-the-art task-specific vision encoders. Our evaluation criteria encompass a variety of dimensions: comprehensive benchmarks [16] , text-oriented Visual Question Answering (VQA) [17, 18] , general VQA [19] , object hallucination [20] , Referring Expression Comprehension (REC) [21] , Referring Expression Segmentation (RES) [21] , and medical VQA benchmark SLAKE [22] . We use the same data for each model.Task MMB DocVQA ChartQA GQA POPE REC RES SLAKE CLIP [11] Image-text Contrastive 64.9 inconsistent performance across tasks and scenarios, which will bias the generalization of multimodal large language models. For instance, MLLMs with a single CLIP vision encoder usually perform poorly on fine-grained tasks such as grounding and optical character recognition (OCR) [12] . Several works have attempted to incorporate extra state-of-the-art vision encoder experts to cope with the challenge. For example, both SPHINX [13] and MoF [14] integrate vision self-supervised learning features of DINOv2 [15] with MLLMs to enhance their visual grounding capabilities. Vary [12] introduces a new vision encoder expert for improved fine-grained document and chart parsing ability. Intuitively, it is necessary to explore the utilization of more task-specific vision encoder experts in MLLMs to promote model generalization across various domains.We aim to start the exploration through empirical analysis of readily available vision experts. In particular, we focus on the multimodal capabilities of seven distinct state-of-the-art vision encoders based on LLaVA-1.5-7B [28] . The results in Table 1 reveal that MLLMs with these task-specific vision encoders achieve optimal performance in their respective area. Concurrently, we note that the plain fusion (concatenation) of vision encoder experts adopted in previous works [13] would not bring consistent improvement compared with the single task-specific vision expert in its proficient task.The inherent bias of each expert introduces biased information and leads to performance degradation in the plain fusion paradigm. For example, DINOv2 serves as an expert in visual grounding but performs poorly at text-oriented tasks. Representation of DINOv2 would be regarded as biased information in text-related scenarios so incorporating DINOv2 for these tasks would inevitably cause performance decrease. Consequently, a flexible method of vision encoder ensemble that dynamically activates and weights context-relevant task-specific vision experts can fully unleash the capacity of these models while avoiding model bias.In this paper, we propose MoVA, a powerful MLLM, adaptively routing and fusing task-specific vision experts with a coarse-to-fine mechanism. Inspired by the powerful tool-use capabilities of LLM [29] , the coarse-grained context-aware expert routing aims to employ LLM to select vision experts with strong relevance to the user's image and instruction from the expert model pool. Thanks to the strong generalization ability of LLM, we also can perform model routing for vision experts in open scenarios. The fine-grained expert fusion facilitates better extraction and integration of expert representations based on multimodal context. Specifically, the expert knowledge extractor in the mixture-of-vision-expert adapter (MoV-Adapter) will extract diverse task-specific knowledge from various vision experts through mixture-of-expert (MoE) cross-attention layers. The dynamic gating network can allocate precise expert-wise soft weights for the integration of extracted taskspecific knowledge. Under the coarse-to-fine paradigm, we provide a flexible and effective manner of leveraging representation from experts based on multimodal context and model expertise, further enhancing the model generalization ability. As presented in  
related work: 
 Multimodal architectures [1, 3, 6, 30, 31, 32, 33, 34] , optimization paradigm [35, 36] , applications [37, 38, 39, 40] , and benchmarks [41, 42, 43, 44, 45, 46, 47] have recently achieved remarkable progress and garnered unprecedented attention within the academic community. Multimodal large language models (MLLMs) usually leverage the alignment from visual features to the linguistic feature space to achieve superior vision-language understanding capabilities based on off-the-shelf LLMs and vision encoders. CLIP vision encoder [11] , which is trained in contrastive learning from billions of diverse image-text pairs [48, 49] , is widely used among these works. For example, LLaVA [3] adopts an MLP projector to align visual tokens from the frozen CLIP vision encoder to the embedding layer of LLM. However, The representation from CLIP exhibits strong discriminative abilities in classification and recognition but only has limited performance on downstream tasks like location and relation understanding [50] . To break through this bottleneck, some works [4, 51] turn to unlock the CLIP vision encoder and further fine-tune the parameter with training data for downstream tasks. For instance, Qwen-VL [6] collected massive training data for grounding and OCR to jointly optimize the CLIP vision encoder and LLM. Recent works propose to involve an extra frozen vision encoder to enhance the performance of MLLMs. SPHINX [13] is one of the pioneers, where grounding capabilities have been significantly improved with the assistance of the DINOv2 [15] . Vary [12] introduces an extra encoder training on large-scale charts and document data to improve the performance on related downstream tasks.3 MoVA Methodology 
overview: 
 MoVA comprises five key components: (i) a pre-trained large language model (LLM) that generates accurate responses given the image tokens and instructions; (ii) a base vision encoder; (iii) vision experts that generate task-specific vision latent features; (iv) mixture-of-vision-expert adapter (MoV-Adapter) that performs fine-grained expert fusion based on the multimodal context.As illustrated in Figure 1 , MoVA consists of two stages: coarse-grained context-ware expert routing and fine-grained expert fusion with MoV-Adapter. First, our coarse-grained context-ware expert routing leverages the tool-use capabilities of LLM, routing the most appropriate experts from N expert candidates via LLM to help the model answer the user's question. In the second stage, we turn to enhance the visual representation with a novel MoV-Adapter module in a fine-grained manner.More specifically, we leverage the mixture-of-expert (MoE) cross-attention layers to extract the taskspecific knowledge of representations from chosen experts. Meanwhile, the dynamic gating network in MoV-Adapter can allocate soft weights to the extracted knowledge of each expert according to the input image and instruction. Then the extracted knowledge can be effectively integrated into the foundational representation of the base vision encoder. Finally, the enhanced visual representation with instruction tokens is fed to the LLM to generate an accurate response. In Section 3.2 and Section 3.3, we will focus on our core contributions, the context-aware expert routing strategy, and the expert fusion with MoV-Adapter. In Section 3.4, we will introduce the training process. The coarse context-aware expert routing is performed in the first stage to select context-relevant experts. Next, we adopt the MoV-Adapter to extract and fuse the task-specific knowledge from these selected experts in a fine-grained manner.Pretrained Vision Encoders and LLM. The vision encoders in MoVA consist of a base encoder and multiple task-specific vision encoder experts. We choose the pre-trained CLIP ViT-L-336px as the base encoder. Our vision experts include several state-of-the-art task-specific encoders: DINOv2, Co-DETR, SAM, Pix2Struct, Deplot, Vary, and BiomedCLIP. The corresponding expertise is presented in Table 1 . For example, both Pix2Struct and Vary will be used when the user asks the MLLM to scan the document image. MoVA is flexible and easy to generalize to all decoder-only LLMs. We mainly consider Vicuna-7B [8] , Llama3-8B3 , and Yi-34B [52] as our language models in this work.Pipeline of Context-aware Routing. The context-aware expert routing strategy aims to employ the impressive tool-use capacity of LLM to select vision experts with strong relevance to the user's image and instruction from a model pool. Specifically, we perform the context-aware expert routing in three steps during inference. First, the input image, user questions, and descriptions of expert models are converted into appropriate instructions that prompt the MLLM to perform expert selection. An example of the prompt instruction input and selection output is shown in Table 2 . Such a routing task does not require image details and high-resolution input images, hence we directly downsample the base encoder's visual feature to obtain a coarse image embedding (e.g., 144 image tokens). The downsampled image tokens and instruction tokens are then fed to the LLM as inputs. Finally, the LLM generates the output text and we parse it to determine which vision expert should be selected for fine-grained knowledge extraction in the second stage. For instance, as depicted in Table 2 , the LLM directly outputs the option's letter of DINOv2 and Pix2Struct, thus we only utilize them for the subsequent extraction. During training, we do not perform context-aware expert routing and replace the routing outputs with our routing annotations to improve efficiency.Routing Data Construction. Compared with other MLLMs, MoVA requires additional routing annotations. We first introduce the formal definition of the data structure for an unambiguous understanding of the routing data. The data structure for expert routing introduces additional routing annotation R to the conventional multimodal data (I, Q, A). Here, I represents the image, Q and A refer to the question-answer pair, and R refers to the expert set which contains the most appropriate ones to solve this question. Then the construction process for routing data can be formulated as (I, Q, A) → R, with the primary objective being to derive vision experts that optimally align with the sample (I, Q, A). Intuitively, the language modeling loss can serve as an effective metric for evaluating how a data sample aligns with the vision expert. Specifically, we can reuse the LLaVA-1.5-7B models with various vision encoders presented in Section 1 to perform loss computation. Here, we denote the model with the base encoder as M 0 and the model with j-th expert among N experts as Table 2 : One example of the instruction-following data for context-aware expert routing. We present the multimodal inputs in the top block and the language response in the bottom block. The detailed model descriptions are released in the Appendix.M j . For the i-th sample (I i , Q i , A i ),we send it to models {M j |j ∈ {0, 1, . . . , N }} and calculate the language modeling loss {L i,j |j ∈ {0, 1, . . . , N }}. The j-th expert is regarded as a useful expert for the i-th sample only if L i,j < L i,0 and will be added to the routing set R i . Note that we only keep up to 3 vision experts to avoid computation costs brought by too many additional experts. All the routing annotations of our training data are generated offline. We can directly parse and input these offline results to the subsequent expert fusion component during training.Routing Data Augmentation. To preserve the expert routing robustness and generalization ability in open scenarios, we only randomly select 2K samples for training, remove the model name in model description, and rewrite the model descriptions using ChatGPT [53] for each expert. We also shuffle the model pool and randomly truncate the model pool during training.We propose the MoV-Adapter to facilitate fine-grained expert representation extraction and integration based on multimodal context. As shown in Figure 2 , the MoV-Adapter consists of L adapter blocks and a text encoder. Each block contains an expert knowledge extractor, a dynamic gating network, and a transformer block. For the i-th block, the input feature is denoted as X i ∈ R C×H×W and we take the CLIP base encoder feature X ∈ R C×H×W as the input feature X 1 of the first block. We use G to indicate the indices of chosen K experts. The expert feature set is {F j |j ∈ G}. The final output feature of L adapter blocks is X L+1 . Additionally, we apply two residual blocks [54] with an average pooling to X L+1 to obtain a coarser image featureX L+1 out ∈ R C× H 2 × W 2, which is further connected to the LLM text embedding space by an MLP layer.Text Encoder. We introduce a pre-trained BERT as the text encoder to extract language context information from the user's instruction. We take the [CLS] token from the output of the text encoder as the text token X T ∈ R C T . It is worth noting that all the adapter blocks share the same text token.Expert Knowledge Extractor. We adopt N cross-attention layers as the expert knowledge extractor to achieve efficient knowledge extraction. Note that only the expert features {F j |j ∈ G} and their corresponding cross-attention layers are involved in the extraction. For each selected expert feature F j ∈ R Cj ×Hj ×Wj , we first align its resolution to X i with bilinear interpolation:Fj = Interpolate(F j , H, W ).(1) For the i-th MoV-Adapter block and the j-th cross-attention layer, we take input feature X i as query, and the aligned expert feature Fj as the key and value: Dynamic Gating Network. We employ a dynamic gating network to contribute to a fine-grained knowledge integration process for the conditional representation {Y i j |j ∈ G}. It is implemented with the softmax over the logits of an MLP layer, processing multimodal representation to generate expert-wise soft weight P i ∈ R K for the output of each cross-attention layer in the extractor. Specifically, the input to the gating network is the concatenated vector of a visual token X i V ∈ R C and the text token X T ∈ R C T . We obtain X i V with a global average pooling operation to X i . Then we concatenate them to compute the gating weights and the expert-wise outputs by computing the weighted sum:EQUATION)Cross Attention Dynamic Gating & ℱ! … Cross Attention & ℱ" Cross Attention & ℱ# Expert Knowledge Extractor Self Attention FFN ( ) $ Transformer Block ) $ ) $%! Adapter Block Text Encoder ×" * $ Global PoolingEQUATIONwhere P i j ∈ (0, 1) is the soft weight for the j-th expert in the i-th block. Transformer Block. The transformer block in the adapter block follows the vanilla design, consisting of a self-attention layer and an FFN layer. Taking the fused visual representation Xi , its output will serve as the input feature X i+1 for the next adapter block.As shown in Figure 3 , the training process of MoVA consists of pretraining and supervised finetuning.Pretraining. To improve multimodal generalization, we first construct 15M visual instruction samples across diverse domains as the training data: (i) Image caption data that covers 4M randomly selected samples from DataComp-1B [55] , ShareGPT4V-PT [51] , and ALLaVA-4V [56] . (ii) Visual grounding and localization dataset that encompasses Objects365 [57] , RefCOCO [21] , VisualGenome [58] , PointQA [59] , and Flickr30K [60] . (iii) Chart understanding data that includes MMC-Instruction [61] , Chart2Text [62] , DVQA [63] , and SciGraphQA [64] . (iv) Text recognition and document parsing data that covers LLaVAR-PT [65] and 3M English document images from Common Crawl4 . (v) LLaVA-Med [66] for biomedical image understanding. During the pretraining phase, we only optimize the MoV-Adapter along with the base vision encoder while preserving the capabilities of the initial large language model. Meanwhile, we leverage the routing annotations generated via the method proposed in Section 3.2 to choose experts and ignore representations from irrelevant ones during training. Supervised Finetuning. We utilize high-quality visual instruction tuning data that build upon LLaVA-665K [28] for finetuning. Additionally, we integrate several visual question answering datasets across various domains, such as DocVQA [17] , ChartQA [18] , InfographicVQA [67] , AI2D [68] , ST-VQA [69] , TextVQA [70] , SynthDoG-en [71] , Geometry3K [72] , PGPS9K [73] , Geo170K [74] , RefCOCO, LLaVA-Med, VQA-RAD [75] , and SLAKE [22] . We also encompass equivalent comprehensive captions [51, 56, 76, 77] generated by the advanced GPT4-V [53] for improved world knowledge. Apart from the above instruction tuning data, we convert the selected 2K routing annotations to instructions and incorporate them into the training data. In the supervised finetuning stage, only task-specific vision experts are frozen and we jointly optimize other components.The objective of supervised fine-tuning is to align the visual representation and the embedding of LLM, boosting its visual instruction-following capabilities. 
implementation details: 
 As mentioned in Section 3.4, our training pipeline consists of two stages. In the pretraining stage, we use the AdamW optimizer with an initial learning rate of 2×10 -4 , a batch size of 1024, and train the model for 1 epoch. We jointly finetune the weights of all components except additional vision experts with a batch size of 128 and an initial learning rate of 2×10 -5 during supervised finetuning. We use 3 transformer blocks (L=3) in the MoV-Adapter and its hidden dimension is 1024, which is consistent with the base vision encoder CLIP. The input resolution of the base vision encoder is 672×672. Two residual blocks with an average pooling are employed in the MoV-Adapter to reduce the number of output image tokens from 2304 to 576. For the experiment performed in Table 1 , we follow the default setting of LLaVA-1.5 but incorporate several additional datasets, including DocVQA [17] , ChartQA [18] , RefCOCO [21] , LLaVA-Med [66] , VQA-RAD [75] , and SLAKE [22] . More details about vision experts, ablations, and analysis are available in Appendix A.1 and A.3.We empirically analyze the multimodal capacity and generalization ability of MoVA on a wide range of challenging MLLM benchmarks in Table 3 . This comprehensive assessment is conducted on MME [81] , MMBench [16] , QBench [82] , MathVista [83] , MathVerse [84] , and POPE [20] .Compared to other open-source MLLMs with similar model complexity, MoVA with Vicuna-7B achieves the best performance across 7 MLLM benchmarks while offering a more favorable balance between training efficiency and performance. For instance, MoVA-7B surpasses the recent state-ofthe-art LLaVA-NeXT-7B [80] with a dynamic high resolution design, processing only 20% image tokens. Furthermore, we adopt Hermes-Yi-34B [85] as the LLM to validate the scaling property of MoVA. As depicted in Table 3 , the performance of MoVA-34B is on par with popular proprietary MLLMs (e.g., Gemini-Pro [78] ) and outperforms Qwen-VL-Plus [6] on 5 MLLM benchmarks. For example, MoVA establishes new records on MMBench and MMBench-CN, even surpassing the GPT-4V [53] . These results suggest that the ensemble of vision experts with adaptive expert routing can serve as an effective dimension for MLLM model scaling.The evaluation results on VQA benchmarks are outlined in Table 4 . We divide these benchmarks into general VQA benchmarks [88, 19, 89 ] and text-oriented VQA benchmarks [70, 18, 17, 68] . Thanks to the dynamic and efficient task-specific knowledge extraction, MoVA achieves state-ofthe-art performances across diverse VQA benchmarks. For general VQA benchmarks, MoVA-7B outperforms SPHINX-2k [4] equipped with Vicuna-13B on VQAv2 [88] and GQA by 4.2% and 1.9%, respectively. Besides, MoVA shows its proficiency in text recognition in various scenarios, encompassing scene text, chart, document, and diagram. For instance, MoVA-7B catches up to the current state-of-the-art generalist CogAgent [86] with 18 billion parameters on these text-oriented benchmarks with smaller model size. The MoVA model with 38B parameters even surpasses the wellestablished specialist model PALI-X-55B [87] by clear margins. These outstanding performances demonstrate MoVA's robust generalization capabilities across diverse domains.We conduct experiments on Referring Expression Comprehension (REC) benchmarks [21] to evaluate the visual grounding ability of MoVA. The results are presented in Table5. The performance of MoVA-7B is on par with the state-of-the-art specialist models that are elaborately designed for This experiment is conducted on popular medical VQA benchmarks VQA-RAD and SLAKE. We directly leverage the medical VQA evaluation metric adopted by LLaVA-Med. Each sample of VQA-RAD and SLAKE is observed only once during the training process of MoVA and LLaVA-1.5. For a fair comparison, we compare MoVA with the LLaVA-Med variant that is finetuned with only 1 epoch on the benchmark. The performance of the LLaVA-Med specialist that is fully finetuned on downstream tasks is also reported. As presented in Table 6 , MoVA-7B consistently yields higher scores than LLaVA-Med and LLaVA-1.5, exhibiting its medical visual chat ability.Component-wise analysis. As presented in Table 7 , we perform an ablation to thoroughly delve into the effect of each component. First, we try to replace the context-aware routing with random routing. Without task-relevant vision experts, the performance drops by a large margin, especially on the textoriented VQA benchmarks. Removing context-aware routing to leverage all vision experts also leads to similar results. It proves that both these modifications introduce biased information from irrelevant vision experts due to the removal of context-aware routing. Then, we ablate the effectiveness of the MoV-Adapter by replacing it with simple linear layers. The removal of fine-grained expert feature fusion downgrades performance across all datasets. These results delineate that each component in MoVA can consistently yield significant gains.Number of activated experts. In the context-aware routing phase, the number of activated experts K is dynamic. We compare such a data-dependent design with other variations of constant K in this experiment. As presented in Table 8 , the overall performance of dynamic K consistently outperforms other models with constant K. This reveals this dynamic implementation can fully exploit the task-specific knowledge of relevant experts while avoiding the incorporation of biased information.Criteria for choosing better experts. To reduce the costs, our method only adopts N 1 models with N various encoders to identify the better vision expert separately. However, we do not explicitly consider the combination of the chosen vision experts. In this experiment, we compare our method with another strategy that considers vision encoders combination and enumerates 9 , our method achieves comparable performance while requiring much less models for data construction.Expert routing in open scenarios. We develop 105 human-verified testing samples that should be answered using novel experts for the expert routing task. These novel experts encompass 7 vision models [93, 71, 94, 91, 95, 54, 96] on various computer vision tasks and each expert corresponds to 15 evaluation samples. We manually check the correctness of the expert routing result. As presented in Table 10 , a lightweight network, such as a MLP classifier fails to generalize to this open-world setting. Besides, increasing the routing training samples and removing data augmentations also lead to severe performance degradation. The results demonstrate our coarse-grained context-aware routing preserves the generalization ability for expert routing in open scenarios.Adapter Design. In this section, we conduct ablation studies on the design of the MoV-Adapter. As presented in Table 11 , we compared the impact of using 2, 3, and 4 adapter blocks on the model's performance. We observed that the baseline with 3 blocks can achieve better performance than other settings. Then, we substituted our multimodal gating for uniform gating to investigate its effectiveness. Each of the experts is assigned the same soft weight in the uniform gating. We find uniform gating brings consistent performance drops in the test benchmarks. It indicates that the lack of the dynamic soft-weight harms the overall performance since it fails to perform precise knowledge extraction.Inference Analysis. As illustrated in Figure 1 , MoVA consists of two stages: coarse-grained contextware expert routing and fine-grained expert fusion with MoV-Adapter. This two-stage inference pipeline can be further broken down into five steps: (i) Data preprocessing. We first process the input image with image processors and convert the input text into a token sequence with the LLM tokenizer. (ii) Base encoder forward. We extract the base image feature using the base CLIP encoder. Note that we only run the base encoder once since its output feature can be preserved and reused in the fourth step. (iii) LLM routing generation. We compress the base image features into 144 image tokens. The LLM generates a concise routing answer based on the compressed image feature and routing instruction. Vision experts and MoV-Adapter forward. (iv) According to the multimodal context and routing results generated in the previous step, we fuse vision features of the base encoder and activated experts in a coarse-to-fine manner. (v) LLM response generation. The LLM generates the final response given the fused vision features and user instructions. To investigate the inference efficiency of each step, we randomly select 200 images from the COCO val2017 dataset and adopt the common image caption instruction: Describe this image. The temperature for generation is 0. The latency is measured using bfloat16 and flash-attention 2 on an A100 80G GPU. We present the average inference latency of each step and show the average sequence length of the routing output and final response. The inference latencies for each step are 0.19s, 0.05s, 0.14s, 0.07s, and 10.24s, respectively. The average length of the routing output is 3.24 tokens, while the average length of the final response is 405.06 tokens. Compared to the LLM response generation (Step 5), the LLM expert routing (Step 3) generates much fewer output tokens and its latency is negligible (0.14s v.s. 10.24s). Therefore, our method does not bring significant inference costs. 
conclusion: 
 In this paper, we reveal that the inherent bias of each vision encoder can diminish its generalization ability across other irrelevant domains by analyzing the performance of individual vision encoders versus the plain fusion of multiple encoders across various tasks. To deal with the problem, we propose MoVA, a powerful MLLM composed of coarse-grained context-aware expert routing and fine-grained expert fusion with MoV-Adapter. Based on multimodal context and model expertise, MoVA fully leverages representation from multiple context-relevant vision encoder experts flexibly and effectively while avoiding biased information brought by irrelevant experts. MoVA can achieve significant performance gains over current state-of-the-art methods in a wide range of benchmarks.Limitations. We acknowledge some limitations in our paper that require attention. One limitation is the hallucination, which refers to the generation of text that appears plausible or coherent but is factually incorrect and misleading. This issue potentially presents in all powerful MLLMs. Additionally, the performance may be affected by failure cases of the context-relevant vision experts, leading to potential degradation. We plan to explore solutions for these limitations in future works. A.3 More Experiments Image segmentation. In this experiment, we aim to investigate if task-specific knowledge can improve MoVA on the segmentation task. Therefore, we introduce a simple design to extend MoVA to segmentation tasks. Unlike segmentation generalists [97] that adopt additional pixel decoder with high-resolution images for high-quality mask generation, we just formulate the referring segmentation task as sequential polygon generation [98] . We finetune MoVA and the baseline with a SAM-Huge [24] backbone on the RefCOCO referring segmentation datasets. MoVA achieves 57.1% gIoU on the testA benchmark, which is 2.6% higher than the 54.5% of baseline. This result indicates that MoVA is capable of exploiting task-specific knowledge to solve segmentation tasks.Effect of LLM for expert routing. In this experiment, we investigate the effect of the LLM in our coarse-grained expert routing. As presented in Table 14 , expert routing with LLM achieves the best performance. When the LLM is substituted for a lightweight MLP classifier and a BERT encoder, we need to increase the number of routing training samples from 2K to 1.6M to preserve model performance. Besides, both MLP classifier and BERT encoder fail to perform expert routing in open scenarios as stated in Table 10 . Therefore, the strong tool-use capacity and generalization ability of LLM is critical to our flexible and effective expert routing.Vision encoder for expert routing. In the coarse-grained expert routing, we only adopt the image feature of base vision encoder CLIP. As presented in Table 15 , the method with CLIP achieves slightly better performance than other methods since such a routing task does not require elaborate expert knowledge. Besides, incorporating other vision experts with plain fusion also brings biased information and increases cost. To achieve a better trade-off between efficiency and performance, we only use CLIP for coarse-grained expert routing.Number of image tokens in expert routing. We ablate the number of image tokens used in the expert routing stage. As shown in Table 16 , routing with 144 tokens can achieve comparable performance to methods with more tokens. Considering the additional cost brought by processing more image tokens, we only use 144 tokens for routing.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: The codes and models will be released when the paper is accepted. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the potential societal impacts of our work in the Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this experiment, we analyze the effectiveness of our routing data construction method. The test split is constructed by randomly selecting 500 samples from the routing data. We first prompt GPT4-V [53] to determine whether the expert routing result is reasonable or not. These evaluation results are then manually checked and we finally compute the routing accuracy using the human-verified evaluation results. Our loss-driven data construction approach achieves 94.6% accuracy (473 samples are regarded as "reasonable" by GPT4-V), demonstrating its effectiveness.We apply MoVA to a wide range of real-world understanding and reasoning tasks to investigate its multimodal comprehension capacity. As presented in Figure 4 and Figure 5 , MoVA successfully solves these diverse and complex cases thanks to the context-relevant expert knowledge.Positive impacts. The positive impact is to use MoVA to solve various open-world tasks in real-world scenarios. For example, MoVA can serve as a useful assistant to help people improve work efficiency and answer their questions.Negative impacts. The potential negative social impact is to use MoVA model to generate misleading and false contents. This issue potentially presents in all multimodal large language models. We will try to improve its safety in future works.User:What is the person with the highest detection confidence score doing?The person with the highest detection confidence score, labeled "person 0.89", is eating ice cream. She is holding a white container, and she appears to be taking a bite of the ice cream. She is wearing a light pink shirt with a red logo on the left chest area, black shorts with a white stripe on the side, and white sneakers with black accents. Her hair is styled in a bob cut, and she is wearing a pair of white earphones around her neck. The person is standing with one hand slightly raised, possibly to hold the container steady while eating. The image captures her in a candid moment, enjoying her snack while walking or standing outdoors.What are all the scene text in the image?The text on the road sign reads:-"CLEMENCEAU AVE" at the top of the sign.-"River Valley Rd" with an arrow pointing right.-"Havelock Rd" with an arrow pointing left.-"Chin Swee Rd" with an arrow pointing left.User:Can you explain this meme?The image is a humorous meme that compares the experiences of different parties involved in online learning during the COVID-19 pandemic. The meme is divided into four panels, each representing a different entity: Zoom, the Wifi company, the teacher, and the students.In the first panel, a man is shown throwing money into the air, symbolizing the financial success of Zoom, a popular video conferencing platform that has seen a surge in usage during the pandemic. The text "Zoom" is written above the image.The second panel shows a man lying on a couch, holding a laptop, with a relaxed and content expression on his face. This represents the teachers who have been able to work from home and enjoy a more flexible lifestyle. The text "Teacher" is written above the image.In the third panel, a man is shown throwing money into the air, similar to the first panel, but this time the text "Wifi company" is written above the image. This suggests that the companies providing internet services have also benefited from the increased demand for online services during the pandemic.The fourth and final panel shows a clown with a sad expression, standing in front of a mirror with the words "PUT ON A HAPPY FACE" written on it. This represents the students who have had to adapt to online learning and may be feeling overwhelmed or stressed. The text "Students" is written above the image. Overall, the meme humorously highlights the different experiences and impacts of online learning during the pandemic, with Zoom and the Wifi company profiting, teachers enjoying a more relaxed lifestyle, and students feeling the pressure to adapt to a new learning environment.User: • The answer NA means that the paper does not include experiments.Imagine• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.https://github.com/meta-llama/llama3https://commoncrawl.org 


Decision: This paper introduces a new MLLM called MoVA, which is composed of coarse-grained context-aware expert routing and fine-grained expert fusion with MoV-Adapter. By utilizing multimodal context and model expertise, MoVA fully leverages representation from multiple experts. The experimental results validate the model's effectiveness across multiple benchmarks.

The paper was evaluated by four reviewers. Three reviewers recommended acceptance (1x accept, 1x borderline accept, 1x weak accept), while one reviewer suggested a borderline rejection. Overall, this work presents a novel and effective MLLM solution that has the potential to inspire further research in the field. 

The authors are encouraged to incorporate their responses to the reviewers' comments in the revised version to enhance the paper's clarity and comprehensiveness.

Reviews: review: paper_summary: paper_summary: In this paper, the authors propose MoVA, a powerful MLLM composed of coarse-grained context-aware expert routing and fine-grained expert fusion with MoV-Adapter. Based on multimodal context and model expertise, MoVA fully leverages representation from multiple context-relevant vision encoder experts flexibly and effectively while avoiding biased information brought by irrelevant experts. MoVA can achieve significant performance gains over current state-of-the-art methods in a wide range of benchmarks.
strength_weakness: strength_weakness: 1. Despite the proposed MoVA in this paper achieving state-of-the-art performance across multiple tasks, leveraging multiple visual encoders in MLLM design does not introduce novelty. Additionally, the design of MoV-Adapter also lacks innovation.
2. This paper lacks a comprehensive evaluation regarding the routing selection of multiple visual encoders using LLMs, necessitating a more targeted and comprehensive assessment from the authors.
questions: questions: Please refer to Weakness Section for more details.
limitations: limitations: The authors have clearly illustrated the weakness of this work by pointing out the hallucination and the model performance may be affected by failure cases of the context-relevant vision experts.
paper_summary: paper_summary: This paper considers a critical issue in vision language models, i.e., the vision encoder design. The authors proposes MoVA, a coarse to fine routing strategy, adaptively routing and fusing task specific vision experts which best suit the task. Experiments conducted on a wide range of MLLM benchmark demonstrate the advantages of the proposed MoV adapter.
strength_weakness: strength_weakness: 1.	As for OCR and related tasks, it has been validated that high resolution is much more import than model size, and image split like LLaVA Next, Monkey, Ferret etc. have been widely used. I wonder the importance of including task specific models. It looks redundancy, and importantly, we cannot include all models that cover different tasks exhaustively

2.	As shown in Table 1, the main improvement results from the OCR related Doc and Chart task, which may resolved by current MLLMs baseline such as LLaVa Next, I wonder the  necessity of the using MoE experts, as other module, adding resolution is more reasonable. In Table 3, since the PT, SFT and pretrained models are different, it is hard to validate the gains are from the MoV experts.  

3.	Does the method introduce error propagation, say, the first stage LLM routing is not correct, which may harm the performance of the whole framework. On the other hand, if the coarse routing is simple, could we say that this stage we do not need to carefully design, and a tiny LLM or even rule based according to the prompt is enough? Since current framework seems complex using two LLMs, the comparison is not fair, and I wonder if there is a better routing strategy.
questions: questions: The main concern is the rationality of using MoV on vision side, while two step routing is interesting, it adds the complexity (using two LLMs), and it is not clear the advantages of MoV over the latest MLLM benchmark.
limitations: limitations: None
paper_summary: paper_summary: This paper proposes MoVA, a powerful and novel MLLM, adaptively routing and fusing task-specific vision experts with a coarse-to-fine mechanism. MoVA first leverages the tool-use capabilities of LLM, routing the most appropriate experts from expert candidates and then uses the MoV-Adapter module to enhance the visual representation.
strength_weakness: strength_weakness: 1. Is there an adapter structure in MoVA? Why do we call it MoV-Adapter?
2. The experiments use SFT data from different MLLMs should be aligned (the same). The comparison is not fair with different SFT data.
3. More vision encoder enhancement methods should be compared using the same SFT data, including

[1] VCoder: Versatile Vision Encoders for Multimodal Large Language Models.

[2] MoF (Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs)

[3] MouSi: Poly-Visual-Expert Vision-Language Models

[4] BRAVE: Broadening the visual encoding of vision-language models

4. The inference speed of LLM is very slow. I think MoVA using LLM as a router selection will decrease the speed of the whole process. How to solve this problem?
questions: questions: In line 102, I just found four components, but the paper claims five.
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: The paper proposes a MoVA to extract and fuse task-specific knowledge from different visual encoder. Besides, the paper designs a coarse-to-fine mechanism, which dynamically selects the most suitable vision experts, then extracts and fuses task-specific knowledge from various experts. The experimental results are encouraging and validate the performance of the proposed method.
strength_weakness: strength_weakness: 1. The number of components in line 102 does not match the serial numbers.
2. During the inference phase, the process involves two rounds of LLM inference and two Base Encoder, which results in significant inference time and GPU memory usage. This poses a challenge for the practical application of MLLMs.
3. This is a well-completed article, but I am concerned about its novelty. The method seems similar to the adapter+agent approach described by Wang et al. [1]. Could you explain the differences and advantages of your method compared to theirs?

[1] Wang X, Zhuang B, Wu Q. Modaverse: Efficiently transforming modalities with LLMs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 26606-26616.
questions: questions: Why is the last row of MoVA results in Table 1 different from those in Table 3? In other words, The caption of Table 1 shows that the same data was used to train each model. Does this mean that all models used LLaVA-1.5 data? If so, I think this table will inspire the community and I would like to ask if you can release the checkpoints of these models.
limitations: limitations: 1. As mentioned in the conclusion, the performance may be affected by failure cases of the context-relevant vision experts, leading to potential degradation. Additionally, it is influenced by the choice of encoder results in the first stage of inference.
2. The model's complex inference structure increases the inference burden.

Comments: title: Thanks for your comments
Response to Authors
Looking forward to your response
Looking forward to your post-rebuttal feedback
Response to follow-up questions
About the inference speed
Thanks for your comments
Response to follow-up questions
Question about the experiment results and novelty
comment: We deeply thank you for the kind support of our work!
After checking the peer review comments and the author's responses, I decided to maintain the given score for this work.
Dear Reviewer gqRq:

We sincerely appreciate your time and efforts in reviewing our paper. We have provided corresponding responses and results, which we believe have covered your follow-up concerns. We hope to further discuss with you whether or not your concerns have been addressed. Please let us know if you still have any unclear parts of our work. If your concerns have been well addressed, please consider raising your rating, thanks.

Best,

Authors
Dear Reviewer dPiw:

We thank you for the precious review time and valuable comments. We have provided corresponding responses and results, which we believe have covered your concerns. We hope to further discuss with you whether or not your concerns have been addressed. Please let us know if you still have any unclear parts of our work. If your concerns have been well addressed, please consider raising your rating, thanks.

Best,

Authors
Thanks for your question. We will address your concern below.

We show the inference latency of each step in Q3 of our global response.

| Step 1 | Step 2 | Step 3 | Step 4 | Step 5 | Routing output | Final response |
| - | - | - | - | - | - | - |
| 0.19s | 0.05s | 0.14s | 0.07s | 10.24s | 3.24 tokens | 405.06 tokens |

The final response generation of LLM (Step 5) takes **$\frac{10.24}{405.06}=0.0253$ seconds to generate a token**, and the output speed of the LLM routing (Step 3) is  **$\frac{0.14}{3.24}=0.0432$ s/token**. Considering we adopt different prompt templates and image features for routing and final response generation, such speed value difference is reasonable. Specifically, we use the prompt template in Table 2 of our paper for expert routing and keep the same Vicuna template for response generation.

Therefore, the expert routing is so fast since MoVA only generates a few tokens (the average length is 3.24 tokens in our experiments) for routing. The routing designs described in the Q4 of the rebuttal for reviewer gqRq also decrease the inference latency.
Thanks for your response. I wonder why expert selection (0.14s) in MoVA is so fast. I think the inference time for LLM should be longer.
We sincerely thank the reviewer for the kind support of our work! We will incorporate the details into our final version.
Thank you for the response. It has addressed all of my concerns and I will raise my rating to 7.
Thanks for your follow-up questions. I'm delighted to address your concerns.

**Q1: SFT data**

We use the same SFT data as the original VCoder [1], MoF [2], and MouSi [3] implementations. Therefore, we directly report their performance scores in the paper.

1. VCoder. We employ the same COST dataset as the VCoder-7B to train the MoVA-7B model.

2. MoF and MouSi. Both of them use LLaVA-665k as the SFT data. To be specific, we institute the SFT data described in the line 210 for LLaVA-665k in our experiments.

3. The SFT data mentioned in the line 230 is only used for the model training in Table 1.

[1] VCoder: Versatile Vision Encoders for Multimodal Large Language Models.

[2] MoF (Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs)

[3] MouSi: Poly-Visual-Expert Vision-Language Models

**Q2: Inference comparison**

We perform inference comparisons among MoVA, VCoder, MoF, and MoE-LLaVA [5] since the codes of both MouSi and BRAVE [4] are still publicly unavailable.

For a fair comparison, we adopt the same number of output tokens (400 tokens). The benchmark setting is identical to the aforementioned benchmarking in the rebuttal.

| Model | Average inference latency |
| - | - |
| VCoder-Vicuna-7B | 10.25s |
| MoF-Vicuna-7B | 10.33s |
| MoVA-Vicuna-7B | 10.47s |
| MoE-LLaVA-Phi2-2.7B | 22.40s | 

The results reveal that the latency values of VCoder, MoF, and MoVA exhibit negligible differences. Moreover, MoE-LLaVA runs much slower than other frameworks even though it is equipped with a smaller LLM Phi2-2.7B.

[4] BRAVE: Broadening the visual encoding of vision-language models

[5] MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

**Q3: The typical MoE methods have the expert selection mechanism**

Selecting $K$ experts from $N$ experts is a common practice of the mixture of experts (MoE) and it is not our contribution. Our paper focuses on **how to select $K$ vision experts** based on the current task in multimodal scenarios. Specifically, we explore how to leverage the reasoning ability of LLM and the prior knowledge of vision encoders to flexibly select the most $K$ appropriate vision experts. This idea has not been explored in the current mulitmodal community.

**Q4: Comparison with MouSi**

We compare our method with MouSi in the table:

| Model | Motivation | Paradigm | Routing | Projector |
| - | - | - | - | - |
| MouSi | This paper proposes to leverage multiple experts to synergizes the capabilities of individual visual encoders. Experiments are conducted to demonstrate simple ensemble of multiple encoders can beat a single vision encoder. | Plain fusion | Experts are all activated. Relations between model expertise and the current task are ignored. | Linear layers |
| MoVA | As discussed in the Q1 of global response, leveraging multiple encoders is not our contribution and our method is motivated by the limitations of these plain fusion methods. We observe the performance degradation occurs if we directly use and fuse all experts in a plain fusion manner. Such observation is not studied in MouSi. | Multimodal context-aware | We leverage the reasoning and tool-use ability of LLM to select the most appropriate experts considering model expertise, image, and instruction. | MoV-Adapter |

**We will discuss and cite these works [1-4] in our revised paper.**

[1] VCoder: Versatile Vision Encoders for Multimodal Large Language Models.

[2] MoF (Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs)

[3] MouSi: Poly-Visual-Expert Vision-Language Models

[4] BRAVE: Broadening the visual encoding of vision-language models
Thanks for your response. I found the results you provided in the rebuttal period, including VCoder-7B, MoF-7B, MouSi-7B were the same as the results in their paper. **I strongly doubt that the authors did not use the same SFT data for a fair fine-tuning**. In our paper lines 230-232, the authors mention that they use additional SFT data. Please explain this, thanks!


For Q4, I think comparing with LLava-Next is not appropriate, it is better to show the inference speed of MoE structure methods in [1-5].

[1] VCoder: Versatile Vision Encoders for Multimodal Large Language Models.

[2] MoF (Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs)

[3] MouSi: Poly-Visual-Expert Vision-Language Models

[4] BRAVE: Broadening the visual encoding of vision-language models

[5] MoE-LLaVA: Mixture of Experts for Large Vision-Language Models


Furthermore, after reading the reviews from Lntx, FVeU. I have the same question about the paper's novelty. 
About the general response Q1(1,2), **the typical MoE methods have the expert selection mechanism**, i.e. choosing K from N experts (e.g. Switch Transformer). For Q1(3), **it has been studied in Mousi [3] and is not the main contribution of this paper**.