MoVA: Adapting Mixture of Vision Experts to Multimodal Context
Authors: Zhuofan Zong
Bingqi Ma
Dazhong Shen
Guanglu Song
Hao Shao
Dongzhi Jiang
Hongsheng Li
Yu Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: Multimodal large language model
Vision encoder
Mixture-of-expert
Primary area: deep_learning_architectures

Abstract: As the key component in multimodal large language models (MLLMs), the ability of the visual encoder greatly affects MLLM's understanding on diverse image content. Although some large-scale pretrained vision encoders such as vision encoders in CLIP and DINOv2 have brought promising performance, we found that there is still no single vision encoder that can dominate various image content understanding, e.g., the CLIP vision encoder leads to outstanding results on general image understanding but poor performance on document or chart content. To alleviate the bias of CLIP vision encoder, we first delve into the inherent behavior of different pre-trained vision encoders and then propose the MoVA, a powerful and novel MLLM, adaptively routing and fusing task-specific vision experts with a coarse-to-fine mechanism. In the coarse-grained stage, we design a context-aware expert routing strategy to dynamically select the most suitable vision experts according to the user instruction, input image, and expertise of vision experts. This benefits from the powerful model function understanding ability of the large language model (LLM). In the fine-grained stage, we elaborately conduct the mixture-of-vision-expert adapter (MoV-Adapter) to extract and fuse task-specific knowledge from various experts. This coarse-to-fine paradigm effectively leverages representations from experts based on multimodal context and model expertise, further enhancing the generalization ability. We conduct extensive experiments to evaluate the effectiveness of the proposed approach. Without any bells and whistles, MoVA can achieve significant performance gains over current state-of-the-art methods in a wide range of challenging multimodal benchmarks.

MoVA: Adapting Mixture of Vision Experts to Multimodal Context
Zhuofan Zong 2.+
Bingqi Ma
Dazhong Shen
Guanglu Song?
Hao Shao
Dongzii Jiang'
Hongsheng Li1.3,4.
Yu Liu
CUHK MMLab
Tense'Time Research
Shanghai Al Laboratory
ACPI under InnoHK
Abstract
As the key component in multimodal large language models (MLLMs), the ability the visual encoder greatly affects MLLM's understanding on diverse image content. Although some large-scale pretrained vision encoders such vision encoders in CLIP and DINOv2 have brought promising performance, we found that there is still no single vision encoder that can dominate various image content understanding. e.g., the CL.IP vision encoder leads to outstanding results on general image understanding but poor performance on document or chart content. To alleviate the bias of CLIP vision encoder, we first delve into the inherent behavior of different pre-trained vision encoders and then propose the Mo powerful and novel MLLM, adaptively routing and fusing task-specific vision experts with coarse-to coarse-to-fine fine mechanism. In the coarse- sarse--rained stage, we design context- aware expert routing strategy to dynamically select the most suitable vision experts according to the user instruction, input image, and expertise of vision experts. This benefits from the powerful model function understanding ability of the large language model (LLM).. In the fine-grained stage, we elaborately conduct the mixture-of- mixture-of-vision-exype vision-exe--e adapter (MoV-Adapter) to extract and fuse task-specific knowledge from various experts. This coarse-to-fing to-fine paradigm effectively leverages representations from experts based multimodal context and model expertise, further enhancing the generalization ability. We conduct extensive experiments evaluate the effectiveness of the proposed approach. Without any bells and whistles, MoVA can achieve significant performance gains over current state-of- the-art methods in wide range of challenging multimodal benchmarks. Codes and models are available at wihhtps:/ //github. tiuu..uu.... com comm/TemleXX98//00
Introduction
Significant achievements in multimodal large language models (MLLMs) 12 have been witnessed due their remarkable proficiency in solving open open-world tasks. MLLMs acquire visual perception capacity while inheriting sophisticated reasoning abilities and knowledge from large lan- guage models (LLMs)) g 10] The core idea behind MILLMs is projecting the vision representation into an LLM through projector, facilitating general- -urppse multimodal understanding.
General multimodal understanding requires comprehending complex image contexts across various tasks and scenarios. The CLIP [1] vision encoder, pre-trained on large-scale image-text pairs with a contrastive loss, is widely considered as flexible and popular choice among the latest leading MLLMs. However, training data and optimization target of the vision encoder determine its
"Equal contribution.
t Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table 1: Comparison of CLIP vs. state-of-the-art task-specific vision encoders. Our evaluation
criteria encompass variety of dimensions: comprehensive benchmarks [16], text-oriented Visual
Question Answering (VQA) 17 18 general VQA [9], object hallucination 20 Referring Expres-
sion Comprehension (REC) 21. Referring Expression Segmentation (RES) 21 and medical VQA
benchmark SLAKE 22
We use the same data for each model.
Vision Encoder
Task
MMB Doc
ChariQA GQA
POPE
REC RES
SLAKE
CLIP M
Image-text Contrastive
64.9
35.6
35.3
62.5
85.7
81.5
43.3
63.7
DINOv2 [15]
Visual Grounding
57.5
14.7
15.9
63.9
86.7
86.1
47.5
59.4
Co-DETR [23
Object Detection
48.4 48.4
14.2
14.8
58.6
88.0
82.1
48.6
55.3
SAM 24
Image Segmentation
40.7
13.9
15.0
54.0
82.0
79.2
49.3
57.7
Pix2Struct [25]
Text Recognition
41.9
57.3
53.4
51.0
78.1
59.2
32.2
44.0
Deplot 226
Chart Understanding
36.2
40.2
55.8
48.1
75.6
51.1
27.0
44.5
Vary [2]
Document Chart Parsing
28.1
47.8
41.8
42.6
69.1
21.6
16.0
40.9
BiomedCLIIP
27
Biomedical Contrastive
40.0
15.3
16.8
50.8
76.9
57.8
27.4
65.1
Plain fusion
63.4
46.5
48.9
63.0
86.4
85.7
45.3
64.7
MoVA
65.9
59.0
56.8
64.1
88.5
86.4
49.8
66.3
inconsistent performance across tasks and scenarios, which will bias the generalization of multimodal large language models. For instance, MLLMs with a single CLIP vision encoder usually perform poorly on fine-grained tasks such grounding and optical character recognition (OCR) [12] Several works have attempted to incorporate extra soeee-of-tee-aee vision encoder experts to cope with the challenge. For example, both SPHINX [ and MoF [14] integrate vision self-supervised learning features of DINOv2 [15 with MLLMs to enhance their visual grounding capabilities. Vary [12 introduces new vision encoder expert for improved fine-graines document and chart parsing ability. Intuitively, it is necessary to explore the utilization of more task-specific vision encoder experts in MILMs promote model generalization across various domains.
We aim to start the exploration through empirical analysis of readily available vision experts. In particular, we focus on the multimodal capabilities of seven distinct ofte-of-theear vision encoders based on LLaVA--15577 vision encoders achieve optimal performance in their respective area. Concurrently, we note that the plain fusion (concatenation) of vision encoder experts adopted in previous works [3] would not bring consistent improvement compared with the single task-specifie vision expert in its proficient task. The inherent bias of each expert introduces biased information and leads to performance degradation in the plain fusion paradigm. For example, DINOv2 serves as an expert in visual grounding but performs poorly at text-oriented tasks. Representation of DINOv2 would be regarded as biased information in text-related scenarios incorporating DINOv2 for these tasks would inevitably cause performance decrease. Consequently, flexible method of vision encoder ensemble that dynamically activates and weights context-relevant task-specific vision experts can fully unleash the capacity of these models while avoiding model bias.
[28]. The results in Table I reveal that MLLMs with these task-specific
In this paper, we propose MoVA, powerful MILM, adaptively routing and fusing task-specift vision experts with a coarse-to-fin mechanism. Inspired by the powerful tool-use capabilities of LLM [2)]. , the marse-grained context-aware expert routing aims to employ LLM to select vision experts with strong relevance to the user's image and instruction from the expert model pool. Thanks to the strong generalization ability of LLM,, also can perform model routing for vision experts in open scenarios. The fine-grained expert fusion facilitates better extraction and integration of expert representations based on multimodal context. Specifically, the expert knowledge extractor in the mixture-of-visse -ii-o---vvio-eeeeerrre adapter (MoV-Adapter) will extract diverse task-specific knowledge from various vision experts through mixture-of-expprr (MoE) cross-attention layers. The dynamic gating network can allocate precise expert-wise soft weights for the integration of extracted task- specific knowledge. Under the coarse-to- toorse-tt-finee fine paradigm, we provide flexible and effective manner of leveraging representation from experts based on multimodal context and model expertise, further enhancing the model generalization ability. As presented in Table I MoVA can preserve the optimal performance of single relevant vision encoder by ignoring non-relevant experts on the GQA, POPE, and REC task. Besides, MoVA can further boost performances via the fine-grained fusion of multiple relevant vision experts other tasks.
2

---

We conduct comprehensive experiments on various benchmarks to evaluate the effectiveness of MoVA, including MLLM benchmarks, visual question answering (VQA), visual grounding, and biomedical understanding. Without any bells and whistles, Mo can achieve significant perfformaneeeommmnc performance gains over current state-of-the-an methods.
The contributions of this work are three-fold: (i) By amalyzing the performance of individual vision encoders versus the plain fusion of multiple encoders across various tasks, we reveal that the inherent bias of each vision encoder can diminish its generalization ability across other irrelevant domains. (ii) We propose MoVA, a powerful MLLM composed of coarse-gras messe--raindd context-awarr expert routing and fine- thee-graied expert fusion with MoV-Adaple Based on multimodal context and model expertise, MoVA fully leverages representation from multiple context-relevant vision experts flexibly while avoiding biased information of irrelevant experts. (iii) We demonstrate the effectiveness of each component MoVA by elaborate ablation studies. MoVA can achieve significant performance gains over state-of- of-the-art methods in wide range of challenging benchmarks.
2 Related Work
Multimodal architectures 0 30
32
optimization paradigm [3]]]6]] applications [37
38 39 40 411, and benchmarks 42,
48) have recently achieved remarkable progress and garnered unprecedented attention within the academic community. Multimodal large language models (MLLMs) usually leverage the alignment from visual features to the linguistic feature space to achieve superior vision- lision-language understanding capabilities based on off-the-shelf LLMs and vision encoders. CLIP vision encoder [II]. which is trained in contrastive learning from billions of diverse image-text pairs [49 50), is widely used among these works. For example, 3 adoptss MIP projector to align visual tokens from the frozen CLIP vision encoder to the embedding layer of L.M.. However, The representation from CLJP exhibits strong discriminative abilities in classification and recognition but only has limited performance like location and relation understanding [51]. To break through this bottleneck, some works [ 52) turn to unlock the CLIP vision encoder and further fine-tune the parameter with training data for downstream tasks. For instance, Qwen-VL VL [6] collected massive training data for grounding and OCR jointly optimize the CLIP vision encoder and LLM. Recent works propose involve an extra frozen vision encoder to enhance the performance of MLLMs. SPHINX [3] is one of the pioneers, where grounding capabilities have been significantly improved with the assistance of the DINOv2 [15]. Vary [12] introduces an extra encoder training on large-scale charts and document data to improve the performance on related downstream tasks.
43
44
45
46
47
LLaVA
downstream tasks
3 MO Methodology
3.1 Overview
MoVA comprises five key components: (i) pre-trained large language model (LLM) that generates accurate responses given the image tokens and instructions; (ii) base vision encoder; (iii) vision experts that generate task-specific vision latent features; (iv) -o-ire-o--vviioo--ee adapter (MoV. Adapter) that performs ene-grai expert fusion based on the multimodal context.
As illustrated in Figure [ MoVA consists of two stages: coarse-grained context-ware expert routing and fine-grained expert fusion with M-V-Adatter First, our coarse-grained context- ware expert routing leverages the tool-use capabilities of LLM,, routing the most appropriate experts from N expert candidates via LLM to help the model answer the user's question. In the second stage, we turn to enhance the visual representation with novel MoV-Adapter module in a fine-grained manner. More specifically, we leverage the mixture-of-expert (MoE) cross-attention layers extract the task- specific knowledge of representations from chosen experts. Meanwhile, the dynamic gating network in MoV-Adapter can allocate soft weights the extracted knowledge of each expert according to the input image and instruction. Then the extracted knowledge can be effectively integrated into the foundational representation of the base vision encoder. Finally, the enhanced visual representation with instruction tokens is fed to the L.M to generate an accurate response. In Section 3.2 and Section 3.3 we will focus on our core contributions, the context-aware expert routing strategy, and the expert fusion with Mo V- Adaater.. In Section 3.4 will introduce the training process.
3

---

Step 1:
Step 2:
Cane expert routing
Expert fusion with MoV-Adapter
Large Language Model
Large Language Model
Setection Expert 1, Expert 2
Downsample
<eysten prompt> descriptions> Here user questions
Expert 1
MoV-Adapter
@ Where is the red sign and what does it say?
visual grounding
Base Encoder
Where the sign and what does 1t say? AMB
Expert 2 poNt nocognitiinn
Base Encoder
Idenciry and select ect models that will best...
Expert chart
STOP
Nhere the sign and what doss it say?
STOP
Vision Experts
Figure 1: The pipeline of VA, MoVA performs coarse-to searse-to-fine routing to solve given question. The coarse context-aware expert routing is performed in the first stage select context-relevant experts. Next, we adopt the MoV-Adapter to extract and fuse the task-specific knowledge from these selected experts in fine-grained manner.
Pretrained Vision Encoders and LLM. The vision encoders in MoVA consist of a base encoder and multiple task-specific vision encoder experts. We choose the preitrained CLIP ViT-L-336px as the base encoder. Our vision experts include several of-te-of-theearr task spec-fpe encoders: DINOv2, Co- DETR, SAM, Pix2Struct, Deplot, Vary, and BiomedCLIP. The corresponding expertise is presented in Table II For example, both Pix2Struct and Vary will be used when the user asks the MLLM to scan the document image. MoVA is flexible and easy to generalize to all decoder-only LLMs. We mainly consider Vicuna-7B g. Llama3-8B and Yi-34B (53
language models in this work.
3.2 Coarse-grained Context-aware Expert Routing
Pipeline of Context-aware Routing. The context-aware expert routing strategy aims to employ the impressive tool-use capacity of LLM to select vision experts with strong relevance to the user's 's image and instruction from a model pool. Specifically, we perform the context-awe aware expert routing in three steps during inference. First, the input image, user questions, and descriptions of expert models are converted into appropriate instructions that prompt the MILM to perform expert selection. An example of the prompt instruction input and selection output shown in Table 2] Such routing task does not require image details and high-resolution input images, hence we directly downsample the base encoder's visual feature obtain a coarse image embedding (e.g., 144 image tokens). The downsampled image tokens and instruction tokens are then fed to the LLM as inputs. Finally, the LLM generates the output text and we parse determine which vision expert should be selected for fine-grained knowledge extraction in the second stage. For instance, as depicted in Table 2 the LLM directly outputs the option's letter of DINOv2 and Pix2Struct, thus we only utilize them for the subsequent extraction. During training, we do not perform context-aware expert routing and replace the routing outputs with our routing annotations to improve efficiency.
Routing Data Construction. Compared with other MLLMs, MoVA requires additional routing annotations. We first introduce the formal definition of the data structure for an unambiguous understanding of the routing data. The data structure for expert routing introduces additional routing annotation R the conventional multimodal data (I, Q, A). Here, I represents the image, and A refer to the question-answer pair, and R refers to the expert set which contains the most appropriate ones solve this question. Then the construction process for routing data can be formulated as (I, A) + R, with the primary objective being to derive vision experts that optimally align with the sample (I, Q. A). Intuitively, the language modeling loss can serve as an effective metric for evaluating how a data sample aligns with the vision expert. Specifically, we can reuse the LLaVA-1.5- 7B models with various vision encoders presented Section [ to perform loss computation. Here, we denote the model with the base encoder as Mo and the model with j-th expert among N experts as
aatpss//github..ommmett -aaanttalmmmmmmaaaaaa

---

Table 2: One example of the cntrruction-oolloiinn data for context-aware expert routing. We present the multimodal inputs in the top block and the language response the bottom block. The detailed model descriptions released in the Appendix.
Routing Prompt Input
You are helpful assistant router. Based on the visual content, questions, model pool the user provides, you need to consider the expertise of these models to select the most 3 suitable models to help you answer the questions. Answer with the model's letter from the given choices directly. If models selected, just answer 'none". Model pool:
A. <DINOv2 model description>
B. <Co-DETR model description>
C. <SAM model description>>
D. <Pix2Struct model description
<Deplot model description>
STOP
F. <Vary model description>
G. <BiomedCLIP model description>
Question:
Where the red sign and what does say?
Routing Prompt Output
A.
Mj. For the i-th sample (I, As), we send it models {Mylj E {0, 10.1...., N}} and calculate The j-th expert is regarded as a useful expert
the language modeling loss [fr E 10, N)). for the i-th sample only if LJ < C,o and will be added to the routing set R, Note that we only keep up to vision experts to avoid computation costs brought by too many additional experts. All the routing annotations of training data are generated offline. We can directly parse and input these offline results to the subsequent expert fusion component during training.
Routing Data Augmentation. To preserve the expert routing robustness and generalization ability in open scenarios, we only randomly select 2K samples for training, remove the model name in model description, and rewrite the model descriptions using ChatGPT [54] for each expert. We also shuffle the model pool and randomly truncate the model pool during training.
3.3 Fine-grained Expert Fusion with MoV-Adapter
We propose the Mo MoV-Adapter to facilitate fine-grained expert representation extraction and integration based on multimodal context. As shown in Figure 2 the MoV-Adapte consists of L adapter blocks and a text encoder. Each block contains an expert knowledge extractor, a dynamic gating network, and transformer block. For the i-th block, the input feature denoted X E Hx and take the CLIP base encoder feature X E RCC as the input feature X of the first block. We G indicate the indices of chosen K experts. The expert feature set is {Fy LJ G). The final output feature of L adapter blocks is XL+1 Additionally, we apply residual blocks (55) with an average pooling X I++1 to obtain coarser image feature X XI++1 RB / W connected to the LLM text embedding space by an MLP layer.
which is further
Text Encoder. We introduce pre-trained BERT as the text encoder to extract language context information from the user's instruction. We take the [CLS] token from the output of the text encoder as the text token Xy E CC It is worth noting that all the adapter blocks share the same text token.
Expert Knowledge Extractor. We adopt cross-attention layers as the expert knowledge extractor to achieve efficient knowledge extraction. Note that only the expert features {Jilj E G} and their corresponding cross-attention layers are involved in the extraction. For each selected expert feature F, E C xW; we first align its resolution X with bilinear interpolation:
f Interpolate (F, H,
(I)
5

---

Stage 1: Pretraining
Stage 2: Supervised Finetuning
Sapertiion
Superna on
Large Language Model
Large Language Model
Vision Experts
Vision Experts
Hork Adaperr
Base Erooddr
The training strategy of MoVA. We enhance the task-specific knowledge extraction capacity in the first stage. Then, we excite model multimodal capacities in the last stage.
Figure 3:
For the i-th MoV-Adapter block and the j-th cross-attention layer, we take input feature X as query, and the aligned expert feature F; as the key and value:
x' Attention X ft).
Dynamic Gating Network. We employ a dynamic gating network to contribute to fine-grained knowledge integration process for the conditional representation {Y) j E G}. It is implemented with the softmax over the logits of an MLP layer, processing multimodal representation to generate expert-wise soft weight p I&K for the output of each cross-attention layer in the extractor. Specifically, the input to the gating network the concatenated vector of visual token X Re and the text token X, RC We obtain X with global average pooling operation X Then concatenate them to compute the gating weights and the expert-wiss wise outputs by computing the weighted sum:
Expart Knawledss Daraster
Oytemic
Isc.
Traraformur
azsiraan
Adapter Bloch
Figure 2: MoV-Adapter architecture.
X
Σ Pj. jEG
where Pj E (0, 1) is the soft weight for the j-th expert in the i-th block.
Transformer Block. The transformer block in the adapter block follows the vanilla design, consisting of a self-attention layer and an FFN layer. Taking the fused visual representation X its output will serve as the input feature Xi+1 for the next adapter block.
3.4 Training Paradigm
As shown in Figure 3, the training process of MoVA consists of pretraining and supervised finetuning.
Pretraining. To improve multimodal generalization, we first construct 15M visual instruction samples
across diverse domains as the training data: (i) Image caption data that covers 4M randomly selected
samples from DataComp-1 [56]. ShareGPT4V-PTT [52] and ALLaVA-4V (77). (ii) Visual grounding
and localization dataset that encompasses Objects365 [88. RefCOCO 21, VisualGienome [59].
PointQA [60], and Flickr30K [61]. (iii) Chart understanding data that includes MMC-Instruucion [62],
Chart2Text 633 DVQA (64, and SciGraphQA (65). (iv) Text recognition and document parsing data that covers LLaVAR-PT [66] and 3M English document images from Common Crawl (v) LLaVA- Med [67 for biomedical image understanding. During the pretraining phase, we only optimize the MoV-Adapter along with the base vision encoder while preserving the capabilities of the initial large language model. Meanwhile, we leverage the routing annotations generated via the method proposed in Section 3.2 to choose experts and ignore representations from irrelevant ones during training.
Supervised Finetuning. We utilize high-quality visual instruction tuning data that build upon LLaVA-665K 28 for finetuning. Additionally, we integrate several visual question answering
'ntps:/coommoncrww.orgg

---

Table Performance comparison with current state-of-the-art frameworks on popular MLLM benchmarks. PT and SFT indicate the number of multimodal training samples in pretraining and finetuning stage. #IMG means the number of image tokens processed by LLM.
Model
LLM
PT SIT #IMG| MME MMB MMBCN QBench MathVista Vethe POPE
Proprietary MLLMs
Owen-VI Vlu-Plus
6
66.2
68.0
66.0
43.3
11.8
Qwen- Owen-VI--Mai Va.-Max
6
77.6
75.1
73.6
51.0
24.8
Gemini-Pro
79
73.6
74.3
68.2
45.2
22.3
GPT-4V
54
75.8
73.9
74.5
49.9
38.3
Open-source MILMs
Qwen-VL
Qwen-7B
1.4B
50M
256
38.2
7.4
59.4
Qwen-VLL Vonn-VL-hhat
Qwen-7B
1.4B
50M
256
1488/361
60.6
56.7
LLaVA-1.5 28
Vicuna-78 7B
558K
665K
576
1/111/316
64.3
58.3
58.7
14.3
85.9
LLaVA-1.5
Vicuna-13B
558K
665K
576
1531/295
67.7
63.6
27.6
17.0
85.9
mPLUG-OWl2 80
LLaMA2-7B
348M
1.2M
64
1450/-
64.5
62.9
85.8
SPHINX-2K 13
Vicuna- 13B
115M
2880
1471/327
65.9
57.9
27.8
87.2
LLaVA-NeXT
81
Vicuna-7B
558K
760K
2880
1S19/332
67.4
60.6
34.6
86.5
LLaVA-NeXT
81
Hermes- Yi-34B
558K
760K
2880
1631/397
79.3
79.0
46.5
23.4
87.7
MoVA
Vicuna-7B
15M
1.6M
576
1562/371
70.4
63.7
69.3
37.6
19.7
88.6
MoVA
Llama3-8B
1596/348
75.3
67.7
37.7
21.4
89.3
MoVA
Hermes-Yi- Yi-34B
15M
1.6M
576
1603/455
81.3
79.0
70.7
44.3
23.7
88.3
datasets across various domains, such as Doc 17 ChartQA 18 Infographic
68
A12D [69], ST-VQA [70), TextVQA (71), SynthDoG-or [2). Geometry (73,, PGPS9K (74).
Geo170K (75). RefCOCo, LLaVA-Med. VQA-RAD 76.. and SLAKE (22). We also encompass equivalent comprehensive captions [52 57 77 78 generated by the advanced GPT4-V [54] for
improved world knowledge. Apart from the above instruction tuning data, convert the selected 2K routing annotations to instructions and incorporate them into the training data. In the supervised fine- tuning stage, only task-specific vision experts are frozen and we jointly optimize other components. The objective supervised fin-tuning is to align the visual representation and the embedding of LLM, boosting its visual instruction-oolowinn capabilities.
4 Experiments
4.1
Implementation Details
As mentioned in Section 3.4, our training pipeline consists of two stages. In the pretraining stage, use the Adam optimizer with an initial learning rate of 2 10-1, 10-1, a batch size of 1024, and train the model for epoch. We jointly finetune the weights of all components except additional vision experts with a batch size of 128 and an initial learning rate of 2 10 during supervised finetuning. We use transformer blocks (L=3) in the MoV-Adapter and its hidden dimension is 1024, which is consistent with the base vision encoder CLIP. The input resolution of the base vision encoder is 672x672. Two residual blocks with an average pooling are employed in the MoV-Adapter to reduce the number of output image tokens from 2304 to 576. For the experiment performed in Table [ we follow the default setting LLaVA- 1.5 but incorporate several additional datasets, including VQA [17 ChartQA 18], RefCOCO (1I, LLaVA-Med [67]. VQA-RAD (76), and SLAKE [22
More details about vision experts, ablations, and analysis are available in Appendix A. and A.3
4.2 MILM Benchmarks
We empirically analyze the multimodal capacity and generalization ability of Mo on a wide range of challenging MLLM benchmarks in Table 3 This comprehensive assessment is conducted
on MME 82 MMBench [16), QBench 83). Math [84], Math Verse [85, and POPE [20].
Compared other open- open-source MLLMs with similar model complexity, MoVA with Vicuna-7B achieves the best performance across 7 MLLM benchmarks while offering a more favorable balance between training efficiency and performance. For instance, MoVA- surpasses the recent state-of- the-art ILaVA-NexT-7B [8]] with dynamic high resolution design, processing only 20% image tokens. Furthermore, we adopt Hermes-Yi-22 (86) as the L.LM to validate the scaling property of MoVA. As depicted in Table the performance of Mo 34B is par with popular proprictary

---

Table 4: Performance comparison VQA benchmarks.
We present the number of model denotes zero-shot evaluation.
parameters of each MLLM for clear complexity comparison.
General VQA
Text-oriented VQA TextVQA ChartQA DocVQA A12D
Model
LLM
Params
VQA GQA SQA!
Generalist models
Qwen-Vi VI.
Qwen-7B
10B
79.5
59.3
67.1*
63.8
65.7
65.1
62.3
Qwen- VL-Chat
Qwen-7B
78.2
57.5
68.2
61.5
66.3
62.6
57.7
LLaVA-115
Vicuna-7B
7B
78.5
62.0
66.8*
58.2*
LLaVA-1.5
228
Vicuna- 13B
7B
80.0
63.3
71.6"
61.3"
SPHINX-2K [13
Vicuna- 13B
16B
80.7
63.1
70.6*
61.2
65.1
Vary-base
12
Qwen-7B
7B
65.3
76.3
CogAgent
88
Vicuna-7B
18B
83.7
76.1
68.4
81.6
Specialist models
Pix2Struct-Large 25
1.3B
58.6
76.6
42.1
PALI-X-55B 88
55B
86.0
71.4
70.9
80.0
81.2
MoVA
Vicuna-7B
10B
83.5
64.8
74.4"
76.4
68.3
81.3
74.9
MoVA
Llama388B
11B
83.5
65.2
74.7"
77.1
70.5
83.4
77.0
MoVA
Hermes- Hermes-Yi-34B
38B
82.3
63.9
79.0"
77.8
73.8
84.2
83.0
Table 5: Performance comparison (Acc@ 0.5) on RefCOCO REC task. Specialists specifically designed for the grounding task or finetuned on RefCOCO data.
RefCOCO
ReiCOCO+ test-A
ReiCOCOg
Type
Model
val
test-A
test
test-B
Shikra- 13B
5
87.83
91.11
81.81
82.89
87.79
74.41
82.64
83.16
Ferret- 138
91
89.48
92.41
84.36
82.81
88.14
75.17
85.83
86.34
Qwen-VI. VI.
6
89.36
92.26
85.34
83.12
88.25
77.21
85.58
85.48
Generalist
SPHINX-2k [3
91.10
92.88
87.07
85.51
90.62
80.45
88.07
88.65
MoVA-7B
92.55
94.50
88.81
87.70
92.05
82.94
89.28
89.70
MoVA-8B
92.18
94.75
88.24
88.45
92.21
82.82
90.05
90.23
MoVA-34B
93.38
94.66
90.58
89.64
92.53
84.03
91.09
90.78
G-DINO-I. 992
90.56
93.19
88.24
82.75
88.95
75.92
86.13 13.13
87.02
Specialist
UNINEXT-H 993
92.64
94.33
91.46
85.24
89.63
79.79
88.73
89.37
MLLMs (e.g., Gemini-Pro 79) and outperforms Qwen-VL-Plus [6] on MLLM benchmarks. For example, MoVA establishes new records on MMBench and MMBench-CNN, even surpassing the GPT-4V [54]. These results suggest that the ensemble of vision experts with adaptive expert routing can serve as an effective dimension for MLLM model scaling.
4.3 Visual Question Answering
The evaluation results on VQA benchmarks are outlined in Table 4 We divide these benchmarks into general VQA benchmarks 88 19 90] and text-oriented VQA benchmarks [71 I8 17 69]. Thanks to the dynamic and efficient task-specific knowledge extraction, MoVA achieves state-of- the-art performances across diverse VQA benchmarks. For general VQA benchmarks, MoVA-7B outperforms SPHINX-2K 4 equipped with Vicuna-13B on VQAv2 [9) and GQA by 4.2% and 1.9%, respectively. Besides, MoVA shows its proficiency in text recognition in various scenarios, encompassing text, chart, document, and diagram. For instance, VA-BA catches up the current oftte-ofttheart generalist CogAgent benchmarks with smaller model size. The Mo VA model with 38B parameters
(87) with 18 billion parameters on these text-oriented surpasses the well- established specialist model PALI-X-55B 88 by clear margins. These outstanding performances demonstrate MoVA's robust generalization capabilities across diverse domains.
4.4 Visual Grounding
We conduct experiments on Referring Expression Comprehension (REC) benchmarks 21l to evaluate the visual grounding ability of Mo.A. The results presented in Tabl The performance of

---

Table 6: Comparisons the Table 7: Results of component wise ablation studies.
Table 8: Results of K
biomedical VQA datasets.
varying from to 3.
Model
Open
VQA-RAD
Close
Open
SLAKE
Close
Design
GQA
ChanQA
DocVQA
K
GQA
ChartQA
MoVA
64.8
68.3
81.3
Dynamic
64.8
68.3
LLaVA-Med
28.6
56.3
70.6
54.6
LLaVA-155
35.3
68.9
73.1
63.7
Random routing
63.1
60.4
21.6
64.0
MoVA
38.3
68.9
78.2
68.8
routing
63.4
62.5
73.7
2
63,5
66.7
LLaVA Med
61.5
84.2
83.1
85.3
w/o MoV-Adageer
62.7
65.2
77.1
3
63.2
Table 9: Comparisons of expert routing criteria.
Table 10: Open-world ex- pert routing results.
Table 11: Performance of vari- ous MoV-Adapter variants.
Design
#Models
POPE
ClanQA
Design
#Samples
Accuracy
Design
MMBF
POPE
Separate
64.8
Combination
14
64.5
68.7
Angmentation MLP chassiferr
2K
28
0%
gaing
1521
MoVA-7B is on par with the state-of-the-art specialist models that are elaborately designed for grounding tasks. For example, MoVA-7B achieves a score of 90.22% on RefCOCO+ val, which is 2.46% higher than the score UNINEXT-H [93]. Our largest model MoVA-34B further pushes the performance bound of visual grounding on these benchmarks. These impressive results demonstrate MoVA's remarkable visual grounding capacity.
4.5 Medical Visual Question Answering
This experiment conducted on popular medical VQA benchmarks VQA-RAD and SLAKE. We directly leverage the medical VQA evaluation metric adopted by LLaVA-Med Each sample of VQA-RAD and SLAKE is observed only once during the traiming process of MoVA and L.aVA- For fair comparison, epoch on the benchmark. The performance of the L.L.VA-Med specialist that is fully finetuned on downstream tasks is also reported. As presented in Table 6 MoVA-7B consistently yields higher scores than VL-Med and LLaVA-1.5, exhibiting its medical visual chat ability.
compare MoVA with the ILLVA-Med variant that is finetuned with only 1
4.6 Ablation Study
Component-wise analysis. As presented in Table we perform an ablation to thoroughly delve into the effect of each component. First, we try to replace the context-aware routing with random routing. Without task-relevant vision experts, the performance drops by large margin, especially on the text- oriented VQA benchmarks. Removing context-aware routing to leverage all vision experts also leads to similar results. It proves that both these modifications introduce biased information from irrelevant vision experts due to the removal of context-an aware routing. Then, we ablate the effectiveness of the MoV-Adapter by replacing with simple linear layers. The removal of Ane-grained expert feature fusion downgrades performance across all datasets. These results delineate that each component in Mo can consistently yield significant gains.
Number of activated experts. In the context-aware routing phase, the number of activated experts K is dynamic. We compare such a data-dependent design with other variations of constant K in this experiment. As presented Table g the overall performance of dynamic K consistently outperforms other models with constant K. This reveals this dynamic implementation can fully exploit the task-specific knowledge relevant experts while avoiding the incorporation of biased information.
Criteria for choosing better experts. To reduce the costs, our method only adopts ( models with N various encoders to identify the better vision expert separately. However, we do not explicitly consider the combination of the chosen vision experts. In this experiment, we compare our method
with another strategy that considers vision encoders combination and enumerates Σi= (?) models for routing data construction. Specifically, we set N = 4 and employ a smaller model pool of DINOv2, Co-DETR, Pix2Struct, and Deplot to reduce training costs. As shown in Table 9 our method achieves comparable performance while requiring much less models for data construction.

---

Expert routing in open scenarios. We develop 105 human- ueman--errfied testing samples that should be answered using novel experts for the expert routing task. These novel experts encompass 7 vision models [94
55 97 on various computer vision tasks and each expert corresponds to 15 evaluation samples. We manually check the correctness of the expert routing result. As presented in Table TQ] lightweight network, such MLP classifier fails to generalize to this open-world setting. Besides, increasing the routing training samples and removing data augmentations also lead to severe performance degradation. The results demonstrate our coarse-grained context-aware routing preserves the generalization ability for expert routing in open scenarios.
72 95
92
96
Adapter Design. In this section, we conduct ablation studies on the design of the MoV-Adaple As presented in Table we compared the impact of using 2, 3, and adapter blocks on the model's performance. We observed that the baseline with 3 blocks can achieve better performance than other settings. Then, we substituted our multimodal gating for uniform gating to investigate its effectiveness. Each of the experts is assigned the same soft weight in the uniform gating. We find uniform gating brings consistent performance drops in the test benchmarks. It indicates that the lack of the dynamic soft-weight harms the overall performance since it fails to perform precise knowledge extraction.
Inference Analysis. As illustrated in Figure D Mo consists of two stages: coarse-grained context- ware expert routing and fine-grained expert fusion with MoV-Adapter This two-stage inference pipeline can be further broken down into five steps: (i) Data preprocessing. We first process the input image with image processors and convert the input text into a token sequence with the LLM tokenizer. (ii) Base encoder forward. We extract the base image feature using the base CLIP encoder. Note that we only run the base encoder once since its output feature can be preserved and reused in the fourth step. (iii) LLM routing generation. We compress the base image features into 144 image tokens. The LLM generates concise routing answer based on the compressed image feature and routing instruction. Vision experts and MoV-Adapter forward. (iv) According to the multimodal context and routing results generated in the previous step, we fuse vision features of the base encoder and activated experts in a coarse-to-fine manner. (v) LLM response generation. The LLM generates the final response given the fused vision features and user instructions. To investigate the inference efficiency of each step, we randomly select 200 images from the COCO val2017 dataset and adopt the common image caption instruction: Describe this image. The temperature for generation is O. The latency is measured using biloat and flash-attention 2 on an A100 80G GPU. We present the average inference latency of each step and show the average sequence length of the routing output and final response. The inference latencies for each step are 0. 0.19,, 0.05s, 0.14s, 0.07s, and 10.24 10.24s, respectively. The average length of the routing output is 3.24 tokens, while the average length of the final response is 405.06 tokens. Compared to the LLM response generation (Step 5), the LLM expert routing (Step 3) generates much fewer output tokens and its latency is negligible (0.14s v.s. 10.24s). Therefore, our method does not bring significant inference costs.
5 Conclusion
In this paper, we reveal that the inherent bias of each vision encoder can diminish its generalization ability across other irrelevant domains by analyzing the performance of individual vision encoders versus the plain fusion of multiple encoders across various tasks. To deal with the problem, we propose MoVA, a powerful MLLM composed of noarse-grained context- expert routing and fine-grained expert fusion with MoV-Adapter. Based on multimodal context and model expertise, MoVA fully leverages representation from multiple context-relevant vision encoder experts flexibly and effectively while avoiding biased information brought by irrelevant experts. MoVA can achieve significant performance gains over current state-of-the th-the- methods in wide range of benchmarks.
Limitations. We acknowledge some limitations in our paper that require attention. One limitation is the hallucination, which refers to the generation of text that appears plausible or coherent but is factually incorrect and misleading. This issue potentially presents in all powerful MILMMs Additionally, the performance may be affected by failure cases of the context-relevant vision experts, leading to potential degradation. We plan to explore solutions for these limitations in future works.
Acknowledgments and Disclosure of Funding
The work was supported by the National Key R&D Program of China under Grant 202 1 IZD0201300.
10

---

References
[1] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742 PMLR, 2023.
[2] Jean-Bi Bean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 1123776-23736,, 2022.
[3] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[4] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguisic tasks. arXiv preprint arXiv:2312.14238, 2023.
[5] Keqin Chen, Zhao Zhang, Weili Zeng. Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023.
[6] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
[7] Wenliang Dai, Junnan Li, Dongxu Li, AnthonyMeng Huat, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards purpose vision- language models with instruction tuning. arXiv preprint 2xXiv:3305.06500 2023.
[8] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[9] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.166000 2023.
[10] Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, et al. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprint arXiv:2405. 19327 2024.
[11) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763. PMLR, 2021.
[12] Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang. Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision- language models. arXiv preprint arXiv:2312.061109 2023.
[13] Ziyi Lin, Chris Liu, Renrui Zhang. Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575. 2023.
[14] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint arXiv:2401.06209 2024.
[15] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint 2xX1v:2304.07193, 2023.
[16] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.0628/ 2023.

---

[17] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200-2209 2021.
[18] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
[19] Drew A Hudson and Christopher Manning. Gqa: new dataset for real-world visual reasoning and compositional question answering. In Proceedingy of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[20] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
[21] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69-85. Springer, 2016.
[22] Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically- labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650-1654, IEEE, 2021.
[23] Zhuofan Zong, Guanglu Song, and Yu Liu, Detrs with collaborative hybrid assignments training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6748-6758, 2023.
[24] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. arXiv preprint arXiv:2304.026433 2023.
[25] Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hcxiang Hu, Fangyu Liu, Julian Martin Eisensch- los, Urvashi Khandelwal, Peter Shaw, Ming-Wo Meng-Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Confer- ence on Machine Learning. pages 18893-1888 18912. PMLR, 2023.
[26] Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. Deplot: One-shot visual language reasoning by plot-to-table translation. arXiv preprint 2rXiv:2212.105000 2022.
[27] Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, et al. Large-scale -opain- pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915 2023.
[28] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv: 2023.
[29] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint ar 23t.:2377.117899 2023.
[30] Hao Shao, Yuxuan Hu, Letian Wang, Steven I. Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. arXiv preprint arXiv:2312.07488, 2023.
[31] Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models. arXiv preprint arXiv:2403. 16999, 2024.
[32] Yang Jiao, Shaoxiang Chen, Zequn Jie, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Lumen: Unleashing versatile vision-centric capabilities of large multimodal models. arXiv preprint arXiv:240..03304, 2024.
12

---

[33] Bingqi Ma, Zhuofan Zong, Guanglu Song, Hongsheng Li, and Yu Liu. Exploring the role of large language models in prompt encoding for diffusion models, 2024.
[34] Hongcheng Guo, Jian Yang, Jiaheng Liu, Liqun Yang, Linzheng Chai, Jiaqi Bai, Junran Peng. Xiaorong Hu, Chao Chen, Dongfeng Zhang, et al. Owl: A large language model for it operations. arXiv preprini urXiv:2309.000 09298, 2023.
[35] Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modali- ties in vision large language models via preference fine-tuning. arXiv preprint 1rXiv:2402./1411, 2024.
[36] Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models. arXiv preprint MXXx:24405.1622 2024.
[37] Dongzhi Jiang, Guanglu Song, Xiaoshi Wu, Renrui Zhang, Dazhong Shen, Zhuofan Zong, Yu Liu, and Hongsheng Li. Comat: Aligning text-to-image diffusion model with image-to-text concept matching, arXiv preprint arXiv:2404.03653 2024.
[38] Zhuofan Zong, Dongzhi Jiang, Binggi Ma, Guanglu Song, Hao Shao, Dazhong Shen, Yu Liu, and Hongsheng Li. Easyref: Omni generigenrallieed group image reference for diffusion models via multimodal Ilm. arXiv preprint arXiv:2412.09618, 2024.
[39] Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. Rule: Reliable multimodal rag for factuality in medical vision language models. arXiv preprint arXivv2407.05513 2024.
[40] Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. Mmed-rag: Versatile multimodal rag system for medical vision language models. arXiv preprint arXiv:2410. 13085, 2024.
[41] Zaiquan Yang, Yuhao Liu, Jiaying Lin, Gerhard Hancke, and Rynson WH Lau. Boosting weakly-supervised referring image segmentation via progressive comprehension. arXiv preprint arXiv:2410.01544, 2024.
[42] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9556-9567, 2024.
[43] Yian Li, Wentao Tian, Yang Jiao, Jingjing Chen, and Yu-Gang Jiang. Eyes can deceive: Benchmarking counterfactual reasoning abilities of multi-modal large language models. arXiv preprint arXiv:2404. 12966, 2024.
[44] Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Jingjing Chen, and Yu-Gang Jiang. Eventhallusion: Diagnosing event hallucinations in video 1lms. arXiv preprint 10Xv:2240.16659,, 2024.
[45] Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Jiayi Lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Guanglu Song, Peng Gao, et al. Mmsearch: Benchmarking the potential of large models as multi- modal search engines. arXiv preprint arXiv:2409. 0984009./999 /2959, 2024.
[46] Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, al. Cares: A comprehensive benehmark of trustworthiness in medical vision language models. arXiv preprint 240.::::2006000007 2024.
[47] Peng Xia, Siwei Han, Shi Qiu, Yiyang Zhou, Zhaoyang Wang, Wenhao Zheng, Zhaorun Chen, Chenhang Cui, Mingyu Ding, Linjie Li. et al. Mmie: Massive multimodal interleaved comprehension benchmark for large vision- vision-language models. arXiv preprint arXiv:24 10.10110 10139, 2024.
[48] Zekun Moore Wang. Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, et al. Rolellm: Benchmarking. eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv:2310.00746. 2023.
13

---

[49] Christoph Schubmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m Open dataset of clip-filterd 400 million image-text pairs. arXiv preprint 1r 1.02//1111144 14,00/1/11111 2021.
[50] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image- models. Advances in Neural Information Processing Systems, 25:25278-25221 2022.
[51] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A compre- hensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
[52] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023.
[53] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403. 04652, 2024.
[54] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.00 08774, 2023.
[55] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[56] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
[57] Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Hamessiing gpt4v- synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.1 1684, 2024.
[58] Shuai Shao, Zeming Li, Tianyuan Zhang. Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the JEEE/CVF international conference on computer vision, pages 1430-8439 2019.
[59] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 13:322733, 2017.
[60] Arjun Mani, Nobline Yoo, Will Hinthorn, and Olga Russakovsky, Point and ask: Incorporating pointing into visual question answering. arXiv preprint arXiv:2011.16611, 2020.
[61] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.
[62] Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. Mme: Advancing multimodal chart understanding with large-scale instruction tuning, arXiv preprint arXiv:2311.1077 2023.
[63] Shankar Kantharaj. Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. Chart-to-text: A large-scale benchmark for chart summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4005-4023 2022.
14

---

[64] Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648-5656, 2018
[65] Shengzhi Li and Nima Tajbakhsh. Scigraphqa: A large-scale synthetic multi-turn question- answering dataset for scientific graphs. arXiv preprint aa:::2388.033449 2023.
[66] Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv::306. / 17107, 2023.
[67] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and- assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024.
[68] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawa- har. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697-1706, 2022.
[69] Aniruddha Kembhavi, Mike Salvato. Eric Kolve, Minjoon Seo, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part IV 14, pages 231-251. Springer, 2016.
[70] Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291-4301, 2019.
[71] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
[72] Geewook Kim, Teakgyu Hong, Moonbin Yim, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Donut: Document understanding trans- former without ocr. arXiv preprint arXiv:2111. 155664, 7:15, 2021.
[73] Pan Lu, Ran Gong, Shibiao Jiang. Liang Qiu. Siyuan Huang, Xiaodan Liang, and Song- Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2105.04110 2021.
[74] Ming-Liang Zhang, Fei Yin, and Cheng-Lin Liu. A multi-modal neural geometric solver with textual clauses parsed from diagram. arXiv preprint arXiv:2302. /tti::2302./199999 1097, 2023.
[75] Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al. G-llava: Solving geometric problem with multi-modal large language model. arXiv preprint arXiv:2312.113700 2023.
[76] Jason Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018
[77] LAION. Gpt-4v dataset. http:://wwgggaaaceecc//eeeee/eep 2023.
[78] Carter Jimmy. Textocr-gpt4v.
textocr-gpte gpt44 2024.
[79] Gemini Team, Rohan Anil. Sebastian Borgeaud, Yonghui Wu, Jean- -aptist Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arX /805, 2023.
[80] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311.04257, 2023.
15

---

[81] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[82] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprini arXiv:2306. 13394, 2023.
[83] Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. Q-bench: benchmark for general-purpose foundation models on low-level vision. arXiv preprini 2rXi::2309.4411 2023.
[84] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255: 2023.
[85] Renrui Zhang, Dongzii Jiang, Yichi Zhang. Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403. 14624, 2024.
[86] 01-AI. Yi. hatts::/huuginggaaet http:///uuugnnggaaa...ooooollll
co/01-a1 2023.
[87] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang. Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv preprint w:Xxx:2312.08911 2023.
[88] Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sehastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305. 18565, 2023.
[89] Yash Goyal, Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Flevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913. 191,, 2017.
[90] Pan Lu. Swaroop Mishra. Tony Xia, Liang Qiu, Kai-Wei Chang, Song- Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Process- ing Systems (NeurIPS), 2022.
[91] Haoxuan You, Haotian Zhang. Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Fhih-Fu Chang, and Yinfci Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint 23 2023.
[92] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.cm 05509999 2023.
[93] Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception object discovery and retrieval. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 15325-15336, 2023.
[94] Zhuofan Zong, Dongzhi Jiang, Guanglu Song, Zeyue Xue, Jingyong Su, Hongsheng Li, and Yu Liu. Temporal enhanced training of multi-view 3d object detector via historical object prediction. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3781-3790, 2023.
[95] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. 2022.
arXiv preprint arXiv:2203.030m
[96] Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furn Wei. Layoutmmv: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM International Conference Multimedia, pages 4083-4091,
2022. 2022.
16

---

[97] Zhuofan Zong, Kunchang Li, Guanglu Song, Yali Wang. Yu Qiao, Biao Leng, and Yu Liu. Self-slimmed vision transformer. European Conference on Computer Vision. pages 432 448. Springer, 2022.
[98] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint 0rXiv:2308.00622 2023.
[99] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, al. Visionllm: Large language model is also open-ended decoder for vision-centric tasks, Advances in Neural Information Processing Systems, 36, 2024.
17

---

A
Appendix supplemental material
Table 12: Vision expert model configurations of vision experts in Mo Methods with use a convolution layer to compress the output feature.
Model
Params Resolution
Width
Depth
Output shape
DINOv2-giant
13
1.1B
518x518
1536
40
1536x37x37
Co-DETR-large
23
304M
1280 1280
1024
24
256x80x80
SAM-huge" [24
632M
1024x 1024
1280
32
256x64x64
Pix2Struct-large
25
513M
720
1536
18
1536x45x45
Deplot-base 26
92M
720x
768
12
768x45x45
Vary-base* [5]
86M
1024x 1024
768
12
512x32x32
BiomedCLIP-bern base [15
86M
224x224
768
12
768x 16
Table 13: Introduction of datasets used in the MoV-Adapter pretraining stage. The <class> placeholder represents the object category in the object detection task. The <expr> placeholder represents the expression in the REC task. The <bhox> placeholder denotes the bounding box coordinates. The <point> placeholder denotes the coordinate of point. We directly use the original question as the instruction for IMC-Instruction and ScigraphQA.
Tisk
Dataset
Tisk templatc
Please deseribe this image.
Datacomp
56
Provide one-sentence caption provided image.
Image Caption
ShacGPPTy-PPT
Can you elaborate on the elements of the picture provided?
Write a derailed description of the given image.
you claborate
elements of picture provided??
ALLaVA-4V
Write detailed description of the given image.
Objocts365
SS
Detect all objects among sclasss in the image.
Perform object detection given image within colasss,
RefCOCO
11
Locate the region this sentence describes: <eepr>. Please provide the bounding box coardinates.
Please generate short and spodlightd mention of the <hhox>> seen in the photo.
Grounding and Localization
the region sentence describes: <capr>, Please provide bounding coordinates.
Genome
Please generate short and spollighted meniion of the <bbox> part seen the photo.
PaintQA 60
How of these objects <hbox> picture?
many
objects <peint> in picture?
Flick:10K
61
Take look the image and give me the location details for any mentioned items.
Unravel the aspocts of the image and give the bounding box for the mentioned items.
AMC-Insttuction
Chart Understanding
Cham 61
What signifkant details and coaclusions can be drawn from this chart?
Can you extract the data points in this image?
ScigraphQA
LLaVAR- PT 66
Report on any text that can be clearly read in the image. Idcutify any visiblc in
provided.
Document Parsing
English Documents
every piece text
image.
I request you to apply optical character recognition to this image.
Biomedical Understanding
LLaVA-Med
Write tense but informative summary the picture.
Share comprehensive nundown the presented image.
A.1
Vision Experts
Model Configuration. We present the detailed model configurations of our task-specific vision experts in 12
We adopt the official checkpoint weights that are publicly available.
Model Description. The model descriptions used in the routing prompt released in Table 17
A.2 Training Data Details
The training process of MoVA consists of two stages. In the Appendix, we present the training datasets with corresponding task templates of the first stage in Table For the training data of the second stage, we follow the prompt format of --. (28 The MoVA models with Vicuna-7B and L.Lam3-8BB are pretrained using 64 A A100 80G GPUs for 2 days, and finetuned using 32 1000 80G GPUs for day. The MoVA with 34B I.M pretrained using 128 A 100 80G GPUs for 5 days and finetuned using 64 A100 80G GPUs for days.
18

---

Table 14: Performance of various routing component.
Table 15: Results of various vision coder combination for routing.
Table 16: Effects of routing image tokens.
Design
@Samples
MME
MMB
Design
MMB
GQA
DocVQA
FIMG
MMB
ChantQ
LLM
2K
1562/371
70.4
CLIP
70.4
64.8
81.3
144
70.4
68.3
BERT
1.6
1520/326
68.8
+DINOv2
70.1
65.1
80.5
256
68.4
MLP
1.6M
1483/305
68.1
1DINOw2+Pi2SSmma
69.5
64.4
80.9
576
70.6
68.7
A.3
More Experiments
Image segmentation. In this experiment, we aim to investigate if task-specific knowledge can improve MoVA on the segmentation task. Therefore, we introduce a simple design to extend MoVA segmentation tasks. Unlike segmentation generalists [98] that adopt additional pixel decoder with high-resolution images for high-quality mask generation, we just formulate the referring segmentation task sequential polygon generation [99]. We finetune MoVA and the baseline with SAM-Huge [24) backbone on the RefCOCO referring segmentation datasets. MoVA achieves 57.1 % gloU the testA benchmark, which is 2.6% higher than the 54.5% of baseline. This result indicates that MoVA is capable of exploiting task-specific knowledge to solve segmentation tasks.
Effect of LLM for expert routing. In this experiment, investigate the effect of the LLM in coarse-grand coarse-grained expert routing. As presented in Table [4] expert routing with LLM achieves the best performance. When the LLM is substituted for lightweight MLP classifier and a BERT encoder, we need to increase the number of routing training samples from 2K to 1.6M to preserve model performance. Besides, both MLP classifier and BERT encoder fail to perform expert routing in open scenarios as stated in Table TO Therefore, the strong tool-use capacity and generalization ability of LLM is critical to our flexible and effective expert routing.
Vision encoder for expert routing. the coarse- coarse-grained expert routing, we only adopt the image feature of base vision encoder CLIPP As presented in Table T5] the method with CLIP achieves slightly better performance than other methods since such a routing task does not require elaborate expert knowledge. Besides, incorporating other vision experts with plain fusion also brings biased information and increases cost. To achieve a better trade-off between efficiency and performance, we only use CLIP for coarse coarse-grained expert routing.
Number of image tokens in expert routing. We ablate the number of image tokens used in the expert routing stage. As shown in Table routing with 144 tokens can achieve comparable performance to methods with more tokens. Considering the additional cost brought by processing more image tokens, we only use 144 tokens for routing.
Data construction. In this experiment, we analyze the effecttivnness our routing data construction method. The test split is constructed by randomly selecting 500 samples from the routing data. We first prompt GPT4-V [54] determine whether the expert routing result is reasonable not. These evaluation results are then manually checked and we finally compute the routing accuracy using the human- verified evaluation results. Our loss-driven data construction approach achieves 94.6% accuracy (473 samples are regarded as "reasonable" by GPT4-V), demonstrating its effectiveness.
A.4 Qualitative Results
We apply Mo a wide range of real-world understanding and reasoning tasks investigate its multimodal comprehension capacity. As presented in Figure 4 and Figure MoVA successfully solves these diverse and complex cases thanks to the context-relevant expert knowledge.
A.5 Potential Societal Impacts
Positive impacts. The positive impact is to use Mo to solve various open-world tasks in real- woaldwrldd scenarios. For example, Mo can serve useful assistant to help people improve work efficiency and answer their questions.
Negative impacts. The potential negative social impact is to use MoVA model to generate misleading and false contents. This issue potentially presents in all multimodal large language models. We will try to improve its safety in future works.
19

---

Multinodal Commonsense
Useer:
What is the person with the highest dutection confidence dsing?
Moll::
The person with the highest dctoccion confatence scont, "persss she eating cream, tolding taking ice crean, She is veariig a Light pirk shirt with rnd topo left chest area,
shorts ts with accents, vearing patr white hardi st siihhtly rateed, pace thll to hald the conta lose steady whille extt eg- The lmage captares standing outdburs.
on
styled and white
heir
cat, and
candie ansent,
enjoyingg
Scene Text Recognition
CLEMENCEAU
the
in
River Valley Rd
MoVA:
Havelock 9d Chin Swee
text road sign reade:
"ou AVE
"Rixer valley agt with arrew psinting rights
"Ohle ke" arriee psinting left. left.
Jake and Nene
liser: Can you explain this more?
Studying online
The trage hamorzus nepe that comparws the aeperiances of different partias insotved learning during the coVra-19 pasdente. neee sivided into four panels, anstty:
Zoom
Teacher
coepany.
studesti,
first pansl, ean is shown throwing msrey the atr, symheltring the Fisancial
conforancing
ia usage durlng the pandonic. The text "Zoce" is written abore the sooge. shou
The sccond content repression his face. from hows and enjoy more floxible Lifestyle. The test "Teacter" is wratter above the
luptop, lying represents the teachers dio been able work
company
Students
Tn The mindd pansl, a man is shewn throwing morey into the atr, statlar to the firer test
panel,
srätten above inapo. This supessts fiattted
the
providing
tervices have
desand for ostine services duriag the gandenle.
aurvor with the vords "PUT au HAPPY race" written it, This represetts students who have aad adopt
ine learniag may be feeling avervtelsed stressef. The
Overall, the neee hanoressly highlightss the different experlences and Inpacts of osline learsing during the pandenic, with zoon the coepany protating, texcters relased ltfestyle, stuzents tealing teaminn anviromeent.
rew
Notification Understanding
User:
Jeagiiee that you are a rebot operating corputer, Like how harons operate msuse,
type
sare teats with the seyboard.
below what you see en computer strmer. Describe the image and prodict locatton and the scon as ceteslee as possible.
Hol/A:
This screenstot cosputer desktop with varlous icons and a vindow possially ses occan, The bridge is a torg, straigh: structure cornection ing The water cees blse, and the lard is racky wits is catling you, suggentin tini that sareuse nesed kevin Lin is trying to reach the user, desktop applicttimn. The lncluding the betton sam aniue time 11:18 and the date 7/22/2813. There are aiso iears applications,
BADG
browser,
screes
rabat, world eet have personst preterenzes rxeds, my nest eove tot is the browser, would click or the browser icon in the tas-borr wore ansver the matification of the tosabar. If I vere asked to open a specific and licttorr,, would click on the cor perrespoccing icon on the dcsktop. If the tandber.
If
ican
sonething.
Figure 4: Qualitative multimodal understanding results of Mo VA.
20

---

soltimenal Visual Referring Prompting
BeCo: CortenCe leogg--mggg Gemeration
liser:
part the poster nighlights?
Text-te-Inag poster bigh Lights "hetor beginn-Cnttrllle desigred to peseraate cagges free text iesersptiana, The poster cut unaa the coppoents
precise region costrol, improved sementi correctness, and letter lmage quality. also exphasizes the sugpectinn thit the system can create smoges that are both visually sopea and serenticall by accurate.
Table Understanding and Reasoning
liser:
In the bocaDA tast, how eany polats is the redel with the highest
Sep
MBc
Theuar
11
144
NoVA:
The model with the highest score the Becida task is "Fix2Stract "essart" with scare 53.2. difference soares 76.6 63.2 points.
10 45
M8
103
184
Flow Chart Understanding
User:
Start
Can you tranatate the Flouchart to Cre code?
MoVA:
Gure, here's paentble Zee code that implements logie Floactart:
+
#reclute clostreare.
Input num2
mxtnt)
Let nust, mun):
Trun
stdticin 1182:
rum2:
False
Doepley
} *hee larger nimter tor nuat sendl;
larger
reture
This code prompts the user to taput two numbers, suel and nue?, and then conpares display nesti otheriise, will display nunz.
Stop
Dense Text Recognition
User: Please recegnize the text in the image.
Absiract
MoVA:
the vhad mosker puaty
cooppeeecc mutiteeiil
motel DELLMMO, shittty internnddeg theme trage
Atstract
modden and & uee cuglt proweiig pefforeace,
Albangh
anodee
vinus? escoder greatly affects iN iPs inderstarding diverse tsag corteat. pretrained visian such
Although
aolestazing og. be CLUP viice moode lnall eutrradig stalls prama
single vision excoder that can doninate varions rage costent unserstaniinn inng, d.s.. content. allevsate the CLiP visior excoder, ve first delive intt the Interent benasor different pre-tratied
aloeas be bas of CLIP Mase cacoker dhe dchi ans bi ideccst boherke
and teea M.M.M. ataehilly mailng anll havee tnd upu iis visee regers with
pvre rokkng ststeg bnbuanedll y whlut rost sarable vise copers anoering teer
routing and fustng mesc-peciffic visior experts with morss-tooffie archarise. Is coare st-ggriine stage, desigr antsxt-aaars expert rauting strategy
image expertee
exper..
baneta
bspaage
rags alberrir condust
ststriction, input smage, expertise viston expetts. This benetats free the elacorately ability
ofeamoof-ntae centiae--rneeeeee repers counce
ratred
the eisture-ct soure-ot-riiin tase-specific knnsleddg sortrraitt-eess ecceret slupter experts.
Fatler milaaving be premalation Letio abaity wendict exteaiive esperaadd
(Pot-aaaster?
valatos Nork ca adiine dipaiiedd periettates aato ever carvel ialood-
nu Ltteddl context and model exportiso, farther erhancing the gererallaation auility. experiments tuste affectiveness significan propased
aithout bells
over current state the retheds in side range of chollenging lerging muttincea!
Figure 5: Qualitative multimodal understanding results of Mo
21

---

Table 17: Model descriptions used in the context-aware expert routing. We describe the pros and cons each expert model in the routing prompt. We only present captions for each expert here.
DINOv2 description
(I) This model demonstrates exceptional prediction capabilities across a range of image-related tasks, including image classification, object detection, segmentation, and image retrieval. The model leverages advanced self supervisdd leaming techniques achieve high performance without relying heavily on labeled data.
(2) This model shows very strong prediction capabilities on tasks such as image classification, detection, segmentation, and image retrieval. However, it encounters challenges accurately reading text within images.
(3) This model can effectively extract the accurate spatial and semantic information from natural images.
Co-DETR description
(I) This model is state-of-thc-arr object detector pretrained natural images. It can enable models solve object-centric problems. Nonetheless, this model struggles with processing background elements natural scenes.
(2) This model is cutting-edge object detection model that can accurately detect objects in images. However, it struggles with identifying text in images. (3) This model stale-of-the-art object detector that can identify objects in images.
SAM description
(I) This model is an image segmentation model.
This model can segment the precise location of either specific objects image every object in image.
(2) This model is leading image segmentation framework and achieves strong zero-shot segmenta- tion performance.
(3) This model is promotable segmentation system with zero-shot generalization to unfamiliar objects and images.
Pix2Struct description
(1) This model excels in text recognition, achieving statc-of-the-art text analysis results across distinct domains: documents, illustrations, user interfaces, natural images containing text, and images of charts.
(2) This model demonstrates exceptional proficiency in text recognition, delivering cutting-edge text analysis performance across various domains. (3) This model can automate the extraction of information from scanned documents, making it casier to digitize and manage large volumes of paperwork.
Deplot description
(1) This model specialized model designed achieve state-of-the-art plot and chart understanding performance.
(2) This model is fine-tuned version of an existing text recognition model. It has been specifically trained achieve superior performance in plot and chart understanding tasks. (3) This model can help detect the text within the input document, diagram, and chart images.
Vary description
(1) This model achieve more fine-grained vision perception for images with text, such document-leved Chinese/English OCR, book image to markdown or LATEX, Chinese/English chart understanding.
(2) This model can handle images with text effectively and accurately, enabling advanced tasks such document OCR and chart understanding.
(3) This model accurately process images with text, enabling tasks such OCR. However, it cannot process natural images without text.
Biomed CLIP description
(1) This model is foundation model designed for biomedical rision-language processing.
(2) This model capable of biomedical images, such as chest X-ray and radiology images.
(3) This model is state-of-thh-ar biomedical vision-language model. has been shown achieve
significant improvements biomedical image-text tasks.
22

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We clarify our main claims and contribution in the abstract and introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
23

---

Justification:
We discuss the limitations of MoVA in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully release the implementation details and experiment settings in the Section Experiments and Section Appendix.
Guidelines:

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
The codes and models will be released when the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
soteeeeiubmmiiioooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https CodeeSumiiii wwwiiisssooosiiiiio
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
25

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We present the experiment details in Section Experiments and Section Ap- pendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We conduct experiments only once following the previous baseline LLaVA 28 It would be too computationally expensive to conduct the pretraining and finetuning multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We release the experiments compute resources in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
26

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttp::/naurriss.c http://aaarrr..ccooaaaaaaccccaaa
Answer: [Yes]
Justification: The research conducted in the paper conforms, every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss the potential societal impacts of our work in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
27

---

Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please see Section Experiments.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: Please see Section Experiment and supplementary material. Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
28

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "7KLkwsrLAd", "reviewer": "Reviewer_Lntx", "summary": "In this paper, the authors propose MoVA, a powerful MLLM composed of coarse-grained context-aware expert routing and fine-grained expert fusion with MoV-Adapter. Based on multimodal context and model expertise, MoVA fully leverages representation from multiple context-relevant vision encoder experts flexibly and effectively while avoiding biased information brought by irrelevant experts. MoVA can achieve significant performance gains over current state-of-the-art methods in a wide range of benchmarks.", "questions": "Please refer to Weakness Section for more details.", "limitations": "The authors have clearly illustrated the weakness of this work by pointing out the hallucination and the model performance may be affected by failure cases of the context-relevant vision experts.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors comprehensively investigate various vision encoders and analyze the performance of individual vision encoders versus the plain fusion of multiple encoders across various tasks.\n2. The authors propose MoVA, a powerful MLLM composed of coarse-grained context-aware expert routing and fine-grained expert fusion with MoV-Adapter.\n3. The proposed MoVA can achieve significant performance gains over state-of-the-art methods in a wide range of challenging benchmarks.", "weaknesses": "1. Despite the proposed MoVA in this paper achieving state-of-the-art performance across multiple tasks, leveraging multiple visual encoders in MLLM design does not introduce novelty. Additionally, the design of MoV-Adapter also lacks innovation.\n2. This paper lacks a comprehensive evaluation regarding the routing selection of multiple visual encoders using LLMs, necessitating a more targeted and comprehensive assessment from the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Lntx,\n\nThank you for appreciating our approach. We will address your concerns below.\n\n**Q1: Leveraging multiple visual encoders in MLLM design does not introduce novelty. Additionally, the design of MoV-Adapter also lacks innovation.**\n\nLeveraging multiple visual encoders is not our novelty. Please refer to the Q1 of our global response.\n\n**Q2: This paper lacks a comprehensive evaluation regarding the routing selection of multiple visual encoders.**\n\n1. We have ablated the design of routing selection in Table 9, Table 10, and Table 14. We verify the effectiveness of our routing data in the line 598.\n2. We present some routing cases in the pdf file of the global response.\n3. We calculate the activation probability for each expert on several benchmarks:\n\n| Benchmark | DINOv2 | Co-DETR | SAM | Vary | Pix2Struct | Deplot | BiomedCLIP |\n| - | - | - | - | - | - | - | - |\n| MME | 63.6% | 12.9% | 2.8% | 26.2% | 29.3% | 28.5% | 0% |\n| MMBench | 54.6% | 15.8% | 3.2% | 21.9% | 22.6% | 22.6% | 0% |\n| DocVQA | 0% | 0% | 0% | 97.9% | 99.8% | 98.3% | 0% |\n| POPE | 100% | 100% | 0% | 0% | 0% | 0% | 0% |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qnOmGjfS3p", "reviewer": "Reviewer_dPiw", "summary": "This paper considers a critical issue in vision language models, i.e., the vision encoder design. The authors proposes MoVA, a coarse to fine routing strategy, adaptively routing and fusing task specific vision experts which best suit the task. Experiments conducted on a wide range of MLLM benchmark demonstrate the advantages of the proposed MoV adapter.", "questions": "The main concern is the rationality of using MoV on vision side, while two step routing is interesting, it adds the complexity (using two LLMs), and it is not clear the advantages of MoV over the latest MLLM benchmark.", "limitations": "None", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe motivation is clear, and the method is easy to follow\n\n2.\tThe proposed coarse to fine routing strategy is reasonable", "weaknesses": "1.\tAs for OCR and related tasks, it has been validated that high resolution is much more import than model size, and image split like LLaVA Next, Monkey, Ferret etc. have been widely used. I wonder the importance of including task specific models. It looks redundancy, and importantly, we cannot include all models that cover different tasks exhaustively\n\n2.\tAs shown in Table 1, the main improvement results from the OCR related Doc and Chart task, which may resolved by current MLLMs baseline such as LLaVa Next, I wonder the  necessity of the using MoE experts, as other module, adding resolution is more reasonable. In Table 3, since the PT, SFT and pretrained models are different, it is hard to validate the gains are from the MoV experts.  \n\n3.\tDoes the method introduce error propagation, say, the first stage LLM routing is not correct, which may harm the performance of the whole framework. On the other hand, if the coarse routing is simple, could we say that this stage we do not need to carefully design, and a tiny LLM or even rule based according to the prompt is enough? Since current framework seems complex using two LLMs, the comparison is not fair, and I wonder if there is a better routing strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer dPiw,\n\nThanks for giving so many constructive suggestions for our paper, I will clarify the settings.\n\n**Q1: The rationality of using task-specific vision models.**\n\n1. We have conducted extensive experiments to benchmark different vision encoders in Table 1. We find a single CLIP encoder cannot generalize to various image domains, such as text recognition tasks.\n2. Besides, a task-specific model can achieve optimal performance in its proficient task. It is reasonable to incorporate task-specific vision experts to improve performances on their relevant tasks. We also find the \"plain fusion\" of multiple vision experts can bring performance gains to the baseline with a single CLIP encoder in Table 1.\n4. Many previous works [1-5] have demonstrated that the CLIP encoder fails to generalize to some domains and the visual capacity of MLLM can be enhanced by introducing additional vision experts.\n\n[1] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. CVPR 2024.\n\n[2] VCoder: Versatile Vision Encoders for Multimodal Large Language Models. CVPR 2024.\n\n[3] SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models. ICML 2024.\n\n[4] SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models. ECCV 2024.\n\n[5] Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models. ECCV 2024.\n\n\n**Q2: We cannot include all models that cover different tasks exhaustively.**\n\n1. We agree with this idea but we do not aim to exhaustively incorporate extensive vision encoders to cover all the tasks in this paper. \n2. Instead, we aim to reveal the common limitations of previous mixture of vision experts methods and propose a new paradigm. This paradigm provides a new perspective for the community that we can dynamically incorporate relevant vision experts for specific tasks with poor performances in real-life scenarios.\n\n\n\n**Q3: Adding resolution is more reasonable.**\n    \n1. We agree that enlarging the image resolution can bring performance gains on OCR tasks. We also find **both adding resolution and adding vision experts (MoVA) are orthogonal approaches to enhance visual capacity in our experiments.** For example, we train a MoVA-7B model with the image split technique of LLaVA-NeXT (MoVA-HD) and compare it with the baseline on OCR tasks. Specifically, we divide an input image into 4 patches. The MoVA-HD can improve the score on the DocVQA benchmark from 81.3 to 84.0. This indicates such a high resolution setting also benefits MoVA.\n2. The experiments in Table 1 have demonstrated the text recognition expert, such as Pix2Struct, can surpass the CLIP encoder on text-oriented VQA benchmarks by clear margins with aligned image token settings.\n3. As described in Q2 of our global response, MoVA can achieve superior performance to LLaVA-NeXT with high resolution technique under the same settings.\n\n\n**Q4: It is hard to validate the gains are from the MoV experts.**\n\nPlease refer to the Q2 of our global response.\n\n**Q5: Does the method introduce error propagation?**\n\nThe incorrect routing results may harm the performance of the MLLM. In some cases, we find the multimodal gating in MoV-Adapter can assign low scores to these experts that are not relevant but incorrectly activated. We release some routing cases in the pdf file of global response.\n\n**Q6: Routing strategy design.**\n\n1. In the Table 14 of paper appendix, we discuss the effect of LLM for expert routing. Specifically, we compare the LLM router with BERT router and MLP router. For smaller router variants, they require more routing training data and fail to attain comparable performance.\n2. We only use 1 LLM for expert routing and response generation. Using a tiny LLM in the routing stage can bring addtional model parameters.\n3. To ablate the routing performance of tiny LLM router (InternLM2-1.8B) and standard LLM router (Vicuna-7B), we mutually label a subset of our selected 500 routing evaluation samples as \"easy\" or \"hard\". The routing performance of MoVA on these 500 samples have been discussed in Appendix A.3. Finally, We obtain 50 easy samples and 20 hard samples. The routing accuracy is obtained by human evaluation. As shown in the table, the tiny LLM router achieves comparable accuracy on simple evaluation samples while failing to catch up to the standard LLM router on hard evaluation samples.\n\n| Router | Simple | Hard |\n| - | - | - |\n| InternLM2-1.8B | 96% | 55% |\n| Vicuna-7B | 98% | 80% |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A2i4wfmAFB", "reviewer": "Reviewer_gqRq", "summary": "This paper proposes MoVA, a powerful and novel MLLM, adaptively routing and fusing task-specific vision experts with a coarse-to-fine mechanism. MoVA first leverages the tool-use capabilities of LLM, routing the most appropriate experts from expert candidates and then uses the MoV-Adapter module to enhance the visual representation.", "questions": "In line 102, I just found four components, but the paper claims five.", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The writing is easy to follow.\n2. The idea of using LLM to route the most appropriate experts is interesting.", "weaknesses": "1. Is there an adapter structure in MoVA? Why do we call it MoV-Adapter?\n2. The experiments use SFT data from different MLLMs should be aligned (the same). The comparison is not fair with different SFT data.\n3. More vision encoder enhancement methods should be compared using the same SFT data, including\n\n[1] VCoder: Versatile Vision Encoders for Multimodal Large Language Models.\n\n[2] MoF (Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs)\n\n[3] MouSi: Poly-Visual-Expert Vision-Language Models\n\n[4] BRAVE: Broadening the visual encoding of vision-language models\n\n4. The inference speed of LLM is very slow. I think MoVA using LLM as a router selection will decrease the speed of the whole process. How to solve this problem?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer gqRq,\n\nThanks for your comments. We will address your concerns below.\n\n**Q1: Is there an adapter structure in MoVA? Why do we call it MoV-Adapter?**\n\nSimilar to conventional MLLMs, MoVA comprises a pretrained LLM, a base vision encoder, multiple vision experts, and a projector that connects the LLM and vision encoders.\nWe call this projector \"MoV-Adapter\" since it also can adapt output features of various activated vision experts to the base vision encoder feature space by elaborate expert knowledge extractor.\n\n**Q2: The comparison is not fair with different SFT data.**\n\nPlease refer to the Q2 of our global response.\n\n**Q3: More vision encoder enhancement methods should be compared using the same SFT data.**\n\nWe compare our method with VCoder [1], MoF [2], and MouSi [3] under the same SFT data and LLM (Vicuna-v1.5-7B) settings. We directly follow their corresponding evaluation benchmarks.\n\n| Model | Seg CS($\\uparrow$) | Seg HS($\\downarrow$) | Ins CS($\\uparrow$) | Ins HS($\\downarrow$) | Pan CS($\\uparrow$) | Pan HS($\\downarrow$) |\n| - | - | - | - | - | - | - |\n| VCoder-7B | 88.6 | 10.4 | 71.1 | 26.9 | 86.0 | 12.8 |\n| MoVA-7B | 89.1 | 10.0 | 73.8 | 24.7 | 87.6 | 11.4 |\n\n| Model | MMVP | POPE |\n| - | - | - |\n| MoF-7B | 28.0 | 86.3 |\n| MoVA-7B | 30.0 | 87.8 |\n\n| Model | VQAv2 | GQA | SQA | TextVQA | POPE | MMB | MMB_CN |\n| - | - | - | - | - | - | - | - |\n| MouSi-7B | 78.5 | 63.3 | 70.2 | 58.0 | 87.3 | 66.8 | 58.9 |\n| MoVA-7B | 80.1 | 65.1 | 71.2 | 68.5 | 87.8 | 67.2 | 58.4 |\n\n**Note that the authors of BRAVE [4] did not publicly release the codes, models, and data.** Due to the limited time of the rebuttal period, it is hard to reproduce its performance during rebuttal. We will add this comparison in the revised manuscript.\n\n[1] VCoder: Versatile Vision Encoders for Multimodal Large Language Models.\n\n[2] MoF (Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs)\n\n[3] MouSi: Poly-Visual-Expert Vision-Language Models\n\n[4] BRAVE: Broadening the visual encoding of vision-language models\n\n**Q4: Using LLM as a router will decrease the inference speed. How to solve this problem?**\n\nAs highlighted by the general response, the expert routing only brings marginal inference costs thanks to our elaborate designs:\n\n1. We use 144 compressed image tokens for expert routing. Note that the current state-of-the-art baseline LLaVA-NeXT employs the LLM to process 2880 image tokens.\n2. Only the vision feature of the base vision encoder is used for expert routing.\n3. During expert routing, we prompt the LLM to directly generate the option's letter of selected vision experts from the given model pool (e.g., \"A\" for DINOv2). As presented in the general response, the routing answer is short and **the average length of the routing output token sequence is only 3.24**.\n\n**Q5: In line 102, I just found four components, but the paper claims five.**\n\nThis is a typo and it should be \"comprises four key components\". We will fix this typo in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zN2f8fB60B", "reviewer": "Reviewer_FVeU", "summary": "The paper proposes a MoVA to extract and fuse task-specific knowledge from different visual encoder. Besides, the paper designs a coarse-to-fine mechanism, which dynamically selects the most suitable vision experts, then extracts and fuses task-specific knowledge from various experts. The experimental results are encouraging and validate the performance of the proposed method.", "questions": "Why is the last row of MoVA results in Table 1 different from those in Table 3? In other words, The caption of Table 1 shows that the same data was used to train each model. Does this mean that all models used LLaVA-1.5 data? If so, I think this table will inspire the community and I would like to ask if you can release the checkpoints of these models.", "limitations": "1. As mentioned in the conclusion, the performance may be affected by failure cases of the context-relevant vision experts, leading to potential degradation. Additionally, it is influenced by the choice of encoder results in the first stage of inference.\n2. The model's complex inference structure increases the inference burden.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. This paper proposes a new perspective, which is to improve the multi-task reasoning ability of MLLMs based on the different perception ability of the different encoders. \n2. The paper is fluent and easy to understand.\n3. The extensive experiments evaluate the effectiveness of the proposed approach.", "weaknesses": "1. The number of components in line 102 does not match the serial numbers.\n2. During the inference phase, the process involves two rounds of LLM inference and two Base Encoder, which results in significant inference time and GPU memory usage. This poses a challenge for the practical application of MLLMs.\n3. This is a well-completed article, but I am concerned about its novelty. The method seems similar to the adapter+agent approach described by Wang et al. [1]. Could you explain the differences and advantages of your method compared to theirs?\n\n[1] Wang X, Zhuang B, Wu Q. Modaverse: Efficiently transforming modalities with LLMs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 26606-26616.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer FVeU,\n\nThanks for appreciating our work and your advice. We will address your concerns below.\n\n**Q1: The number of components in line 102 does not match the serial numbers.**\n\nThere are four components in MoVA. We will fix this typo in the revised version.\n\n**Q2: The process involves two rounds of LLM inference and two Base Encoders, which results in significant inference time and GPU memory usage.**\n\n1. As described in the general response, MoVA only runs the base encoder once during inference since its base vision feature can be reused.\n2. Please refer to the Q3 of our global response.\n\n**Therefore, we think the inference cost is not the bottleneck for MoVA.**\n\n**Q3: The method seems similar to the adapter+agent approach described by Wang et al [1].**\n\nBoth MoVA and ModaVerse are MLLM frameworks but they are different:\n\n| Model | Motivation | Setting | Method (input side) | Method (output side) |\n| - | - | - | - | - |\n| MoVA | We reveal that the feature fusion of fixed mixture of vision experts fails to achieve the optimal performance and leads to performance degradation if we do not consider the relations between model expertise and the current task. | MoVA focuses on the visual component design of large vision-language models | MoVA first uses the LLM to dynamically select $K$ experts that are most relevant to current task according to the model expertise and context. Then the vision features of selected experts and the base encoder are fused by an elaborate MoV-Adapter. | MoVA generates text outputs by a LLM. |\n| ModaVerse | ModaVerse operates directly at the level of natural language. It aligns the LLM’s output with the input of generative models, avoiding the complexities associated with latent feature alignments, and simplifying the multiple training stages of existing MLLMs into a single, efficient process. | ModaVerse is capable of interpreting and generating data in various modalities. | ModaVerse employs a single encoder to encode diverse multi-modal inputs and aligns them with LLM through a set of adapters. | ModaVerse treats the LLM as an agent and this agent produces a meta-response to activate generative models for generating the final response. \n\n**Advantages:**\n\n1. ModaVerse uses a single encoder to encode inputs of various modalities. However, MoVA can dynamically select the most appropriate vision experts that are relevant to current inputs and tasks.\n2. ModaVerse adopts a set of linear projection layers to align multimodal inputs and the LLM. In our paper, we propose the MoV-Adapter as the projector and demonstrate it can attain better performance than conventional linear layers.\n\n[1] ModaVerse: Efficiently Transforming Modalities with LLMs. CVPR 2024.\n\n**Q4: Why is the last row of MoVA results in Table 1 different from those in Table 3? Do all models in Table 1 use LLaVA-1.5 data? If so, I would like to ask if you can release the checkpoints of these models.**\n\n1. The MoVA model in Table 3 uses more training data than the models in Table 1. The detailed data settings of Table 3 MoVA model are presented in the line 210.\n2. As described in the line 229, we additionally incorporate several datasets (DocVQA, ChartQA, RefCOCO, LLaVA-Med, VQA-RAD, and SLAKE) to the LLaVA-1.5 665k SFT data to train the models in Table 1. \n3. We will definitely release our codes and models to facilitate the community if the paper will be accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Global response: We would like to extend our sincere gratitude to all reviewers for devoting their valuable time and expertise to reviewing our paper. We are delighted to hear that the reviewers have generally acknowledged and appreciated the contributions made in our work, which include
 
1. Clear motivation (dPiw) and reasonable method (dPiw, gqRq).
2. A new perspective to improve the reasoning ability of MLLMs (FVeU).
3. Comprehensive and extensive experimental results (Lntx, FVeU).
4. State-of-the-art performance (Lntx).


We express our sincere appreciation to all the reviewers for their insightful suggestions. To address the reviewers' concerns, we provide the following summaries and discuss them in rebuttal:

1. Paper novelty (Lntx, FVeU).
2. Performance comparison (dPiw, gqRq).
3. Inference cost (gqRq, FVeU).
4. Expert routing design and evaluation (Lntx, dPiw).
5. Necessity of vision experts (dPiw).

We will address the raised concerns and incorporate these improvements into our paper once we can edit it on OpenReview.
We make a global response for the Q1, Q2, Q3:

**Q1: The novelty of our paper**

1. Given $N$ vision encoders, previous works of the community all focus on how to directly fuse features generated by these $N$ **fixed mixture of vision experts** to solve every task without considering model expertise.
2. We provide a new perspective that we can first leverage the reasoning and tool-use ability of LLM to select $K$ ($K<N$) vision experts that are most relevant to the current task and then fuse these $K$ **dynamic mixture of vision experts**. 
3. We conduct comprehensive experiments to reveal that the feature fusion of fixed mixture of vision experts fails to achieve the optimal performance and leads to performance degradation if we do not consider the relations between model expertise and the current task.

**Q2: Performance comparison using the same data**

We conduct experiments on LLaVA-1.5-7B and LLaVA-NeXT-7B with the same data and LLM as MoVA-7B. The performance comparison can be found in the pdf file of the global response. 
As presented in the table, our method still outperforms LLaVA-1.5 and recent state-of-the-art LLaVA-NeXT on a wide range of benchmarks under the same settings.
More importantly, MoVA with 576 image tokens can beat LLaVA-NeXT with image split technique and high-resolution input image on text-oriented benchmarks.


**Q3: Inference analysis**

As illustrated in Figure 1 of our paper, MoVA consists of two stages: coarse-grained context-ware expert routing and fine-grained expert fusion with MoV-Adapter. This two-stage inference pipeline can be further broken down into 5 steps:
1. (Stage 1) Data preprocessing. We first process the input image with image processors and convert the input text into a token sequence with the LLM tokenizer.
2. (Stage 1) Base encoder forward. We extract the base image feature using the base CLIP encoder. Note that we only run the base encoder once since its output feature can be preserved and reused in the fourth step.
3. (Stage 1 ) LLM routing generation. We compress the base image features into 144 image tokens. The LLM generates a concise routing answer based on the compressed image feature and routing instruction.
4. (Stage 2) Vision experts and MoV-Adapter forward. According to the multimodal context and routing results generated in the previous step, we fuse vision features of the base encoder and activated experts in a coarse-to-fine manner.
5. (Stage 2) LLM response generation. The LLM generates the final response given the fused vision features and user instructions.

To investigate the inference efficiency of each step, we randomly select 200 images from the COCO val2017 dataset and adopt the common image caption instruction: *Describe this image*. The temperature for generation is 0. The latency is measured using bfloat16 and flash-attention 2 on an A100 80G GPU. The benchmark results of model inference are presented here:

| Step 1 | Step 2 | Step 3 | Step 4 | Step 5 | Routing output | Final response |
| - | - | - | - | - | - | - |
| 0.19s | 0.05s | 0.14s | 0.07s | 10.24s | 3.24 tokens | 405.06 tokens |

We present the average inference latency of each step and show the average sequence length of the routing output and final response.
**Compared to the LLM response generation (Step 5), the LLM expert routing (Step 3) generates much fewer output tokens and its latency is negligible (0.14s *vs* 10.24s).** Therefore, our method does not bring significant inference costs.

Moreover, we compare the inference efficiency of MoVA with the common **fixed mixture of vision experts** paradigm. We use DINOv2, Co-DETR, SAM, Vary, Pix2Struct, Deplot, and BiomedCLIP as the mixture of vision experts. Specifically, we do not present the latency of LLM final response generation (Step 5) since the only difference is the visual component.

| Mixture of vision expert | Step 1 | Step 2 | Step 3 | Step 4 | Total |
| - | - | - | - | - | - |
| MoVA | 0.19s | 0.05s | 0.14s | 0.07s | 0.45s
| Fixed | 0.19s | 0.05s | N/A | 0.43s | 0.67s |

The results reveal that dynamically selecting the most $K$ relevant experts from $N$ experts can accelerate model inference while boosting performances.

We also compare the memory usage of MoVA and MoVA without expert routing. As shown in the following table, the LLM routing only increases the memory usage from 25165 MB to 27667 MB.

| MoVA | MoVA w/o routing |
| - | - |
| 27667 MB | 25165 MB |

Meta-review: This paper introduces a new MLLM called MoVA, which is composed of coarse-grained context-aware expert routing and fine-grained expert fusion with MoV-Adapter. By utilizing multimodal context and model expertise, MoVA fully leverages representation from multiple experts. The experimental results validate the model's effectiveness across multiple benchmarks.

The paper was evaluated by four reviewers. Three reviewers recommended acceptance (1x accept, 1x borderline accept, 1x weak accept), while one reviewer suggested a borderline rejection. Overall, this work presents a novel and effective MLLM solution that has the potential to inspire further research in the field. 

The authors are encouraged to incorporate their responses to the reviewers' comments in the revised version to enhance the paper's clarity and comprehensiveness.
Decision: Accept (poster)