Poly-Visual-Expert Vision-Language Models

Authors: xiaoran  fan
tao  ji
changhao  jiang
shuo  li
senjie  jin
sirui  song
junke  wang
lu  hong
guodong  chen
ming  zheng
caishuang  zhang
rui  huang
zhiheng  zheng
yuhao  xi
shihan  zhou
junjie  dou
hang  ye
tao  yan
  gui
qi  zhang
xipeng  qiu
xuanjing  huang
zuxuan  wu
yu-gang  jiang
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 14:06:45

Abstract: Current large vision-language models (VLMs) frequently face challenges such as the limited capabilities of a single visual component and the excessive length of visual tokens. These issues can limit the model's ability to interpret complex visual information and over-lengthy contextual information accurately. Tackling these challenges is crucial for enhancing the performance and applicability of VLMs. This paper proposes leveraging the ensemble experts technique to synergize the capabilities of individual visual encoders, including those skilled in image-text matching, image segmentation, OCR, etc. This method introduces a fusion network that consolidates the outputs from different visual experts while bridging the gap between image encoders and pre-trained LLMs. In addition, we explore different positional encoding schemes to mitigate the waste of positional encoding caused by lengthy image feature sequences, effectively addressing the issue of position overflow and length limitations. For instance, in our implementation, this technique significantly reduces the positional occupancy in models like SAM, from a substantial 4096 to a more efficient 64 or even down to 1. Experimental results show that VLMs with multiple experts consistently outperform isolated visual encoders, with notable performance improvements as more experts are integrated. Our codes are available on our project website.

Main paper: 
 introduction: 
 Current large vision-language models (VLMs) demonstrate significant potential in tasks requiring joint visual and linguistic perception, such as image captioning (Agrawal et al., 2019) , visual question answering (Antol et al., 2015) , visual grounding (Yu et al., 2016) , and autonomous agents (Durante et al., 2024; Xi et al., 2023) . VLMs harness large language models (LLMs) as cognitive foundation models to empower various vision-related tasks, while one vision component, such as CLIP (Radford et al., 2021) , typically serves as auxiliary modules that provide additional visual perception (Liu et al., 2023b) . However, the perception abilities of the individual vision models still lag behind, even in simple tasks like counting. (Yamada et al., 2022; Thrush et al., 2022; Yuksekgonul et al., 2022) . This gap highlights a significant limitation in these models' capacity to process and understand visual information as effectively as they handle linguistic data. According to the operation of the vertebrate visual system, with each functional unit encoding different visual aspects in parallel, retinal ganglion cells transmit distinct features to the brain (Baden et al., 2016) . This biological mechanism suggests a model structure where the varied visual information should be parallelly encoded by diverse perception channels.Figure 1 : Left: Comparing InstructBLIP, Qwen-VL-Chat, and LLaVA-1.5-7B, our poly-visualexpert achieves SoTA on a broad range of nine benchmarks. Right: Performances of the best models with different numbers of experts on nine benchmark datasets. Overall, triple experts are better than double experts, which are better than a single expert.To this end, the community has verified that each model, with its unique vision processing approach contributes differently to understanding visual content (Chen et al., 2023a) . CLIP, with its contrastive learning approach, excels in aligning images with textual descriptions, providing a robust semantic understanding (Radford et al., 2021) . DINO v2 , through its self-supervised learning at both the image level and patch level, offers significant advances in robust and stabilized feature extraction without relying on labeled data (Oquab et al., 2023) . LayoutLM v3 's specialization in document AI tasks demonstrates the power of visual text processing (Huang et al., 2022) . (Wang et al., 2023a) empirically investigated different visual tokenizers pre-trained with dominant methods (i.e., DeiT (Touvron et al., 2021) , CLIP, MAE (He et al., 2021) , DINO (Caron et al., 2021)) , and observed that CLIP could capture more semantics, whereas the other models excelled at fine-grained perception. However, on the multimodal leaderboard organized by OpenCompass2 , the visual encoders of all opensource VLMs are based on the pre-trained CLIP encoder family. Many researchers have pointed out the shortcomings of the CLIP encoder, such as the inability to reliably capture even basic spatial factors of images (Kamath et al., 2023) , suffering from object hallucination (Li et al., 2023c) , and so on. In light of the distinct capabilities and limitations of these diverse vision models, a key question emerges: How can we incorporate the strengths of multiple visual experts so that they work in synergy to improve overall performance?Drawing inspiration from biology, we take on the poly-visual-expert perspective and design a novel model, similar to how the vertebrate visual system operates. Consequently, in the process of developing VLMs with poly-visual experts, three problems are in major concern:(1) whether the poly-visual experts are effective; (2) how to better integrate multiple experts; and (3) how to avoid exceeding the LLM's maximum length with multiple visual experts?In order to verify whether multiple visual experts are effective for VLMs, we construct a candidate pool consisting of six well-known experts, including CLIP, DINO v2 , LayoutLM v3 , Convnext (Woo et al., 2023) , SAM, and MAE. Using LLaVA-1.5 as the base setup, we explored single-expert, double-expert combinations, and triple-expert combinations in nine benchmarks. The results, as shown in Figure 1 , indicate that as the number of visual experts increases, the VLMs acquire richer visual information (due to more visual channels), and the upper limit of the multimodal capability improves across the board.In existing single visual channel VLMs, visual signals are transmitted using either the MLP network (Liu et al., 2023a; Wang et al., 2023b) or the Q-Former network (Bai et al., 2023; Dai et al., 2023b) . To support multi-channel signal transmission from multiple experts, we modified both methods for poly-expert fusion networks separately. The proposed method also compresses the local visual information by multi-patch-one-token for better transmission efficiency and reduces the quadratic computational cost of subsequent processing of VLMs.In position-aware VLMs, vision tokens consume a staggering amount of positional embeddings. Taking a single-turn multimodal dialogue in VQA as an example, with the SAM expert, the number of vision tokens (about 4096) is more than 500 times higher than the number of text tokens (about 8.7). Inspired by the fact that visual experts already have positional encodings, we believe it is redundant to again assign a VLM position embedding to each visual token individually. Therefore, we explore different positional encoding schemes to effectively address the issue of position encoding waste. The results show that the two schemes: sharing one position for all patches and 2D positional encoding (rows plus columns) are able to reduce the position consumption (in the case of CLIP, the PE used drops from 576 to 24 or even 1), while the performance is still comparable.Our contributions can be summarized as follows:• We introduce a poly-visual-expert VLM that synergistically combines the strengths of various visual encoders to improve the overall capabilities of VLMs. • We tackle the challenge of vision token overflow in VLMs by proposing multi-patchsingle-token projection and efficient positional encoding solutions.  
the overview: 
 When a user uploads an image of wind pollination in a conical inflorescence and asks "Which cones make pollen?" the image is processed in sequence through the encodings of the CLIP expert, the SAM expert, and the LayoutLM expert, yielding three sets of visual representations. Subsequently, a poly-expert fusion network compresses the multi-channel visual information and aligns it multimodally to the vision input tokens for our poly-visionexpert model. The user's question is processed into text tokens by the LLMs' Embedding layer. Finally, our model generates the correct answer "Male cones make pollen." by employing its VQA capabilities to understand the vision-language question, and its OCR capabilities to recognize the answer text from the image.Res In order to accomplish the above task, we propose the poly-vision-expert model, which consists of three fundamental components:1. a multi-expert visual encoder, which combines the experts selected from a pool; 2. a poly-expert fusion network, which is implemented as a simple projection fusion method or a Q-Former fusion method (Li et al., 2023b) ; 3. a pre-trained open-source LLM (e.g., Vicuna v1.5).Figure 2 shows an overview of our poly-vision-expert model. The core of a Vision-Language Model is typically an LLM which is pre-trained on large-scale textual corpus. To perceive the visual signals, a vision encoder and vision-language connection layer are adopted to separately extract the visual features and align them to the semantic space of LLM.The VLM takes as input a sequence comprised of interleaved text and image segments, denoted as X = (. . . , T 1 , I 1 , T 2 , I 2 , . . . ), where text fragments T are processed by the tokenizer and embedding layer of the LLM, and image segments I are fed to the vision encoder. To ensure the universality and generalizability of the vision encoder, it is common practice to freeze its pre-trained parameters. In this paper, we rethink the design of the visual encoder in VLMs and aim to improve its capability by ensembled experts.After extensive investigation, we choose six vision encoders skilled in different domains, including CLIP (2021), DINO v2 (2023), LayoutLM v3 (2022), Convnext (2023), SAM (2023), and MAE (2021). They differ significantly from each other in terms of input resolution, hidden size, model type, model size, pre-training tasks, and training methods, as shown in Table 1 . Their detailed descriptions can be found in Appendix A.Given a image I in the input sequence and a vision expert encoder e i (•), we can obtain the representation vectors of n image patches:EQUATIONAssuming we have three experts(e i (•) ∈ R n i ×d i , e j (•) ∈ R n j ×d j , e k (•) ∈ R n k ×d k ), the final sequence of image representations V I is a concatenation of the three output sequences.EQUATIONIt is worth noting that each expert outputs a different number (n i vs. n j vs. n k ) and dimension (d i vs. d j vs. d k ) of vectors, and we will handle these differences in the fusion network.Since the dimension and number of output sequences are often different for different visual experts, a fusion network needs to be designed to unify the processing. Following LLaVA (Liu et al., 2023b) and BLIP (Li et al., 2022) , we propose an MLP projection fusion network and a Q-Former fusion network, respectively. Given a specific expert e i (•), the first layer network is defined as MLP(1) iEQUATIONk (e k (I))EQUATIONIn practice, multiple experts output a large number of vision tokens, which not only increases the computational cost and memory usage of the VLM but also tends to exceed the maximum length limit during inference. Therefore, we propose multi-patches-one-token projection to proportionally reduce the number of tokens output by each expert. Since image signals have local or sparse properties, it is practical to use one token to represent neighboring patches. Take m-patch-one-token for example, we make the input dimension of the first layer of the network m times (MLP (1) : d in × m → d hidden ), and its hidden layer output vectors h i 1 , h i 2 , . . . are defined as follows:h i 1 = MLP (1) [v i 1 ⊕ • • • ⊕ v i m ] , h i 2 = MLP (1) v i m+1 ⊕ • • • ⊕ v i 2m , . . .where the [⊕ • • • ⊕] notation denotes concatenation over the vector dimension. The final number of vision tokens is reduced to 1 m of the original. In practice, m is typically set from 2 to 8, which reduces cost while usually not losing performance on downstream tasks. If m is set too large, the information of the image might be lost.Q-Former network is a trainable Querying Transformer module and proposed to bridge the gap between a frozen image encoder and a pre-trained LLM. It extracts a fixed number of output features from the vision encoder, independent of input image resolution. We create a set number of learnable query embeddings as input to the Q-Former. The queries interact with each other through self-attention layers, and interact with frozen image features e i (I) through cross-attention layers. The output queries of the last layer are projected to the input layer of the LLM. We use the pre-trained parameters in BLIP-2 as initialization to accelerate convergence and, similar to the second layer MLP network, share the parameters among all experts. Since the dimension of query embeddings is equal to 768, we add an additional linear transformation (W i ∈ R d i ×768 ) for each expert.V I = Q-Former W i (e i (I)) ⊕ W j e j (I) ⊕ W k (e k (I))The ablation study in Section 3.2.1 shows that the MLP fusion network fuses better than the Q-Former despite having fewer parameters and not being pre-trained.Inspired by the fact that visual experts already have positional encodings (e.g., 2D position encoding in ViT (Wang & Liu, 2019 )), we believe it is redundant to assign a VLM position embedding (PE) again to each visual token individually. Figure 3 illustrates our exploration of three encoding schemes to optimize PE assignments:1. all vision tokens of an image share a PE (share-all); 2. one PE shared by the same row of vision tokens (share-by-row); 3. one PE shared by the same row of vision tokens, plus a set of learnable columns PEs (share-by-row&col).Among the three methods, share-all can reduce the original O(N 2 ) PE cost to O(1), while the share-by-row and share-by-row&col can reduce the cost to O(N). All of them can significantly alleviate the out-of-maximum-length problem, but the question is how much do they affect the performance of VLM? We report ablation results in Section 3.2.2. 
main results: 
 We conduct explorations of single-expert, double-expert, and triple-expert ensembles. Following LLaVA-1.5 (Liu et al., 2023a) , our training pipeline consists of two stages. In the pre-training stage, we freeze the text-only LLM and the multi-expert encoder, and train the poly-visual fusion network from scratch to align the representation space of both. After training on a large-scale weakly-supervised (with noise) dataset, the text-only LLM is already capable of multimodal input and comprehension. In the fine-tuning stage, we unfreeze the LLM and further train it together with the poly-visual fusion network on diverse and high-quality supervised fine-tuning (SFT) datasets.We use the same datasets, evaluation, and hyperparameters as in LLaVA-1.5 (Liu et al., 2023a) . The details can be found in Appendix C.We present case studies on seven tasks in Appendix D. Our model can successfully follow multimodal instructions, enabling flexible human interaction. The current mainstream open-source VLMs have only one visual encoder, i.e., a single visual channel. Among six evaluated vision experts, the CLIP expert achieves the best performance in 8/9 benchmarks. However, multimodal tasks are diverse, and different tasks require different visual signals. We investigate whether dual-channel, i.e., double visual experts can outperform single experts on various tasks. We combine the strongest CLIP expert with other experts to construct a total of five double-expert encoders.Table 3 shows the performance of the double-expert encoders and their respective single expert. The results show that the "DINO v2 +CLIP", "LayoutLM v3 +CLIP", and "Con-vNeXt+CLIP" three double-expert encoders outperform the arbitrary single encoder in almost all cases (23/27). This demonstrates that using two visual channels enhances multimodal capabilities, confirming the feasibility of multi-expert collaboration.Comparing the performance between double-expert methods, we found that the best doubleexpert is DINO v2 improvement when collaborating with CLIP, performing the best on four benchmarks and ranking overall just behind DINO v2 +CLIP. For SAM or MAE, even if their pre-training tasks are Image Segmentation, Patch-level Denoising, respectively, which are mainly optimization objectives from the image perspective. "SAM+CLIP" and "MAE+CLIP" double-expert encoders outperform the arbitrary single-encoder in almost all cases (13/18). Therefore, we can conclude that when paired with the versatile visual expert CLIP, other experts can focus on capturing supplemental visual information to further enhance performance.As shown in Table 4 , the triple-expert approach wins in 4/6 cases in comparison with the two-expert at the data size of LLaVA-1.5. The best-performing three-expert is LayoutLM v3 +DINO v2 +CLIP, followed by ConvNeXt+LayoutLM v3 +CLIP, and finally ConvNeXt+DINO v2 +CLIP. Among them, model LayoutLM v3 +DINO v2 +CLIP has the largest number of parameters, reaching 8.8 billion. Based on the triple-expert experiments, we further conceptualize scenarios with more experts. The different types of vision encoders we chose for the pre-training tasks contain Image-Text Matching, Image Classification, and Patch-level Denoising, which fall into three categories: weakly-supervised, self-supervised, and supervised. We have already achieved high performance in training triple-expert, and continuing to increase the number of experts has limited performance improvement. Additionally, the integration of extra experts leads to an excessive total token count, causing the length of the training process to be too long, increasing the cost of training and inferencing. The MLP projection and Q-Former network are two mainstream methods for connecting vision and language. Which of them can more effectively convey visual signals is a key issue, especially in the context of poly-expert fusion. Table 5 presents the performance of using MLP and Q-Former respectively on three double-expert encoders, including "DINO v2 & CLIP" and "ConvNeXt & CLIP". The results demonstrate that MLP significantly outper-forms Q-Former in all cases, despite having fewer parameters and not utilizing pre-trained parameters like Q-Former, being instead directly initialized randomly. Table 6 shows the results of four positional encoding schemes introduced in Section 2.4.The share-all method (used by CogVLM (Wang et al., 2023b) ) not only saves the most PE but also improves the average performance by 0.8 on top of CLIP. The 2D positional coding (share-row&col) also improves the average performance by 0.6. However, sharerow impairs the performance of the model, probably because row-sharing corrupts the position information of the visual encoder itself. The results support our conjecture that it is redundant to re-assign LLM positional encoding to each vision token that already has positional information.Among poly-visual encoders, an important question is the contribution of different experts to the model's output. Attention mechanisms are commonly used interpretive tools in Transformer networks (Wiegreffe & Pinter, 2019) . Here, we employ a triple-expert encoder to assess the individual contributions of each expert within two multilingual benchmarks: MMB-English and MMB-Chinese. A sample's contribution is measured by the average attention the output token directs toward each expert's representations. By aggregating these averages across the dataset, we determine each expert's overall impact on the encoding process. Table 7 shows the individual contributions of the text prompt, LayoutLM v3 , DINO v2 , and CLIP to the output. The results indicate that the contribution of the text prompt to the answer is significantly higher than that of the visual experts. This is as expected. Firstly, the text prompt defines the format of the VLM's response, necessitating attention to the prompt during output, and secondly, the text has a higher information density than images, hence the average attention is usually higher for text. Comparing the three visual experts, we find that their contributions in descending order are CLIP, DINO v2 , and LayoutLM v3 . CLIP still demonstrates the characteristics of being the dominant eye or the primary visual channel. DINO v2 's contribution is approximately 20% of CLIP's, while LayoutLM's contribution is minimal, at only 1% of CLIP's.An ensuing inquiry is the relevance of visual channels that exhibit minimal contributions within the model's architecture. The detailed description of this section is in Appendix B.We selected 9 of the 12 evaluation benchmarks for LLaVA-1.54 . Table 8 reports the results for LLaVA-1.5 (i.e., single CLIP expert), and our poly-vision-expert model (LayoutLM v3 +ConvNeXt+CLIP / LayoutLM v3 +DINO v2 +CLIP) on above benchmarks. The poly-vision-expert VLMs show improved performance over the single-expert VLM, with average improvements of +1.3/+1.8, while the number of parameters only increased by 600M/1500M. In comparison with established VLMs, our poly-vision-expert model emerges as the top-performing system in 7/8 benchmarks, and secures a close second in the remaining one, thus affirming its robust multimodal assistant capabilities.  
related work: 
 Vision-Language Models (VLMs) integrate linguistic and visual processing, showing promising results in various applications. VisualGPT (Chen et al., 2022) provided foundational work in image captioning, BLIP series (Li et al., 2022; 2023b) extended capabilities to include visual question answering. Flamingo (Alayrac et al., 2022) and Kosmos-1 (Huang et al., 2023) demonstrated effective multi-modal understanding. LLaVA (Liu et al., 2023b) and MiniGPT-4 (Zhu et al., 2023) utilize projection for connecting vision and language.CogVLM (Wang et al., 2023b ) replicated close to double the parameters to build visual experts specializing in visual tokens, while similar to our exploration of positional encoding, they used share-by-one rather than the original approach. Qwen-VL and BLIP series (Bai et al., 2023; Dai et al., 2023b) use the Q-Former network to bridge text and image.Multi-Modal Large Language Models (MLLMs) have been evolving rapidly, with models like ImageBind-LLM (Han et al., 2023) and PandaGPT (Su et al., 2023) incorporating richer modality inputs, including audio and video. There is also a growing focus on regionlevel parsing (Chen et al., 2023b ), text-to-image generation (Wen et al., 2023) , and 3D understanding (Xu et al., 2023) . These models show that MLLMs can achieve meaningful performance across a range of tasks. 
conclusion: 
 In this paper, we push the boundaries of VLMs by proposing a novel polyvisual system that closely mirrors the complex and multi-dimensional nature of biological visual processing. Leveraging the unique attributes of diverse visual encoders, our system unifies their strengths to enrich the multimodal understanding of VLMs. Furthermore, we address the challenge of efficiently integrating visual information into language models by introducing techniques such as multi-patch-single-token projection and optimizing positional embeddings. This not only allows us to manage the overflow of vision tokens that typically burdens VLMs but also retains the models' semantic and spatial reasoning capabilities.Through rigorous experiments across a suite of benchmarks, we demonstrate that our polyvisual approach significantly enhances the VLMs' performance, outpacing existing models in accuracy and depth of understanding. These results support our hypothesis that a well-integrated assembly of expert encoders can lead to a substantial improvement in handling complex multimodal inputs.  
 
 Appendix: https://rank.opencompass.org.cn/homeExcluding LLaVA-Bench that rely on unstable GPT4 responses, as well as VisWiz (2018) and MME (2023) for the website crashed.https://huggingface.co/openai/clip-vit-large-patch14-336https://huggingface.co/facebook/dinov2-gianthttps://huggingface.co/microsoft/layoutlmv3-largehttps://huggingface.co/laion/CLIP-convnext_large_d_320.laion2B-s29B-b131K-ft-souphttps://huggingface.co/facebook/sam-vit-hugehttps://huggingface.co/facebook/vit-mae-huge 


Decision: This paper proposes a vision--language model that combines multiple expert vision encoders---similarly to MoE work in the LM space. The authors also proposes a method to reduce the number of positional embeddings used by various models. They show the effectiveness of their approach across a range of benchmarks and perform ablations to examine the effectiveness of different component of their model. This paper contributes to the MoE work in the multimodal space, and the future work can build on it. However, I encourage the authors to address the reviewers' feedback by (1) adding discussion around the related work (2) including analysis of inference time, memory, and the effect of number of parameters (3) discussion around strengths/weaknesses of each expert.

Reviews: review: paper_summary: paper_summary: Current VLM only uses a single visual encoder to obtain image information, which may not capture all the useful hints in the image. This paper proposes an ensemble framework that uses more than one visual encoder to get image information and introduces a fusion network to unify the processing of outputs from different visual encoders. Experiments on downstream tasks show that incorporating more than one visual encoder could significantly increase the model's performance in different aspects.
strength_weakness: strength_weakness: NA
questions: questions: 1. The ensemble model contains more parameters than baselines, it might be unfair for comparison.

2. [minor] It would be better if the strengths and weaknesses of different visual experts could be summarized more clearly, rather than just using case studies to demonstrate them.
paper_summary: paper_summary: In this paper, authors present the use of ensemble experts technique to synergize the capabilities of individual visual encoders, including those skilled in image-text matching, OCR, image segmentation, etc. The proposed approach MouSi have a poly-expert fusion network, a pre-trained open-source LLM and a multi-expert visual encoder. Experiments performed with various benchmarks showcase combing multiple experts provide better performance in contrast to single expert.
strength_weakness: strength_weakness: 1. Mixture-of-experts idea has been explored in the literature, and the proposed approach MouSe is built on the same principle

2. MLP fusion network or Q-Former network has been a standard approach used by VisualLLMs. The proposed Poly-Fusion Network is very similar in architecture.

3. Experimental study combining different available models and approaches. No new novel method proposed.
questions: questions: Comments to Authors

1. Is there an effective way to combine the experts or all known experts for a task is used and the model will figure it out which one to use? Table-2 showed that double vision experts are useful, but these combinations can become overwhelming when there are large number of experts available for a task.

2. Although, Section-3.5 mention different case studies by MouSi. Is MouSi capable of comprehending different languages text also?

3. Although multiple experts boos performance, how effective it is for the time-sensitive applications which require fast inference, still single expert is the approach for them? Some discussion on the trade-off will be useful. Also during inference, even though parameter space has increased, will all parameters will be used?

Minor Comments

1. Table-2 figure is unreadable without zoom. Please make it more visible.
paper_summary: paper_summary: This paper investigates how to ensemble multiple image encoders to provide a visual input to an LLM. The authors explore the use of six different visual encoders (eg. CLIP, DINO, MAE) and find that, often, using three encoders is better than using two encoders, which in turn is better than the best single encoder. To combine the visual tokens from multiple visual encoders, the authors devise a fusion network that reduces the number of tokens per visual encoder while mapping the image embedding to the LLM’s input dimension. Experiments on 9 datasets show that the proposed approach improves upon using a single image encoder as well as on related baselines.
strength_weakness: strength_weakness: 1. Albeit being generally well written, the paper is not easy to follow. Several key results are delegated to the appendix, without much discussion in the main paper. For example, it is said that triple-encoders outperform other methods (ie. are the strongest proposed model) but are however not discussed in the main paper. Another example is at the beginning of Section 2.2, where you said that you chose six image encoders “after extensive investigation”; which is not shared with the readers. Lastly, you add a whole Section 3.4 on “data enhancement” without ever saying what you do in the main body, letting the reader wonder what this might be about.
2. The performance gains are within 1 percentage point for most datasets. We can observe larger gains on SQA and MMB, but the overall boost between the three-expert model and the CLIP-only model is less than 1 percentage point in the final “Data enhancement” results. 
3. Moreover, the results include evaluation on both MMB and MMB-CN (a version of MMB translated into Chinese). If the goal is to assess the ability of the model to perform multiple tasks, having MMB-CN is redundant as it assesses the same ability as MMB. If the goal is to evaluate multilingual capabilities, there are far better benchmarks to do so (eg. MaRVL, Liu+EMNLP’21; IGLUE, Bugliarello+ICML’22; XM3600, Thapliyal+EMNLP’22).
4. One of your main contributions was to tackle the “challenge of visual token overflow”. I was not aware of this challenge, was it introduced by someone else before (if so, please cite) or was it something you first encountered in this study? In the latter case, I would rather see it as an ablation of positional encoding more than a challenge itself, as one can simply not add positional encodings (again) after the visual embeddings, since these normally include positional encodings in their respective encoders (which was your final recommendation too)
5. The related work would benefit from discussing other approaches for ensembling (both for VLMs and not) as well as for token merging/dropping.
questions: questions: 1. In Section 2.1, you state that “the image is processed in _sequence_ through the [visual] encodings”; do you mean in parallel (looking at Figure 2)?
2. In Table 2, could you please add the absolute performance of a CLIP-based model in the first row for better comparability?
3. The model size increases by 1.5B parameters when using three encoders. How does this influence inference time/memory in practice? This is a key information (along with accuracy) to decide whether/when to use this method.
paper_summary: paper_summary: This paper proposes a way to fuse together multiple vision encoders within a vision-language model setup. They show that a) using two and three vision encoders (that are pre-trained on complementary tasks and datasets like image-text contrastive learning, image segmentation, and OCR) can improve upon using a single vision encoder (i.e. CLIP), b) a simple MLP does better than a more complicated transformer-based network (Q-former) in fusing the image and text modalities from the image encoder and the LLM, and c) improving the efficiency of VLMs with respect to input tokens by eliminating a lot of redundancy in the positional embedding space for images.
strength_weakness: strength_weakness: * would have been great to see further analysis on Table 2 and Table 8. For example in Table 2, why is VQA anomalous? In Table 8, we see results are a lot more mixed with 3 experts. Is this a limitation of the approach that could be highlighted in the main text?
* it doesn't seem like, compared to "origin", the parameter sharing schemes in 3.2.2 have much effect on anything? The differences with the baseline are small, in fact smaller than the effect of expert order (Table 9)

Comments: title: Response to Reviewer
Acknowledgement of Rebuttal
Acknowledgement of Rebuttal
Looking Forward to Further Discussion
Looking Forward to Further Discussion
Looking Forward to Further Discussion
Rebuttal acknowledgement
comment: Dear Reviewer RrkK,

Thank you again for your valuable comments and constructive suggestions. We will follow your suggestions: (1) to evaluate English and Chinese tasks separately instead of mixing them up, and (2) to supplement more Chinese evaluation sets.
Dear authors,

Thank you for replying to my questions. I have also read the other reviews and responses.
In light of them, I have increased my score.

It would however be much better if you (1) did not average results on MMB-CN along with other English tasks, and (2) evaluated on other multilingual benchmarks (even if just on the CN split) as other benchmarks have been created for this purpose. In this aspect, just because one paper (LLaVA) had suboptimal experimental settings, you should not blindly follow them.
Thanks for the details about the concerns raised in the review. It address the concerns. I have acknowledged the response and have kept my score as is.
Response to the Reviewer BvJn

We sincerely appreciate the detailed feedback you provided on our work. 
We have addressed each of your concerns with **additional inference time and memory usage experiments** and clarification of the contribution of our work:

- **We performed additional inference time and memory usage experiments**. Our results show that: **our inference cost is even lower than that of a single encoder with the same parameter size, demonstrating the efficiency of the m-patch-one-token method and parallel encoding with multi-visual-experts.**

- Our primary goal of the Poly-Fusion network is not to innovate the standard MLP or Q-Former networks, but rather 1) to demonstrate how MLP or Q-Former networks can be adapted from a single vision expert to multiple vision experts, and 2) to determine which network, MLP or Q-Former, is more suitable for multiple experts.

- MouSi has demonstrated the effectiveness of Poly-Visual-Expert, we will add a trainable routing module of experts, allowing Poly-Visual-Fusion networks to sparsely activate corresponding experts for specific tasks, potentially reflected in MouSi-2.

- MouSi can generalize to some other languages (such as French or German).

- We agree and will supplement the Related Work section with MoE-related literature. We'd like to emphasize that, to the best of our knowledge (also mentioned by Reviewer oxQg), we are the first to apply this idea to LVLMs.


As the rebuttal process draws to a close, we eagerly await your response to ensure that our rebuttal adequately addresses your concerns. If you have any further questions, we stand ready to respond before the end of the rebuttal period. Thank you for your time and consideration.
Response to the Reviewer a4mu

We sincerely thank you for the thoughtful and constructive feedback.
We have addressed each of your concerns with **additional experiments for fair comparison** and we analyzed the experimental results in more depth to yield some valuable conclusions.

As the rebuttal process draws to a close, we eagerly await your response to ensure that our rebuttal adequately addresses your concerns. If you have any further questions, we stand ready to respond before the end of the rebuttal period. Thank you for your time and consideration.
Response to the Reviewer RrkK

We sincerely appreciate the detailed feedback you provided on our work. 
We have addressed each of your concerns with **additional inference time and memory usage experiments** and clarification of the contribution of our work. The details in our response are summarized as follows:

- **We performed additional inference time and memory usage experiments**. Our results show that: **our inference cost is even lower than that of a single encoder with the same parameter size, demonstrating the efficiency of the m-patch-one-token method and parallel encoding with multi-visual-experts.**

- We elaborated that our performance gains are **comparable to (or even higher than)** other recently published works.

- Our experiments confirmed that vision tokens don’t degrade baseline performance even without second PE. We effectively mitigated this issue by proposing multi-patch-single-token projection and Share-all PE.

- **We answered all your questions and addressed your concerns**, and we promise to add the necessary results and descriptions to enhance readability.

As we approach the end of the rebuttal process, we eagerly await your response to ensure our rebuttal has adequately addressed your concerns.  Should you have any more questions, we are ready to respond before the end of the rebuttal period.  We appreciate your time and consideration.
I have acknowledged the response and have kept my score as is.