Poly-Visual-Expert Vision-Language Models
Authors: Xiaoran Fan
Tao Ji
江常皓
Shuo Li
Senjie Jin
Sirui Song
Junke Wang
Boyang Hong
Lu Chen
Guodong Zheng
Ming Zhang
Huangcaishuang
Rui Zheng
Zhiheng Xi
Yuhao Zhou
Shihan Dou
Junjie Ye
Hang Yan
Tao Gui
Qi Zhang
Xipeng Qiu
Xuanjing Huang
Zuxuan Wu
Yu-Gang Jiang
Venue: COLM (2024)
Keywords: Vision-Language Models, Multi-modal Models
TL;DR: Poly-Visual-Expert Vision-Language Models

Abstract: Current large vision-language models (VLMs) frequently face challenges such as the limited capabilities of a single visual component and the excessive length of visual tokens. These issues can limit the model's ability to interpret complex visual information and over-lengthy contextual information accurately. Tackling these challenges is crucial for enhancing the performance and applicability of VLMs. This paper proposes leveraging the ensemble experts technique to synergize the capabilities of individual visual encoders, including those skilled in image-text matching, image segmentation, OCR, etc. This method introduces a fusion network that consolidates the outputs from different visual experts while bridging the gap between image encoders and pre-trained LLMs. In addition, we explore different positional encoding schemes to mitigate the waste of positional encoding caused by lengthy image feature sequences, effectively addressing the issue of position overflow and length limitations. For instance, in our implementation, this technique significantly reduces the positional occupancy in models like SAM, from a substantial 4096 to a more efficient 64 or even down to 1. Experimental results show that VLMs with multiple experts consistently outperform isolated visual encoders, with notable performance improvements as more experts are integrated. Our codes are available on our project website.

Published as a conference paper at COLM 2024
Poly- Poly-Visual-Expert Vision-Language Models
Xiaoran Fan', Tao Ji", Changhao Jiang", Shuo Li", Senjie Jin"" Sirui Song Junke Wang, Boyang Hong, Lu Chen, Guodong Zheng, Ming Zhang, Caishuang Huang Rui Zheng, Zhiheng Xi, Yuhao Zhou, Shihan Dou, Junjie Ye, Hang Yan
Tao Gui Qi Zhang/ Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang Fudan NLP Lab & Fudan Vision and Learning Lab Shanghai 200438, China
xrfan23@m. fudan. edu. cn, (tgui, qz)efudan. edu..e.u.. cn
Abstract
Current large vision-language models (VLMs) frequently face challenges such as the limited capabilities of single visual component and the ex- cessive length of visual tokens. These issues can limit the model's ability to interpret complex visual information and over-lengthy contextual infor- mation accurately. Tackling these challenges is crucial for enhancing the performance and applicability of VLMs. This paper proposes leveraging the ensemble experts technique to synergize the capabilities of individual visual encoders, including those skilled in image-text matching, image segmentation, OCR, etc. This method introduces fusion network that con- solidates the outputs from different visual experts while bridging the gap between image encoders and pre-trained LLMs. In addition, we explore different positional encoding schemes to mitigate the waste of positional encoding caused by lengthy image feature sequences, effectively address- ing the issue of position overflow and length limitations. For instance, in our implementation, this technique significantly reduces the positional occupancy in models like SAM, from a substantial 4096 to a more efficient 64 or even down to 1. Experimental results show that VLMs with multi- ple experts consistently outperform isolated visual encoders, with notable performance improvements as more experts are integrated. Our codes are available on our project website
1 Introduction
Current large vision-language models (VLMs) demonstrate significant potential in tasks requiring joint visual and linguistic perception, such image captioning (Agrawal et al.) 2019), visual question answering (Antol et al. 2015), visual grounding (Yu et 2l.) and autonomous agents (Durante et al. 2024 Xi et al. 2023). VLMSs harness large language models (LLMs) as cognitive foundation models to empower various vision-related tasks, while one vision component, such as CLIP (Radford et al. 2021), typically serves as aux- iliary modules that provide additional visual perception (Liu et 2 However, the perception abilities of the individual vision models still lag behind, even in simple tasks like counting. (Yamada et al., 2022 Thrush et al. 2022 Yuksekgonul et al.) 2022). This gap highlights significant lim lation in these models capacity to process and understand visual information as effectively as they handle linguistic data. According to the operation of the vertebrate visual system, with each functional unit encoding different visual aspects in parallel, retinal ganglion cells transmit distinct features to the brain (Baden et al. 2016). This biological mechanism suggests a model structure where the varied visual information should be parallelly encoded by diverse perception channels.
Equal contributions.
t Corresponning author.
https://gi https///ggttuub..cm/uuuaaanil wwt.://aaw//.e.aeeecocoo Www.nnnNNLLAA/MSSSSSSSSSS

---

Published as a conference paper at COLM 2024
IrstuctELP
Owen VL-
MazSi
Best Single
Dust Double Eccerts
Sest Triple Exparts
TextVQA
Te
61.5
S0A IMG
NM Vet
SQA-IM0
15,
33.6
69.9
21,
56
30-1
50,
GOA 52.6
POPP
GOA 63.6
POPE
R27.
23.7
01
70.5
78.2
78.5
63.6 65.
86,0
387
79.1
VQA-V2
MHiBench
VQA-22
51.4
MMBench
66.2
26.7
06.2
195
67.5
1010
SEED-Bench
MMBench-CN
BEED-Bench
HMBesch-CN
Figure 1: Left: Comparing InstructBLIP, Qwen-VL-Chat, and LLaVA-1.5-7B. our poly-visual- expert achieves SoTA on a broad range of nine benchmarks. Right: Performances of the best models with different numbers of experts on nine benchmark datasets. Overall, triple experts are better than double experts, which are better than a single expert.
To this end, the community has verified that each model, with its unique vision processing approach contributes differently to understanding visual content (Chen et al. 2023a). CLIP, with its contrastive learning approach, excels in aligning images with textual descriptions, providing a robust semantic understanding (Radford et al., 2021). DINO through its self-supervised learning at both the image level and patch level, offers significant advances in robust and stabilized feature extraction without relying on labeled data (Oquab et al. 2023). LayoutLM specialization in document AI tasks demonstrates the power of visual text processing (Huang et al., 2022). (Wang et al. 2023a) empirically investigated different visual tokenizers pre-trained with dominant methods (i.e., DeiT (Touvron et al. 2021), CLIP, MAE (He et al. 2021], DINO (Caron et al. 2021)), and observed that CLIP could capture more semantics, whereas the other models excelled at fine-grained perception. However, on the multimodal leaderboard organized by OpenCompase the visual encoders of all open- source VLMs are based on the pre-trained CLIP encoder family. Many researchers have pointed out the shortcomings of the CLIP encoder, such as the inability to reliably capture even basic spatial factors of images (Kamath et al. 2023), suffering from object hallucination (Li et al. 2023c), and so on. In light of the distinct capabilities and limitations of these diverse vision models, a key question emerges: How can we incorporate the strengths of multiple visual experts so that they work in synergy to improve overall performance?
Drawing inspiration from biology, we take on the poly-visual-exper perspective and design a novel model, similar to how the vertebrate visual system operates. Consequently, in the process of developing VLMs with poly-visual experts, three problems are in major concern: (1) whether the poly-visual experts are effective; (2) how to better integrate multiple experts; and (3) how to avoid exceeding the LLM's s maximum length with multiple visual experts?
In order to verify whether multiple visual experts effective for VLMs, construct candidate pool consisting of six well-known experts, including CLIP, DINO LayoutL.M*3 Convnext (Woo et al. 2023), SAM, and MAE. Using L.L.VA-1.5 as the base setup, we explored single-expert, double-expert combinations, and triple-expert combinations in nine benchmarks. The results, as shown in Figure [ indicate that as the number of visual experts increases, the VLMs acquire richer visual information (due more visual channels), and the upper limit of the multimodal capability improves across the board.
In existing single visual channel VLMs, visual signals are transmitted using either the MLP network (Liu et al. 2023a Wang et al. (023b) or the Q-Former network (Bai et al. 2023 Dai et al. 2023b). To support multi-channel signal transmission from multiple experts, we modi- fied both methods for expe-e fusion networks separately. The proposed method also
org. wrw..c/hom..........o cn/home
2

---

Published as a conference paper at COLM 2024
compresses the local visual information by multi-patch-one-token for better transmission efficiency and reduces the quadratic computational cost of subsequent processing of VLMs.
In position-awaare VLMs, vision tokens consume staggering amount of positional embed- dings. Taking a single-turn multimodal dialogue in VQA as an example, with the SAM expert, the number of vision tokens (about 4096) is more than 500 times higher than the number of text tokens (about 8.7). Inspired by the fact that visual experts already have positional encodings,
believe it is redundant to again assign a VLM position embed- ding to each visual token individually. Therefore, we explore different positional encoding schemes effectively address the issue of position encoding waste. The results show that the two schemes: sharing one position for all patches and 2D positional encoding (rows plus columns) are able to reduce the position consumption (in the case of CLIP,, the PE used drops from 576 to 24 or even 1), while the performance is still comparable.
Our contributions can be summarized as follows:
We introduce a poly-visual-expet VLM that synergistically combines the strengths of various visual encoders to improve the overall capabilities of VLMs.
We tackle the challenge of vision token overflow in VLMs by proposing multi-patch- single-token projection and efficient positional encoding solutions.
By experimenting with different combinations of experts, our results demonstrate en- hanced performance (+1.8 with fair comparison) in multimodal tasks.
2 Architecture
Monši
Male renes make pollon
Image
OUR
Odhers
Vislon-Language Models
Fibbuiling
Poly-Expert Networkk
User:
cones
CLP
SAM Imerr tacoerr
LawvetL-ed bmae lanakr
Madalty Taesber
Tmagy Treade
Semantic Expert
Segmentation Expert
OC R Expert
Other Expert
Figure An overview of our model structure. The poly-vision-exper model supports the integration of visual experts with various types and capabilities.
2.1 The Overview
When a user uploads an image of wind pollination in a conical inflorescence and asks "Which cones make pollen?" the image is processed in sequence through the encodings of the CLIP expert, the SAM expert, and the LayoutLM expert, yielding three sets of visual representations. Subsequently, a poly-expert fusion network compresses the multi-channel visual information and aligns it multimodally to the vision input tokens for our poly-vision- expert model. The user's question processed into text tokens by the LLMs' Embedding layer. Finally, our model generates the correct answer "Male cones make pollen." by employing its VQA capabilities to understand the vision-language question, and its OCR capabilities to recognize the answer text from the image.
3

---

Published as a conference paper at COLM 2024
Pre-training
Expert
Res.
Param.
d_hid
#Patch
Type
Tasks
Images
CLIP
336
300M
1024
576
VIT
Image- Text Matching
400M
DINOv2
224
1.1B
1536
256
VIT
UNO++iBOTSSWAN
142M
LayoutLMv3
224
368M
1024
196
ViT
Document OCR
11M
ConvNeXt
384
200M
768
1024
CNN
Image Classification
2B
SAM
1024
637M
1280
4096
VIT
Image Segmentation
11M
MAE
224
630M
1280
256
VIT
Patch-level Denoising
1.3M
Table 1: Comparison of six pre-trained visual experts. Res. indicates image resolution, hid indicates hidden dimension and Param. indicates the number of parameters.
In order to accomplish the above task, we propose the poly-vision-expert model, which consists of three fundamental components:
1. multi-expert visual encoder, which combines the experts selected from pool; a poly-expert fusion network, which implemented as a simple projection fusion method or Q-Former fusion method (Li et al. 2023b);
a pre-trained open-source LLM (e.g,, Vicuma v1.5).
Figure shows overview of our poly-vision-exper model. The core of a Vision-Language Model is typically an LLM which is pre-trained on large-scale textual corpus. To perceive the visual signals, a vision encoder and vision-language connection layer adopted to separately extract the visual features and align them to the semantic space of LLM.
The VLM takes input sequence comprised of interleaved text and image segments, denoted as X (..., Ti, J1, T2, 12,...),, where text fragments T are processed by the tokenizer and embedding layer of the LLM, and image segments I are fed the vision encoder. To ensure the universality and generalizability of the vision encoder, it is common practice to freeze its pre-trained parameters. In this paper, we rethink the design of the visual encoder
in VLMs and aim to improve its capability by ensembled experts.
2.2 Multi-Expert Vision Encoder
After extensive investigation, we choose six vision encoders skilled in different domains, including CLIP (2021), DINO (2023), LayoutLM'3 (2022), Convnext (2023), SAM (2023), and MAE [2211) They differ significantly from each other in terms of input resolution, hidden size, model type, model size, pre-training tasks, and training methods, as shown in Table Their detailed descriptions can be found in Appendix
Given image I in the input sequence and vision expert encoder e;(-), obtain the representation vectors of n image patches:
ot ,r,,...,,,,, b/ =
(1)
Assuming we have three experts (e)(-) R" ei(-) Rdxxd, ex(.) RR)xd.), the final sequence of image representations V, is concatenation of the three output sequences.
V1 (1) @ e1) @ (1) jol,..,, o py of
(2)
is worth noting that each expert outputs different number (n vs. nj vs. ny) and dimension (d; vs. dj vs. dı) of vectors, and we will handle these differences in the fusion network.
2.3 Poly-Expert Fusion Network
Since the dimension and number of output sequences are often different for different visual experts, fusion network needs to be designed to unify the processing. Following LLaVA (Liu et al. 2023b) and BLIP (Li et al.) 2022), we propose an MLP projection fusion network and Q-Former fusion network, respectively.
4

---

Published as a conference paper at COLM 2024
Q-Fommer
I LLM
LLM
text, text, text, TeNt,
P
from L.M
Q-Forner
Q-Former
5
S
S@
$
MR pl
MILIS
SILPP
SILP
SILE
10
5
5
5
6
6
6
6
5
d
7
Esparti
Espart
Imtuge
Hhure-b--row
(c) sture- -row
Figure 3: Left: Examples of two poly-expert fusion methods. We show how the MLP method compresses visual information with "2-patches-1-toeen" and how the Q-Former method compresses information with trainable queries. The modules with color gradients represent the sharing of parameters among multiple experts to transfer knowledge. Right: Diagram of the four positional encoding schemes. The @ operator indicates that the row position embedding and column position embedding are summed.
MLP projection is a 2-layer (dn, - - dout) multilayer perceptron. To simplify the processing and to share the knowledge among multiple experts, we set the hidden dimension (dnideen) and the output dimension (dow)) equal to the dimension (dmode!) of the LLM, and the second layer network (MLP(2) dridden + dout) parameters are shared among all experts.
Given specific expert e;(,,, the first layer network defined as MIP!!" d dnidden-
H₃ M (e:(1)) @ MLP (ej(1) + MLP (ex(1)
Vi MLP (Ht)
In practice, multiple experts output large number of vision tokens, which not only increases the computational cost and memory usage of the VLM but also tends to exceed the maximum length limit during inference. Therefore, we propose multi-patchess-ne-toiie to proportionally reduce the number of tokens output by each expert. Since image signals have local or sparse properties, it practical use one token to represent neighboring patches. Take m--atch--nee-toen for example, we make the input dimension of the first layer of the network m times (MIP(1), d i + dmidden), and its hidden layer output vectors h liy... are defined as follows:
projection
I1 MMP((1) (lo (of .... . v'nl). 11 =LP(L)()) (p @ @V2
where the [e (+... ++) notation denotes concatenation over the vector dimension. The final of the original. In practice, is typically set from
number of vision tokens reduced to ir to 8, which reduces cost while usually not losing performance on downstream tasks. If m is set too large, the information of the image might be lost.
Q-Former network a trainable Querying Transformer module and proposed to bridge the gap between frozen image encoder and pre-trained LLM. It extracts a fixed number of output features from the vision encoder, independent of input image resolution. We create set number of learnable query embeddings input to the Q-Former. The queries interact with each other through self-attention layers, and interact with frozen image features (T) through cross-attention layers. The output queries of the last layer are projected to the input layer of the LLM. We use the pre-trained parameters in BLIP-2 as initialization to accelerate convergence and, similar to the second layer MLP network, share the parameters among all experts. Since the dimension of query embeddings is equal to 768, we add an additional linear transformation (W; R:77768) for each expert.
Q-Former (W: (e)(1)) @ Wj (e;(1) @ We (ex
The ablation study in Section 3.2.1 shows that the MLP fusion network fuses better than the Q-Former despite having fewer parameters and not being pre-trained.

---

Published as a conference paper at COLM 2024
2.4 Different Positional Encoding Schemes
Inspired by the fact that visual experts already have positional encodings (e.g., 2D position encoding in ViT (Wang Liu 2019),, we believe redundant to assign a VLM position embedding (PE) again to each visual token individually. Figure illustrates our exploration of three encoding schemes to optimize PE assignments:
1. all vision tokens of an image share PE (share-all); one shared by the same row of vision tokens (share-by-row),
3. one PE shared by the same row of vision tokens, plus set of learnable columns PEs
Among the three methods, share-all can reduce the original O(N2) PE cost to 0(1), while the share-by-row and share-by- bolloo can reduce the cost to O(N). All of them can significantly alleviate the out-of-ma tttt--f-axxmmmmmeeneeee problem, but the question is how much do they affect the performance of VLM? We report ablation results in Section 3.2.2
3 Experiments
3.1 Main Results
We conduct explorations of single-expert, double-expert, and triple-expert ensembles. Fol- lowing LLaVA-1.5 (Liu et al. (202a),, our training pipeline consists of two stages. In the pre-training stage, we treeze the text-only LLM and the multi-expert encoder, and train the poly-visual fusion network from scratch to align the representation space both. Af- ter training on large-scale
(with noise) dataset, the text-only LLM already capable of multimodal input and comprehension. In the fine-tuning stage, we unfreeze the LLM and further train it together with the poly-visual fusion network diverse and high-quality supervised fine-tuning (SFT) datasets.
Datasets & Evaluation & Hyperparameters We use the same datasets, evaluation, and hyperparameters in LLaVA-1.5 (Liu al. 2023a). The details can be found in Appendix
Case Study We present case studies on seven tasks in Appendix D Our model can successfully follow multimodal instructions, enabling flexible human interaction.
3.1.1
Single Vision Expert
Model
Param VQA
GQA SQA!
VQAT POPE MMB MMBCN
SEED
MM Vet
Avg
Single Expert
CLIP
7.3B
78.5
62.0
66.8
58.2
85.9
63.0
57.4
66.2
30.5
63.2
DINO
8.1B
74.9
61.7
66.1
46.2
84.6
57.9
48.7
63.4
23.4
58.5
LayoutLM*
7.4B
44.9
40.0
62.8
43.6
59.1
29.0
19.8
34.8
11.8
38.4
ConvNeXt
7.2B
75.1
60.5
65.0
56.3
85.6
63.3
55.0
61.5
26.0
60.9
SAM
7.6B
64.7
55.8
63.9
44.1
82.0
43.7
33.9
51.9
17.7
50.9
MAE
7.6B
62.0
53.2
63.3
44.5
79.7
41.6
33.0
49.4
16.5
49.2
Table 2: Comparison of six vision experts on benchmarks.
Table compares the performance of all six VLMs with single vision expert. The CLIP expert achieves the best performance 8/9 benchmarks, fully explaining why has become the dominant choice of vision encoder for VLMMs. Comparing the different attributes of the experts, CLIP ranked 5th in terms the number of parameters, 3rd in terms of image resolution, and 2nd on the size of the pre-training data, none of which had an absolute lead. Therefore, we guess that its main advantage lies in its image-text matching pre-training task, which has multimodal alignment capability in advance. Overall, the performance ranking of the six experts is roughly CLIP>
SAM> MAADMMAAA-LLyLL..M LayoutLMM In

---

Published as a conference paper at COLM 2024
addition, LayoutLM is an undisputed expert in OCR and SAM in image segmentation but performs poorly as single visual encoder in VLM. A natural question is whether poly-expert fusion can activate their capabilities in their specialized fields?
3.1.2 Double Vision Experts
OUP
03
WID Book
Hi
Model
Param VQA
GQA SQA VQA
POPE MMB MMBCN
SEED
MM
Avg
Double Experts
DCLIP
8.4B
79.0
63.1
69.8
57.7
86.4
67.0
60.5
66.9
32.0
64.7
ADINO
4.1
1.5
3.7
11.5
1.8
9.1
11.8
3.5
8.6
ACLIP
0.5
1.1
3.0
4.0
0.7
1.5
L
7.7B
79.2
62.4
68.5
58.9
86.1
67.0
59.9
66.8
33.0
64.6
ALayoutLM*
34.3
22.4
5.7
15.3
27.0
38.0
40.1
32.0
21.2
ACLIP
0.7
0.4
1.7
0.7
0.2
4.0
2.5
0.6
2.5
CoNYNeXI+CLIP
7.5
78.7
61.9
69.9
57.8
86.1
65.5
59.2
66.1
32.1
64.1
AConvNeXt
3.6
1.4
4.9
1.5
0.5
2.2
4.2
4.6
6.1
ACLIP
0.2
0.1
3.1
0.4
0.2
2.5
1.8
0.1
1.6
SAM+CLIP
7.9B
78.9
63.5
70.3
57.9
86.3
66.3
58.8
66.7
32.7
64.5
ASAM
14.2
7.7
6.4
13.8
4.3
22.6
24.9
14.8
15.0
ACLIP
0.4
1.5
3.5
0.4
3.3
1.4
0.5
2.2
MAE+CLIP
7.9B
78.4
62.7
67.8
56.9
87.0 0
66.2
56.6
66.2
29.3
63.5
AMAE
16.4
9.6
4.5
12.4
7.3
24.6
23.6
16.8
12.8
ACLIP
0.1
0.7
1.0
1.3
1.1
3.2
0.8
0.0
1.2
Table 3: Comparison of different double-expert encoders. The A-marked rows are compared to the single-expert methods. Where blue cells mean the double-expert model is better, and red cells mean the single-expert model is better.
The current mainstream open-source VLMs have only one visual encoder, i.e., single visual channel. Among six evaluated vision experts, the CLIP expert achieves the best performance in 8/9 benchmarks. However, multimodal tasks are diverse, and different tasks require different visual signals. We investigate whether dual-channel, i.e., double visual experts can outperform single experts on various tasks. We combine the strongest CLIP expert with other experts to construct total of five double-expert encoders.
Table 3 shows the performance of the double-expert encoders and their respective sin- gle expert. The results show that the "DINO "LayoutLM**+ and "Con- vNeXt+CLIP" three double-expert encoders outperform the arbitrary single encoder in almost all cases (23/ This demonstrates that using two visual channels enhances multi- modal capabilities, confirming the feasibility of multi-expert collaboration.
Comparing the performance between double-expert methods, we found that the best double- expert is 2 rather than the ensemble of the best single expert CLIP and the second-best ConvNeXt. indicates that superior performance as a single expert does not necessarily imply optimality when ensembled. Since ConvNeXt and CLIP have considerable overlap in their training methods and training corpora, leading the extraction of similar visual information, whereas the self-supervised DINO'2 and the weakly-supervised CLIP complement each other, resulting in more effective ensemble. Furthermore, it is worth men- tioning that LayoutLMM3 which performed the worst as a single expert, shows significant
7

---

Published as a conference paper at COLM 2024
Model
Param VQA GQA SQA' VQA POPE MMB
MMBCN
SEED MM
Avg
Triple Experts
Con/NeXt+Layoutt M
7.98
78.5
63.3
70.2
58.0
87.3
66.8
58.9
66.0
32.2
64.6
AConvNeXt+CLIP
0.2
1.4
0.3
0.2
1.2
1.3
0.3
0.1
0.1
MLayoutLM**ICLI
0.7
0.9
0.9
1.7
1.2
0.2
1.0
0.8
0.8
ConvNeext+DINO*><CLL
8.6
78.6
63.2
69.2
57.8
86.5
66.6
58.9
67.1
32.9
64.5
AConvNNet=<CLIP
0.1
1.3
0.7
0.0
0.4
1.1
0.3
1.0
0.8
ADINO"
0.4
0.1
0.6
0.1
0.1
0.4
1.6
0.2
0.9
8.8B
79.1
69.0
86.5
67.4
60.0
M
1.2
0.5
0.4
0.7
0.6
ADINO*2+CLIP
0.5
0.7
0.6
1.6
Table 4: Performance comparison of different triple-expert methods. The A-marked rows
are compared to the double-expert method. Where blue cells indicate the triple-expert model is better, and red cells indicate the double-expert model better.
improvement when collaborating with CLIP, performing the best on four benchmarks and ranking overall just behind DINI For SAM or MAE, even if their pre-training tasks are Image Segmentation, Patch-level Denoising, respectively, which are mainly optimization objectives from the image perspective. "SAM+CLIP" and "MAE+CLIP" double-expert encoders outperform the arbitrary single-encoder in almost all cases (13/18). Therefore, we can conclude that when paired with the versatile visual expert CLIP, other experts can focus on capturing supplemental visual information to further enhance performance.
3.1.3 Triple Vision Experts and More Vision Experts
As shown in Table 4 the triple-expert approach wins in 4/6 cases in comparison with the two-expert at the data size of LLaVA-1.5. The best-performing three-expert is ConNNeXt-DINO +CLIP. Among them, model
followed by ConvNext=LayouttMM +CLIP, and finally has the largest number parameters, reaching 8.8 billion. Based on the triple-expert experiments, we further conceptualize scenarios with more experts. The different types of vision encoders we chose for the pre-training tasks contain Image-Text Matching, Image Classification, and Patch-level Denoising, which fall into three categories: weakly-supervised self-suuprvised, and supervised. We have already achieved high performance in training triple-expert, and continuing to increase the number of experts has limited performance improvement. Addi- tionally, the integration of extra experts leads to an excessive total token count, causing the length of the training process to be too long, increasing the cost of training and inferencing.
3.2
Ablation Study
3.2.1
Effect of Fusion Methods
Model
Param
VQAº2
GQA
SQA!
VQAT
POPE
MMB MMBCN
SEED!
MM
Avg
8.4B
79.0
63.1
69.8
57.7
86.4
67.0
60.5
66.9
32.0
64.7
DINO +CLIP- -CLIP-Q-FForer
8.5B
60.4
50.9
66.7
45.1
45.2
52.7
44.8
51.8
20.5
48.7
ConyNeXt+CLIP-a MLP
7.5B
78.7
61.9
69.9
57.8
86.1
65.5
59.2
66.1
32,1
64.1
Con/NeXtCCLI+---omme
7.6B
65.8
52.6
68.7
45.6
77.0
59.7
49.8
53.2
22.1
54.9
Table 5: Performance comparison of different poly-expert fusion methods.
The MLPP projection and Q-Former network are two mainstream methods for connecting vision and language. Which of them can more effectively convey visual signals is key issue, especially in the context of poly-expert fusion. presents the performance of using MLP and Q-Former respectively on three double-expert encoders, including "DINOv2 CLIP" and "ConvNeXt & CLIP" The results demonstrate that ML.P significantly outper-
8

---

Published as a conference paper at COLM 2024
forms Q-Former in all cases, despite having fewer parameters and not utilizing pre- pre-trained parameters like Q-Former, being instead directly initialized randomly.
3.2.2
Effect of Different Position Schemes
Model
VQA
GQA
SQA!
VQA
POPE
MMB
MMBCN
SEED!
MM
Avg
Origin
78.5
62.0
66.8
58.2
85.9
64.3
58.3
66.2
30.5
63.4
Share-all
79.0
62.4
68.4
58.4
86.3
67.4
58.2
65.7
31.7
64.2
Share-by-row
75.0
57.2
63.4
51.7
86.1
46.4
43.4
55.6
31.9
56.7
Share-by-row&col
79.0
62.6
68.3
58.1
86.3
66.0
58.8
66.2
30.6
64.0
Table 6: Comparison of four positional encoding schemes on 9 benchmarks.
the results of four positional encoding schemes introduced in Section 2.4 The share-all method (used by CogVLM (Wang et al.; 2023b)) not only saves the most PE but also improves the average performance by 0.8 on top of CLIP, The 2D positional coding (share-row&ccll) also improves the average performance by 0.6. However, share- row impairs the performance of the model, probably because row-sharing corrupts the position information of the visual encoder itself. The results support our conjecture that it redundant to re-assign LLM positional encoding to each vision token that already has positional information.
3.3 Analysis
Among poly-visual encoders, an important question is the contribution of different experts to the model's output. Attention mechanisms are commonly used interpretive tools in Transformer networks (Wiegreffe & Pinter 2019). Here, we employ a triple-expert encoder to assess the individual contributions of each expert within two multilingual benchmarks: MMB-English and MMB-Chinese. A sample's contribution is measured by the average attention the output token directs toward each expert's representations. By aggregating these averages across the dataset, we determine each expert's overall impact on the encoding process. Table 7 shows the individual contributions of the text prompt, LayoutL DINO and CLIP to the output. The results indicate that the contribution of the text prompt to the answer is significantly higher than that of the visual experts. This is as expected. Firstly, the text prompt defines the format of the VLM's response, necessitating attention to the prompt during output, and secondly, the text has a higher information density than images, hence the average attention usually higher for text. Comparing the three visual experts, find that their contributions in descending order are CLIP, DINO'2, and LayoutLM CLIP still demonstrates the characteristics of being the dominant eye or the primary visual channel. DINO'2's contribution is approximately 20% of CLIP's, while LayoutLM's contribution is minimal, at only 1 1 of CLIP's.
An ensuing inquiry the relevance of visual channels that exhibit minimal contributions within the model's architecture. The detailed description of this section is in Appendix B
3.4 VLM Benchmark results
We selected 9 of the 12 evaluation benchmarks for LLaVA-1.5 Table reports the results for LLaVA-1.5 (i.e., single CLIP expert), and our poly-vision-expers model L LwwSCCLP/LLLouLMMMMM L
on above benchmarks. The poly-vision-expert VLMMs show improved performance over the single-expert VLM, with average improvements of +1.3/+1.8, while the number of parameters only increased by 600M/1500M. In comparison with established VLMs, our poly-vision-xxpee as the top-performing system in 7/8 benchmarks, and secures a close second the remain- ing one, thus affirming its robust multimodal assistant capabilities.
model emerges
'Excluding I.I. L.VAA-Bencl that rely on unstable GPT4 responses, well as VisWiz 2018 and MME 2023) for the website crashed.

---

Published as a conference paper at COLM 2024
Table 7: Average attention prob-
Prompt
LayoutLMva
DINO
CLIP
ability (%) allocation of our triple-
MMB
61.1%
0.14%
2.76%
11.1%
MMBCN
58.8%
0.16%
2.92%
10.7%
expert model's output each input.
Model
Param
VQA
GQA
SQA'
VQA
POPE MMB MMB
SEED'
MM Avg
BLIP-2
2023b
14.1B
41.0
41.0
61.0
42.5
85.3
46.4
22.4
InstructBLIP 2023a
8.2B
49.2
60.5
50.1
36.0
23.7
53.4
26.2
VisualGLM
[2022
8.0B
37.6*
35.5
47.0*
14.8
Shikra 22023
7.3B
77.4
58.8
PandaGP'I-13B 2023
13B
45.4* 32.0*
47.67
19.6
Qwen-VL-Chat 2023
9.6B
78,2
57.5
68.2
61.5
60.6
56.7
58.2
mPLUG-Ow12 2212
8.2B
79.4
56.1
68.7
54.3
66.5t
59.5*
64.5
35.71
Monkey 22024)
9.8B
69.4
59.6*
54.77
64.3
38.1
CLIP LL. (2023a)
7.3B
78.5
62.0
66.8
58.2
85.9
63.0
57.4
66.2
30.5
63.2
Ours LayouLLM" -Con-NeXI-CLIP
7.9B
78.5
62.9
69.8
57.6
86.6
66.6
59.8
66.2
32.5
64.5
Ours 2
8.8B
79.1
63.6
69.0
58.4
86.5
67.4
60.0
67.5
33.6
65.0
Table 8: The effect of default data on nine benchmarks. Param, indicates the number of pa- rameters. The +/$ marks denote the results from MMMB/OpenCompaant official, respectively.
4 Related Work
Vision-Language Models (VLMs) integrate linguistic and visual processing, showing
promising results in various applications. VisualGPT (Chen et al. 2022) provided founda- tional work in image captioning, BLIP series (Li et al., 2022: 2023b extended capabilities to include visual question answering. Flamingo (Alayrac et al. 2022) and Kosmos-1 1 (Huang et al. 2023) demonstrated effective multi-modal understanding, LLaVA (Liu et al. 20236) and MiniGPT-4 (Zhu et al. 2023) utilize projection for connecting vision and language. CLgVLM (Wang et al., 2023b) replicated close to double the parameters to build visual experts specializing in visual tokens, while similar our exploration of positional encoding, they used share-by-one rather than the original approach. Qwen-VL VL and BLIP series (Bai Dai et al. 2023b) use the Q-Former network to bridge text and image.
et al. 2023
Multi-Modal Large Language Models (MLLMs) have been evolving rapidly, with models like ImageBind-LLM (Han et al., 2023) and PandaGITT (Su al., 2023) incorporating richer modality inputs, including audio and video. There is also growing focus on region- level parsing (Chen et 2023b), text-to-image generation (Wen al.) 2023),, and 3D understanding (Xu et al. 2023). These models show that MLLMs can achieve meaningful performance across range tasks.
Conclusion
In this paper, we push the boundaries of VLMs by proposing novel polyvisual system that closely mirrors the complex and multi-dimensional nature of biological visual pro- cessing. Leveraging the unique attributes of diverse visual encoders, our system unifies their strengths to enrich the multimodal understanding of VLMs. Furthermore, we address the challenge of efficiently integrating visual information into language models by intro- ducing techniques such as embeddings. This not only allows us to manage the overflow of vision tokens that typically burdens VLMs but also retains the models' semantic and spatial reasoning capabilities. Through rigorous experiments across suite of benchmarks, we demonstrate that our polyvisual approach significantly enhances the VLMs' performance, outpacing existing models in accuracy and depth of understanding, These results support hypothesis that a well-integrated assembly of expert encoders can lead to a substantial improvement in handling complex multimodal inputs.
projection and optimizing positional
10

---

Published as a conference paper at COLM 2024
6
Acknowledgements
The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of China (No.62441602, 62206057, 62076069), Shanghai Rising-Star Program (23QA1400200), Natural Science Foun- dation of Shanghai (23ZR1403500), Program of Shanghai Academic Research Leader under grant 22XD1401100.
References
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
8948-8957, 2019.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 15-23716-23736, 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Tom Baden, Philipp Berens, Katrin Franke, Miroslav Román Rosón, Matthias Bethge, and Thomas Euler. The functional diversity of retinal ganglion cells in the mouse. Nature, 529 (7586):345-383 350, 2016.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint ar ArXi::2308.122 129666, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 9650-9660, 2021.
Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Xu. Vlp: A survey on vision-language pre-training Machine Intelligence Research, 20(1): 38-56, 2023a.
Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny, Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 18030-18040, 2022.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. ar preprint ar arXiv:2306.15195, 2023b.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: lowards general-purpose Vision-language models with instruction tuning, 2023a.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023b.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Gim: General language model pretraining with autoregressive blank infilling. In Proceed- ings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 320-335, 2022.
11

---

Published as a conference paper at COLM 2024
Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, et al. Agent ai: Surveying the horizons of multimodal interaction, ar preprint ar XrXi::240..03568, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. A comprehensive evaluation benchmark for multimodal large language models. CoRR, abs/2306. 1 13394, 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1904-6913, 2017.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEÉE conference on computer vision and pattern recognition, pp. 3608-3000000 10088-361,,00 2018.
Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. Imagebind-llm: Multi-modality instruction tuning. arXiv preprint XrXiv2230..03900 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045, 2023.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking, 2022.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's" up" with vision-language models? investigating their struggle with spatial reasoning. arviv preprint arXivv2310.11778,, 2023,
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. Xiv preprint ar arXiv:2304.02643 2023.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal Ilms with generative comprehension. arv preprint arXin:2307.16125 2023a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888-12909 PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2:
Bootstrapping language- image pre-training with frozen image encoders and large language models. ar Xio preprint ar 7Xix:2301.12597 2023b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXio preprint ar Xii:23055.10355 2023c.
Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models, 2024.
12

---

Published as a conference paper at COLM 2024
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: is your multi-modal model an all-around player? ar preprint ar Xiv:2300 006010, 2023c.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song- Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arvi preprint ar arXio:2304.07193, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, PP. 8748-8763, PMLR, 2021.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arviv preprint ar xrXie22305.1635. 2023.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5238-5245 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, Pp. 10347-10357, PMLR, 2021.
Guangzhi Wang, Yixiao Ge, Xiaohan Ding, Mohan Kankanhalli, and Ying Shan. What makes for good visual tokenizers for large language models? arXiv preprint ar arXie:2305.12223, 2023a.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. ar Xrv preprint ar XrXiv2231..0309, 2023b.
Zelun Wang and Jyh-Charn Liu. Translating math formula images to latex sequences using deep neural networks with sequence-level training, 2019.
Song Wen, Guian Fang, Renrui Zhang, Peng Gao, Hao Dong, and Dimitris Metaxas. Im- proving compositional text-to-image generation with large vision-language models. arXiv preprint ar xrXio:2310.06111, 2023.
Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation, 2019.
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoen- coders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16133-16142, 2023.
13

---

Published as a conference paper at COLM 2024
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. ar Xio preprint ar Xrx::2209.00866
2023.
Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointilm: Empowering large language models to understand point clouds. arXiv preprint drXio:2308.16911, 2023.
Yutaro Yamada, Yingtian Tang, and IIker Yildirim. When are lemons purple? the concept association bias of clip. arv preprint arXit:2212.12045 2022.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration, 2023.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 89-85. Springer, 2016.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. Xiv preprint ar xrXiv:2308.02990 2023.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2022.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. Xiv preprint ar Xiv: 10592, 2023.
14

---

Published as a conference paper at COLM 2024
A
Vision Experts
CLIP learns the image-text alignment through contrastive learning. It is pre-trained on a large-scale dataset consisting of 400M noisy image-text pairs sourced from the internet. The vision encoder of CLIP is Vision Transformer (ViT) with 300M parameters. The input resolution is fixed to 336 336, and the feature dimension is 1024
DINO trains a student network to mimic the behavior of a more powerful teacher network, without the need for any training labels. Two objective functions utilized for self-supervised pretraining: an image-level object that constrains the CLS tokens from the network and teacher network, and a patch-level object that is applied the extracted representations of masked input. The DINO vision encoder is a Vision Transformer (ViT) with 1.1B parameters. The input image is preprocessed to 224 x 224 resolution and the hidden dimension is 153
LayoutLM pre-trains multimodal Transformers for Document Al with unified text and image masking. The simple unified architecture and training objectives make LayoutL. general-purrpse model for both text-centric and image-centric Document AT tasks. The LayoutLM vision encoder a VIT architecture with 368M parameters. The input image is first preprocessed to the resolution of 224 224 and then encoded to 1024-dimension patch embeddings
ConvNeXt is a purely convolutional network (ConvNet) that introduces fully convolu- tional masked autoencoder framework (FCMAF) and new global response normalization (GRN) layer to ConvNeXt. ConvNeXt underwent pretraining on the ImageNet-221 dataset, significantly enhancing the performance of the pure ConvNet across various recognition benchmarks. The ConvNeXt vision encoder used has 200M parameters. The input resolution is 384 x 384 and the feature dimension is 768
SAM is trained on large-scale segmentation dataset, comprising 11 million images and over billion masks, and achieves impressive zero-shot generalization. It is designed to efficiently predict object masks from images with different types of prompts, e.g. text or point. SAM also adopts ViT as a vision encoder with 637M parameters. The input resolution and hidden dimension are both larger, i.e. 1024x 1024 and 1280, respectively""
MAE aims reconstruct the original image given only partial observations (25% of the patches). The ViT-Huge encoder paired with MAE achieved new record at the time on the ImageNet-1K dataset with an accuracy 87.8% and generalized very well. The MAE vision encoder has 630M parameters, while input resolution and hidden dimension are 1024x 1024 and 1280
15

---

Published as a conference paper at COLM 2024
B Analysis
QI: Where dog in the picture?
A1: The dog sitting on top of wooden table the picture.
The dog is sitting on top of wooden table
mask DINO
The dog is sitting on top of table.
mask CLIP
The dog is sitting on top wooden table...
mask Lay
Q2: How many dogs are there in the picture? What color are they?
A2: There four dogs in the picture, and they respectively golden yellow, white. brown. and black.
There are three dogs... They are brown and white.
mask DI
There are three dogs... They are black white,
mask CLIP
There are three dogs..., and they are all brown.
mask LayoutLM
Figure 4: The perturbation experiments on the triple-expert + model, the specific perturbation is to mask all output of the corresponding vision expert.
Figure 4 presents perturbation experiments conducted on the same triple-expert. The outputs of each expert were individually masked to assess their influence on the model's response. In Case our model queried with straightforward question: "Where is the dog in the picture?" The experiment demonstrates that, irrespective of which visual expert masked, the two unmasked channels sufficiently identify the location as "on top of". However, the presence of CLIP experts enriches the details by specifying "wooden table" over a generic "table". Case poses a more complex question our model: "How many dogs are there in the picture? What colors are they?" Here, the perturbation tests reveal that the collective operation of all three experts is necessary for an accurate response. The omission of any single expert leads to erroneous answers, underlining the diverse informational scope captured by each visual channel within the poly-visual-exper VLM. This underscores the importance of channel integration in multimodal tasks, as certain nuances can be missed when relying on solitary channel or single-expert VLM.
16

---

Published as a conference paper at COLM 2024
C Datasets & Evaluation & Hyperparameters
Datasets During the pre-training stage, we utilized the LCS-558K datase which com- prises ~558K image-text pairs from the LAION-CC-SBU, annotated with -enerrree captions. During the SFT stage, We use the default SFT data from LLaVA-,.5, which
~665k 10 containing VQA, OCR, region-level VQA, visual conversation, and language conversation data.
Evaluation Including VQA v2 2017); GQA ; SQA! ScienceQA-IMG 2022); VQAT: TextVQA 2019); POPE (023c); MMB & MMB MMBench & CMMBench-Chiness dev results SEED SEED- (2023a); MM-Vet (2023).
Hyperparameters For main results, keep all training hyperparameters roughly the same as the LLaVA series (Liu et al. 2023b We present a detailed description of the hyperparameters in Table For the MLP husion network, we set i in mn-patches-one-tokee from 1 to 16 to avoid exceeding the maximum length for training and inference. For the Q-Former fusion network, we set the number of queries per expert match the number of outputs from the MLP fusion network. The parameters of the Q-Former fusion network are initialized using the pre-training parameters of BLIP-2 (Li al. 2023b).
Hyperparameter
Pretrain
Finetune
batch size
256
128
1e-3
2e-5
Ir schedule
cosine decay
warmup ratio
0.03
weight decay epoch
optimizer
AdamW
DeepSpeed stage
2
3
Table 9: Hyperparameters our model's pretrain and finetune.
17

---

Published as a conference paper at COLM 2024
D
Case Study
Complex imagle Captioning
OCR Interpreting
Visual Math Frob lem Soliving
Resiopgl
Pleaer memoee Nociii
BICON FRISE
D Solva pusikar wriding oxpaniias
Moeii:
limule for sess sggle li
Ave laggl Wata
ths??
Maafi
3 Jokk lie paecr andlip obonaii vith grice zot
ln bem heun This to la memunas mne move dunnn wilh time Seyenbee cue
Non dur ally tne enta
"2le0"" ualld postia pap aul aree lad pumplic gour seing deooniee of leaf.
sen hupud oypttor
27
or
ComplexxContingg
Visual Text Generating
You Woman FEARS LORD fe pradsed
How born there the imaqee?
Whor & the meeazing of the wards the image?
hare lnsia lrnge.
The image oueair now Porrett 3100, whath
O How clare boiss well and bow mny
White a story based on the images
Thi note puncill lager parsig But descthorrs eagant Bua tuaaar tee bur casp proiinee Bir todd
Meute
Ti san have
mas mas
frme inreticten and wothor to many The
Reasoning World Knowledge
Grounding
eouits lies cn raps tua alaract the cienee ballarees
wlanc dicaao gropp yoong milid,,
bere ieplre tur
sse andiy us dlom lise wtadine offike
ue cour is slai culle la saukines the
anoud
0 Franee bicecas
boen les opracios
Thensl
bikeen norld
locoro ascicald catraric ctrce donic
Medka
As sarla, hu paal umu loanu discc dhit brouth feees and quilligg muaan cruatans
abnicalli be Loavro and plas pranii bieen
witl move andrastt mite retor the movers
Figure 5: Qualitative examples generated by our poly-vision-expert model.
Figure shows the case study of our model on seven tasks, including Complex Image Captioning, Visual Text Generating, OCR Interpreting Reasoning with World Knowledge, Visual Math Problem Solving, Complex Counting, and Visual Grounding. Our model is able to successfully follow variety of multimodal instructions, allowing for flexible interaction with humans.
18


Reviews:
[{"review_id": "StMKjIoWHV", "reviewer": "Reviewer_a4mu", "summary": "Current VLM only uses a single visual encoder to obtain image information, which may not capture all the useful hints in the image. This paper proposes an ensemble framework that uses more than one visual encoder to get image information and introduces a fusion network to unify the processing of outputs from different visual encoders. Experiments on downstream tasks show that incorporating more than one visual encoder could significantly increase the model's performance in different aspects.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging comments and helpful feedback. We have addressed each of the issues you raised. We will make the revisions to our manuscript.\n\n---\n\n> **Response to Q1: The ensemble model contains more parameters than baselines, it might be unfair for comparison.**\n\n- Thank you for your comments. First, we would like to emphasize that the ensemble model has the *same* number of *learnable parameters* as the baseline because the visual encoders, whether there is one or multiple, are always frozen (as in the case with most LVLMs).\n\n- Second, to make the total parameters of triple-experts and single-experts comparable, we did our best to find a 1.9B CLIP encoder (CLIP-ViT-bigG-14-laion2B-39B-b160k), and trained a new single-expert baseline (8.9B). The results show that our method remains state-of-the-art with a fair comparison. Additionally, using a 1.9B CLIP expert even hurt the baseline's performance, demonstrating that the capability of visual experts is not related to the number of their parameters (as also evidenced by Table 7 in Appendix C of our paper).\n\n  |Model|Param.|VQAv2|GQA|VQA-T|POPE|MMB|MMB-CN|SEED-I|MM-Vet|Avg@8|\n  |-----|------|----------------|---|--------------|----|---|--------------|-------------|------|-----|\n  |Single-expert(0.3B CLIP-Res-336)|7.3B|78.5|62.0|58.2|85.9|63.0|57.4|66.2|30.5|62.7|\n  |Single-expert(1.9B CLIP-Res-224)|8.9B|73.1|52.8|50.4|82.2|64.7|58.6|59.0|27.9|58.6|\n  |Triple-expert|8.8B|79.1|63.6|58.4|86.5|67.4|60.0|67.5|33.6|**64.5**|\n\n---\n\n> **Response to Q2: [minor] It would be better if the strengths and weaknesses of different visual experts could be summarized more clearly, rather than just using case studies to demonstrate them.**\n\n- Inspired by your and reviewer oxQg's insightful comments, we believe that an in-depth analysis of the experimental results can yield some valuable conclusions. \n\n- Using the TextVQA results in Table 2 as an example, we found that it differs from other benchmarks as it emphasizes the model's ability to read and reason about text information in images. When the CLIP expert collaborated with the **LayoutLM** expert, the performance on TextVQA improved by 0.7, benefiting from LayoutLM's document-level OCR pre-training. Conversely, combining with other experts decreased performance (ranging from -0.3 to -1.3) because their pre-training processes did not emphasize text information within images. This clearly demonstrates **LayoutLM**'s strength in text OCR tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a simple but effective method to take advantage of different pre-trained visual encoders. Experiments show the proposed method could significantly outperform the method with only a single visual encoder.\n\n2. The improvements of the ensemble method further illustrate that the information obtained by different visual encoders is different. Future work might consider how to help one visual expert encode more details.\n\n3. The ablation study is comprehensive and is very helpful in understanding the contribution of different components.", "reasons_to_reject": "NA", "questions_to_authors": "1. The ensemble model contains more parameters than baselines, it might be unfair for comparison.\n\n2. [minor] It would be better if the strengths and weaknesses of different visual experts could be summarized more clearly, rather than just using case studies to demonstrate them.", "extra_scores": {}}, {"review_id": "MnQqSVqDjX", "reviewer": "Reviewer_BvJn", "summary": "In this paper, authors present the use of ensemble experts technique to synergize the capabilities of individual visual encoders, including those skilled in image-text matching, OCR, image segmentation, etc. The proposed approach MouSi have a poly-expert fusion network, a pre-trained open-source LLM and a multi-expert visual encoder. Experiments performed with various benchmarks showcase combing multiple experts provide better performance in contrast to single expert.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments and helpful feedback. \n\n---\n\n> **Response to Q1 & Q3: Mixture-of-experts idea has been explored in the literature & no new novel method proposed**\n- We agree and will supplement the Related Work section with MoE-related literature. We'd like to emphasize that, to the best of our knowledge (also mentioned by Reviewer oxQg), we are the first to apply this idea to LVLMs.\n\n---\n\n> **Response to Q2: Poly-Fusion network is very similar to the standard MLP or Q-Former network**\n- We agree and would like to clarify that the primary goal of the Poly-Fusion network is not to innovate the standard MLP or Q-Former networks, but rather 1) to demonstrate how MLP or Q-Former networks can be adapted from a single vision expert to multiple vision experts, and 2) to determine which network, MLP or Q-Former, is more suitable for multiple experts (the conclusion is that MLP is significantly better).\n\n---\n\n> **Response to Q4: Is there an effective way to combine the experts for a task and the model will figure out which one to use?**\n-  MouSi has demonstrated the effectiveness of Poly-Visual-Expert, but we have also observed negative transfer effects (potential conflicts between experts). This suggests that expert combinations should be flexible and changeable when handling different tasks. A feasible solution is to add a trainable routing module of experts, allowing Poly-Visual-Fusion networks to *sparsely activate* corresponding experts for specific tasks. This will be a focus of our future work, potentially reflected in MouSi-2.\n\n---\n \n> **Response to Q5: Is MouSi capable of comprehending different languages text also?**\n- MouSi's pre-training dataset is only in English, while the SFT dataset includes both Chinese and English. Since the LLM module (Vicuna-1.5) has multilingual capabilities, we found that MouSi can generalize to some other languages (such as French or German), but it is not always successful.\n\n---\n\n> **Response to Q6: How effective it is? Also during inference, will all parameters be used?**\n\n |Model|Param|Memory(G)|Inference time(s)/Example|\n |-|-|-|-|\n |1-expert(0.3B-CLIP)|7.3B|15.9 / 1.00x|0.120 / 1.00x|\n |1-expert(1.9B-CLIP)|8.9B|19.3 / 1.21x|0.177 / 1.48x|\n |2-expert-ours|8.4B|16.9 / 1.06x|0.145 / 1.21x|\n |3-expert-ours|8.8B|18.2 / 1.14x|0.149 / 1.24x|\n\n- The increased inference time and memory on three experts are *not significant*, benefiting from the proposed Poly-Fusion network. \n- During inference, all parameters are used.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Proposed method compresses the local visual information by multi-patch-one-token for better transmission efficiency and reduces the quadratic computational cost of subsequent processing of VLMs.\n\n2. Explores different positional encoding schemes to effectively address the issue of position encoding waste. To be specific, sharing one position for all patches and 2D positional encoding.\n\n3. Experiments are showcased with different combination of experts (i.e., VLMs)", "reasons_to_reject": "1. Mixture-of-experts idea has been explored in the literature, and the proposed approach MouSe is built on the same principle\n\n2. MLP fusion network or Q-Former network has been a standard approach used by VisualLLMs. The proposed Poly-Fusion Network is very similar in architecture.\n\n3. Experimental study combining different available models and approaches. No new novel method proposed.", "questions_to_authors": "Comments to Authors\n\n1. Is there an effective way to combine the experts or all known experts for a task is used and the model will figure it out which one to use? Table-2 showed that double vision experts are useful, but these combinations can become overwhelming when there are large number of experts available for a task.\n\n2. Although, Section-3.5 mention different case studies by MouSi. Is MouSi capable of comprehending different languages text also?\n\n3. Although multiple experts boos performance, how effective it is for the time-sensitive applications which require fast inference, still single expert is the approach for them? Some discussion on the trade-off will be useful. Also during inference, even though parameter space has increased, will all parameters will be used?\n\nMinor Comments\n\n1. Table-2 figure is unreadable without zoom. Please make it more visible.", "extra_scores": {}}, {"review_id": "VghBvVuBID", "reviewer": "Reviewer_RrkK", "summary": "This paper investigates how to ensemble multiple image encoders to provide a visual input to an LLM. The authors explore the use of six different visual encoders (eg. CLIP, DINO, MAE) and find that, often, using three encoders is better than using two encoders, which in turn is better than the best single encoder. To combine the visual tokens from multiple visual encoders, the authors devise a fusion network that reduces the number of tokens per visual encoder while mapping the image embedding to the LLM’s input dimension. Experiments on 9 datasets show that the proposed approach improves upon using a single image encoder as well as on related baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your insightful comments. \n\n---\n\n> **Response to W1&W5: Several key results are delegated to the appendix & More discussion on related work**\n- We apologize for moving key results to the appendix due to page limits. Fortunately, noticed by the PCs' email that there will be an extra page. We promise to add the necessary results and descriptions to enhance readability. Thanks for your helpful suggestions.\n\n---\n\n> **Response to W2: Most performance gains less than 1**\n- Our performance gains are comparable to (or even higher than) other recently published works, e.g., compared to LLaVA-1.5 on 6/8 shared datasets, the average gains of Monkey and mPLUG-Owl2 are +0.0 (59.6) and +0.7 (60.3). While our default and data enhanced models are +2.3 (61.9) and +0.5 (63.8).\n---\n\n> **Response to W3: Evaluation on MMB-CN**\n- Since SFT data contains both EN and CN, we evaluated on MMB-CN (in line with LLaVA).\n\n---\n\n> **Response to W4: Challenge of visual token overflow**\n- Since LVLMs apply LLM's position encoding(PE) again to vision tokens(i.e., double PE), they are prone to encounter out-of-max-length issues, especially when handling multiple visual experts or multiple input images. Our experiments confirmed that vision tokens don’t degrade baseline performance even without second PE. We effectively mitigated this issue by proposing multi-patch-single-token projection and Share-all PE.\n\n--- \n\n> **Response to Q1: Do you mean in parallel**\n- Yes, multiple experts encode images independently and in parallel. We will revise it.\n\n---\n\n> **Response to Q2: Absolute performance of CLIP**\n- We will add it.\n\n---\n\n> **Response to Q3: Inference time, Memory**\n\n |Model|Param|Memory(G)|Inference time(s)/Example|\n |-|-|-|-|\n |1-expert(0.3B-CLIP)|7.3B|15.9 1.00x|0.120 1.00x|\n |1-expert(1.9B-CLIP)|8.9B|19.3 1.21x|0.177 1.48x|\n |2-expert-ours|8.4B|16.9 1.06x|0.145 1.21x|\n |3-expert-ours|8.8B|18.2 1.14x|0.149 1.24x|\n\n- We conducted inference time and memory usage experiments with MMB. For fair comparison, we added a single encoder(1.9B-CLIP) baseline with a similar number of total parameters.\n\n- Compared to LLaVA, our selected(from paper) 2/3 experts' inference time increased by 1.21x/1.24x, memory increased by 1.06x/1.14x. The additional inference cost is not significant.\n\n- Our inference cost is even lower than that of a single encoder with the same parameter size, demonstrating the efficiency of the m-patch-one-token method and parallel encoding with multi-visual-experts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors study how to effectively and efficiently incorporate multiple image encoders with different strengths to improve visual understanding for multimodal tasks\n2. The proposed approach is simple yet often yields better results than using a single image encoder, especially on SQA and MMB", "reasons_to_reject": "1. Albeit being generally well written, the paper is not easy to follow. Several key results are delegated to the appendix, without much discussion in the main paper. For example, it is said that triple-encoders outperform other methods (ie. are the strongest proposed model) but are however not discussed in the main paper. Another example is at the beginning of Section 2.2, where you said that you chose six image encoders “after extensive investigation”; which is not shared with the readers. Lastly, you add a whole Section 3.4 on “data enhancement” without ever saying what you do in the main body, letting the reader wonder what this might be about.\n2. The performance gains are within 1 percentage point for most datasets. We can observe larger gains on SQA and MMB, but the overall boost between the three-expert model and the CLIP-only model is less than 1 percentage point in the final “Data enhancement” results. \n3. Moreover, the results include evaluation on both MMB and MMB-CN (a version of MMB translated into Chinese). If the goal is to assess the ability of the model to perform multiple tasks, having MMB-CN is redundant as it assesses the same ability as MMB. If the goal is to evaluate multilingual capabilities, there are far better benchmarks to do so (eg. MaRVL, Liu+EMNLP’21; IGLUE, Bugliarello+ICML’22; XM3600, Thapliyal+EMNLP’22).\n4. One of your main contributions was to tackle the “challenge of visual token overflow”. I was not aware of this challenge, was it introduced by someone else before (if so, please cite) or was it something you first encountered in this study? In the latter case, I would rather see it as an ablation of positional encoding more than a challenge itself, as one can simply not add positional encodings (again) after the visual embeddings, since these normally include positional encodings in their respective encoders (which was your final recommendation too)\n5. The related work would benefit from discussing other approaches for ensembling (both for VLMs and not) as well as for token merging/dropping.", "questions_to_authors": "1. In Section 2.1, you state that “the image is processed in _sequence_ through the [visual] encodings”; do you mean in parallel (looking at Figure 2)?\n2. In Table 2, could you please add the absolute performance of a CLIP-based model in the first row for better comparability?\n3. The model size increases by 1.5B parameters when using three encoders. How does this influence inference time/memory in practice? This is a key information (along with accuracy) to decide whether/when to use this method.", "extra_scores": {}}, {"review_id": "8g9hM0XrBt", "reviewer": "Reviewer_oxQg", "summary": "This paper proposes a way to fuse together multiple vision encoders within a vision-language model setup. They show that a) using two and three vision encoders (that are pre-trained on complementary tasks and datasets like image-text contrastive learning, image segmentation, and OCR) can improve upon using a single vision encoder (i.e. CLIP), b) a simple MLP does better than a more complicated transformer-based network (Q-former) in fusing the image and text modalities from the image encoder and the LLM, and c) improving the efficiency of VLMs with respect to input tokens by eliminating a lot of redundancy in the positional embedding space for images.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your providing insightful suggestions. We have addressed each of the issues you raised. We will make the revisions to our manuscript.\n\n> **Response to Q1: would have been great to see further analysis on Table 2 and Table 8. For example in Table 2, why is VQA anomalous? In Table 8, we see results are a lot more mixed with 3 experts. Is this a limitation of the approach that could be highlighted in the main text?**\n- Thank you for pointing out some anomalies that require further analysis. Using the TextVQA results in Table 2 as an example, we found that it differs from other benchmarks as it emphasizes the model's ability to read and reason about text information in images. When the CLIP expert collaborated with the LayoutLM expert, the performance on TextVQA improved by 0.7, benefiting from LayoutLM's document-level OCR pre-training. Conversely, combining with other experts decreased performance (ranging from -0.3 to -1.3) because their pre-training processes did not emphasize text information within images.\n- This *negative transfer effect* indicates that, in a multi-expert system, the specialties and capabilities of different experts may lead to conflicts or contradictions. It is a limitation as you mentioned of our current approach, suggesting that expert combinations should be flexible and changeable when handling different tasks. This will be a focus of our future work, potentially reflected in MouSi-2.\n\n> **Response to Q2: it doesn't seem like, compared to \"origin\", the parameter sharing schemes in 3.2.2 have much effect on anything? The differences with the baseline are small, in fact smaller than the effect of expert order (Table 9)**\n- There may be a misunderstanding here, so we'd like to clarify that the **Different Position Schemes** we propose aim to resolve the out-of-max-length issue in VLMs, particularly when handling multiple vision experts (e.g., triple-experts) or input images (e.g., video). The aim is not to improve the performance of downstream tasks. As a result, the \"Share-all\" method consumed the fewest positional encodings (one position per image) while maintaining performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* I believe this might be the first VLM that combines multiple vision encoders (trained on different objectives and tasks), and they do show by and large improvement on a wide variety of tasks\n* evaluation in multiple languages", "reasons_to_reject": "* would have been great to see further analysis on Table 2 and Table 8. For example in Table 2, why is VQA anomalous? In Table 8, we see results are a lot more mixed with 3 experts. Is this a limitation of the approach that could be highlighted in the main text?\n* it doesn't seem like, compared to \"origin\", the parameter sharing schemes in 3.2.2 have much effect on anything? The differences with the baseline are small, in fact smaller than the effect of expert order (Table 9)", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a vision--language model that combines multiple expert vision encoders---similarly to MoE work in the LM space. The authors also proposes a method to reduce the number of positional embeddings used by various models. They show the effectiveness of their approach across a range of benchmarks and perform ablations to examine the effectiveness of different component of their model. This paper contributes to the MoE work in the multimodal space, and the future work can build on it. However, I encourage the authors to address the reviewers' feedback by (1) adding discussion around the related work (2) including analysis of inference time, memory, and the effect of number of parameters (3) discussion around strengths/weaknesses of each expert.
Decision: Accept