Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders
Authors: Min Shi
Fuxiao Liu
Shihao Wang
Shijia Liao
Subhashree Radhakrishnan
Yilin Zhao
De-An Huang
Hongxu Yin
Karan Sapra
Yaser Yacoob
Humphrey Shi
Bryan Catanzaro
Andrew Tao
Jan Kautz
Zhiding Yu
Guilin Liu
Venue: ICLR 2025 Spotlight (2025)
Keywords: LLM
Vision-Language Model
Vision-Centric Multimodal LLM
Primary area: foundation or frontier models, including LLMs
TL;DR: This work presents a thorough design space exploration to strengthen multimodal LLM perception with mixture of vision encoders.

Abstract: The ability to accurately interpret complex visual information is a crucial topic of multimodal large language models (MLLMs). Recent work indicates that enhanced visual perception significantly reduces hallucinations and improves performance on resolution-sensitive tasks, such as optical character recognition and document analysis. A number of recent MLLMs achieve this goal using a mixture of vision encoders. Despite their success, there is a lack of systematic comparisons and detailed ablation studies addressing critical aspects, such as expert selection and the integration of multiple vision experts. This study provides an extensive exploration of the design space for MLLMs using a mixture of vision encoders and resolutions. Our findings reveal several underlying principles common to various existing strategies, leading to a streamlined yet effective design approach. We discover that simply concatenating visual tokens from a set of complementary vision encoders is as effective as more complex mixing architectures or strategies. We additionally introduce Pre-Alignment to bridge the gap between vision-focused encoders and language tokens, enhancing model coherence. The resulting family of MLLMs, Eagle, surpasses other leading open-source models on major MLLM benchmarks.

Published as a conference paper at ICLR 2025
EAGLE: EXPLORING THE DESIGN SPACE FOR MULTI- MODAL LLMS WITH MIXTURE OF ENCODERS
Min Shi Fuxiao ,
Shihao Wang', Shijia Liao', Subhashree Radhakrishnan', Yilin Zhao De-An Huang', Hongxu Yin', Karan Sapra', Yaser Yacoob', Humphrey Shi?, Bryan Catanzaro', Andrew Tao', Jan Kautz*, Zhiding Yu Guilin Liu"!
NVIIIA Georgia Tech UMD
4HKPU
NNYU
http://gitub.coo///aaaasse
Eagle
HANDA
Highlights
Strong per pormommnnc
Thorough space exploration
FACA
open-inucee
Better recipe
Vision-enttii Design
Moture of vision encodars
Mar
Ensentile
extendable tusion
Encoder
Optimal enooder combination
6 Prealigmmet
TectOA
Achanced Training Recipas
cOMbrah
High resolution actepacion
Cintrtur-1
Exg
Design Space Exploration (Vicuna-7B)
Viito expert pre-aligrmen
Figure 1: Overview of Eagle. Eagle is a family of multimodal large language models (MLLMs) with a mixture of vision encoders. Left: comparisons between Eagle and existing competitive MLLMs with Vicuna-131 (Chiang et al., 2023), with Eagle achieving favorable results on all 13 benchmarks. Middle: an evolutionary road map of the design space and advanced training recipes leading to consistent and significant improvements. Right: highlights and core features of Eagle.
ABSTRACT
The ability to accurately interpret complex visual information is a crucial topic of multimodal large language models (MLLMs). Recent work indicates that enhanced visual perception significantly reduces hallucinations and improves performance eesolution-sensitive tasks, such optical character recognition and document analysis. A number of recent MLLMs achieve this goal using a mixture of vision encoders. Despite their success, there lack of systematic comparisons and detailed ablation studies addressing critical aspects, such as expert selection and the integration of multiple vision experts. This study provides an extensive exploration of the design space for MILMS using mixture of vision encoders and resolutions. Our findings reveal several underlying principles common to various existing strategies, leading to streamlined yet effective design approach. We discover that simply concatenating visual tokens from a set of complementary vision encoders is effective as more complex mixing architectures or strategies. We additionally introduce Pre-Alignmení to bridge the gap between vision-focused encoders and language tokens, enhancing model coherence. The resulting family of MLLMs, Eagle, surpasses other leading open-source models on major MI.LM benchmarks.
INTRODUCTION
The success of large language models (LLMs) has triggered significant interest in enabling their visual perception capability, such that they could see, understand, and reason in the real world. At the center of these multimodal large language models (MLLMs) (Fei al., 2024) is a typical design
"qqual contribution. Work done during an internship at NVIDIA.
I Equal advising. Corresponding authors: [guilinl, zhidingyjennn jenvid wwweeeiiiiiiccoom dia.

---

Published as a conference paper at ICLR 2025
where images are converted into series of visual tokens by the vision encoders and appended with the text embeddings. CLIP (Radford et al., 2021) is often chosen as the vision encoder since its visual representation is aligned with the text space by pre- pre-training on image- pairs. Depending on the architectures, training recipes, and the way how vision tokens are injected into the language model, there exist various notable families of MILMs such as Flamingo (Alayrac al., 2022). BLIP (Li et al., 2022; 2023d; Dai et al., 2024). PaLI (Chen et al., 2023e), PaLM-E (Driess et al., 2023) and LLaVA (Liu et al., 2023d;c). Most of these works keep relatively low input resolutions due to the limits on pre-trained vision encoders and LLM sequence length.
Recent studies (Li al., 2024c: Liu et al., 2024a) show that stronger vision encoder design is important for mitigating MLLM hallucinations (Liu et al., 2023a; Wu et al., 2024) and improving issolution-sensitiiii tasks like optical character recognition (OCR). constellation of works thus focuses on enhancing the capability of the vision encoder. For example, scaling up the pre-training data and parameters of vision encoder (Chen et al., 2023f) or dividing images into low-resolution patches (Liu et al., 2024a; Shi et al., 2024). However, these approaches usually introduce large training resources. An efficient yet powerful strategy is to mix visual encoders pre-trained with different tasks and imput resolutions, either fusing higher resolution encoders with the CLIP encoder (Luo et al., 2024; Li et al., 2024b), sequentially appending features from different encoders (Fan et al., 2024; Lin et al., 2023b; Karamcheti et al., 2024; Tong et al., 2024), adopting more complex fusion and routing strategies to make the best of different encoders (Lee et al., 2024; Zong et al., 2024). In addition, Prismatic VLM (Liu et al., 2024b) incorporates multiple vision encoders with channel concatenation as part of their design space exploration. These mnixture-of-vision--xe are shown be effective. However, a detailed study focusing on their designs still lacking.
strategies
To address the above questions, our work systematically investigates the siatre----viiio-ennooeee design space for improved MLLM perception. As shown in Fig. 1, our exploration of the design space consists of the following steps: 1) Benchmarking various vision encoders and searching recipes for higher resolution adaptation: 2) "Apples to apples" comparison between vision encoder fusion strategies; 3) Progressive identification of the optimal combination of multiple vision encoders; 4) Improved vision expert pre-alignment and data mixture. Our study covers the performance of vision encoders pre-trained on different tasks and resolutions (e.g., lasoun-anguuag alignment (Ilaacc et al., 2021; Cherti et al., 2023; Radford et al., 2021; Schuhmann et al., 2022), self-supervised learning (Oquab et al., 2023), detection (Fang et al., 2023b;a), segmentation (Kirillov et al., 2023), and OCR (Lee et al., 2023)). We use a round-robin approach to incorporate additional vision experts. Starting with the basic CLIP (Radford al., 2021) encoder, we add one additional expert each time with the best improvement in each round.
Our work is not the first one to leverage multiple vision encoders in MLLM. However, the systematic study leads to several interesting new findings under this setting:
Unlocking the vision encoders during MLLM training matters. This is in sharp contrast to the LLaVA (Liu et al., 2023d;c) family and many works that consider multiple vision encoders or teachers (Lin et al., 2023b; Liu et al., 2024b; Fan et al., 2024; Kar et al., 2024; Ranzinger et al., 9024; et al., 2024), where freezing the vision encoders has been a common choice.
Some recently proposed fusion strategies (Luo et al., 2024; Li et al., 2024b) do not show significant advantages despite their advanced designs. Instead, we find that straightforward channel concatenation stands out as simple yet competitive fusion strategy, offering the best efficiency and performance.
Incorporating additional vision experts leads to consistent gain, making it promising path to systematically enhance MLLM perception besides scaling up vision encoders. The improvement is particularly pronounced when vision encoders are unlocked.
We propose a pre-alignment stage where non-text-alignee vision experts individually fine- tuned with a frozen L.MM before being trained together. This stage is found to enhance the MILLM performance significantly under the cnxture-o--vision-emee
design.
We finally conclude our findings into a family of MLLMs termed Eagle. Eagle is evaluated on series of benchmarks, including visual question answering, OCR/documentt-rlated tasks, and benchmarks tailored for MLLMs. Our model attains state-of-the-ar performance across different benchmarks and demonstrates obvious advantages on OCR and document understanding tasks. Using
2

---

Published as a conference paper at ICLR 2025
the same pre-train and supervised fine-tuning data from Cambrian-1 (Tong al., 2024) concurrent family of vision-centric MLLMs sharing similar design spirits, Eagle models overall achieve better performance. We hope that the Eagle can provide highly performant and easy-to-reproduce MLLM solution to the community.
DESIGN SPACE EXPLORATION
In this section, we show how to utilize the advantages of different vision encoders via step-by-step investigations, yielding the Eagle model family. Unlike previous methods focusing on new fusion strategies architectures among vision encodes, our goal is to identify set of minimalistic design to fuse different vision encoders supported with detailed ablations, removing any unnecessary parts. As shown in Fig. we start by extending the basic CLIP encoder (Radford al., 2021) to a set of vision experts with different architectures, pre-training tasks, and resolutions. With these experts, we then compare different fusion architectures and methods and study how to optimize the pre-training strategies given more encoders. We also give detailed analysis of how to select the vision encoders to be integrated. Finally, we put all the findings together and further extend to multiple expert vision encoders with different resolutions and domain knowledge.
2.1 BASE SETUP
We adopt LLaVA-1.5's (Liu et al., 2023c) model architecture as the basis, which consists of large language model (Vicuna-vI., 1,5 7B (Chiang et al., 2023)), a vision encoder, and projection layer. The projection layer projects the visual embedding from the vision encoder into the text embedding space.
Base training data. We adopt the same pre- pre-training data (LL5VA-595k) as LLaVA-1.5 (Liu et al., 2023c) for the first pre- pre-training stage, which consists of 595k image text pairs. To fully examine the potential of different vision experts and fusion methods, instead of using the SFT data from I.LVA-1.55 (Liu et al., 2023c), we collect data from a series of tasks and convert them into multimodal conversations for the supervised fine-tuning (SFT) stage, denoted as Eagle1 1.8M in Table 1.
Table 1: Composition of the base supervised fine-tuning data (Eagle1.8M)
Tottl Data Size
Data Source
LLM -15 (665k) (Liu et al., 2023c), DocVQA (39)) (Mathew et al., 2021), synDog EN (50k) (Kim et al., 2022),
CharQA (28k) (Masry al., 2022), DVQA (25k) (Kafle et a 2018), A12D (15k) (Kembhavi et al,, 2016n),
1,809k
ShareGPT 4V (10Dk) (Chen et al., 2023b), laion GPT4V (1Ik) (lai, 2023), LVIS Instrucid (220k) (Wang et al., 2023s), I.V.Instruct (150k) (Liu et al., 2023b), Geol 170k (120k) (Gao al., 2023), LLaVAR (20k) (Zhang et al,, 2023), Visual7W (70k) (Zhu et al,, 2016), Open-Hermes 2.5 (300k) (Toknium, 2023)
Base training recipes. We start from the LLaVA-I.5 (Liu et al., 2023c) recipe where the model is first pre-trained with image-fext pairs for one epoch with a batch size of 256. The whole model is frozen and only the projector layer is updated in this pre-training stage. In the second stage, we further fine-tune the model on the multi-modal conversation data for one epoch with batch size of 128. The learning rates are set to be 1e-3 for the first stage and 2e-5 for the second stage, respectively.
Base evaluation. To conduct comprehensive comparison of various methods, we adopt 11 distinct benchmarks that span multiple tasks. These benchmarks include 1) General VQA tasks: GQA (Hud- son & Manning, 2019), VizWiz (Gurari et al., 2018), MME (Fu et al., 2023), SEED (Li et al., 2023c); 2) OCC//occment/chart understanding: OCRBench (Liu et al., 20231), Doc VQA (Mathew et al., 2021), ChartQA (Masry et al., 2022); 3) vision-centric tasks: POPE (Li et al., 2023e), Real- tasks: ScienceQA (Saikh et al., 2022), A12D (Kembhavi al., 2016b). To obtain an average score, we normalize each benchmark total score of 1,000 and then calculate the average score across all benchmarks.
WorldQA (xAI, 2024); 4) knowledge-basee
2.2 STRONGER CLIP ENCODER
We start our exploration by upgrading the vanilla CLIP Radford et al. (2021) model since it has become a standard choice for most of the MLLMs Liu et al. (2023d;c). While CLIP models are known to benefit multimodal tasks via the text-image alignment, they also have inherent drawbacks. For instance, many existing MLLMs (Liu et al., 2023c) tend to use the pre-trained CL/P resolutions (such 224 x 224 or 336 x 336) as their input resolutions. In these cases, the encoders often fail to capture fine-graine mne-grained details that are important for resolution-sensitive tasks like OCR and document understanding (Li et al., 2024c).
3

---

Published as a conference paper at ICLR 2025
Step1: Vision Modification Optiiization
Step2: Fusion Paradgg Exploration
Step3:: Training Strmtegy Model Optimization
Malt cocolution
Highteodestare
Visul keatuse:
Fre-algnment
Expert
input Iroges
Large Language Model
LUNE
Vhice Pargeag
Object Detection
Polie
GOG
Sxpert 1:3
Breagaitsa
Sepmentrtion
solation
Enbeadikng
Exprrt
dxpert
Exper
Sequance Appand
Charnal-wskn Concateration
LLAVA-R
Mini- Mini-Gamini
Deformable Attantion
Roickyzon
Roodatin
Lne
Petien
Bisct
Gooda
Adaate
Adarrer
Corm
High Read'' Mon
High Paccution
High Rlcc Ludoa
High hotcstton
High Tasckutie
Figure Overview of the Eagle exploration. this work, we explore the design space of Multi-Modal Large I Language Models (MLLMs) with multiple vision encoders, aiming to identify optimized design choices and enhance MLLM perception. We collect a range of vision experts and adapt them for integration into MLLMs. A systematic comparison of popular fusion paradigms is then conducted under controlled settings. After identifying discrepancies between vision experts pre-trained on different tasks, we optimize the pre-traine pre-training strategy through pre-alignment stage and use round-robin searching to determine the optimal combination of vision encoders.
To handle increased input resolution, common practice is use tiling where input images are divided into tiles and encoded separately (Liu et al., 2024a; Li et al., 2024c), or just interpolate the position embedding of the vision transformer model to fit high-resolution inputs (Chen et al., 2023c;d; Beyer et al., 2024). We compare these two approaches with frozen/unfrozen vision encoders under different resolutions, with the results shown in Table 2. Our findings can be summarized as follows:
Updating the CL/P encoder during SFT significantly improves performance at higher resolutions but slightly reduces it when using the pre- pre-training resolution.
Interpolating CLIP encoder to fit the input size of 448 x 448 offers strong balance between efficiency and performance, trailing the 672 x 672 version with less than half the tokens.
Despite its smaller size (0.3B vs. 5.9B) and less pre-training data, the CLIP encoder gets close with interpolation approaches Intern VL's (Chen et al., 2023f) performance under the same setting.
Based on the results, we can see that Direct inter- polation to 448 x 448 can achieve competitive performance while being more efficient. We thus use the CL/P encoder with 448 x 448 input resolution while unlocking the encoder during the SFT stage.
Table 2: Comparing different high-resolution adaption methods. "#Tok(V)" denotes the num- ber of visual tokens. "#Parames", "FLOPs" and "Img/Sec" denote the model size, complexity and throughput (bs=4) of the vision encoder.
Method
Unfreeze Res. #Tok(V) #Params FLOPs Img/Sce Avg.
2.3 VISION EXPERTS
Original Original
x
336
576
0.38
119G
197.2
616.5
336
576
0.38
119G
197.2
562.6
To better establish the foundation for multi- vision expert fusion, we extend the toolbox with vision experts pre-trained on different tasks and resolutions, and verify our findings on high- resolution adaptation with these experts. This also helps us identify the distinct advantages of different experts. We collect a set of vision ex- perts, including: (1) Eision-Languag Aligument: CLIP (Radford et al., 2021) and ConvNeXt (Liu
Interpolate Interpolate
X
448
1024
0.38
214G
119.5
589.7
448
1024
0.38
214G
119.5
670.5
Interpolate
672
2304
0.38
480G
56.3
674.2
Tiled-inpua
672
2304
0.38
476G
51.6
673.9
InterVVL.
448
1024
5.9B
S669G
13.52
661.9
InterVLL
448
1024
5.9B
5669G
13.52
671.5
et al., 2022) from OpenCLIP (Ilharco et al., 2021; Schuhmann et al., 2022). (2) Object- Centric: EVA-02 (Fang et al., 2023b;a) pre-trained on detection datasets. (3) OCR: Pix2Struct (Lee et al.,
(42 Segmentation: SAM (Kirillov et al., 2023). (5) Self-supervised: DINOv2 (Oquab et al., 2023). We resize the output 2D feature maps of each vision encoder using bilinear interpolation or pixel shuffle (Shi al., 2016) to ensure that the visual token number equals 1024.
4

---

Published as a conference paper at ICLR 2025
Results in Table 3 show that un- freezing the vision experts again leads to consistent improvement, which is aligned with Sec. 2.2. In addition, results in Table 10 (see Appendix A) further demon- strate that MLLMs with these task- specific vision encoders achieve optimal performance in their pre- training domains. EVA-02 excels in the object hallucination evalua- tion benchmark POPE and general visual question answering bench- mark GQA. CLIP and ConvNeXt perform well across all benchmarks, benefiting from their training on large-scale image-text pairs using contrastive loss. Conversely, while Pix2Struct excels in text recognition, it shows limited capability in object recognition and general VQA tasks, like POPE and GQA. DINOv2 and SAM, pre-trained with self-supervised learning and segmentation, struggle with text recognition tasks.
Table 3: Comparison between different vision experts as the MLLM encoders.
Calegory
Vision Encoder
Res.
Post-process
Unfreeze
Avg.
Model Link
Al.lfimmment
ComNecKi
1024
None
654.6
CunvNeXI-XXL
682.1
Segmentuatio
Pizel Unshuille
x
486.2
SAM LaT-Laee
510.5
Object Derection
EVA 02
1024
Resize
x
543.7
EVA-02-L-Det
639.1
Revoggnition
2ir2Struct
1024
Resize
X
598.
Pio2Simat-12-4aag arge
606.2
Self Sep-Superdd
DMNOv2
448
None
x
520.7
ViT-I/14-Rgg
$37.3
2.4 FUSION STRATEGY
Existing MILM frameworks have proposed various -oextureoo--viiion--ee hope that their domain-speciin strengths can be leveraged. In all cases, improvements in MLLM performance have been reported with the fusion of vision encoders. However, the roles of the fusion strategies as part of their MLLM architecture innovations, have not been decoupled and clearly studied under an "apples to apples" comparison. It is thus not entirely clear how much improvement is from the fusion strategies themselves versus the improved representations from various encoders.
strategies, with the
We notice that existing popular fusion strategies, despite their variations in designs, can be broadly represented by the following several categories: (1) Sequence Append: directly appending the visual tokens from different backbones as a longer sequence (Fan et al., 2024; Kar et al., 2024); (2) Channel Concatenation: concatenating the visual tokens along the channel dimension without increasing the sequence length (Lin et al., 2023b; Karamcheti et al., 2024); (3) LLaVA-IIR injecting high- resolution features into low-resolution vision encoders using mixxure--f-resollutin adapter (Luo et al., 2024); (4) Mini-Gemini: using the CLIP tokens as the low-resolution queries to cross-attend another high-resolution vision encoder in the co-located local windows (Li et al., 2024b). (5) Deformable Attention: a new baseline we introduce on top of Mini-Gemini, where the vanilla window attention replaced with deformable attention (Zhu et al., 2021). Fig. gives a detailed illustration of these fusion strategies. To better study them, we choose "CLIP+ComNeXI" and "CLIP+ComWeXX-SAMM as the base multi-encoder combinations to perform comparisons.
Our study in Table 4 shows that Channel Concate- nation stands out with the best performance, ex- pandability, and efficiency. The "injection-basee methods, such as LLaVA-HR, Mini-Gemini and De- formable Attention, are in general less competi- tive on TextVQA (Singh et al., 2019) and OCR- Bench (Liu al., 2023f), performing worse than using ConvNeXi alone as the vision encoder. Al- though sequence append shows comparable perfor- mance to channel concatenation, it faces the chal- lenge of handling significantly increased sequence lengths with additional vision encoders.
Table 4: Comparison of different fusion meth- ods for different vision experts. "#Token(V)" denotes the number of visual tokens. "#To- kens/s" denotes the inference throughtput of the whole pipeline.
Vision Enueders
Fusion
#TokennV) 4Tokems' 4Purams Avg.
Seg. Append
2048
46.1
1200M
Chanari Cunzar.
1034
47.3
11843
681.5
CLIF + ComeNeXi
HLVA--IR
1024
47.0
1319M
678.7
Mou Gemint
1024
45.3
1201M
672.5
Deformable Ann.
1024
47.3
1201M
674.3
CLI ComeNeX)
Sey. Appead
40.3
1520M
686
SAM
Chansel Concar.
1024
46.3
1495M
690.4
2.5 VISON-LANGUAGE PRE-ALIGNMENT
As shown in Table 3, encoders pre-trained exclusively vision tasks (e.g., detection, OCR, and segmentation) are less competitive compared to those pre-trained on vision language alignment. This is possibly due to representational inconsistencies when integrated with large language models.
5

---

Published as a conference paper at ICLR 2025
Stagel: Alignment
Stage2: Joint- Projectoo Training
Stage3: Supervised Finetuning
Volon A Kboon
Volan Export e Hoine Lergugee Abgerwer
Visuta 7D
Tot fenegrioo
Large Lang uaguage Madel
Large Language Model
Visce Ancigaiian
Voual Festeres
Welen Expet Tex Plooge Vinn
Festuras
Veien Expert B
Vicura 7B
Object Oeiectoon
Vteen Expet
+
Vieana 7B
Vidaon Expart Opper Deanciion
+
Vidlen Expert C
Vidds
Seeoch Seytintioe Suriee
Wieee Ixpirt Semantts Stpmomation
Expert
Projecor
Generative loss
Figure 3: The proposed training strategy of Eagle. It consists of three progressive stages, including vision-language pre-alignment training, joint-project training and supervised fine-tuning. These stages effectively leverage public data from diverse sources, ranging from noisy image-text pairs on the web high-quality caption, VQA, and multimodal dialogue datasets.
Additionally, when combining different encoders, there is a gap between these encoders, creating difficulties in the training process. To address this feature inconsistency, propose Pre-Aligument training stage that first aligns each individual vision encoder with the same large language model, fostering better synergy between visual and linguistic capabilities.
Fig. depicts our pre-alignment strategy. Instead of training a projector to simultaneously align multiple vision experts as LLVVA's (Liu et al., 2023c) original pre-training strategy, we first align the representation of each individual expert with a smaller language model (Vicuna-7B in practice) using weexx-token--rreiieee process consists of three steps: 1) training each pre-trained vision expert with their own projector, while keeping the language model frozen; 2) combining all vision experts from the first step and training both the projector and vision experts; 3) training the whole model on SFT data.
supervision. As shown in Fig. 3. with pre-ali pre-alignment, the whole training
To verify the proposed method, we compare the pre- alignment strategy with the normal two-stage training strategy Table 5, considering both freezing and un-
Table 5: The effectiveness of Pre-alignment.
CLIP
Vision Expert (X) Unfreeze Pre-align Avg.
freezing vision experts for comparison. As shown in Table 5, although unfreezing the vision experts dur- ing SFT helps improve performance by updating the
X
x
630.6
X
/
648.5
CLIP-448
SAM-1024
V
X
662.5
vision experts to fit the language model, the Pre-Align strategy more effectively mitigates the inherent biases of each vision expert and stabilizes the training pro- cess, subsequently improving overall performance.
/
V
672.3
X
X
652.0
X
V
670.1
CLIP-448
ComNeex-1024
X
68 6.5
V
V
686.2
X
X
653.5
2.6 EXTENSION TO MULTI-EXPERTS
665.7
CLIP-448 Pix2Struct1102
X
673.7
With the optimized strategies and training recipes of incorporating individual vision experts, we con- sider the incorporation of even more vision experts push the limit. To conduct the search in systematic manner, we adopt step-by-step greedy strategy to incorporate additional vision experts.
V
/
680.4
X
630.2
X
V
645.2
CLIP-448 EVA-02-L-1024
V
X
659,2
/
668.2
We consider the vision experts discussed in Section 2.3 for experiments. We mark CLIP, ConvNeXt, SAM, DINOv2, Pix2Struct, and EVA-02-L as CL, CN, SA, DI, PS, and EV, respectively. A round-robin scheme, as shown in Fig. 4. is adopted. We first use the two top- performing vision encoders, CLIP and ConvNeX1, as the basis and gradually add one more vision encoder each time. In each round, the best-performing vision encoder combination retained for the next round.
Fig. 4 reveals several insights. Generally, introducing additional vision encoders enhances the performance. This indicates that the distinct advantages of different encoders can be preserved and utilized; for example, integrating the EVA-02 encoder improves metrics on the POPE benchmark. Although individual metrics may vary, the aggregated performance shows an upward trend, as evidenced by normalized average metrics, suggesting that the overall efficacy of the system is enhanced with more encoders. Also, Fig. shows that the best combination of vision experts are CLIP, ConyNeXt, SAM, Pix2Struct, and EVA-02. We will use this recipe in the final model.

---

Published as a conference paper at ICLR 2025
Mincoder
Encoder Combination
Config #Param (M)
FLORs ImpiSe
Avg.
2
CV
X2
1155.2
3347.2
18.1
681.5
0011
CL+
148016
3659.9
15.1
685.
666
604
C1+
690
6907
C1+ CN
1669.6
4313.2
6.9
68511
CV+
865
C7++ PV
1765.
4591.6
8.3
6880
1768.4
689.
CV P/P
X4
1974.1
5306.9
5,0
694.6
675
C)++VV RV 45
2270.5
$619.5
4.7
684.7
6702
CL+ PS
Enender
CV SA 6
X6
2588.2
6930.1
3.6
6863
666
Figure 4: Results of vision expert selection process. CL, CN, EV, PS, SA and DI denote CLIP, Con- WNeXt, EVA-02, Pix2Struct, SAM and DINOv2, respectively. (Left) The performance of various vision encoder combinations is presented, highlighting how different pairings influence overall effectiveness. "#fParames", "FLOPs" and "Img/Sec" denote the model size, complexity and throughput (bs=4) of the vision encoder. (Right) The curve illustrates the average score as the number of vision encoders increases. Each point on the curve represents the best-performing combination for the corresponding number of vision encoders. No Pre-Alignment is used in this comparison.
3 EXPERIMENTS
In this section, we take the findings and the best-explored design from Section 2 and compare them against the current state-of-thes thettrrrt MLLMs different tasks.
3.1 IMPLEMENTATION DETAILS
Language models. We use Vicuna-v1.5-7B (Chiang et al., 2023), Llama3-8B (AI@Meta, 2024) and Vicuna-v1.5-133 (Chiang et al., 2023) as the language models.
Vision encoders. We follow the best X4 and X5 configurations, where the interpolated CLIP-448 and pre-aligned vision experts are channel-comcatenated, and trained following the exact best training recipes Figure and Table 5.
Training recipe. On Eagle1.8M, we follow the base recipe in Section 2.1 with encoder learning rate the same as SFT (2e-5). On Cambrian data, we follow Tong et al. (2024) with PT/SFT bs=1024.
3.2 MAIN RESULTS
Evaluation visual question answering tasks. We compare Eagle model series across three Visual Question Answering (VQA) benchmarks, including GQA (Hudson & Manning, 2019), VQAv2 (Goyal all.,2217 and VizWiz (Gurari et al., 2018). As shown in Table 6, Eagle-X5 achieves state-of-the-art performance on GQA and VQAv2, underscoring the advantages with additional vision experts.
Evaluation on OCR and chart understanding tasks. To evaluate the OCR, document, and chart understanding capabilities of Eagle, we benchmark our model on OCRBench (Liu et al., 2023f), TextVQA (Singh et al., 2019), and ChartQA (Masry et al., 2022). As illustrated in Table our model significantly surpasses competitors on TextVQA, benefiting from its high- resolutsoo architecture and integration of different vision encoders. Notably, Eagle maintains straightforward design, supporting up to 1024x 1024 resolution without requiring complex tile decomposition of images.
Fig. shows some examples of OCR and document understanding cases. With high-resolution adaptation and more vision experts, our model can identify small text within images and accurately extract information according to the users' instructions. To better understand the benefits intro- ducing experts pre-trained other vision tasks, visualize the results of model with only the ConvNeXt and CLIP vision encoders, compared to the results Eagle-X5 X5 in Fig. 5. With the full set of vision encoders, the model can successfully correct mistakes, showing that even when equipped with high- high-resolttion vision encoders pre-trained on language alignment, the model's abilities still be enhanced by integrating additional vision experts pre-trained on diverse vision tasks.
7

---

Published as a conference paper at ICLR 2025
Table 6: Main results with base training data. SQA! denotes image split of ScienceQA.
MME
MMB
SEED
MathVista
MMMU
DOPE
SQA!
GOA
VieWiz
VQA2
OCR
TexVVQ
CharqQ
Model
Vicuna- 7B & Quven- 7B
LLaVA-I.5 (Liu et al., 2023c)
1510
64.3
85.9
66.8
62.0*
50.0
78.5
297
58.2
MaVA-NeXr (Liu at, 2024a)
1519
67.4
70.2
34.6
35.8
86.5
70.1
64.2*
57.6
80.0
490
64.9
InterVVL (Chen et al., 2023l)
1525
65.4
86.4
62.9*
52.5
79.3*
57.0
LLAVA-HR (Luo et al., 2024)
1554
64.2
87.6
65.1
64,2*
48.7
81,9*
67.1
Monkey
2024c)
60.7+
61.2%
80.3*
67.6
Mini-Gemini (Li et al., 2024b)
1523
65.8
32.2
36.8
71.1
64.5*
65.2
Eagle-X5
1528
68.4
73.9
37.0
36.3
88.8
70.0
64,9'
54.4
83.4
529
71.2
67.7
Eagle XS (+Pre-Align)
1582
69.7
73.7
38.2
38.0
88.7
71.9
64.6*
58.7
83.6
566
71.9
69.3
LLaVA-155 (Liu al., 2023c)
1531
67.7
36.4
85.9
71.6
63.3*
53.6
80.0*
331
61.3
MaVA-NeXi (1.iu at, 2024a)
1575
70.0
71.9
35.3
36.2
86.2
73.5
65.4
60.5
82.8*
514
67.1
62.2
InterVVI. et al., 2023/)
1546
87.1
63.9*
54.6
80.2*
517
58.7
Vicuna- 138
MaVA-UHD (Xu al., 2024)
1535
68.0
89.1
72.0
65.2*
56.1
81.7*
67.7
LL LLVA--HR (Luo al,, 2024)
1540
87.8
68.1
64.8*
57.9
82.6*
68.1
Mini-Gemini (Li et al., 2024b)
1565
68.6
70.6
37.0
37.3
71.9
65.8*
466
65.9
56.6
Eagle-X5
1609
69.2
74.1
38.8
36.6
87.8
72.7
66.2*
59.3
83.8*
574
74.2
69.9
E5gle-X5 (4 (ree-Align)
1605
71.6
74.9
42,7
38.5
89.2
75.5
64,6*
60.9
84,5
598
73.3
72.1
Evaluation on multimodal benchmarks. We evaluate Eagle on seven benchmarks for MLLMs to demonstrate its capabilities from different perspectives, including MME (Fu et al., 2023), MM- Bench (Liu et al., 2023e), SEED (Li et al., 2023c), Math (Lu et al., 2024), MMMU (Yue et al., 2024), ScienceQA (Saikh et al., 2022), and POPE (Li et al., 2023e). Specifically, MME, MMBench, and SEED assess the overall performance on various real- world tasks based on reasoning, recognition, knowledge, and OCR. MMMU focuses challenging problems from diverse domains that require college-level knowledge. POPE evaluates the visual hallucinations of MI.MMs. The metrics used in our paper adhere to the default settings of these benchmarks. We report the perception score for MME, the en_dev split for MMBench, the i image split of SEED, the test-mini split of Math Vista, the val split of MMMU, the F1-score of POPE, and the image split of SQA to align with the reported scores from other models.
From the data presented in Ta- ble 6, Eagle consistently sur- passes existing models across various MILMs on SEED and MME, demonstrating the com- prehensive knowledge and rea- soning abilities of Eagle. With the help of vision encoders object-centric tasks, Eagle also achieves the best performance the POPE benchmark. Addition- ally, the Pre-Aligument strategy discussed in Sec. 2.5 has been found to further enhance perfor- mance when integrating multiple task-specific vision backbones. This approach not only mitigates the inherent biases of each vision expert and the synergy between different modalities but also establishes a robust framework for multiple-expert fusion.
Table 7: Comparison between different training strategies. "I epoch" means train Eagle for I epoch in the supervised fine- tuning stage. "unlock*" means we unlock vision encoders in the pre-training stage. More details in Table. 13.
Config Summary
Pre-ulign
Pre-train
Fine-tune
Avg.
epoch
LLAVA-595K
Eagle1.M
697.1
epoch
LLaVA-595K
Eagle18M
698.3
spoch, unlock
x
LLaVA-595K
Eagle18M
698.0
epoch, unlock?
LLaVA-595K+Eagk11 8M
Eagle1.8M
699.5
epach
EaglcI.SM
LLaVA-595K
Eagle18M
epoch, unlock+
Eagle1 LMM
LLaVA-595K
EagleI8M
707.1
epoch, unlock*
LLaVA.555K-Eaglee...
LL5V
Eagle1.M
707.8
epoch, unlock+
LL.VA LLaVA.
LLaVA.955K+Euggel...
EagleI8M
708.9
dy-eby experiments to study the training recipes. We found that the best recipe is to first pre-align each vision expert on LLaVA-595K + Eagle/ .8M. In the pretraining stage, combine all vision experts from the first step and training both the projector and vision experts on LL5VA-595K + Eaglel. R&M. Finally, we train the whole model the E.gleI.8M.
Study more advanced training recipes. Table. 7 presents
Comparison with Cambrian-l. Using the same pre-training and supervised fine-tuning datasets from Cambrian-I (Tong et al., 2024), Eagle demonstrates superior performance across all the evaluated
8

---

Published as a conference paper at ICLR 2025
Table 8: Results using the same training data as Cambrian-I (Tong et al., 2024). SQA' denotes ScienceQA-IMG (Saikh et al., 2022). RWQA denotes the RealworldQA (xAI, 2024).
Model
Knowledge
General
OCR and Chart
Vision-Centris
MMMU
MathVista
ChanQA
TexVQA
DoeVQA
MMVP
RWQA
ANe SQA'
A12D
Avg
MME
MMB
SEED
GQA
Avg
OCR
Avg
Llama3
MGM-HD
55.7
75.1
37.3
37.0
73.5
72.7
1606
72.7
73.2
64.5
62.9
59.1
47.7
70.2
74.6
40.4
18.7
62.1
Cambrian-1
61.3
80.4
42.7
49.0
73.0
73.1
1547
75.9
74.7
64.6
71.3
73.3
62.4
71.7
77.8
57.6
51.3
64.2
Eagle-X5
65.2
84.1
43.5
56.9
76.2
74.0
1587
75.5
76.5
64.9
77.0
80.7
62.6
76.7
87.1
59.6
52.0
67.2
Vicana-138
MGM-HD
54.1
71.9
37.3
37.0
70.1
70.7
1597
68.6
70.6
63.7
60.8
56.6
46.6
70.2
69.8
38.4
19.3
57.5
Cumbrian--
60.2
79.3
48.0
73.6
73.7
1610
75.7
64.3
71.3
73.8
72.8
76.8
52.2
63.0
Eagle-X5
63.8
82.6
42.2
54.6
73.8
74.6
1651
75.7
75.0
65.0
75.7
78.6
62.4
74.9
86.7
54.8
44.6
65.0
Yi-34B
MGM HD
62.4
77.7
48.0
43.4
80.5
76.2
16599
80.6
75.3
65.8
68.1
67.6
51.8
74.1
78.9
52.3
37.3
67.2
Combrian-1
67.0
85,6
49.7
53.2
79.7
76.8
1689
81.4
75.3
65.8
71.9
75.6
60.0
76.7
75.5
60.3
52.7
67.8
X5
68.6
85.5
53.2
57.9
79.1
76.3
1677
81.0
75.6
64.9
75.4
77.2
62.4
78.8
83.0
59.8
50.0
69.5
benchmarks without bells and whistles. As shown in Table 8, Eagle outperforms the Cambrian- I counterparts considerably for the OCR and Chart category. Consistent improvements are also observed for the General, Knowledge, and Vision- categories, showing the robustness and generalization ability of the improved perception design in Eagle.
RELATED WORK
Multimodal large language models. Our work is related to the general architecture design of multimodal large language models. Besides the line of representative open-source research mentioned in the introduction section, other notable families of MLLMs include, but are not limited to MiniGPT- 4 (Zhu et al., 2023; Chen al., 2023a), Lynx (Zeng et al., 2023), Otter (Li et al., 2023b;a), Qwen- VL (Bai et al., 2023), CogVLM (Wang et al., 2023b; Hong et al., 2024), VILA (Lin et al., 2023a), GPT-4V (Achiam et al., 2023), Gemini (Team et al., 2023), and Llama 3.1 (Dubey et al., 2024). Depending on how vision signals are integrated into the language model, MILMSs can be broadly categorized into "cross-modal attention" ones and "prefix-tuning"" ones (Yin al., 2024). The former injects visual information into different layers of L.Ms using mooa attention (Alayrac et al., 2022; Li et al., 2023b), whereas the latter views the visual tokens as part of the language token sequence and directly append them with text embeddings (Liu et al., 2023d; Li et al., 2022; 2023d), Our model belongs to the prefix-tuni tunnng family by following a LLa VA-styled multimodal architecture. Considering that MILM is a mast-grooiig field, readers are recommended to refer to more detailed studies and surveys such as Yin et al. (2024); Li et al. (2024a); Fei et al. (2024).
Vision encoder design for MLLMs. Our work is closely related research focusing on improved vision encoder designs for MILMM. Early works (Liu et al., 2023d; Li et al., 2022) usually adopted vision encoder pre-trained on vision-language alignment tasks such as CLIP (Radford et al., 2021) and EVA-CLIP (Sun et al., 2023). Stronger vision encoders such as SigL/P (Zhai et al., 2023) and InternVL (Chen et al., 2023f) have been proposed to improve vision-language tasks with improved designs, larger model sizes, and better training recipes. Considering that the models are often pre- trained on low-resolution images and lack the ability to encode fine- ane-grained details, higher resolution adaptation is often performed to increase the MILM input resolution (Chen et al., 2023c;d; Beyer et al., 2024; Chen et al., 2024).
Besides higher resolution adaptation, models such LLaVA-NeXT (Liu et al., 2024a), LLaVA- UHD (Xu et al., 2024), Monkey (Li et al., 2024c), InternLM-XComposs (Dong et al., 2024), and InternVL (Chen et al., 2024) use adaptive tiling to divide high-resolution input into lower resolution patches that are processed separately. Although the ability to handle higher resolution is similarly made possible with the introduction of additional vision experts, its spirit is slightly orthogonal to the tiling techniques in terms of diversifying the visual representations. Both techniques are compatible and can be combined together.

---

Published as a conference paper at ICLR 2025
Effectiveness of more vision experts
User: Tall me the locations of the number 10 jerseys in the sentences
ANDHIK
picture Answer quastions
Tolerii
CLIP ConuNeXt baseline; there are two number 10 jerseys in the picture One is on the side
and the other is on the right vide
center players wezring jerseys with the number 20 One other pontioned towords right side the group, Both players are part of the bosketholl trom, as indieated by thek uniforms and the context of the image.
Eagle XS: the image, there of them standing
grous,
User: What's opening time restaurant Sunday?
User: What's the value the row "low" and column "referents" ??
CLIP ConuNelt baseline:
11:30-2:00
CLIP a Conffeettt baseline: 2012-13 Eagle-XS: 10,970 (60N)
Eagle (more experts]: riowed
Qualitative Examples
Eleutrode reuction
Standard electride pocential
(V)
Asl 3 An
+1.430
Oel 42i Ca
+0.340
What's the total Fat size (Mt Sinal) in the -05 year?
Nil 2eeeoo-N NI
-0.250
PV" +2"
-0.440
Eagle: $35,804
Na' +- Nu
-2.94
User:
(eelow are shown, for five standlerd electrode potential valces. Which of these metals is the lorst reactive? explain your answers distal.
reduction reactions and
A
User: was committeee strength for the first meeting? Eagle:
Option: (AJAu (8) Cu (c) NiD) Fe (2) Na
omong the given options Explonation: The standard electrade potentoll (t") is measue of the tendency of metii to lase electron and undergo redurtion reaction. The higher the 5n value, the more rooctise the metal is the
Eagle: least reactive
the standard electrode potential values (iy
follows.
N--0 (1) Pec- Comporing these salues, we see the metal with the nighest F'valoe (+1,420 V) Au, which indicates that It is the least reactive among the glven options The other metoti have lower E" values, making them more reactive than
(AJaur
Figure 5: Qualitative comparison of different numbers of vision experts. Baseline means Eagle with only CLIP+ConvNeNt. More Vision Experts denotes the Eagle-X5 model. We highlight a failure case in RED. BLUE indicates the correct answers. With more vision experts, Eagle can more precisely capture the information in the image and provide the correct answer.
Our work is most related to existing models using multiple vision encoders for improved perception. (Li et al., 2024b) and LLaVA-HHE (Luo et al., 2024) propose to fuse high-resolution visual features into the low-resolution visual tokens. Apart from the resolution issue, these pre-trained vision encoders may lack specific abilities such as reading text and localizing objects. Hence, series of works have integrated vision models pre-trained on different tasks for more comprehensive capabilities. For example, Mousi (Fan al., 2024), Prismatic VI.M (Karamcheti al., 2024) and Brave (Kar et al., 2024) fuse visual tokens from different vision encoders by concatenating along the channel or token direction. There are also more complex approaches, including knowledge distillation Ranzinger et al. (2024), augmenting the input prompt with the information output by the vision experts (Lee et al., 2024; He et al., 2024; Liu al., 2024b), or using a routing network assign input to proper vision experts (Zong et al., 2024). In particular, Prismatic VLM (Karamcheti et al., 2024) systematically explores the design space of MLLMS across various dimensions, including data, training recipes, and notably. It also includes the vision encoder ensemble part of its enhancement strategics. However, it lacks a comprehensive ablation study and discussion addressing the challenges associated with combining multiple vision encoders.
5 CONCLUSION
We conduct an in-depth analysis study on the design space for integrating vision encoders for multimodal large language models. Unlike previous works that focus on designing novel fusion paradigms, we find systematic design choice matters and discover a series of useful techniques. Step by step, we optimize the training recipes of individual vision encoders. identify an extendable and efficient fusion method, and gradually combine vision encoders with different domain knowledge.
The results show the importance of basic design space. We hope our work can serve as a new basis and bring new inspiration for the vision encoder design for MI..M.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENTS
The team would like to thank Yunhao Fang and Jason Lu for sharing the data and training recipes, and Yawen Luo for the assistance on figure editing. We thank Wei Ping, Zhuolin Yang. Wenliang Dai, Nayeon Lee. Boxin Wang, Ilia Karmanov, Lukas Voegtle, Philipp Fischer, Matthieu Le and Tuomas Rintamaki for their assistance and input from Zhiqi Li, Guo Chen, Shilong Liu. Jihao Liu, Ming- Saan-Chang Chiu, Yunze Man, Shiyi Lan, Nadine Chang, Maying Shen, Vibashan VS, Jenny Schmalfuss, Jose Alvarez, Amala Sanjay Deshmukh, Mike Ranzinger, Greg Heinrich, Pavlo Molchanov, Vidya Murali, Parthasarathy Sriram, Mohammad Shoeybi, Song Han, Ofri Masad, Osvald Nitski, Qing Miao, Yao Xu, Jane Scoweroft, Dmitry Chichkov and Padmavathy Subramanian. Finally, the team would like to thank the Hugging Face Team that for the support of ZERO GPU demo, and the NVIDIA infrastructure team for their prompt and helpful assistance. Min Shi is partly supported by NSF Award #2427478 and #2229873.
the internal codebases. We also thank the valuable discussions
REFERENCES
LAION-GPT4< dataset. http:://wwgugnnaoc...coooaattstt
2023. 3 Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv:2303.08774. 2023. 9
co
AI@ Meta. Llama 3 model card, 2024. URL https://github.. cow www.co/t//mtcteec.cc 11ama 1lama3/bloblb Biin/MODEL_CARD md. 7
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurlPS, 2022. 2,9
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VLe VL: frontier large vision-language model with versatile abilities. arXiv:2308. 12966, 2023. 9
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang. Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv:2407.077230 2024.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. MiniGPT-v2: Large language model as a unified interface for vision langual multi-task learning. 1rXiv:2310.09478, 2023a. 9
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. ShareGPT4V: Improving large multi-modal models with better captions. arXiv:2311.12793, 2023b. 3
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. PaLI-X: On scaling up a multilingual vision and language model. arXiv:2305. /8565, 2023c. 4,9
Xi Chen, Xiao Wang. Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. PaL1-3 vision language models: Smaller, faster, stronger. arXiv:2310.09199, 2023d. 4,
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pal.: A jointly-scaled multilingual language-image model. ICLR, 2023e.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. InterVV:: Scal- ing up vision foundation models and aligning for generic visual- -ingusiin tasks. Xxxx:23322.14223 2023f. 2, 8,
11

---

Published as a conference paper at ICLR 2025
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong. Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv:2404. 16821, 2024. 9
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. CVPR, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* $ chatgpt quality. https://lmst org/blog/ 2023-03-38 30---33--vvnaaa/-- March 2023. 3,
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale N Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision- language models with instruction tuning. In NeurIPS, 2024. 2
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. M-xoommMXXoommrrre2ee large vision-language model handling resolutions from 336 pixels to 4k hd. arXiv:2404.06512, 2024. 9
A pioneering
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM-E: An embodied multimodal language model. arXiv:2303.03378, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama herd of models. arXiv:2407.2178. 2024. 9
Xiaoran Fan, Tao Ji, Changhao Jiang, Shuo Li, Senjie Jin, Sirui Song, Junke Wang, Boyang Hong, Lu Chen, Guodong Zheng, et al. MouSi: eely-- vision-language models. arXiv:2401.17221, 2024. 2, 10
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EVA-02: A visual representation for neon genesis. arXiv:2303, 200.23333.133,, 2023a. 2,
Yuxin Fang, Wen Wang. Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EVA: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023b. 2,
Hao Fei, Yuan Yao, Zhuosheng Zhang, Fuxiao Liu, Ao Zhang, and Tat-Seng Chua. From multimodal I1m to human-leved ai: Modality, instruction, reasoning, efficiency and beyond. In LREC-Coling Tutorials, 2024. 1,9 9
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulci Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang. Xiawu Zheng, al. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv:2306. 13394, 2023. 3,8 8
Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye. Wanjun Zhong, Yufei Wang, Langing Hong, Jianhua Han, Hang Xu, Zhenguo Li, and Lingpeng Kong. G-llava: Solving geometric problem with multi-modal large language model. 1.. 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Stur Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Flevating the role of image understanding in Visual Question Answering. In CVPR, 2017.7 7
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, 2018. 3.7
Xin He, Longhui Wei, Lingxi Xie, and Qi Tian. Incorporating visual experts to resolve the information loss in multimodal large language models. arXiv:2401.03/00 2024. 10
12

---

Published as a conference paper at ICLR 2025
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. CogAgent: A visual language model for gui agents. In CVPR, 2024.
Drew Hudson and Christopher D Manning. GQA: new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 3, 7
Gabriel Iharco,, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL htttps://dol org/10. 5281/zenodo. 5143773. If you use this software, please cite it as below. 2,
Kushal Kafle, Scott Cohen, Brian Price, and Christopher Kanan. DVQA: Understanding data visualizations via question answering. In CVPR, 2018. 3
Oguzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. BRAVE: Broadening the visual encoding of vision-language models. arXiv:2404.07204, 2024. 2, 5, 10
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vims: Investigating the design space of tisually-onndttioned language models.
arXiv preprint arXiv:2402.07866, 2024. 2, 5,
Aniruddha Kembhavi, Michael Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. 1rXiv:1603.07396 2016a. 3
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min Joon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision EBsV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pp. 235-251, 2016b. 3
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeong Yeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun. Dongyoon Han, and Seunghyun Park. OCR-Free document understanding transformer. In ECCV, 2022. 3
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yes Yen Lo, et al. Segment anything. In ICCV, 2023. 2, 4
Byung- Kwan Lee, Beomchan Park, Chae Won Kim, and Yong Man Ro. MoAl: Mixture of all intelligence for large language and vision models. 1:Xi::2403.07500 2024. 2, 10
Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisensch- los, Urvashi Khandelwal, Peter Shaw, Ming-We Wei Chang, and Kristina Toutanova. Pix2Struct: Screenshot parsing as pretraining for visual language understanding. In ICML, 2023. 2, 4
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwci Liu. MIMIC-IT: Multi-modal in-context instruction tuning. arXiv:2306. 05425, 2023a.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Jingkang Yang. and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. 20x5:22305.03726, 2023b. 9
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge. Yixiao Ge, and Ying Shan. Seed-Bench Bench- marking multimodal Iims with generative comprehension. arXiv:2307.1612. 2023c. 8
Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao, et al. Multimodal foundation models: From specialists to general-purposs assistants. Foundations and Trends@ in Computer Graphics and Vision, 2024a. 9
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified lisiuaelanguage understanding and generation. In ICML, 2022. 2, 9
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping aanguage-imagge pre-training with frozen image encoders and large language models. In ICML, 2023d. 2,
13

---

Published as a conference paper at ICLR 2025
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong. Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-Gemini: Mining the potential multi-modality vision language models. arXiv:2403.18814, 2024b. 2,5, 8, 10
Yifan Li, Yifan Du, Kun Zhou, Jimpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. 23Xiv:2305..033: 2023e. 3, 8
Zhang Li. Biao Yang. Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In CVPR, 2024c. 2, 2, 4,
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Moham- mad Shoeybi, and Song Han. VII.A: On pre-training for visual language models. arXiv:23 2023a. 9
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xíao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. SPHINX: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv:2311.07575 2023b. 2,5
Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. HallusionBench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), 1lava-1.5, and other multi-modality models. arXiv:2310. 14566, 2023a.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv:23066. 14565, 2023b. 3
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. arXiv:2310.03744, 2023c. 2, 6,
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NewrIPS, 2023d. 2, 3,9
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaVA-NeXT: Improved reasoning, ocr, and world knowledge. https://llava-v vl. g..git thubthu... io/ blog/2025 2004-01222----1 00-30- lllval next/, January 2024a. 2, 4, 8,
Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandkumar. Prismer:
A vision-language model with an ensemble of experts. TMLR, 2024b. 2, 10
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li. Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is your multi-modal model an all-around player? urXiv:2307.0628) 2023e. 8
Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. arXiv:2305.07895, 2023f. 3, 5,7
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. In CVPR, 2022. 4
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista. Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, 2024. 8
Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: cfxture-of-resolutii adaptation for multimodal large language models. 00xh0:2403.03003, 2024. 2, 5, 8, 10
Ahmed Masry, Do Xuan Long. Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. arXiv:2203.10244 2022. 3,
14

---

Published as a conference paper at ICLR 2025
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. DocVQA: A dataset for vqa document images. In WACV, 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin Bl-Nouby, al. DINOv2: Learning robust visual features without supervision. arXiv:2304.07193, 2023. 2,
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In ICML, 2021. 2, 3, 9,9
Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. AM-RADIO: Agglomerative vision foundation model reduce all domains into one. In CVPR, 2024. 2, 10
Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. ScienceQA: A novel resource for question answering on scholarly articles. International Journal Digital Libraries, 2022. 3, 8,
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. In NeurlPS Datasets and Benchmarks Track, 2022. URL htttps://ppenreieet net/forummid- M3Y74vmsMcY.. 2, 4
Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do we not need larger vision models? arXiv:2403. /3043, 2024. 2
Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, 2016.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In CVPR, 2019. 5,
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: Improved training techniques for clip at scale. arXiv:2303. 15389,, 2023. 9
Gemini Team, Rohan Anil, Sehastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv:2312.1 1805, 2023.
Teknium. OpenHermes 2.5: An open dataset of synthetic data for generalist LLM assistants. https::/huggingff.. httpe//nnnacc..occtttteeeeeeeeee wwt.tttttts//aaeo///oooo.o http///aaeetseenninn/eeeeeeeeeee www.eiimeppeeeereeeeeee 2.5, 2023.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-I: A fully open, vision-centric exploration of multimodal 1lms. 1xXiv:2406.1686 2024. 2. 7. 8.9
Junke Wang, Lingcchn Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu- Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv:2311.07574 2023a. 3
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang. Lei Zhao, Xixuan Song. al. CogVLM: Visual expert for pretrained language models. arXi::2311.03079 2023b. 9
Xiyang WU, Ruiqi Xian, Tianrui Guan, Jing Liang, Souradip Chakraborty, Fuxiao Liu, Brian Sadler, Dinesh Manocha, and Amrit Singh Bedi. On the safety concerns of deploying IIms/vlms in roboties: Highlighting the risks and vulnerabilities. arXiv preprint arXiv:2402. 10340, 2024." 2
xAI. Grok- 1.5 Vision Preview. https:
1.5v, 2024. 9, 17
XAI. Grok-1.5v. https://x.ai//llgg/ 2024. Accessed: 2024-09-28.
2024.
15

---

Published as a conference paper at ICLR 2025
Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. LLaVA-UHD: an Imm perceiving any aspect ratio and high-resolution images. arXiv:2403.11703, 2024. 8,9
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen, A survey on multimodal large language models. IEEE Trans. PAMI, 2024. 9
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In CVPR, 2024. 8
Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, and Tao Kong. What matters training gpt4-style language model with multimodal inputs? arXiv:2307.02469, 2023. 9
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023. 9
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. LAR: Enhanced visual instruction tuning for text-rich image understanding. arXiv:2306. /7107, 2023. 3
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv:2304. 10592, 2023.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: Deformable transformers for end-to-end object detection. In ICLR, 2021. 5
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7W: Grounded Question Answering in Images. In CVPR, 2016.
Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and Yu Liu. MoVA: Adapting mixture vision experts to multimodal context. arXiv:2404. 13046, 2024. 2, 10
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
A. BENCHMARK DETAILS
In this section, we provide the additional benchmark details for the tables Section 2. The detailed comparison of different adaptation methods for the CLIP encoder are shown in Table 9. Table 10 shows the comparison between different vision encoders on all the adopted benchmarks. Table 11 list the detailed results on the vision encoder fusion methods. Table 12 shows the comparison between different vision encoder combinations on each benchmark.
Table 9: Comparison of different high-resolution adaption methods to strengthen CLIP model (336x336), RWQA denotes the RealworldQA (xAI, 2024).
Vision Encoder
Uniieoze
Res
6QA
VizWiz
MME
SEED
OCR
DocVQA
ChanQA
AI2D
POPE
RWQA
SQA
Avg.
Original
336
63.2
55.1
1574
70.2
354
59.6
42.1
71.1
86.7
58.0
72.5
616.5
Original
336
56.4
562.6
Interpolate
X
448
62.7
53.8
1470
69.1
299
58.4
37.7
71.2
85.7
55.3
69.6
589.7
Interpolate
V
448
65.6
57.8
1534
73.7
526
65.0
61.1
73.7
87.3
$7.7
71.5
670.5
Interpolate
672
61.9
55.7
1503
7244
509
61.6
62.0
72.2
87.1
71.2
671.2
Tiled--ngut
672
hnsmVV
448
63.6
56.9
1537
71.7
529
65.0
58.6
72.9
81.4
59.2
70.2
661.9
Table 10: Comparison between different vision experts as the MILM encoders.
Category
Vision Faceder
Unfreeze
Ros
GQA
VizWiz
MMF
SERD
OCR
DocVQA
CharQA
AEDD
POPE
RWQA
SQA
Avg.
VT. Aliggmmnnt
1024
63.3
53.5
1526
70.6
404
70.4
60.8
71.6
87.5
57.1
68.6
635.0
Segmentation
1024
56.9
20.1
174
84.3
49.2
66.8
471.1
1024
60.2
51.5
1291
65.9
35
31.2
17.8
70.7
86.4
54,1
65.7
494.7
Objert Detection
EVA-02
1024
63.1
51.1
1359
69.2
123
25.6
25.2
71.2
88.5
57.9
66.1
523.5
1024
1449
72.7
57.1
57.5
72.2
88.3
59.6
67.7
614.4
Recognition
Pi22Strcct
1024
53.1
48.1
1296
53.4
460
71.0
61.0
69.6
79.2
46.7
65.5
578.7
1024
54.9
1262
55.1
472
72.5
62.0
68.7
80.0
49.3
66.6
$84.8
Self Supervized
DINOV2
448
64.2
55.5
1466
71.8
45
20.4
17.5
71.4
87.4
57.3
67.8
518.0
Table 11: Comparison of different fusion methods for difffernt vision experts. "#Tokens(V)" denotes the number of visual tokens.
Vision Encoders
Pusion
(Tokets(V)
GQA
VizWiz
MME
SEED
OCR
DocVQA
CharQA
A12D
POPE
RWQA
Avg.
Siy, Appond
2048
64.8
54.5
1563
73.4
77.7
67.6
72.4
87.9
61.2
68.8
590.5
Channel Concar.
63.2
73.5
71.7
72.4
CLIP + ComNeNi
LLaVA-HR
1024
64.5
57.2
1538
72.0
498
74.5
63.8
72.3
87.7
59.2
68.7
578.7
Deformable Ann.
1024
64.0
57.3
1504
72.7
463
69.5
64.4
75.3
87.4
62.8
68.9
674.3
ComNNeXt
Append
72.3
SAM
Channel Concor.
1024
63.3
55.9
1528
23.3
545
78,9
67.2
72.3
88.4
59.2
70.0
590.4
17

---

Published as a conference paper at ICLR 2025
Table 12: Detailed comparison on vision encoder combinations.
ABocoder
Encodar Combinatoon
CharQQ
AI2D
POPE
Aug.
CL,+
67.0
72.4
88.3
59.1
CL CN
63.3
55.9
1528.0
73.3
545.0
78.9
67.2
72.3
88.4
59.2
70.0
690.4
(1. CN SA
64.6
55.3
1504.0
73.3
526.0
75.7
64.9
72.1
88.3
61.1
70.9
685.4
CL+CN PS
63.2
51.4
1497.0
73.3
550.0
78.5
63.9
73.1
87.7
60.3
70.5
685.1
89.0
CL+ BV DI
1512.0
73.8
547.0
77.0
66.7
73.1
88.9
60.4
69.7
CL CN BV
64.3
1533.0
73.7
521.0
65.3
72.2
88.5
61.1
CL BV PS
64.8
56.5
1561.0
73.4
540.0
78.8
67.5
72.2
88.4
59.9
70.5
694.6
CL CN a BV SA
64.7
59.1
1528.0
73.9
529.0
78.6
67.8
72.9
88.8
62.2
69.7
697.1
CI, RV
64.7
54.1
1506.0
73.7
$41.0
75.1
64.9
72.7
88.3
60.0
70.3
684.7
CL CN BV PS
DI
63.8
57.8
1512.0
73.5
525.0
75.1
65.8
71.8
88.4
61.4
69.9
686.8
Table 13: Comparison between different training strategies. 'I epoch" means we train Eagle for epoch in the supervised fine-tuning stage. "unlock* means we unlock vision encoders in the pre-training stage.
DocVQA
CharQA
GQA
MME
OCR
SoQA
POPE
SEED
Viowiz
A12D
RWQA
Config
Prealign
Pretrain
Finetune
Avg.
1 epach
Blava595k
Eagle1.8M 64.7 1528
$2.9
69.7
88.8
78.6
67.7
73.9
59.1
72.8
62.2
697.1
2 spoch
x
Ilava595k
Eagle1.SM 65.4
1539
51.4
70.3
87.9
79.8
67.9
73.8
58.5
73.5
62.7
698.3
epoch, unlock
x
1lava595k
Fagle1.8M BMd 64.1 1541
54.4
71.5
88.5
79.1
68.5
74.0
5666
72.
61.9
698.0
1 epoch, unlock
x
1lva555kk-Eagle 8M
Eugle1.SM 65.3 1545
54.8
70.5
88.5
78.8
68.4
73.5
57.5
72.0
62.9
699.5
1 epoch
EugleI.SM
1aav5595k
EugleI.SM 64.5 1579
55.4
71.6
88.7
79.3
68.9
73.9
58.9
73.9
63.4
706.6
1 epoch, unlock
Eaglel.M
llav5595k
Eagle1.8M 65.1 1561
56.0
71.3
88.5
79.3
69.5
74,2
59
73.5
63.4
707.1
epoch, unlock"
1lava595k+ Eagle 8M
Nlav5595k
Eagle1 SM 64.5 1570
56.0
71.7
89
79.7
68.9
74.0
58.5
74.0
63.7
707.8
spoch, unlock"
Eavell95k-Enelel 8M Bawa599k= 8M Eagle1.8M 64.6 1582 56.6
71.9
88.7
79.8
69.3
73.7
58.7
73.5
63.9
708.9
18

---

Published as a conference paper at ICLR 2025
A.2 ADDITIONAL QUALITATIVE RESULTS
Fig. 6 demonstrates that our model can perform reasoning based on the knowledge to answer some difficult questions. Fig. 7 shows some examples of OCR and document understanding cases.
Input image:
Imput Image:
Weoneed sllaed printal
-6.200
What sinlanies alffvences betwees there tro pictore?
User: Scloe are aloam, for five octtal, sccletion rearioons and erscoeed elecmode pateenni unbuee Which of thene Option (Ajha Guy1 na (53 lte
Usen: the
Capl: The two picta un thow red pandain a nature settigg resting en thee comet. The simtarttes between the images
he ouqpaed paar Suavio paloes of the padd
willigioes
Saglc reuciie umoog the piicn aptino tendency
lour eechose underga reduction wetttes
naturd ronment.
The de the poalin pedio and ong
ponels
the ruulitue consoecation the puiciig repereets
183
(1) the duay of Juy Phlp ln duaiingig Nis moscier
in flannd rabhaling dane
figher dee wabes lhe mosec vouthee tie mesul
(MM 4ht Covil 340 cleaveee
grers
200 dett 20000/
seatin The fing (boto cag uns the and pands more riburd
red pardle
the brands, looking skitt
where the pertato the royer fendy would nowable by
vactig dipedemnc
Comporing these
we diex mesu with die lest "wite
Eagle: ICI
Philip chellenpinghh
(4,4200 of Au which sufoors shor the
contidrr the mutrinud onee presstions oon painting
rearesves
making there recative
Input Image:
Usen Miic is she pietee & more aerodynevid ? And exgliale your ansoerr in cirtal.
Impat Imragee
the inshlig?
tagle the pictare thet aurocimarii
deunigg, Thi
Garone
(4) atAal
socasiclt in has stoumrd and shaps with nouth and caroed body tuat nalucis ereana stope sloae the car move ettieetly througe other hand has arore restangglle shaps,
(00 Aurteaen
(O Clarevaggee (0)
more slunoound
Eaple
Tode best bertyoomi
Figure 6: Qualitative results of Eagle on VQA tasks requiring college-level knowledge or deliberate reasoning.
Input Image:
égwre
welideess
Far the Madea hare
Eagle:
User: What was the committee stanggth for the first meetin??
1100s East en Eoterre
Eagle:
was the committee strength meeting?
Likens
Tomo
Eagit:
Input Image:
Input Image:
User: Descmbe chert
Eagle The image prrsece ber chart that prowider fernoe the compucin muiler sur tnclire years
evietr year ere iabaned on the e-ccl whit the markst tam represented the eali ailiau
ohe
dout
User: What's the totel cosr of Fat call aoe (et Sinaj) in the -25 year?
The
twes cemedly the
2002, which for pablic indlussing cipeifoaa graatt stis reitie The lownd coe cloud nos
Eagic $35,864
cload mavie.
Figure 7: Qualitative samples on OCR and document understanding tasks. Eagle is able to extract useful information from small text.
19


Reviews:
[{"review_id": "TIkhArhBav", "reviewer": "Reviewer_Y3dK", "summary": "This paper conducts a systematic study on the **mixture of vision encoders** in multimodal large language models (LLMs). Specifically, the paper thoroughly compares the performance differences between various combinations of vision encoders and fusion methods, ultimately finding that simply concatenating the tokens from different encoders outperforms more complex fusion strategies. To bridge the gap between different vision encoders, the authors propose a **pre-alignment** strategy, which enhances the subsequent collaboration between vision encoders by aligning them with a small LLM in advance. The proposed method demonstrates significant improvements across several metrics.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper provides a detailed and systematic study on the **mixture of vision encoders**, offering valuable insights to the research community.", "weaknesses": "1. **Limited novelty**:  \n   While the primary contribution of this paper lies in the extensive comparisons and ablation studies, the method itself lacks significant innovation. Technically, it does not introduce any groundbreaking insights for the community.\n\n2. **Flaws in Experimental Setup**:  \n   There are some flaws in the experimental setup. In the comparison shown in **Table 5**, the configuration using pre-alignment includes an additional **stage 1** of training compared to the configuration without pre-alignment, leading to an imbalance in the amount of training. It is recommended that the authors add a setup where **stage 1** is removed and **stage 3** is trained for an additional epoch to eliminate the unfair advantage of extra training rounds. Furthermore, since the vision encoders are pre-aligned in **stage 1**, why is it necessary to retrain in **stage 3**? What would happen if the vision encoders were simply frozen?\n\n3. **Computational and Memory Costs**:  \n   Since the proposed method involves mixing multiple different vision encoders, the computational and memory costs during training and inference are important factors to consider. When comparing with other methods, these indicators should be included. Additionally, it would be helpful to provide the computational cost and memory usage for each combination in **Table 6** for a more comprehensive evaluation of different vision encoder combinations. **Table 6** shows that the performance difference between five encoders and three encoders is minimal. If the cost is significantly higher, is it necessary to use the combination of five encoders?\n\n4. **Omission of a Key Model**:  \n   **SigCLIP** is one of the widely used vision models for MLLMs. Why is it not included as an option in **Table 2**?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "stNXKBKJwL", "reviewer": "Reviewer_La1k", "summary": "The authors present an extensive study of mixture of encoders, providing ablations and experiments on various design choices. The authors recommend interpolating for higher resolutions, having a higher number of vision encoder experts, concatenating tokens in a channel-wise manner, and also compare training strategies. The method is compared against a lot of recent advances in this space, and has competitive performance on a variety of benchmarks.", "questions": "Please see the weaknesses section above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well articulated, with various steps properly backed up with experiments, and compared against significant number of baselines. The results indicate that the approach performs better than other mixture of vision encoders works. \n- The better performance of interpolation over tiling in the initial steps is an important contribution towards efficiency, and is much appreciated\n- The paper addresses an important issue, with multiple strides in the space of multimodal LLMs, it is essential to incorporate the best set of changes that work well together, and this paper is a good step towards this goal.\n- Well written paper, easy to read", "weaknesses": "I will combine weaknesses and questions here:\n\n(1) My major primary concern is about the efficiency of the approach, and I will list it below as a two-part concern - \n\n- (**Most Imp Concern**) Fig 4 mentions the various combinations tried in the mixture of vision encoders. As compared to the 2 encoder setting (X2), the next mixture (X3) leads to a slight improvements at the expense of 50% reduction in throughput, and going further, X5 leads to more than 4x reductions in throughput. While the results in terms of accuracy improve, as also suggested by Table 6, it comes at a significant compute expense. It would be important to understand the author's perspective on this agenda. Additionally, in Table 6, it would be useful to mention the throughput of other baseline methods, wherever possible. In my humble opinion, if the proposed method has a severely lower throughput (high inference cost), it should also be compared against other methods having comparable throughput when comparing accuracy metrics.\n- The paper suggests that unfreezing leads to better performance, however, it also increases the number of trainable parameters. The authors should provide details on the training overhead that this change leads to, so as to better understand the efficacy of the slight improvement.\n\n(2) The motivation behind introducing the Deformable Attention baseline is unclear and the comparisons made with it are limited. Additionally, to motivate channel concatenation further over sequence concatenation, quantitative comparisons of the implications of handling longer sequence lengths would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7pWH5BoEEK", "reviewer": "Reviewer_1gkn", "summary": "This paper's findings reveal several underlying principles common to various existing strategies, leading to a streamlined yet effective design approach. The paper discovers that simply concatenating visual tokens from a set of complementary vision encoders is as effective as more complex mixing architectures or strategies. Additionally, the paper introduces Pre-Alignment to bridge the gap between vision-focused encoders and language tokens, enhancing model coherence. The resulting family of MLLMs, Eagle, surpasses other leading open-source models on major MLLM benchmarks.", "questions": "## Justification For Recommendation And Suggestions For Rebuttal：\n- Justification For Recommendation：Reference to Paper Strengths.\n\n## Additional Comments For Authors：\nIt would be better to add the results of some other models that also use many vision encoders for comparison in Table 6.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper has confirmed through comprehensive and integrated experiments the necessity of turning on vision encoders during the SFT stage. At the same time, the paper has discovered and validated that straightforward channel concatenation stands out as a simple yet competitive fusion strategy, offering the best efficiency and performance.\n2. This paper proposes a pre-alignment stage where non-text-aligned vision experts are individually fine-tuned with a frozen LLM before being trained together.\n3. The illustrations in this paper are accurate and concise, and the language expression is very refined, making it easy for people to understand.", "weaknesses": "1. Some details of the methods are missing. How are the projectors of each expert in stage 1 of Figure 3 dealt with in stage 2?\n2. Some corrections to details. In Table 4 at line 317 of the text, it looks like it should be referring to Figure 4. The same applies below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fY4xpxhp9F", "reviewer": "Reviewer_GjPo", "summary": "This paper conducts an empirical analysis study on the design space for integrating vision encoders for MLLMs. It optimizes the training recipes of individual vision encoders step by step, identify an extendable and efficient fusion method, and gradually combine vision encoders with different domain knowledge. Built upon these findings, this paper propose Eagle, the results somehow show the importance of basic design space.", "questions": "Please see the comments above.\nMy main concern is weakness 1, for such reasons i don't think this paper's contribution can lead to an acceptance.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strength：\n\n1. Extensive experiments demonstrate the importance of optimizing the design space of vision encoders.\n2. The ablation studies are abundant.\n3. This paper is well-organized and clearly-written.", "weaknesses": "Weakness:\n\n1. The contribution is limited. PrismaticVLM[1] conducts similar empirical study on MLLMs, especially on the vision encoders. Although they only explored the combination of two encoders, they have achieved significant improvements while keeping a good efficiency. This paper extends the mixture of encoders up to six, but the performance only increases from 681.5 to 697 as shown in Fig.4, which is somehow trivial from my perspective. What's worse, it brings double parameters and slows down the inference by more than four times. \n2. Missing related discussion. It is really strange there is not any discussion with PrismaticVLM, considering these two paper focus so closely.\n\n[1] Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models. ICML 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2NdUfwkQLN", "reviewer": "Reviewer_aaRq", "summary": "The paper explores strategies to inject information (visual tokens) from a mixture of pretrained visual encoders (experts) into LLM’s (Vicuna-12B) for finetuning MMLM’s. Namely, they explore how to incorporate: 1) High Resolution inputs which are needed for text understanding. 2) A number of visual encoders. Based on their ablation strategies, they propose the following training recipe.\n\n* Simple linear interpolation for adaptation to high resolution.\n* Unfreezing the image encoder instead of just the projection layer\n* Channel concatenation to incorporate multiple vision encoder outputs.\n* Prealigning the visual encoders with a LLM.\n* Simple greedy strategy to incorporate multiple vision experts.", "questions": "The clarity over some of the details in the paper can be expanded upon.\n\n* It’s a bit worrisome that the benchmark used to arrive at the recipe (Table 2-5 and Fig.4) is the same benchmark used in the final results (Table 6.). This would be great if authors confirm that the validation set is used for Table 2-5 and Fig.4 and the test-set for Table 6 or provide some evidence that the recipe is not overfitting to this particular benchmark.\n* There is a final finetuning stage where the entire model, i.e LLM + projector + vision encoder is finetuned. In Table 7, it is mentioned that this is done on Eagle 1.8M, however on L141-L142, it is mentioned that this is done on multi-model conversation data. Further clarification on this will be helpful.\n* Similarly, in Table 7 the best model is pretrained on LLava-595K + Eagle 1.8M. However, it seems Table 2-5 and Fig 4, the recipe is arrived at by only pretraining on Eagle 1.8M. Can the authors confirm this and clarify this in the paper?\n* What is the effect of unfreezing vs freezing the image encoder on training time? Table 2 shows the number of flops are the same for both freezing and unfreezing. But while unfreezing one has to backpropagate through the encoder , so the flops should be different.\n* What is the difference between 5th row of Table 5 (Unfreeze yes + prealign no + clip + convnext) and 2nd row of Table 4? In both cases, clip + convnext is used with channel concatenation and pre-alignment switched off. So why are the numbers different?\n* The empirical comparison of flops between tiling and interpolation is provided in Table 2. I assume there is a tradeoff in the sense that tiling requires more forward passes but the self-attention might be less expensive because of the lesser number of tokens . So It would also be nice if some theoretical comparisons are also provided between interpolation and tiling with respect to number of tokens and resolution.\n* I may have missed it but was the size of the Clip encoder L? And what resolution was the pretraining done on?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written. They start with a base clip image encoder model and build towards the final recipe using careful ablations. They show that a simple recipe without any fancy tricks can lead to strong performance on a number of benchmarks. The paper also achieves strong performance on VLM benchmarks and is likely to have an impact on multimodal LLMs.", "weaknesses": "The clarity over some of the details in the paper can be expanded upon. See below for detailed feedback.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)