Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

## Abstract

Abstract The ability to accurately interpret complex visual information is a crucial topic of multimodal large language models (MLLMs). Recent work indicates that enhanced visual perception significantly reduces hallucinations and improves performance on resolution-sensitive tasks, such as optical character recognition and document analysis. A number of recent MLLMs achieve this goal using a mixture of vision encoders. Despite their success, there is a lack of systematic comparisons and detailed ablation studies addressing critical aspects, such as expert selection and the integration of multiple vision experts. This study provides an extensive exploration of the design space for MLLMs using a mixture of vision encoders and resolutions. Our findings reveal several underlying principles common to various existing strategies, leading to a streamlined yet effective design approach. We discover that simply concatenating visual tokens from a set of complementary vision encoders is as effective as more complex mixing architectures or strategies. We additionally introduce Pre-Alignment to bridge the gap between vision-focused encoders and language tokens, enhancing model coherence. The resulting family of MLLMs, Eagle , surpasses other leading open-source models on major MLLM benchmarks.

## 1 Introduction

The success of large language models (LLMs) has triggered significant interest in enabling their visual perception capability, such that they could see, understand, and reason in the real world. At the center of these multimodal large language models (MLLMs) [] is a typical design where images are converted into a series of visual tokens by the vision encoders and appended with the text embeddings. CLIP [] is often chosen as the vision encoder since its visual representation is align with the text space by pre-training on image-text pairs. Depending on the architectures, training recipes and the way how vision tokens are injected into the language model, there exist various notable families of MLLMs such as Flamingo [], BLIP [, , ], PaLI [], PaLM-E [] and LLaVA [, ]. Most of these works keep relatively low input resolutions due to the limits on pre-trained vision encoders and LLM sequence length.

Recent studies [, ] show that stronger vision encoder design is important for mitigating MLLM hallucinations [, ] and improving resolution-sensitive tasks like optical character recognition (OCR). A constellation of works thus focuses on enhancing the capability of the vision encoder. For example, scaling up the pre-training data and parameters of vision encoder [] or dividing images into low-resolution patches [, ]. However, these approaches usually introduce large training resources. An efficient yet powerful strategy is to mixing visual encoders pre-trained with different tasks and input resolutions, either fusing higher resolution encoders with the CLIP encoder [, ], sequentially appending features from different encoders [, ], or adopting more complex fusion and routing strategies to make the best of different encoders [, ]. Such a “mixture-of-vision-experts” mode is shown to be effective. However, a detailed study of its design space with rigorous ablation is still lacking, which motivates us to revisiting this area. Questions such as which vision encoder combination to choose, how to fuse different experts together, and how to adjust training strategies with more vision encoders remain to be answered.

To address the above questions, our work systematically investigates the mixture-of-vision-encoders design space for improved MLLM perception. As shown in Fig. 1, our exploration of the design space consists of the following steps: 1) Benchmarking various vision encoders and searching recipes for higher resolution adaptation; 2) “Apples to apples” comparison between vision encoder fusion strategies; 3) Progressive identification of the optimal combination of multiple vision encoders; 4) Improved vision expert pre-alignment and data mixture. Our study covers the performance of vision encoders pre-trained on different tasks and resolutions (e.g., vision-language alignment [, , , ], self-supervised learning [], detection [, ], segmentation [], and OCR []). We use a round-robin approach to incorporate additional vision experts. Starting with the basic CLIP [] encoder, we add one additional expert at a time with the best improvement in each round.

Our work is not the first one to leverage multiple vision encoders in MLLM. However, the systematic study leads to several interesting new findings under this setting:

- •
Unlocking the vision encoders during MLLM training matters. This is in sharp contrast to the LLaVA [, ] family and many works that consider multiple vision encoders or teachers [, , , , , ], where freezing the vision encoders has been a common choice.
- •
Some recently proposed fusion strategies [, ] do not show significant advantages despite their advanced designs. Instead, we find that straightforward channel concatenation stands out as a simple yet competitive fusion strategy, offering the best efficiency and performance.
- •
Incorporating additional vision experts leads to consistent gain, making it a promising path to systematic enhance MLLM perception besides scaling up single encoders. The improvement is particularly pronounced when vision encoders are unlocked.
- •
We propose a pre-alignment stage where non-text-aligned vision experts are individually fine-tuned with a frozen LLM before trained together. This stage is found to enhance the MLLM performance significantly under the mixture-of-vision-encoder design.

We finally conclude our findings into a family of MLLMs termed Eagle. Eagle is evaluated on a series of benchmarks, including visual question answering, OCR/document related tasks, and benchmarks tailored for MLLMs. Our model attains state-of-the-art performance across different benchmarks and demonstrates obvious advantages on OCR and document understanding tasks. Using the same pre-train and supervised fine-tuning data from Cambrian-1 [] - a concurrent family of vision-centric MLLMs sharing similar design spirits, Eagle models overall achieve better performance. We hope that the Eagle can provide a highly performant and easy-to-reproduce MLLM solution to the community.

## 2 Design space exploration

In this section, we show how to utilize the advantages of different vision encoders via step-by-step investigations, yielding the Eagle model family. Unlike previous methods focusing on new fusion strategies or architectures among vision encodes, our goal is to identify a set of minimalistic design to fuse different vision encoders supported with detailed ablations, removing any unnecessary parts. As shown in Fig. 2, we start from extending the basic CLIP encoder [] to a set of vision experts with different architectures, pre-training tasks and resolutions. With these experts, we then compare different fusion architectures and methods, and study how to optimize the pre-training stratgies given more encoders. We also give a detailed analysis in how to select the vision encoders to be integrated. Finally, we put all the findings together and further extend to multiple expert vision encoders with different resolutions and domain knowledge.

### 2.1 Base setup

We adopt LLaVA’s [] model architecture as the basis, which consists of a large language model, a vision encoder, and a projection layer. The projection layer projects the visual embedding from the vision encoder into the text embedding space. We mainly focus on the vision encoder design.

Figure: Figure 2: Overview of the Eagle exploration pipeline. We investigate the MLLM design space with multiple vision encoders, aiming to identify optimized design choices and improve the MLLM perception. We include various types of vision experts pre-trained on different tasks and resolutions. We then systematically compare a series of popular fusion strategies under controlled settings, including Sequence Append, Channel Concatenation, LLaVA-HR [], Mini-Gemini [], and Deformable Attention []. Finally, the optimal fusion design is further extended to multiple vision encoders to form a strong MLLM perception.
Refer to caption: x2.png

Base training data. We use the same pre-training data as LLaVA-$1.5$ [], which consists of $595$k image text pairs. For the supervised fine-tuning stage, we collect data from a series of tasks and convert them into multimodal conversations, including: LLaVA-1.5 [], Laion-GPT4V [], ShareGPT-4V [], DocVQA [], synDog-EN [], ChartQA [], DVQA [], and AI2D [], resulting in 934k samples.

Implementation details.
We first pre-train the model with image-text pairs for one epoch with a batch size of 256, where the whole model is frozen and only the projector layer is updated. In the second stage, we further fine-tune the model on the supervised fine-tuning data for one epoch with a batch size of 128. For this exploration, we employ Vicuna-7B as the underlying language model. The learning rates are set to 1e-3 for the first stage and 2e-5 for the second stage, respectively.

### 2.2 Stronger CLIP encoder

We start our exploration with the CLIP model since it has become the main choice for many MLLMs. While CLIP models are known to benefit multimodal tasks, their drawbacks have also been well-studied. For instance, many existing MLLMs [] tend to use the pre-trained CLIP resolutions (such as $224\times 224$ or $336\times 336$) as their input resolutions. In these cases, the encoders often fail to capture fine-grained details that are important for resolution-sensitive tasks like OCR and document understanding [].

To handle increased input resolution, a common practice is to use tiling where input images are divided into tiles and encoded separately [, ]. Another simpler method is to directly scale up the input resolution and interpolate the position embeddings of the vision transformer model if needed [, , ]. We compare these two approaches with frozen/unfrozen vision encoders under different resolutions, with the results shown in Table 1. Our findings can be summarized as follows:

- •
Unfreezing the CLIP encoder leads to significant improvement when interpolating to a higher MLLM input resolution that is different from the CLIP pre-training resolution. There is also no performance degradation when resolutions remain the same.
- •
When the CLIP encoder is frozen, directly adapting it to a higher MLLM input resolution considerably hurts the performance.
- •
Among the compared strategies, directly interpolating to $448\times 448$ with an unfrozen CLIP encoder is shown to be both effective and efficient in terms of performance and cost.
- •
The best CLIP encoder gets close to InternVL in performance despite the significantly smaller model size (300M vs. 6B) and less pre-training data.

It is worth noting that CLIP-448 allows us to match the setting with LLaVA-HR [] and InternVL [] where the CLIP encoders are similarly adapted to take $448\times 448$ input and output $1024$ patch tokens(^1^11The latest InternVL 1.0 and InternVL 1.2 models also adopt a pixel shuffle module. We remove it in our experiment to ensure full alignment. See [https://github.com/OpenGVLab/InternVL](https://github.com/OpenGVLab/InternVL) for details.). In the following investigation, we stick to this simple strategy of scaling up the input resolution and unlocking the vision encoder during training.

**Table 1: Comparison of different high-resolution adaption methods. Direct interpolation is more efficient and effective. “Res” denotes input image resolution. “#Token/s” denotes the inference speed of the whole framework. “#Token (V)” denotes the number of visual feature tokens fed into LLM. “OCR” denotes OCRBench []. “Avg” denotes average. Note that CLIP-448 gets very close to InternVL in terms of performance, despite the significantly smaller model size (300M vs. 6B).**
| Method | Unfreeze | Res | #Token/s | #Token (V) | GQA | MME | MMMU | OCR | SQA | POPE | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Original | ✗ | 336 | 47.2 | 576 | 62.46 | 1488 | 36.1 | 369 | 72.79 | 86.77 | 615 |
| Original | ✓ | 336 | 47.2 | 576 | 63.68 | 1496 | 36.6 | 369 | 72.22 | 85.9 | 617 |
| Interpolate | ✗ | 448 | 47.9 | 1024 | 62.45 | 1484 | 34.4 | 285 | 72.27 | 86.54 | 598 |
| Interpolate | ✓ | 448 | 47.9 | 1024 | 64.62 | 1531 | 35.7 | 496 | 72.77 | 87.88 | 645 |
| Interpolate | ✓ | 672 | 44.5 | 2304 | 64.51 | 1492 | 36.2 | 492 | 72.86 | 87.49 | 641 |
| Tiled-input | ✓ | 672 | 44.8 | 2304 | 63.59 | 1455 | 34.8 | 445 | 71.94 | 87.83 | 625 |
| InternVL [] | ✓ | 448 | 46.2 | 1024 | 65.12 | 1521 | 36.2 | 527 | 72.46 | 87.13 | 649 |

**Table 2: Detailed setting of the pre-trained vision experts.**
| Expert | Category | Resolution | Token Number | Post-Processing | Variant Hyper-links |
| --- | --- | --- | --- | --- | --- |
| CLIP | Image-Text Matching | 448 | 1024 | None | [VIT-L](https://huggingface.co/openai/clip-vit-large-patch14-336) |
| ConvNeXt | Image Classification | 1024 | 1024 | None | [ConvNeXt-XXL](https://huggingface.co/laion/CLIP-convnext_xxlarge-laion2B-s34B-b82K-augreg-soup) |
| EVA-02 | Object Detection | 1024 | 1024 | Resize | [EVA-02-Large](https://huggingface.co/Yuxin-CV/EVA-02/blob/main/eva02/det/eva02_L_coco_det_sys_o365.pth) |
| Pix2Struct | Text Recognition | 1024 | 1024 | Resize | [Pix2Struct-02-Large](https://huggingface.co/google/pix2struct-large) |
| DINOv2 | Self-supervised | 448 | 1024 | None | [dinov2_vitl14_reg](https://github.com/facebookresearch/dinov2/blob/main/MODEL_CARD.md) |
| SAM | Image Segmentation | 1024 | 1024 | Pixel-unshuffle | [SAM-Large](https://huggingface.co/facebook/sam-vit-large) |

### 2.3 Vision experts

To better establish the foundation for multi-vision expert fusion, we extend the toolbox with vision experts pre-trained on different tasks and resolutions, and verify our findings on high-resolution adaptation with these experts. This also helps us identify the distinct advantages of different experts. We collect a set of vision experts, including: (1) Vision-Language Alignment: CLIP [] and ConvNeXt [] from OpenCLIP [, ]. (2) Object-Centric: EVA-02 [, ] pre-trained on detection datasets. (3) OCR: Pix2Struct [].(4) Segmentation: SAM []. (5) Self-supervised: DINOv2 []. The detailed input resolution and checkpoint of each vision encoder can be found in Table 2. We resize the output 2D feature maps of each vision encoder using bilinear interpolation and pixel shuffle [] to ensure that the visual token number equals $1024$.

The results in Table 3 reveal that MLLMs with these task-specific vision encoders achieve optimal performance in their pre-training domains. EVA-02 excels in the object hallucination evaluation benchmark POPE [] and general visual question answering benchmark GQA []. CLIP and ConvNeXt perform well across all benchmarks, benefiting from their training on large-scale image-text pairs using contrastive loss. Conversely, while Pix2Struct excels in text recognition, it shows limited capability in object recognition and general VQA tasks, like POPE and GQA. DINOv2 and SAM, pre-trained with self-supervised learning and semantic segmentation respectively, struggle with text recognition tasks. Note that the conclusion made in Sec. 2.2 aligns with the results here. Both ConvNeXt and EVA-02 show significant performance gains when the vision encoders are updated during training.

**Table 3: Comparison between different vision experts as the MLLM encoders. “Res” denotes resolution. “OCR” denotes OCRBench [].**
| Category | Vision Tower | Unfreeze | Res | GQA | MME | MMMU | OCR | SQA | POPE | TextVQA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| VL Alignment | ConvNeXt | ✗ | 1024 | 63.71 | 1473 | 34.2 | 402 | 71.92 | 87.42 | 66.95 |
| ✓ | 1024 | 63.71 | 1433 | 37.0 | 527 | 72.01 | 87.91 | 71.66 |  |  |
| Segmentation | SAM | ✗ | 1024 | 54.87 | 1193 | 34.8 | 32 | 70.34 | 83.31 | 45.04 |
| ✓ | 1024 | 60.05 | 1305 | 35.9 | 38 | 70.64 | 86.62 | 45.71 |  |  |
| Object Detection | EVA-02 | ✗ | 1024 | 63.54 | 1422 | 35.8 | 58 | 70.86 | 88.50 | 47.86 |
| ✓ | 1024 | 64.93 | 1474 | 35.7 | 387 | 71.78 | 88.96 | 59.79 |  |  |
| Text Recognition | Pix2Struct | ✗ | 1024 | 52.47 | 1217 | 35.7 | 443 | 69.61 | 77.54 | 56.56 |
| ✓ | 1024 | 54.05 | 1220 | 34.3 | 425 | 70.29 | 79.21 | 57.05 |  |  |
| Self-supervised | DINOv2 | ✗ | 448 | 60.81 | 1269 | 34.0 | 39 | 70.76 | 86.42 | 45.71 |
| ✓ | 448 | 64.06 | 1371 | 34.2 | 37 | 71.35 | 87.29 | 46.77 |  |  |

### 2.4 Fusion strategy

Existing MLLM frameworks have proposed various fusion strategies to ensemble the encoders in Table 3, with the hope that their domain-specific strengths can be leveraged. In all cases, improvements in MLLM performance have been reported with the fusion of vision encoders. However, the roles of the fusion strategies as part of their MLLM architecture innovations, have not been decoupled and clearly studied under an “apples to apples” comparison. It is thus not entirely clear how much improvement is from the fusion strategies themselves versus the improved representations from various encoders.

We notice that existing popular fusion strategies, despite their variations in designs, can be broadly represented by the following several categories: (1) Sequence Append: directly appending the visual tokens from different backbones as a longer sequence [, , ]; (2) Channel Concatenation: concatenating the visual tokens along the channel dimension without increasing the sequence length [];
(3) LLaVA-HR: injecting high-resolution features into low-resolution vision encoders using mixture-of-resolution adapter [];
(4) Mini-Gemini: using the CLIP tokens as the low resolution queries to cross-attend another high-resolution vision encoder in the co-located local windows []. (5) Deformable Attention: a new baseline we introduce on top of Mini-Gemini, where the vanilla window attention is replaced with deformable attention []. Fig. 2 gives a detailed illustration of these fusion strategies. To better study them, we choose “CLIP+ConvNeXt” and “CLIP+ConvNeXt+SAM” as the base multi-encoder combinations to perform comparisons.

Our study in Table 4 shows that Channel Concatenation achieves the best average performance while maintaining better throughput compared to sequence append. The “injection-based” methods, such as LLaVA-HR, Mini-Gemini and Deformable Attention, are in general less competitive on TextVQA [] and OCRBench [], performing worse than using ConvNeXt alone as the vision encoder. A plausible explanation is that the CLIP features continue to play a dominant role in the visual tokens. Although sequence append shows comparable performance to channel concatenation, it faces the challenge to handle more vision encoders due to the increasing sequence length. Hence, we choose direct channel concatenation as our fusion strategy considering its performance, expandability, and efficiency.

**Table 4: Comparison of different fusion methods for different vision experts. SA, CC, LH, MG and DA denote the fusion strategies: Sequence Append, Channel Concatenation, LLaVA-HR [], Mini-Gemini [] and Deformable Attention, respectively. “Token (V)” denotes the number of vision feature tokens fed into LLM. “#Tokens/s” denotes the speed of the whole pipeline. “Avg” denotes average. Note that Channel Concatenation achieves both the best performance and better throughput.**
| Vision Encoder(s) | Fusion | Token (V) | #Tokens/s | GQA | MME | MMMU | OCR | SQA | POPE | TextVQA | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP | – | 1024 | 47.9 | 64.6 | 1531 | 35.7 | 496 | 72.8 | 87.9 | 64.0 | 644.4 |
| ConvNeXt | – | 1024 | 46.8 | 63.7 | 1433 | 37.0 | 527 | 72.0 | 87.9 | 71.7 | 652.3 |
| CLIP<br>+ ConvNeXt | SA | 2048 | 46.1 | 64.6 | 1482 | 35.3 | 536 | 72.5 | 88.3 | 71.8 | 657.2 |
| CC | 1024 | 47.3 | 64.0 | 1486 | 36.0 | 533 | 72.7 | 88.6 | 71.9 | 658.2 |  |
| LH | 1024 | 47.0 | 62.9 | 1488 | 36.8 | 521 | 72.6 | 88.9 | 68.7 | 652.1 |  |
| MG | 1024 | 45.2 | 64.9 | 1481 | 36.7 | 474 | 72.8 | 88.2 | 68.6 | 646.7 |  |
| DA | 1024 | 47.3 | 63.6 | 1497 | 35.3 | 459 | 71.9 | 88.0 | 69.1 | 640.9 |  |
| CLIP<br>+ ConvNeXt + SAM | SA | 3072 | 40.3 | 64.9 | 1439 | 35.0 | 504 | 72.7 | 88.2 | 71.3 | 649.2 |
| CC | 1024 | 46.3 | 64.6 | 1497 | 35.2 | 558 | 71.7 | 87.9 | 72.2 | 660.4 |  |

### 2.5 Vison-language Pre-Alignment

Figure: Figure 3: The proposed training strategy of Eagle. It consists of three progressive stages, including vision-language pre-alignment training, joint-project training and supervised fine-tuning. These stages effectively leverage public data from diverse sources, ranging from noisy image-text pairs on the web to high-quality caption, VQA, and multimodal dialogue datasets.
Refer to caption: x3.png

As shown in Table 3, vision backbones pre-trained exclusively on vision tasks (e.g., detection, OCR, and segmentation) are less competitive compared to encoder pre-trained on vision language alignment. This is possibly due to representational inconsistencies when integrated with large language models. Additionally, when combining different encoders, there is a gap between these encoders, creating difficulties in the training process. To address this feature inconsistency, we propose a Pre-Alignment training stage that first aligns each individual vision encoder with the same large language model, fostering better synergy between visual and linguistic capabilities.

Fig. 3 depicts our pre-alignment strategy. Instead of training a projector to simultaneously align multiple vision experts as in LLaVA’s [] original pre-training strategy, we first align the representation of each individual expert with a smaller language model (Vicuna-7B in practice) using next-token-prediction supervision. As shown in Fig. 3, with pre-alignment, the whole training process consists of three steps: 1) training each pre-trained vision expert with their own projector on SFT data, while keeping the language model frozen; 2) combining all the vision experts from the first step and training only the projector with image-text pairs data; 3) training the whole model on the SFT data.

To verify the proposed method, we compare the pre-alignment strategy with the normal two-stage training strategy in Table 5, considering both freezing and unfreezing vision experts for comparison.
As shown in Table 5, although unfreezing the vision experts during SFT helps improve performance by updating the vision experts to fit the language model, the Pre-Align strategy more effectively mitigates the inherent biases of each vision expert and stabilizes the training process, subsequently improving overall performance.

**Table 5: Experiments to show the effectiveness of Pre-Align. “OCR” denotes OCRBench [].“Avg” denotes average.**
| CLIP | Vision Expert (X) | Unfreeze | Pre-Align | MME | MMMU | OCR | SQA | POPE | TextVQA | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP-448 | SAM-1024 | ✗ | ✗ | 1495 | 35.1 | 292 | 71.8 | 85.6 | 59.7 | 593 |
| ✓ | ✗ | 1504 | 36.9 | 482 | 73.5 | 87.7 | 63.2 | 641 |  |  |
| ✓ | ✓ | 1554 | 36.7 | 505 | 73.8 | 88.2 | 62.9 | 649 |  |  |
| CLIP-448 | ConvNext-1024 | ✗ | ✗ | 1499 | 35.1 | 416 | 72.3 | 86.7 | 67.1 | 629 |
| ✓ | ✗ | 1522 | 36.6 | 550 | 72.7 | 87.9 | 71.9 | 667 |  |  |
| ✓ | ✓ | 1538 | 36.3 | 556 | 71.9 | 87.9 | 71.6 | 667 |  |  |
| CLIP-448 | Pix2Struct-1024 | ✗ | ✗ | 1500 | 35.8 | 467 | 71.9 | 86.1 | 60.9 | 627 |
| ✓ | ✗ | 1529 | 36.2 | 535 | 73.1 | 87.1 | 64.5 | 651 |  |  |
| ✓ | ✓ | 1564 | 35.3 | 541 | 73.3 | 86.6 | 63.9 | 652 |  |  |
| CLIP-448 | EVA-02-L-1024 | ✗ | ✗ | 1484 | 35.8 | 305 | 72.0 | 86.0 | 59.9 | 597 |
| ✓ | ✗ | 1537 | 37.0 | 474 | 72.6 | 87.5 | 62.6 | 639 |  |  |
| ✓ | ✓ | 1586 | 37.3 | 495 | 73.1 | 88.9 | 64.2 | 653 |  |  |

### 2.6 Extension to multi-experts

With the optimized strategies and training recipes of incorporating individual vision experts, we consider the incorporation of even more vision experts to push the limit. To conduct the searching in a systematic and principled manner, we adopt a step-by-step greedy strategy to incorporate additional vision experts.

We consider the vision experts discussed in Section 2.3 for experiments. We mark CLIP, ConvNeXt, SAM, DINOv2, Pix2Struct, and EVA-02-L as A, B, C, D, E, and F, respectively. A round-robin scheme, as shown in Table 6, is adopted. We first use the two top performing vision encoders, CLIP and ConvNeXt, as the basis and gradually add one more vision encoder each time. In each round, the best-performing vision encoder combination is retained for the next round.

**Table 6: Results of vision expert selection process. A, B, C, D, E and F denote CLIP, ConvNeXt, SAM, DINOv2, Pix2Struct and EVA-02-L models respectively. “OCR” denotes OCRBench []. “Avg” denotes average. Under the same number of encoders, the best scores of various combination variants are bold.**
| #Encoders | Encoder Combination | GQA | MME | MMMU | OCR | SQA | POPE | TextVQA | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 2 | A + B | 64.0 | 1486 | 36.0 | 533 | 72.7 | 88.6 | 71.9 | 658.3 |
| 3 | A + B + C | 64.6 | 1497 | 35.2 | 558 | 71.7 | 87.9 | 72.2 | 660.3 |
| A + B + D | 65.7 | 1506 | 35.0 | 509 | 72.6 | 87.9 | 70.1 | 653.6 |  |
| A + B + E | 65.5 | 1459 | 35.3 | 565 | 73.1 | 87.9 | 70.8 | 660.1 |  |
| A + B + F | 64.7 | 1506 | 35.9 | 562 | 73.2 | 88.3 | 72.1 | 665.3 |  |
| 4 | A + B + F + C | 65.9 | 1498 | 36.6 | 540 | 72.0 | 88.8 | 71.6 | 662.6 |
| A + B + F + D | 65.6 | 1521 | 35.8 | 508 | 73.0 | 89.0 | 70.0 | 657.5 |  |
| A + B + F + E | 65.3 | 1539 | 37.1 | 552 | 73.0 | 87.4 | 71.3 | 666.1 |  |
| 5 | A + B + F + E + C | 65.7 | 1511 | 37.6 | 548 | 72.7 | 88.6 | 71.6 | 666.5 |
| A + B + F + E + D | 65.6 | 1512 | 36.0 | 528 | 72.9 | 88.4 | 70.5 | 660.0 |  |
| 6 | A + B + F + E + C + D | 65.18 | 1504 | 35.3 | 534 | 71.9 | 87.6 | 70.86 | 656.3 |

Table 6 reveals several insights. Generally, introducing additional vision encoders enhances the performance. This indicates that the distinct advantages of different encoders can be preserved and utilized; for example, integrating the EVA-02 encoder improves metrics on the POPE benchmark. Although individual metrics may vary, the aggregated performance shows an upward trend, as evidenced by normalized average metrics, suggesting that the overall efficacy of the system is enhanced with more encoders. Also, Table 6 shows that the best combination of vision experts are CLIP, ConvNeXt, SAM, Pix2Struct, and EVA-02. We will use this recipe in our final model.

**Table 7: The advanced training data used in the supervised fine-tuning stage.**
| Note | Data Source |
| --- | --- |
| Total Number (1,809k) | LLaVA-1.5 [] (665k), DocVQA [] (39k), synDog-EN [] (50k),<br>ChartQA [] (28k), DVQA [] (25k), AI2D [] (15k), ShareGPT-4V [] (100k),<br>laion-GPT4V [] (11k), LVIS-Instruct4V [] (220k), LRV-Instruct [] (150k),<br>Geo170k [] (120k), LLaVAR [] (20k),<br>Visual7W [] (70k), Open-Hermes 2.5 [] (1809k) |

## 3 Experiments

After meticulously developing our recipes, we have formulated the following principals for our model, Eagle: (1) integrating more vision experts with optimized training recipe; (2) combining multiple vision experts through direct channel concatenation; (3) pre-training the vision experts separately via pre-alignment.
In this section, to further show the advantages of the Eagle models, we add more training data and compare Eagle against the current state-of-the-art MLLMs on different tasks.

### 3.1 Implementation details

Models. We use Vicuna-v1.5-7B [], Llama3-8B [] and Vicuna-v1.5-13B [] as the language model. For vision encoder, following the results in Sec. 2.6, we denote the model with four vision encoders: CLIP, ConvNeXt, Pix2Struct and EVA-02 as Eagle-X4 and the model with an additional SAM vision encoder as Eagle-X5.

Augmented training data. Considering the training cost, we only use 934K data during the investigation and ablations, as listed in Sec. 2.1. In this section, inspired by recent practices [, ], we introduce more datasets, including LVIS-INSTRUCT4V [], LRV [], Geo170K [], LLaVAR [], Visual7w [], and OpenHermes-2.5 [], resulting in a total of 1,809k data for the supervised fine-tuning stage. See Table 7 for more details on this data mixture. To better perform an apple-to-apple comparison with recent open-source MLLM projects, we also train Eagle-X5 with the same data as Cambrian-1 [], including 2.5 million and 7 million data for pre-training and supervised fine-tuning.

### 3.2 Main results

Evaluation on visual question answering tasks.
We compare Eagle model series across three Visual Question Answering (VQA) benchmarks, including GQA [], VQAv2 [] and VizWiz []. As shown in Table 8, Eagle-X5 achieves state-of-the-art performance on GQA and VQAv2, underscoring the advantages of incorporating additional vision experts.

**Table 8: Results with the advanced training data recipe. SQA^I denotes ScienceQA-IMG [].**
|  | Model | MME | MMBench | SEED | MathVista | MMMU | POPE | SQA^I | GQA | VizWiz | VQAv2 | OCRBench | TextVQA | ChartQA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Vicuna-7B & Qwen-7B | LLaVA-1.5 [] | 1510 | 64.3 | 58.6 | - | - | 85.9 | 66.8 | 62.0* | 50.0 | 78.5* | 297 | 58.2 | - |
| LLaVA-NeXt [] | 1519 | 67.4 | 70.2 | 34.6 | 35.8 | 86.5 | 70.1 | 64.2* | 57.6 | 80.0* | 490 | 64.9 | - |  |
| InternVL [] | 1525 | - | 65.4 | - | - | 86.4 | - | 62.9* | 52.5 | 79.3* | - | 57.0 | - |  |
| LLaVA-HR [] | 1554 | - | 64.2 | - | - | 87.6 | 65.1 | 64.2* | 48.7 | 81.9* | - | 67.1 | - |  |
| Monkey [] | - | - | - | - | - | - | - | 60.7* | 61.2* | 80.3* | 514 | 67.6 | 65.1 |  |
| Mini-Gemini [] | 1523 | 65.8 | - | 32.2 | 36.8 | - | 71.1 | 64.5* | - | - | 477 | 65.2 | - |  |
| Eagle-X4 | 1561 | 67.8 | 73.4 | 37.3 | 34.9 | 88.4 | 70.4 | 64.8* | 50.8 | 83.4* | 540 | 70.9 | 67.5 |  |
| +Pre-Align | 1561 | 68.1 | 73.4 | 38.8 | 36.8 | 88.7 | 70.8 | 64.9* | 51.0 | 83.7* | 544 | 71.9 | 67.9 |  |
| Eagle-X5 | 1528 | 68.4 | 73.9 | 37.0 | 36.3 | 88.8 | 70.0 | 64.9* | 54.4 | 83.4* | 529 | 71.2 | 67.7 |  |
| +Pre-Align | 1579 | 68.8 | 73.5 | 39.0 | 36.4 | 88.5 | 71.6 | 64.0* | 56.0 | 83.6* | 554 | 71.9 | 67.0 |  |
| Vicuna-13B | LLaVA-1.5 [] | 1531 | 67.7 | 61.6 | - | 36.4 | 85.9 | 71.6 | 63.3* | 53.6 | 80.0* | 331 | 61.3 | - |
| LLaVA-NeXt [] | 1575 | 70.0 | 71.9 | 35.3 | 36.2 | 86.2 | 73.5 | 65.4* | 60.5 | 82.8* | 514 | 67.1 | 62.2 |  |
| InternVL [] | 1546 | - | - | - | - | 87.1 | - | 63.9* | 54.6 | 80.2* | 517 | 58.7 | - |  |
| LLaVA-UHD [] | 1535 | 68.0 | - | - | - | 89.1 | 72.0 | 65.2* | 56.1 | 81.7* | - | 67.7 | - |  |
| LLaVA-HR [] | 1540 | - | 64.5 | - | - | 87.8 | 68.1 | 64.8* | 57.9 | 82.6* | - | 68.1 | - |  |
| Mini-Gemini [] | 1565 | 68.6 | 70.6 | 37.0 | 37.3 | - | 71.9 | 65.8* | - | - | 466 | 65.9 | 56.6 |  |
| Eagle-X4 | 1627 | 69.9 | 74.4 | 37.6 | 36.9 | 87.7 | 73.1 | 66.3* | 56.2 | 83.7* | 561 | 73.9 | 70.5 |  |
| +Pre-Align | 1621 | 69.9 | 74.9 | 39.4 | 37.3 | 89.1 | 73.5 | 64.9* | 55.9 | 84.5* | 581 | 74.3 | 72.2 |  |
| Eagle-X5 | 1609 | 69.2 | 74.1 | 38.8 | 36.6 | 87.8 | 72.7 | 66.2* | 59.3 | 83.8* | 574 | 74.2 | 69.9 |  |
| +Pre-Align | 1604 | 70.5 | 74.9 | 39.7 | 38.0 | 88.2 | 73.1 | 64.4* | 60.9 | 84.5* | 573 | 73.9 | 71.0 |  |

Evaluation on OCR and chart understanding tasks.
To evaluate the OCR, document and chart understanding capabilities of Eagle, we benchmark our model on OCRBench [], TextVQA [], and ChartQA []. As illustrated in Table 8, our model significantly surpasses competitors on TextVQA, benefiting from its high-resolution architecture and integration of different vision encoders. Notably, Eagle maintains a straightforward design, supporting up to 1024 tokens without requiring complex tile decomposition of images.

Fig. 5 shows some examples of OCR and document understanding cases. With high-resolution adaptation and more vision experts, our model can identifies small text within images and accurately extracts information according to the users’ instructions. To better understand the benefits of introducing experts pre-trained on other vision tasks,we visualize the results of a model with only the ConvNeXt and CLIP vision encoders, compared to the results of Eagle-X5 in Fig. 4. With the full set of vision encoders, the model can successfully correct mistakes, showing that even when equipped with high-resolution vision encoders pre-trained on vision-language alignment, the model’s abilities can still be enhanced by integrating additional vision experts pre-trained on diverse vision tasks.

Evaluation on multimodal benchmarks.
We evaluate Eagle on seven benchmarks for MLLMs to demonstrate its capabilities from different perspectives, including MME [], MMBench [], SEED [], MathVista [], MMMU [], ScienceQA [], and POPE [].
Specifically, MME, MMBench, and SEED assess the overall performance on various real-world tasks based on reasoning, recognition, knowledge, and OCR. MMMU focuses on challenging problems from diverse domains that require college-level knowledge. POPE evaluates the visual hallucinations of MLLMs. The metrics used in our paper adhere to the default settings of these benchmarks.
We report the perception score for MME, the en_dev split for MMBench, the image split of SEED, the test-mini split of MathVista, the val split of MMMU, the F1-score of POPE, and the image of SQA to align with the reported scores from other models.

From the data presented in Table 8, Eagle consistently surpasses existing models across various MLLMs on SEED and MME, demonstrating the comprehensive knowledge and reasoning abilities of Eagle. With the help of vision encoders on object-centric tasks, Eagle also achieves the best performance on the POPE benchmark. Additionally, the Pre-Alignment strategy discussed in Sec. 2.5 has been found to further enhance performance when integrating multiple task-specific vision backbones. This approach not only mitigates the inherent biases of each vision expert and the synergy between different modalities but also establishes a robust framework for the multiple-expert fusion.

Comparison with Cambrian-1.
Using the same pre-training and supervised fine-tuning datasets from Cambrian-1 [], Eagle demonstrates superior performance across all the evaluated benchmarks without bells and whistles. As shown in Table 9, Eagle outperforms the Cambrian-1 counterparts considerably on the OCR and Chart category. Consistent improvements are also observed on the General, Knowledge and Vision-Centric categories, showing the robustness and generalization ability of the improved perception design in Eagle.

**Table 9: Results using Cambrian-1 training data. The Eagle-X5 models are trained using the same pre-training and supervised fine-tuning data as Cambrian-1 []. SQA^I denotes ScienceQA-IMG []. RWQA denotes RealworldQA [].**
| Model | Knowledge | General | OCR and Chart | Vision-Centric |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Avg | SQA^I | MMMU | MathVista | AI2D | Avg | MME | MMBench | SEED | GQA | Avg | ChartQA | OCRBench | TextVQA | DocVQA | Avg | MMVP | RWQA |
| Llama3-8B |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| MGM-HD | 55.7 | 75.1 | 37.3 | 37.0 | 73.5 | 72.7 | 1606 | 72.7 | 73.2 | 64.5 | 62.9 | 59.1 | 47.7 | 70.2 | 74.6 | 40.4 | 18.7 | 62.1 |
| Cambrian-1 | 61.3 | 80.4 | 42.7 | 49.0 | 73.0 | 73.1 | 1547 | 75.9 | 74.7 | 64.6 | 71.3 | 73.3 | 62.4 | 71.7 | 77.8 | 57.6 | 51.3 | 64.2 |
| Eagle | 64.2 | 84.3 | 43.8 | 52.7 | 76.1 | 73.8 | 1559 | 75.9 | 76.3 | 64.9 | 76.6 | 80.1 | 62.6 | 77.1 | 86.6 | 69.1 | 71.6 | 66.5 |
| Vicuna-13B |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| MGM-HD | 54.1 | 71.9 | 37.3 | 37 | 70.1 | 70.7 | 1597 | 68.6 | 70.6 | 63.7 | 60.8 | 56.6 | 46.6 | 70.2 | 69.8 | 38.4 | 19.3 | 57.5 |
| Cambrian-1 | 60.2 | 79.3 | 40.0 | 48.0 | 73.6 | 73.7 | 1610 | 75.7 | 74.4 | 64.3 | 71.3 | 73.8 | 61.9 | 72.8 | 76.8 | 52.2 | 41.3 | 63.0 |
| Eagle | 63.0 | 82.0 | 41.6 | 54.4 | 74.0 | 74.6 | 1651 | 75.7 | 74.8 | 65.3 | 75.1 | 77.6 | 61.9 | 75.5 | 85.4 | 61.4 | 58.0 | 64.8 |
| Yi-34B |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| MGM-HD | 62.4 | 77.7 | 48 | 43.4 | 80.5 | 76.2 | 1659 | 80.6 | 75.3 | 65.8 | 68.1 | 67.6 | 51.8 | 74.1 | 78.9 | 52.3 | 37.3 | 67.2 |
| Cambrian-1 | 67.0 | 85.6 | 49.7 | 53.2 | 79.7 | 76.8 | 1689 | 81.4 | 75.3 | 65.8 | 71.9 | 75.6 | 60.0 | 76.7 | 75.5 | 60.3 | 52.7 | 67.8 |
| Eagle | 68.6 | 85.5 | 53.2 | 57.9 | 79.1 | 76.3 | 1677 | 81.0 | 75.6 | 64.9 | 75.4 | 77.2 | 62.4 | 78.8 | 83.0 | 68.3 | 67.0 | 69.5 |

Figure: Figure 4: Qualitative comparison of different numbers of vision experts. Baseline means Eagle with only CLIP+ConvNext. More Vision Experts denotes the Eagle-X5 model. We highlight a failure case in RED. BLUE indicates the correct answers. With more vision experts, Eagle can more precisely capture the information in the image and provide the correct answer.
Refer to caption: x4.png

Figure: Figure 5: Qualitative samples on OCR and document understanding tasks. Eagle is able to extract useful information from small text.
Refer to caption: x5.png

Figure: Figure 6: Qualitative results of Eagle on VQA tasks requiring college-level knowledge or deliberate reasoning.
Refer to caption: x6.png

## 4 Related work

### 4.1 Multimodal large language models

Our work is related to the general architecture design of multimodal large language models. Besides the line of representative open-source research mentioned in the introduction section, other notable families of MLLMs include, but are not limited to MiniGPT-4 [, ], Lynx [], Otter [, ], Qwen-VL [], CogVLM [, ], VILA [], GPT-4V [], Gemini [], and Llama 3.1 []. Depending on how vision signals are integrated into the language model, MLLMs can be broadly categorized into “cross-modal attention” ones and “prefix-tuning” ones []. The former injects visual information into different layers of LLMs using cross-modal attention [, ], whereas the latter views the visual tokens as part of the language token sequence and directly append them with text embeddings [, , ]. Our model belongs to the prefix-tuning family by following a LLaVA-styled multimodal architecture. Considering that MLLM is a fast growing field, readers are recommended to refer to more detailed studies and surveys such as [, , ].

### 4.2 Vision encoder design for MLLMs

Our work is closely related to research focusing on improved vision encoder designs for MLLM. Early works [, ] usually adopted vision encoder pre-trained on vision-language alignment tasks such as CLIP [] and EVA-CLIP []. Stronger vision encoders such as SigLIP [] and InternVL [] have been proposed to improve vision-language tasks with improved designs, larger model sizes, and better training recipes. Considering that the models are often pre-trained on low-resolution images and lack the ability to encode fine-grained details, higher resolution adaptation is often performed to increase the MLLM input resolution [, , , ].

Besides higher resolution adaptation, models such as LLaVA-NeXT [], LLaVA-UHD [], Monkey [], InternLM-XComposer [], and InternVL [] use either tiling or adaptive tiling to handle high resolution input, where input images are divided into lower-resolution patches and process them separately. Although the ability to handle higher resolution is similarly made possible with the introduction of additional vision experts, its spirit is slightly orthogonal to the tiling techniques in terms of introducing the diversity of visual representations. Both techniques are compatible and can be combined together.

Our work is most related to existing models using multiple vision encoders for improved perception. Mini-Gemini [] and LLaVA-HR [] propose to fuse high-resolution visual features into the low-resolution visual tokens. Apart from the resolution issue, these pre-trained vision encoders may lack specific abilities such as reading text and localizing objects. Hence, a series of works have integrated vision models pre-trained on various vision tasks to extend the capabilities of vision encoders. For example, Mousi [], and Brave [] fuse visual tokens from different vision encoders by concatenating along the channel or token direction. RADIO [] proposes a multi-teacher distillation method to unify the abilities of different vision encoders into a unified model. MoAI [], IVE [], and Prismer [] further use the output of vision experts, such as OCR, detection, or depth estimation, to supplement more information for MLLM to generate answers. MoVA [] devises a routing network to assign an optimal vision model conditioned on the given image and instructions.

## 5 Conclusion

We conduct an in-depth analysis study on the design space for integrating vision encoders for multimodal large language models. Unlike previous works that focus on designing novel fusion paradigms, we find systematic design choice matters and discover a series of useful techniques. Step by step, we optimize the training recipe of individual vision encoders, identify an extendable and efficient fusing method, and gradually combine vision encoders with different domain knowledge. The results show the importance of basic design space. We hope our work can serve as a new basis and bring new inspiration for the vision encoder design for MLLM.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weaknesses.
soundness: 3
strengths: This paper provides a detailed and systematic study on the **mixture of vision encoders**, offering valuable insights to the research community.
summary: This paper conducts a systematic study on the **mixture of vision encoders** in multimodal large language models (LLMs). Specifically, the paper thoroughly compares the performance differences between various combinations of vision encoders and fusion methods, ultimately finding that simply concatenating the tokens from different encoders outperforms more complex fusion strategies. To bridge the gap between different vision encoders, the authors propose a **pre-alignment** strategy, which enhances the subsequent collaboration between vision encoders by aligning them with a small LLM in advance. The proposed method demonstrates significant improvements across several metrics.
weaknesses: 1. **Limited novelty**:  
   While the primary contribution of this paper lies in the extensive comparisons and ablation studies, the method itself lacks significant innovation. Technically, it does not introduce any groundbreaking insights for the community.

2. **Flaws in Experimental Setup**:  
   There are some flaws in the experimental setup. In the comparison shown in **Table 5**, the configuration using pre-alignment includes an additional **stage 1** of training compared to the configuration without pre-alignment, leading to an imbalance in the amount of training. It is recommended that the authors add a setup where **stage 1** is removed and **stage 3** is trained for an additional epoch to eliminate the unfair advantage of extra training rounds. Furthermore, since the vision encoders are pre-aligned in **stage 1**, why is it necessary to retrain in **stage 3**? What would happen if the vision encoders were simply frozen?

3. **Computational and Memory Costs**:  
   Since the proposed method involves mixing multiple different vision encoders, the computational and memory costs during training and inference are important factors to consider. When comparing with other methods, these indicators should be included. Additionally, it would be helpful to provide the computational cost and memory usage for each combination in **Table 6** for a more comprehensive evaluation of different vision encoder combinations. **Table 6** shows that the performance difference between five encoders and three encoders is minimal. If the cost is significantly higher, is it necessary to use the combination of five encoders?

4. **Omission of a Key Model**:  
   **SigCLIP** is one of the widely used vision models for MLLMs. Why is it not included as an option in **Table 2**?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weaknesses section above.
soundness: 3
strengths: - The paper is well articulated, with various steps properly backed up with experiments, and compared against significant number of baselines. The results indicate that the approach performs better than other mixture of vision encoders works. 
- The better performance of interpolation over tiling in the initial steps is an important contribution towards efficiency, and is much appreciated
- The paper addresses an important issue, with multiple strides in the space of multimodal LLMs, it is essential to incorporate the best set of changes that work well together, and this paper is a good step towards this goal.
- Well written paper, easy to read
summary: The authors present an extensive study of mixture of encoders, providing ablations and experiments on various design choices. The authors recommend interpolating for higher resolutions, having a higher number of vision encoder experts, concatenating tokens in a channel-wise manner, and also compare training strategies. The method is compared against a lot of recent advances in this space, and has competitive performance on a variety of benchmarks.
weaknesses: I will combine weaknesses and questions here:

(1) My major primary concern is about the efficiency of the approach, and I will list it below as a two-part concern - 

- (**Most Imp Concern**) Fig 4 mentions the various combinations tried in the mixture of vision encoders. As compared to the 2 encoder setting (X2), the next mixture (X3) leads to a slight improvements at the expense of 50% reduction in throughput, and going further, X5 leads to more than 4x reductions in throughput. While the results in terms of accuracy improve, as also suggested by Table 6, it comes at a significant compute expense. It would be important to understand the author's perspective on this agenda. Additionally, in Table 6, it would be useful to mention the throughput of other baseline methods, wherever possible. In my humble opinion, if the proposed method has a severely lower throughput (high inference cost), it should also be compared against other methods having comparable throughput when comparing accuracy metrics.
- The paper suggests that unfreezing leads to better performance, however, it also increases the number of trainable parameters. The authors should provide details on the training overhead that this change leads to, so as to better understand the efficacy of the slight improvement.

(2) The motivation behind introducing the Deformable Attention baseline is unclear and the comparisons made with it are limited. Additionally, to motivate channel concatenation further over sequence concatenation, quantitative comparisons of the implications of handling longer sequence lengths would be helpful.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: ## Justification For Recommendation And Suggestions For Rebuttal：
- Justification For Recommendation：Reference to Paper Strengths.

## Additional Comments For Authors：
It would be better to add the results of some other models that also use many vision encoders for comparison in Table 6.
soundness: 4
strengths: 1. This paper has confirmed through comprehensive and integrated experiments the necessity of turning on vision encoders during the SFT stage. At the same time, the paper has discovered and validated that straightforward channel concatenation stands out as a simple yet competitive fusion strategy, offering the best efficiency and performance.
2. This paper proposes a pre-alignment stage where non-text-aligned vision experts are individually fine-tuned with a frozen LLM before being trained together.
3. The illustrations in this paper are accurate and concise, and the language expression is very refined, making it easy for people to understand.
summary: This paper's findings reveal several underlying principles common to various existing strategies, leading to a streamlined yet effective design approach. The paper discovers that simply concatenating visual tokens from a set of complementary vision encoders is as effective as more complex mixing architectures or strategies. Additionally, the paper introduces Pre-Alignment to bridge the gap between vision-focused encoders and language tokens, enhancing model coherence. The resulting family of MLLMs, Eagle, surpasses other leading open-source models on major MLLM benchmarks.
weaknesses: 1. Some details of the methods are missing. How are the projectors of each expert in stage 1 of Figure 3 dealt with in stage 2?
2. Some corrections to details. In Table 4 at line 317 of the text, it looks like it should be referring to Figure 4. The same applies below.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the comments above.
My main concern is weakness 1, for such reasons i don't think this paper's contribution can lead to an acceptance.
soundness: 3
strengths: Strength：

1. Extensive experiments demonstrate the importance of optimizing the design space of vision encoders.
2. The ablation studies are abundant.
3. This paper is well-organized and clearly-written.
summary: This paper conducts an empirical analysis study on the design space for integrating vision encoders for MLLMs. It optimizes the training recipes of individual vision encoders step by step, identify an extendable and efficient fusion method, and gradually combine vision encoders with different domain knowledge. Built upon these findings, this paper propose Eagle, the results somehow show the importance of basic design space.
weaknesses: Weakness:

1. The contribution is limited. PrismaticVLM[1] conducts similar empirical study on MLLMs, especially on the vision encoders. Although they only explored the combination of two encoders, they have achieved significant improvements while keeping a good efficiency. This paper extends the mixture of encoders up to six, but the performance only increases from 681.5 to 697 as shown in Fig.4, which is somehow trivial from my perspective. What's worse, it brings double parameters and slows down the inference by more than four times. 
2. Missing related discussion. It is really strange there is not any discussion with PrismaticVLM, considering these two paper focus so closely.

[1] Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models. ICML 2024

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: The clarity over some of the details in the paper can be expanded upon.

* It’s a bit worrisome that the benchmark used to arrive at the recipe (Table 2-5 and Fig.4) is the same benchmark used in the final results (Table 6.). This would be great if authors confirm that the validation set is used for Table 2-5 and Fig.4 and the test-set for Table 6 or provide some evidence that the recipe is not overfitting to this particular benchmark.
* There is a final finetuning stage where the entire model, i.e LLM + projector + vision encoder is finetuned. In Table 7, it is mentioned that this is done on Eagle 1.8M, however on L141-L142, it is mentioned that this is done on multi-model conversation data. Further clarification on this will be helpful.
* Similarly, in Table 7 the best model is pretrained on LLava-595K + Eagle 1.8M. However, it seems Table 2-5 and Fig 4, the recipe is arrived at by only pretraining on Eagle 1.8M. Can the authors confirm this and clarify this in the paper?
* What is the effect of unfreezing vs freezing the image encoder on training time? Table 2 shows the number of flops are the same for both freezing and unfreezing. But while unfreezing one has to backpropagate through the encoder , so the flops should be different.
* What is the difference between 5th row of Table 5 (Unfreeze yes + prealign no + clip + convnext) and 2nd row of Table 4? In both cases, clip + convnext is used with channel concatenation and pre-alignment switched off. So why are the numbers different?
* The empirical comparison of flops between tiling and interpolation is provided in Table 2. I assume there is a tradeoff in the sense that tiling requires more forward passes but the self-attention might be less expensive because of the lesser number of tokens . So It would also be nice if some theoretical comparisons are also provided between interpolation and tiling with respect to number of tokens and resolution.
* I may have missed it but was the size of the Clip encoder L? And what resolution was the pretraining done on?
soundness: 3
strengths: The paper is well written. They start with a base clip image encoder model and build towards the final recipe using careful ablations. They show that a simple recipe without any fancy tricks can lead to strong performance on a number of benchmarks. The paper also achieves strong performance on VLM benchmarks and is likely to have an impact on multimodal LLMs.
summary: The paper explores strategies to inject information (visual tokens) from a mixture of pretrained visual encoders (experts) into LLM’s (Vicuna-12B) for finetuning MMLM’s. Namely, they explore how to incorporate: 1) High Resolution inputs which are needed for text understanding. 2) A number of visual encoders. Based on their ablation strategies, they propose the following training recipe.

* Simple linear interpolation for adaptation to high resolution.
* Unfreezing the image encoder instead of just the projection layer
* Channel concatenation to incorporate multiple vision encoder outputs.
* Prealigning the visual encoders with a LLM.
* Simple greedy strategy to incorporate multiple vision experts.
weaknesses: The clarity over some of the details in the paper can be expanded upon. See below for detailed feedback.

Meta-review: It is well-known that different vision encoders have different strengths and weaknesses, which has motivated work on VLMs which consider a mixture of different vision encoders. This paper performs a thorough analysis of these mixtures of vision encoders. Reviewers appreciated the thorough experiments and insights, the quality of the writing and the "pre-alignment" proposed by the authors. Reviewers were initially positive about the paper, and the rebuttal phase, where the authors comprehensively addressed the remaining concerns and revised their paper, improved the reviewers impressions even further. Therefore, the decision is to accept the paper.

Decision: Accept (Spotlight)