Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders
Authors: Yizhou Wang
Song Mao
Yang Chen
Yufan Shen
Pinlong Cai
Ding Wang
Guohang Yan
Zhi Yu
Yinqiao Yan
Xuming Hu
Botian Shi
Venue: ICLR 2026 Poster (2026)
Keywords: Large language Model
MLLMs
Vision Encoder
Machine Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: Adding more vision encoders to MLLMs often yields redundancy or even harms performance.

Abstract: Recent multimodal large language models (MLLMs) increasingly integrate multiple vision encoders to improve performance on various benchmarks, assuming that diverse pretraining objectives yield complementary visual signals. 
However, we show this assumption often fails in practice. Through systematic encoder masking across representative multi-encoder MLLMs, we find that performance typically degrades gracefully—and sometimes even improves—when selected encoders are masked, revealing pervasive encoder redundancy. 
To quantify this effect, we introduce two principled metrics: the Conditional Utilization Rate (CUR), which measures an encoder’s marginal contribution in the presence of others, and the Information Gap (IG), which captures heterogeneity in encoder utility within a model. 
Using these tools, we observe: (i) strong specialization on tasks like OCR \& Chart, where a single encoder can dominate with a CUR >90%, (ii) high redundancy on general VQA and knowledge-based tasks, where encoders are largely interchangeable, (iii) instances of detrimental encoders with negative CUR. 
Notably, masking specific encoders can yield up to 16% higher accuracy on a specific task category and 3.6% overall performance boost compared to the full model.
Furthermore, single- and dual- encoder variants recover over 90% of baseline on most non-OCR tasks. Our analysis challenges the “more encoders are better” heuristic in MLLMs and provides actionable diagnostics for developing more efficient and effective multimodal architectures.

Published as a conference paper at ICLR 2026
INVESTIGATING REDUNDANCY IN MULTIMODAL LARGE LANGUAGE MODELS WITH MULTIPLE VISION ENCODERS
Yizhou Wang1.2.*.1 Song Mao".", Yang 1.3x,, Yufan Shen', Yinqiao Yan*, Pinlong Cai', Ding Wang' Guohang Yan', Zhi Yu², Xuming Hu? Botian Shi!
'Shanghai Artificial Intelligence Laboratory
"The Hong Kong University of Science and Technology (Guangzhou)
ZZhejiang University, *Beijing University of Technology
(wangyizhoul, maosong, chenyang3]@pjlabb
ABSTRACT
Recent multimodal large language models (MLLMs) increasingly integrate multi- ple vision encoders to improve performance on various benchmarks, assuming that diverse pretraining objectives yield complementary visual signals. However, we show this assumption often fails in practice. Through systematic encoder masking across representative multi-encoder MLLMs, we find that performance typically degrades gracefully-ande sometimes even improves-wiie selected encoders are masked, revealing pervasive encoder redundancy. To quantify this effect, we in- troduce two principled metries: the Conditional Utilization Rate (CUR), which measures an encoder's marginal contribution in the presence of others, and the Information Gap (IG), which captures heterogeneity in encoder utility within a model. Using these tools, we observe: (i) strong specialization on tasks like OCR & Chart, where single encoder can dominate with a CUR > 90%, (ii) high redundancy on general VQA and snowledge-baser tasks, where encoders are largely interchangeable, (iii) instances of detrimental encoders with negative CUR. Notably, masking specific encoders can yield up to 16% higher accuracy on a specific task category and 3.6% overall performance boost compared to the full model. Furthermore, single- and dual- encoder variants recover over 90% baseline on most non-OCR tasks with substantially lower training resources and inference latency. Our analysis challenges the "more encoders are better" heuristic in MLLMs and provides actionable diagnostics for developing more ef- ficient and effective multimodal architectures. The project website is available https:/ / tig....t......... com wwt.c/oo//////. www.////oooggg22//M..oooooo Fnccsennnnn
INTRODUCTION
Multimodal large language models (MLLMs) have marked major leap in artificial intelligence (AI), exhibiting remarkable prowess integrating visual and textual information for complex gen- eration and reasoning tasks (OpenAl 2025a DeepMind] 2025; Anthropic, 2024: Bai et al. 2025 Zhu et al. 2025). Their ability to interpret images (Luo et al. (024). answer visual questions (hu
al. 2025 Li al., 2025a). and perform visual reasoning (OpenAl 2025b Peng et al. 2025 has
positioned them at the forefront of AI research.
A prominent architectural trend for enhancing visual capabilities of MLLMs is the incorporation of multiple, distinct vision encoders. The rationale is intuitive: different encoders, pre-trained with varied objectives or architectures, could capture complementary aspects of vision- iaaionnspanning global
semantics (Radford et al., 2021 Zhai et al., 2023) to fine- finegrrineed pixel-level details (Oquab et al.)
2023
Kirillov et al.] 2023), thereby providing a richer representation to the language model (Tong
et al.
2024b Lu et al. 2024a Jiang et al. 2024 Shi et al. 2024 Tong al. 2024a Li et al. 2024
Equal contribution, This work done during internship at Shanghai AT Laboratory.

---

Published as a conference paper at ICLR 2026
Encoder Contribution
LLM
More Encoders #
Better
MLP
Tokenizer
000000
How many dogs this imuge?
Eacedar
Eneake2
Enieder
Excakrr
An illustration of encoder redundancy. Different vision encoders provide similar or conflict visual cues, by ablating one several of them the performance maintain improve.
Figure 1:
However, the assumption that more encoders are always better is increasingly being challenged. Emerging evidence suggests that performance gains from additional encoders are often marginal. and in some cases, multi-encoder models even underperform their counterparts with fewer (Shi al.) 2024 Fan et al. (2024). This counterintuitive outcome reveals critical, yet underexplored issue: coder redundancy. Such redundancy occurs when encoders provide overlapping or conflicting cues, leading to fusion difficulties, distraction from irrelevant signals, and inefficient use of computational Iillustrates the encoder redundancy phenomenon, where a multi-encoder MLLM specific encoder accomplish domain-specifer task.
resources. Figure I
strongly depends
While significant research has focused on designing sophisticated fusion mechanisms for multi- encoder MILMMs (Kar al. 2024 Wang et al.) 2025 Shen al. 2024)), the fundamental question of whether and to what extent each encoder provides unique, non-redundant information remains largely unexplored. To address this gap, we first empirically demonstrate the presence redundancy by systematically masking individual vision encoders in representative multi-encoder MLLMs (c.g., Eagle (Shi et al., 2024)) and measuring the impact on performance across a wide range of bench- marks (Tong et al. (0224a) Second, we introduce the Conditional Utilization Rate (CUR), which quantifies the marginal contribution each encoder given the presence of others. A low or negative CUR indicates that an encoder is redundant or even detrimental. Building on this, we define the In- formation Gap (IG) as the difference between the maximum and minimum CUR values, capturing the disparity in encoder contributions. A large IG signifies poorly balanced encoder set, with some encoders dominating and others underutilized. Finally, we analyze factors such as LLM size and encoder type that influence redundancy.
Our experiments confirm that significant redundancy is prevalent in modern multi melti-encooer MLLMs. For instance, two-encoder subset of Cambrian-1 1 8B surpasses the full model by 1.7%, while mask- ing two encoders of Eagle-X5 7B retains 96% of its five-encoder baseline performance. Beyond ac- curacy, redundancy reduction also improves efficiency: in our setup, training dual-encoder variant is 1.53x x faster than its five-encoder counterpart while retaining 94% performance. Together, these results show that multi-encoder MLLMs often carry substantial redundancy: adding more vision en- coders to MLLMs often yields severely diminishing returns, resulting in performance gains that are negligible compared to the substantial increase in required training resources and inference latency. CUR and IG provide actionable diagnostics for designing more efficient and effective MLLMs by balancing training cost and performance.
In summary, this paper makes the following contributions:
1. We provide the first systematic, quantitative study confirming encoder redundancy and practical inefficiency in multi-encoder MLLMs.
2. We introduce CUR (Conditional Utilization Rate) and IG (Information Gap) as principled, model-agnostic metrics to quantify encoder utility and imbalance.
3. We demonstrate that dual-encoder variants achieve 90% of full performance at signifi- cantly reduced computational cost, offering an actionable framework for efficient MLLM design.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
2.1 MULTIMODAL LARGE LANGUAGE MODELS
The landscape of MLLMs has evolved rapidly, with models demonstrating increasingly sophis- ticated capabilities for understanding and generating multimodal content. Early influential work like Flamingo pioneered the integration of pre-trained vision encoders and Large Language Models (LLMs) by introducing mechanisms like resamplers for token reduction and cross-attention layers for feature fusion (Alayrac et al. 2022). Subsequently, LLaVA presented a simpler yet effective architecture consisting of a vision encoder, an LLM,, and a projection layer, establishing a modu- lar paradigm that facilitated scalability and adaptation (Liu et al. 2024). Efforts to enhance visual processing have included mPLUG's 's visual abstractor for handling high-resolution inputs (Li et al.) 2022) and Intern dynamic aspect ratio matching (Chen et al. 2024). Similarly, Qwen-VL series introduced techniques like 2D-RoPE and M-RoPE to better model inter-modal relationships (Wang) jet al., 2024a, Bai et al., 2025). These advancements underscore a continuous drive towards richer visual understanding and more effective vision-language alignment in MLLMs.
2.2 EMPLOYING MULTIPLE VISION ENCODERS IN MLLMS
Our research directly engages with the growing body of work on multi-encoder MLLMs. The pri- mary motivation behind multi-encoder MLLMs architectures to harness diverse visual features by combining encoders pre-trained with different objectives or on varied data. For instance, DeepSeek- VL (Lu et al., 2024a) integrates SigLIP (Zhai et al., 2023) for semantic understanding and SAM- B (Kirillov al. 2023) for visual grounding. HiLight (Wang al.) (024b), Mini-Gemini (Li al. 2024). and CogAgent (Hong et al. 2024) employ dual encoders to capture features at varying levels of granularity. Other models like SPHINX (Lin et al., (023) and Cambrian-1 (Tong al. 2024a) have explored using up to four distinct encoders. Several works have focused on the architectural aspects of fusing information from multiple encoders. I-MoF (Tong al. 2024b) uses separate projection layers for its two encoders, while Vary (Haoran et al.) 2023) extends the vocabulary to manage inputs from different visual sources. Prismer (Liu et al. al.] (2023a) utilizes an expert resampler for outputs from an ensemble of experts. CoMM (Jiang et al. (2024) investigated effective combina-
tions, finding CLIP (Radford et al., (02T) and DINO (Oquab al. 2023) to be potent, while noting that MAE (He et al.) (022) and DeiT (Tourrn et al., 2021) performed less effectively as visual branches. More recently, CLIP-MOE (Zhang et al. (2024b) proposed a model-agnostic strategy for building CLIP with a miixture-of-expeee
approach. While these studies have significantly advanced the capabilities of MLLMs, their primary focus has been on achieving state-of-the-an performance or enabling new functionalities. The critical question of encoder redundancy, i.e., the extent to which additional encoders provide unique, non-overlapping information has received less direct attention. Although works like Eagle (Shi al., 2024) and Mousi (Fan al., (2024) have reported diminishing returns, our work is the first to propose a formal framework and principled metrics (CUR, IG) systematically quantify and diagnose this redundancy, thereby enabling a deeper un- derstanding of the efficiency and necessity of each one.
2.3
VISUAL TOKEN SELECTION
To better exploit such complementary experts and avoid overwhelming the language model with re- dundant visual tokens, a growing line of work introduces explicit expert and token-selection mech- anisms. Mo VA adaptively routes and fuses task-specific vision experts via a coarse-to-fine mixture- of-vision-experrs adaptor (Zong et al. 2024), while Mixture-of- EisodesEnnoodess style frameworks such as MOVE and Mixpert route each input to the most suitable encoder or expert branch instead of activating all experts uniformly (Skripkin et al., 2025 He al. 2025). Orthogonal to expert routing, LEO-MINI performs conditional token reduction, consolidating large pool of visual tokens into a compact, query-aware subset before feeding them to the LLM (Wang et al. 2025). While these works effectively mitigate inefficiency by either selecting experts or pruning tokens, they largely operate under the implicit assumption that integrated vision encoders inherently provide comple- mentary value-2 a premise we challenge in this study. Through systematic encoder masking, demonstrate pervasive redundancy (and even detrimental effects) among multi-encoder architec- tures, which existing selection strategies fail quantify address fundamentally.
3

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
This section details our formal approach to investigating encoder redundancy. We first define the multi-encoder MLLM architecture (Section 3.1,, then introduce our proposed metrics for quantity- ing redundancy (Section 3.2
3.1 PROBLEM FORMULATION
We consider MILMs based on the prevalent "ViT-adapter--LLM" architecture (Liu et al., 2024 Bai al. 2025 Zhu et al. 2025). As illustrated in Figure I given image / and text prompt T. the output response Y multi-encoder MLLM with a set of n vision encoders E, = {Ei..... E,, is generated as:
Y Je, = [M
F1 (r), (n)),
where fusion (.) combines features from the different encoders (e.g., concatenation (Lu et al. 2024a Tong et al. 2024b) (Shi et al.) 2024) attention- fusion (Li et al. 2024)) and proj( is an adapter that aligns visual features with the LLM's embedding space (Liu et al. 2024).
While multiple encoders can theoretically provide more comprehensive visual information, they also introduce noise, conflicting signals, or critically, redundant information. Such redundancy arises when encoders learn overlapping features when some encoders supplies information al- ready captured by others. We define encoder redundancy as a scenario where including an encoder (or subset of encoders) does yield to meaningful performance improvement, even causes degradation. Formally, encoder redundancy is observed if removing one or more encoders does not harm improve performance. This implies that the information from those encoder is either redundant or detrimental, making their computational cost and architectural complexity unjustified.
3.2 QUANTIFYING ENCODER CONTRIBUTION AND REDUNDANCY
To move beyond observations, we introduce two metrics quantify the utility of each encoder within multi-encoder system.
Conditional Utilization Rate (CUR) The Conditional Utilization Rate (CUR) an encoder E; measures its unique contribution relative to the full encoder set E,;
2 (E,) acc(fe,) acc(ff.
acc(fE.,)
where fe., (EE)) denotes the MLLM with E, masked (e.g.. replaced by zero tensor), and acc(-) is the accuracy on benchmark evaluations (Appendix (A). Since acc(-) E [0, u u(E,) E (-oo, 1]. A large positive u( (.))) indicates substantial unique contribution; values near zero imply redundancy; and negative values show that the encoder is detrimental, introducing conflicting or noisy features.
Information Gap (IG) Building on CUR, we define the information gap Agap for an encoder set E,
iEl...... Agop == max u( min u(E)).
jel...,m
The IG measures disparity in encoder contributions. A small Agap suggests balanced utility across encoders, while large Agap highlights severe imbalance: some encoders are indispensable while others are redundant or harmful. Such imbalance indicates inefficiency in the architecture, redun- dant encoders inflate computational cost without improving performance.
Together, CUR and IG provide rigorous framework for quantifying encoder contributions and characterizing redundancy in multi-encoder MLLMs.
4 EXPERIMENTS
Our experiments are designed to: (1) empirically validate the existence of encoder redundancy sce- narios across different MILLM architectures, and (2) apply our proposed CUR and IG metrics to
4

---

Published as a conference paper at ICLR 2026
quantify the encoder redundancy. We first introduce the experiment setup in Section 4.1 Then, we quantitatively analyze the contribution of each vision encoder or combinations in Section 4.2
Finally, we analyze the key factors that contribute to this phenomenon in Section 4.3
4.1 EXPERIMENTAL SETUP
Baseline Models Eagle (Shi et al. 2024) represents MILLMs designed with larger ensemble of
encoders (typically 4 or 5), including CLIP (Radford et al. 2021). ConvNext (Liu et al. SAM (Kirillov et al., 2023). EVA-02 Fang et al.) 2024). and Pix2Struct (Lee al. Eagle primarily uses channel concatenation for feature fusion. Cambrian-1 (Tong et al. introduces a vision-centric approach with novel fusion mechanism called Spatial Vision Aggre- gator (SVA). Rather than directly feeding all image tokens to the LLM, SVA uses cross-attention with learnable queries to integrate features from multiple encoders, including CLIP (Radford et al.) 2021), ConvNext (Liu et al. 2022). SigLIP IP (Zhai et al., 2023) and D INO (Oquab et al. 2023), which offers contrast to simpler concatenation methods and allows us to study if more sophisti- cated fusion mitigate redundancy. These model architectures provide excellent testbed for investigating redundancy in systems with many specialized encoders, allowing us to calculate CUR for each and assess the overall IG. We provide
2022)
2023
2024a)
detailed introduction in Appendix B
Evaluation Benchmarks To assess MLLM performance and analyze redundancy across diverse capabilities, we adopt the benchmark categorization proposed by Cambrian-1 (Tong et al. 2024a) which groups common benchmarks into four distinct categories based on principal component analysis: (1) General; (2) Knowledge; (3) OCR & Chart; (4) Vision-Centric. Using these categories (detailed in Appendix A) allows for a nuanced understanding of how encoder redundancy might manifest differently depending on the task demands. All evaluations are performed using standard- ized protocols, primarily leveraging VLMEvalKit (Duan et al. 2025) for consistency.
4.2 EVIDENCE OF PERVASIVE ENCODER REDUNDANCY
In this section, we systematically probe the multi-encoder systems to demonstrate that significant redundancy is an inherent characteristic of current architectures.
Performance Resilience to Encoder Ablation. To evaluate redundancy, we consider all 2" en- coder combinations for model with n encoders. When an encoder masked, we replace its output with a zero tensor of the same shape. Figure 2 shows the distribution of overall performance for several multi-encoder MILLMs with respect to the number masked encoders. The results reveal a consistent trend: performance of multi-encoder MILMS degrades gracefully rather than catastroph- ically as specific encoders are removed. For instance, the best-case performance of Eagle-X5 7B de- creases by under 4% when 3 specific encoders are masked; the optimal performance of Cambrian-1 8B is achieved with subset of 3 vision encoders, which is 3.5% higher than the full model. These findings validate that multi-encoder MILMs can maintain most of their capabilities with only subset of encoders, implying that additional encoders often yield diminishing returns and introduce computation inefficiency.
Quantifying Specialization with CUR and IG. We next employ CUR and IG to quantify the unique contribution of each encoder's unique contribution. Figure 3 presents the CUR across bench- mark categories. The results indicate strong specialization for tasks such as OCR & Chart, where CUR values are extremely high. For example, in Eagle-X4 8B, EVA- achieves a CUR of 92.89% on OCR & Chart, and in the Cambrian-1 series, ConvNext contributes with CUR above 70%. This demonstrates that specific encoders dominate OCR-related tasks. In contrast, for Knowledge and General categories, CUR values are much lower, suggesting that encoders provide more homo- geneous semantic features and are largely interchangeable. Notably, some encoders exhibit negative CUR values. For instance, in Cambrian-l 8B, SigLIP attains a CUR 166 on the Vision- Centric category, indicating that its inclusion is detrimental- that the fusion mechanism cannot resolve. To assess disparity more directly, we analyze IG. Table shows that models with more than encoders tend to have larger IG, reflecting greater redun- dancy. This imbalance is most evident in OCR & Chart and Vision-Centric categories, consistent with CUR results: single encoder typically dominates performance for a given task, while others
introducing conflicting signals
5

---

Published as a conference paper at ICLR 2026
HMff 138
Eegle2 9B
DeepSeek-VL 1.30
51
38-
70
48
05
80
45
X
32
55
42
39
0
Ve
Cambrian-1
Cambrian-1
65
56-
60
00
55
55
35
35
30
Cambrian-1 38
Eagle-X4 68
Eagle-X5 78
60
40
36
24
10
36
Number Masked Erooders
Number Masked Encoders
Number Masked Encodors
Max
Min
Mean
mole
Figure 2: Performance of multi-encoder MLLMs with different number of masked vision en- coders. Max, Min and Mean refer to the subset of ablated encoders with best, worst and average performance among all possible subsets respectively.
contribute minimally or not at all. As result shows, MLLMs with more than encoders tend to have larger IG, indicating that a greater number of encoders leads to increased redundancy. This imbalance is emphasized on OCR & Chart and Vision-Centric categories, which is consistent with CUR results, that is, specific encoder dominates the contribution to particular type of task, and this dominance is fixed, meaning that the model relies on this encoder while largely ignoring the others when performing these tasks.
Table 1: Information Gap of vision encoders multi-encoder MILMMs. A higher value indicates higher imbalance across different vision encoders.
Model
General
Knowledge
OCR & Chart
Vision-Centric
Overall
Eagle-X5 7B
9.89%
5.68%
30.17%
17.19%
11.48%
Eagle-X4 8B Plus
85.41%
55.83%
92.89%
50.14%
70.27%
Cambrian-1 3B
7.30%
8.09%
66.47%
8.45%
22.42%
Cambrian-1 8B
4.03%
11.59%
73.07%
21.77%
26.24%
Cambrian-1 13B
9.62%
14.87%
76.22%
12.28%
27.82%
1-MoF 13B
51.64%
6.69%
80.92%
23.30%
40.63%
Eagle2 9B
2
10.50%
0.23%
28.03%
8.79%
2.24%
DeepSeek- VL 7B
1.18%
0.50%
0.51%
2.43%
1.15%
Fewer Encoders, Comparable Performance. Having established redundancy, we next examine whether comparable accuracy can be achieved with fewer encoders. We evaluate Eagle-X5 7B,

---

Published as a conference paper at ICLR 2026
General
Knowledge
HMaf 138
HMof
VeepSeek-VL
DeepSeekVL 7B
Eagle2 98
Eagle2 98
Eagle-X4 86 Plus
Eagle-X4 88 Plus
Eagle-X5
Eag e-xs 78
Cambrian 1 138
Cambrian 138
Cambrian-1
Cambrian-1 88
Cambrian-1
Cambrian-1 38
04
20%
60%
62%
30%
0%
10%
20%
30%
40%
52%
cun (4)
cuR
OCR & Chart
Vision-Centric
I-MoF 138
I-MoF 138
DeepSeek-VL
DeepSeek-VL 78
Eagle2 98
Eagle2 98
Eagle-X4
Eagle-X4
Eagle:X5 78
Eagle:X5 78
Cambrian-1 138
Combrien-1
Cambrian-1
SqUP
Corwrice
DRVO22
CLP
SAM
2VA
Figure 3: CUR of different encoders across different category of benchmarks. higher CUR means larger dependence specific encoders.
Eagle-X: X4 8B Plus, and Cambrian-1 8B, in which EVA-02 (Fang et al. 2024) and ConvNext (Liu al. 2022) are the dominant contributors, respectively. Progressively masking encoders, we mea- sure the resulting performance. As shown in Table B masking two encoders reduces Eagle model performance by only 1%, while the same operation increases Cambrian-1 8B performance by 1.7%. Across non-OCR tasks, single-encoder variants of the Eagle series retain at least 90% of the full model's performance with five encoders, showing that one strong encoder suffices for most ca- pabilities. For OCR & Chart tasks, which demand fine-grained text and structural cues, adding ConvNext as second branch substantially restores accuracy. From a resource perspective, ad- ditional encoders significantly inflate training costs: fine-tuning Eagle-X5 7B with five encoders requires approximately 94.57 A100 GPU hours, whereas removing three encoders reduces training time by 34.6% (See Appendix C for details). These results highlight practical trade-off: em- ploying one or two encoders recovers most accuracy while substantially lowering training time and computational cost.
Our CUR and IG metrics effectively quantify the severe imbalance in encoder contributions. Some encoders are highly specialized and indispensable for certain tasks like OCR & Chart (high CUR, high IG), while for other tasks like Knowledge-basee based tasks, encoders are largely interchangeable and redundant (low CUR, low IG). In summary, our analyses confirm that encoder redundancy both pervasive and predictable, suggesting that careful encoder selection can preserve accuracy while substantially improving efficiency in multi-encoder ML.MSs.
Computation efficiency. Beyond accuracy, removing redundant encoders brings tangible savings in both training and inference cost. Table I7 reports the wall-clock time (on 8 NVIDIA A100 GPUs) for pre-training and fine-tuning Eagle-X5 7B and its variants. Moving from the full five- encoder E5gle-X5 7B its dual-encoder variant Eagle-X2 7B reduces the overall training time by

---

Published as a conference paper at ICLR 2026
34% while retaining over 96% of the original performance. A similar trend holds at inference time. Table 18 reports per-sample latency on MME under different masking configurations. For Eagle-X5 X5 7B, masking three encoders reduces end-to-end latency by 19.5% while maintaining more than 96% of its performance. Table further decomposes the vision FL.OPs of Eagle-X5 7B across encoders where ConvNext dominates the vision-side computation, and the full vision stack requires about 6.62 TFLOPs per image. For a typical yes/no QA with a 64-token prompt and a one-token answer, the vision processing accounts for roughly 31% of the total FLOPs. When we keep only the two most useful encoders identified by our CUR/IG analysis in Eagle-X2 7B, the rision-side FLOPs drop to 61.4% compared with Eagle-X: XS 7B, corresponding a 12.1% reduction in total FLOPs and 19.5% reduction in latency, with less than 4% average performance degradation. These measure- ments quantitatively demonstrate that eliminating redundant encoders yields substantial efficiency gains in both training and inference.
4.3 ANALYZING ON ENCODER REDUNDANCY
Previous experiments validate the existence encoder redundancy. In this section, we investigate its underlying causes.
Table 2: Attention score distribution analysis. KL Divergence of attention score distributions between full model and the model with only one encoder activated. Lower value indicates higher Similarity.
Model
CLIP
ConvNext
SAM
EVA-02
Pix2Struct
SigLIP
DINOv2
Eagle-X5 7B
2.658
3.004
2.537
0.982
2.959
Eagle-X4 8B Plus
1.007
oc
80
0.392
Cambrian-1 8B
0.102
0.080
0.095
128
Attention Analysis. While the IG values for the Eagle and Cambrian-1 series are relatively high, the outstanding CUR of EVA-02 and ConvNext in these two series suggest that the outputs of these encoders dominate the visual representation respectively. To further investigate these dispar- ities, we perform an attention based analysis using the Eagle series and the Cambrian-1 8B model. Specifically, we evaluate each model on MME (Fu al., 2024) with particular encoders selectively activated (n = 1), and extract the attention scores of visual tokens from the final layer of the LLM. Using the full model as baseline, we then compute the Kullback-Leite
(KL. divergence between its attention distribution and those obtained when only single encoder is activated. The results of this analysis are reported in Table For Eagle series, the combination of high IG and the CUR dom- inance of EVA-02 is reflected in the attention maps, which further confirm EVA-02 02 the primary contributor to the visual features. In contrast, for the Cambrian-1 series, ConvNext emerges as the most influential encoder. For Eagle-X4 8B Plus, the infinite KL for ConvNext and SAM indicates support mismatch, where these single-encoder runs place zero mass on positions that the full model attends to, which is consistent with EVA-02-dominatl a clear imbalance of encoder contributions during inference in multi-encoder MLLMs, with certain encoders contributing minimally, suggesting that some may be functionally redundant within the architecture. For more detailed analysis results, please refer to the Appendix
attention pattern. This analysis reveals
The Role of Encoder Pre-training. An encoder's pre-training objective and number of param- eters largely governs the kind and quality of visual evidence it supplies to an MLLM. Firstly, the size of an encoder irrelevant to its final contribution, EVA-02, an encoder with 304M parame- ters, dominates the performance in Eagle-X4 8B Plus compared to P pix2Struct, which has 1 1.2B parameters. Secondly, encoders pretrained with the same objective may ultimately lead to different levels of contribution. ConvNext, CLIP and SigLIP are all pretrained via contrastive way, however, as shown in Figure 3 ConvNext achieves a higher CUR. Finally, different combinations of encoders may perform differently due to differences in model architecture or training data. For non-OCR tasks. SigLIP and CLIP show distinct interaction patterns within the Cambrian- se- ries. These findings highlight design trade-off: assembling diverse, specialized encoders curb
8

---

Published as a conference paper at ICLR 2026
Table Robustness of specific vision encoders with respect masking operation. Performance comparison of Eagle-X5 7B (EVA- 02. + ConvNext; + +Pix2Struct CLIP; + SAM),, Eagle- X4 8B Plus (EVA-02p + ConvNextı + Pix2structz CLIPs), Cambrian-1 8B (ConvNexto + DINOV21 CLIP₂ + SigLIPs) and DeepSeek-VL. 7B (SAMo + SigLIP1) against masking
operation. The subscript such as 0123 refers to retained encoder index.
Model
Y2
General
Knowledge
OCR Chart
Vision-Centric
Overall
Bagle-X5 7B
5
70.77
54.79
66.60
67.55
64.93
X4 0123
70.64 %
54.19
L %
66.55 [ 0
67.39 [ %
64.69
10.3%
X3 012
3
69.87 111 %
53.64
22%
66.02 J %
67.29 1.%%
64.20
11.1%
-X2 01
2
69.04 4. %
52.77
4.4%
62.01 7 %
66.05
2 %
62.48
1338%
-X1
64.60
47.70
113%
10.68 (84%
62.83 7
46.45 28%
Eagle-X4 X4 8B Plus
4
66.49
61.88
71.92
70.62
67.73
-X3 012
65.68 1
61.57 1
71.97 1
70.50 4 %
67.43 10.4%
-X2 01
67.28 %
59 59.83/ 2
70.57 1 %
69.60 0 %
66.82 11.15%
XX g
64.22 3 %
51.21 117%
9.14 183%
66.68 J
47.81 29%
Cambrian-1 8B
67.47
57.88
70.08
56.65
63.02
-X3 012
3
69. 29
58.05 0 %
67.97 3.3 %
65.81
166
65.28 13.6%
-X2 03
f 68.64
58.09 0 %
66.41 5
63.24 1
64.09 11.7%
-X1
57.04 115%
53.72 7 %
60.57 J14%
55.93 %
56.82 19.8%
DeepScck- VL 7B
2
69.84
52.37
53.96
61.83
59.50
X1
60.60 L 13 %
46.38
11
27.53 4.49 %
55.89 110%
47.60 L20%
XI
1
61.42 12 %
46.64
11 %
27.80 4:48 %
57.40 7 7%
48.32 L19%
redundancy on targeted skills, but risks under utilization and inefficiency on broad tasks; conversely, stacking multiple semantically similar encoders amplifies redundancy with limited aggregate gain.
Number of Vision Encoders. The number of encoders should also be considered when studying encoder redundancy. Both Fagle (Shi et al., 2024) and MouSi (Fan et al.) (024) performs ablation studies on number of encoders. According MouSi, MILMS with encoders outperform those with a single encoder in most cases (8/9), However, when extended to three encoders, the winning case ratio drops to 4/6. The results of the experiment in Table shows a similar trend, that is, the dual-encoder architecture is trade-off between performance and efficiency. When increasing the number of encoders, the performance improvement becomes marginal. When adopting only single encoder, performance on specialized tasks such as OCR & Chart drops.
LLM Capacity. Our investigation into the impact of model scale on encoder redundancy, as shown in Table [ reveals that while larger models achieve higher performance, they also exhibit more pronounced redundancy (See Table 15 for details). For Cambrian-1 series, masking single encoder in the 13B model results performance that can either remain high as 98.6% of the full model or drop to 70.2%, depending on which encoder is removed. This wide variance substantially larger than that observed in the 8B and models, which indicates greater disparity in encoder contributions within the larger model. The ability of the 13B model to sustain near-peak performance even when its least important encoder is masked suggests a high degree of informational overlap. Consistently, the IG A Agap increases from 22. 22.22% to 27.82% the LLM size grows. These findings strongly suggest that encoder redundancy becomes more pronounced as LLMS scale up, rendering larger models both more robust to the loss of individual encoders and less efficient in their architectural design.
Ablation Study on Masking Operation. We selected zero-masking which replaces an encoder's output with zero tensor for its simplicity in our main analysis. To validate this choice, we compare with mean-masking which instead uses the feature's mean value. While both methods perform similarly when masking few encoders, mean-masking is significantly more robust in the extreme single-encoder setting (n = 1), where nearly matches the full model's performance (Table The remarkable effectiveness of simple mean value strongly highlights the redundancy of the other

---

Published as a conference paper at ICLR 2026
Table Ablation study on masking operation. Zero masking and mean masking replace specific encoders' output image features with the same-shaped zero tensors and their element- -wise mean, respectively.
Model
72
MMBench
MMVP
ScienceQA
TextVQA
Eagle-X4 X4 8B Plus
4
71.39
71.00
80.16
66.29
X3 012 (Zero)
3
70.10 2 %
70.67 4. %
80.16 1.0 %
66.15 4. %
X2 01 (Zero)
70.62 + %
70.67 4.0
79.64 4
65.91
-X1 (Zero)
62.29 113%
66.00 7 %
72.06 110%
10.67 184%
X3 012 (Mean)
69.85 4 2 %
70.00 4 %
79.97 4
65.90 %
-X2 01 (Mean)
2
69.85 | %
69.00 3 %
79.59 $ %
65.86
%
-X1 0 (Mean)
69.76 $ 2 %
69.00 3
79.97 4
65.96
0 %
encoders. This confirms that our choice of the simpler zero- pero-masking operation serves as an effective albeit more stringent method for quantifying encoder contribution.
Discussion The Platonic Representation Hypothesis Huh et al. 2024 posits that large models trained similar data develop increasingly aligned representation spaces. Since multi-encoder MLLMs use vision encoders trained on overlapping, web-scale corpora, their feature spaces natu- rally converge. Our empirical that adding more encoders yields diminishing returns and that encoders are largely interchangeable-iis a direct manifestation of this representational conver- gence.
LIMITATION AND CONCLUSION
Limitation First, our performance evaluation focuses on standardized benchmark accuracy across four core multimodal task dimensions (General, Knowledge, OCR & Chart, Vision-Centric), without explicitly assessing robustness, out-of-distribution generalization, calibration, or fair- ness diseenionss requiring specialized datasets. While our CUR and IG metrics are metric-agnostic (extensible these axes via task-specific scores). systematic validation of this extensibility demands dedicated protocols, which we leave for future work. Second, our analyses are limited to pre-trained, static multi-encoder MILMs (fixed fusion, no dynamic routing). This choice leverages their status mainstream foundational architectures and stable input-output mappings for reliable redundancy quantification. Dynamic routing, by contrast, causes variable encoder activation weights, destabiliz- framework cannot disentangle intermediate redundancy
ing CUR calculations. We acknowledge from encoder-LLLM alignment in dynamic models. Finally, a priori prediction of multi-encoder MLLM performance e.g., scaling laws for encoder count- unresolved. Our CUR/IG met- rics quantify redundancy post-hoc but not upfront encoder selection, constraining their use in initial architecture design. Notably, they still enable actionable post-training optimization. Establishing pre-training scaling relationships remains key future direction.
Conclusion In this paper, we presented the first systematic, quantitative investigation into encoder redundancy in Multi-encoderr MLLMs. We introduced the Conditional Utilization Rate (CUR) and Information Gap (IG) as principled metries to quantify encoder contribution and utility disparity. Our comprehensive analysis. including combinatorial ablation and efficiency trade-off studies, vealed three core findings: (1) Pervasive Redundancy: Adding more encoders often yields severely diminishing returns, resulting in performance gains that are negligible compared the substantial increase in required training resources (i.e., practical inefficiency): ) (2) Dominance and Specializa- tion: Performance is often dominated by single key encoder small complementary pair, leading to large Information Gap (IG); (3) Efficiency Breakthrough: We demonstrated that dual-encoder variants can consistently recover over 90% of the full model's performance with significantly re- duced computational cost. These results fundamentally challenge the "more is better" heuristic. Our work provides an actionable diagnostic framework (CUR/IG) to guide the design of future efficient, resource-optimized MLLMs.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
The research was supported by Shanghai Artificial Intelligence Laboratory.
ETHICAL CONSIDERATIONS
Ethical considerations Our study analyzes and optimizes multi-encoder MLLMs, which raises several ethical points. First, efficiency gains (e.g., comparable accuracy with fewer encoders) have positive environmental implications by reducing GPU hours and energy use, but they also lower deployment barriers; efficient models could be repurposed for privacy-sensitive applications (e.g,, large-scale surveillance). We therefore discourage uses that violate privacy or local regulations and recommend task app-opprtee safeguards (rate limiting, face/plate filtering, audit logs). Second, pruning or down-weighting weighting encoders can introduce distributional harm: performance regressions may disproportionately affect specific scripts, chart styles, domains (e.g.. OCR for low-resource languages), accessibility use-cases. To mitigate this, advocate reporting per-subset metrics (language, domain, layout) alongside overall scores, and avoiding model changes that degrade crit- ical safety-reled relevant tasks. Third, our evaluation relies public benchmarks and M-baedddd judging; these may encode social and cultural biases and can over- or under-estimate real-world reliability. We encourage complementary human evaluation and transparency about judge models and prompts. Fourth, we use only publicly licensed encoders and datasets and avoid training on personal sensitive data; when working with document images, should be taken exclude PII and respect dataset licenses. Finally, revealing encoder-specific salience (via CUR/IG or atten- tion analyses) could be exploited to craft targeted adversarial inputs; we release results at aggregate levels and recommend standard robustness testing before deployment. Overall, our efficiency rec- ommendations should be paired with privacy, fairness, and robustness checks ensure that benefits do not come at the expense of vulnerable users.
CLAIM ON REPRODUCIBILITY
Claim on Reproducibility We commit that all reported results are fully reproducible, where the code and models will be open-sourced upon acceptance.
USE OF LARGE LANGUAGE MODELS
Use Large Language Models Large Language Models are mainly used for grammar check and polishing in this paper. The prompt used for polishing the paper is: "Please help me to check if there are grammar error, possible, provide some advice on how to improve my academic paper."

---

Published as a conference paper at ICLR 2026
REFERENCES
Man-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Anthropic.
The claude 3 model family: URL
Opus, sonnet, haiku. https://
www. www.anthropic.coom coml 2024.
aethhmannttiiiicco tt com/
debba9hocccabccaab33333 00818880100011186676776766688880 Card Claude 3laude_3.pdl 3.
Mozhgan Nasr Azadani, James Riddell, Sean Sedwards, and Krzysztof Czarnecki. Leo: Boost- ing mixture of vision encoders for multimodal large language models, 2025. URL https:
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint 13X1v22502.13923, 2025.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistc tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 24185-24198, 2024.
DeepMind. Gemini 2.5 pro. pro/ 2025.
https: http:://deeppiaddoooooe/eeeecccp gemiii
Haodong Duan, Xinyu Fang, Jumming Yang, Xiangyu Zhao, Yuxuan Qiao, Mo Li, Amit Agarwal, Zhe Chen, Lin Chen, Yuan Liu, Yubo Ma, Hailong Sun, Yifan Zhang, Shiyin Lu, Tack Hwa Wong, Weiyun Wang, Peiheng Zhou, Xiaozhe Li, Chaoyou Fu, Junbo Cui, Xiaoyi Dong, Yuhang Zang, Pan Zhang. Jiaqi Wang, Dahua Lin, and Kai Chen. Vimevalkit: An open-source toolkit for evaluating large multi-modality models, 2025. URL httpsstts:///
org/ ab2//205
11691
Xiaoran Fan, Tao Ji, Changhao Jiang, Shuo Li. Senjie Jin, Sirui Song, Junke Wang, Boyang Hong, Lu Chen, Guodong Zheng, Ming Zhang. Huangcaishuang. Rui Zheng. Zhiheng Xi, Yuhao Zhou, Shihan Dou, Junjie Ye, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Poly-visualexpper vision-language models. In First Conference on Language Modeling, 2024. URL
wtwp:e/wwwwweeeeeeuueuuueaaaaeaa iid- 7QaE09WYMa
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 149: 105171, September 2024. ISSN 0262-8856. doi: 10.1016/j.mmvis..22 2mavis.2024.10517. URL http: /xx doi .oi.org/
10. 10.1016 imaviiiiii.........m......iii..s 2024. 105171
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL
2336.13399
Yuying Ge. Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. arXiv preprint (xXXix:307.0004: 2023.
Wei Haoran, Kong Lingyu, Chen Jinyue, Zhao Liang, Ge Zheng, Yang Jinrong, Sun Jianjian, Han Chunrui, and Zhang Xiangyu. Vary: Scaling up the vision vocabulary for large vasion-language models. arXiv preprint arXiv:2312.06100 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Xin He, Xumeng Han, Longhui Wei, Lingxi Xie, and Qi Tian. Mixpert: Mitigating multimodal learning conflicts with efficient mixture-of of-urroo vision-expeessss 2025. htt.:htpss///axx.. orq/
abs/2505. 24541
12

---

Published as a conference paper at ICLR 2026
Tuomo Hiippala, Malihe Alikhani, Jonas Haverinen, Timo Kalliokoski. Evanfiya Logacheva, Sera- fina Orekhova, Aino Tuomainen, Matthew Stone, and John A Bateman. Ai2d-rst: A multimodal corpus of 1000 primary school science diagrams. Language Resources and Evaluation, 55:661- 688, 2021.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang. Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2024.
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for geal-world visual reasoning and compositional question answering. In CVPR, 2019.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic represen- tation hypothesis. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 20617-20642. PMLR, 21-27 Jul 2024. URL https: httth:://ttoceeeee httpss///ppcceeeeieeooo www.oceedingg..o ml ww..ainnsssmms..e....e..e .tt.errrrs/.gs....prrss press/
v235/ 22355/uh244..20 hhtm
Dongsheng Jiang, Yuchen Liu, Songlin Liu, XIAOPENG ZHANG, Jin Li, Hongkai Xiong, and Qi Tian. From CLIP DINO: Visual encoders shout in multi-modal large language models, 2024. URL
Oguzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. Brave: Broadening the visual encoding of lisoun-aaggaage models. arXiv preprint arXiv:2404.07204, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference Computer Vision, pp. 4015 2023.
Kenton Lee, Mandar Joshi, Tulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khan- delwal, Peter Shaw, Ming-Wos Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding, 2023. URL aattps://azwe /ttp//xxxi...g.ss//// org/abs/
[2210 1210.00 03347
Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hchong Chen, Guohai Xu, Zheng Cao, et al. mplug: Effective and efficient lision-language learning by cross- modal skip-connections. arXiv preprint arXiv:2205. / /2005, 2022.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun MA, and Chunyuan Tackling multi-image, video, and 3d in large multimodal models.
Li. LLaVVA-neT-inntrrfle In The Thirteenth International Conference on Learning Representations, 2025a. URL https:
7 //pperevvew.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 11003..88114 18814, 2024.
Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang, Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, Nadine Chang, Karan Sapra, Amala Sanjay Deshmukh, Tuomas Rintamaki, Matthieu Le, Ilia Karmanov, Lukas Voegtle. Philipp Fischer, De-An Huang, Timo Roman, Tong Lu, Jose M. Alvarez, Bryan Catanzaro, Jan Kautz, Andrew Tao, Guilin Liu, and Zhiding Yu. Eagle 2: Building post-training data strategies from scratch for frontier vision- language models, 2025b. URL URt/ttpss:// httpp::///axxv...// org/ abs 250..
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint xrXiv:2311.0755..22 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2026
Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandkumar. Pris- mer: A lisionlaagguage model with multi-task experts. arXiv preprint arXiv:2303.02500 2023a.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: your multi-modal model an all-around player? arXiv preprint arXiv:2307.0628/ 2023b.
Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng. Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al. On the hidden mystery ocr in large multimodal models. arXiv preprint arXiv:2305.0788 2023c.
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang. Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint ar 2rXiv:2403.05525, 2024a.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-We Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024b.
Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: Mfxture--ffrrsolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003, 2024.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. In ACL, 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: dataset for vqa on document images. In WACV, 2021.
OpenAl. Gpt-4o system card. httppss/pppeaa..commm//e/ee////p 2025a.
OpenAl. OpenAl 03-mini: Pushing the frontier of cost-effective reasoning. https://opent
com/index: inwiiiidxx/iiii wwneinoopeaiiiiiiiiieio opt.///oaaaammmmiiiae- January 2025b.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini arXiv:2304.071 193, 2023.
Yi Peng, Peiyu Wang, Xiaokun Wang, Yichen Wei, Jiangbo Pei, Weijie Qiu, Ai Jian, Yunzhuo Hao, Jiachun Pan, Tianyidan Xie, Li Ge, Rongxian Zhuang, Xuchen Song, Yang Liu, and Yahui Zhou. Skywork rlv: Pioneering multimodal reasoning with chain-of-thought, 2025. URL https:
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning, 2021. URL https: http:::/app..aamrre.. spp.ssematiiscool.....
arg/CorpusTD:20 23159:23199 445
Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Mome: Mixture of multi- modal experts for generalist multimodal large language models, arXiv preprint 2rXiv:2407.12709 2024.
14

---

Published as a conference paper at ICLR 2026
Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catanzaro, Andrew Tao. Jan Kautz, Zhid- ing Yu, and Guilin Liu. Eagle: Exploring the design space for multimodal Ilms with mixture of encoders. arXiv:2408. 15998, 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019.
Matvey Skripkin, Elizaveta Goncharova, Dmitrii Tarasov, and Andrey Kuznetsov. Move: A mixture-of-vissio--nncdees approach for domain-focused vision-language processing, 2025. URL /tRpss///////
abs/2502. 15381
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-I: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406. 16860, 2024a.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9568 9568-9578, 2024b.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference machine learning, pp. 10347-10357. PMLR, 2021.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing lasionnlanguage model's perception of the world at any resolution. arXiv preprint arXiv:2409.12199 2024a.
Yimu Wang, Mozhgan Nasr Azadani, Sean Sedwards, and Krzysztof Czarnecki. Leo-mini: An efficient multimodal large language model using conditional token reduction and mixture multi- modal experts, 2025. URL
abs 2504. 04653
Zhiting Wang, Qiangong Zhou, Kangjie Yang. Zongyang Liu Mao, al. Hilight: Technical report on the motern ai video language model. arXiv preprint arXiv:2407.07325, 2024b.
XAI. grok, 2024. URL SRLLhttts:://// aaaaalabbllll httbbllggroooo....//
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975-11986, 2023.
Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong- You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, and Yinfei Yang. Ferret-v2: An improved baseline for referring and grounding with large language models. In First Conference on Lan- guage Modeling, 2024a. URL : http//ppwwwee..eeeeeeee. nwt./nneeeeeeeeeeee Btttt//oommmm--PPPP222222
Jihai Zhang, Xiaoye Qu, Tong Zhu, and Yu Cheng. Clip-moe: Towards building mixture of ex- perts for clip with diversified multiplet upcycling, 2024b. URL https://arxiv. org/ abs/
2409. 19291
Jinguo Zhu, Weiyun Wang. Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo. Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang. Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min
15

---

Published as a conference paper at ICLR 2026
Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. In- ternv13: Exploring advanced training and test-time recipes for open-source multimodal models, 2025. URL URtpttps:: / arxiv. org/ abs /2zzgaab//550.10055 200.//0004...999.
Zhuofan Zong, Bingqi Ma. Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and Yu Liu. Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint arXiv:2404. 13046, 2024.
16

---

Published as a conference paper at ICLR 2026
A BENCHMARK AND EVALUATION DETAILS
The evaluated benchmarks, summarized in Table 5 are classified into four categories:
OCR & Chart: Emphasizes text extraction and reasoning from visual documents (e.g., ChariQA (Masry et al. 20022). OCRBench (Liu et al. 2023c), TextVQA (Singh et al.) 2019), Doc (Mathew et al. 2021)).
Knowledge-based VQA: Relies on world knowledge and complex reasoning integrated with visual perception (e.g.. SQA (Lu et al. (002). MMMU (Yue et al. 2024), Math-
Vista (Lu et al. 2024b), A12D (Hippala et al. 2021
Vision-Centric Tasks: Requires detailed visual understanding and focus on visual
attributes (e.g., MMVP (Tong et al. 2024b), RealWorldQA (xAI, 2024), CV-Bench
2D/3D TTogg et al. 2024a)).
General VQA: Assesses overall comprehensive visual understanding and reasoning (e.g.. MME (Fu et al.] 2024), MMBench (Liu et al.] 2023b), SEED-Bench (Ge et al. 2023
GQA (Hudson & Manning 2019)).
For consistency, all benchmarks evaluated via VLMEvalKit (Duan al. 2025) where do_sample is set to False; num_bear, ams is set to 1 and max.new. okens is set to 1024 by default. For MME benchmark,
only report its perception score to be consistent with (Tong al, 2024a). Before averaging, we divide the MME score by 20 and OCRBench score by 10. We use Qwen-Plus as judge model when there is need. Other configurations are kept default.
When compute acc of a model on these four categories, we first evaluate the performance on each benchmarks, then compute the average over these benchmarks, for example, General category, the accuracy is computed as
acc(model) acc(model)Gqa acc(model) MMB + acc(model)MMe + acc(model)sEED-
4
Table 5: Evaluation benchmark details. 15 benchmarks are classified into 4 categories.
Category
Benchmark
metric
remark
GQA (Hudson & Manning
2019
accuracy
General
MMB (Liu et al.
2023b
accuracy
MME
Fu et al.
2024)
score
perception score divided by 20
SEED-I
Ge et al. 2023
accuracy
A12D HHippala et al.
2021
accuracy
Knowledge
Math Vista (Luu et al.
2024b
score
SQA-I (Lu et al.
2022)
accuracy
MMMU Yue et al. 2024)
accuracy
DocVQA Mathew et
2021
accuracy
OCR & Chart
ChartQA Masry et al.
2022
accuracy
OCRBench LLu et al.
2023c
score
divided by 10
TextVQA Singh et al. 2019
accuracy
CV-Bench Tong et al. 2024a)
accuracy
Vision-Centrie
MMVP Tong et al.
2024b)
accuracy
Real World QA (xAl
2024
accuracy
17

---

Published as a conference paper at ICLR 2026
B MULTI-ENCODER MLLMS SUMMARIZATION
bunch of related works combines the information extracted by multiple vision encoders, we sum- marize the MLLMs with multi encoders and their fusion strategies in Table Most multi-encoders adopts CLIP, SigLIP, which are trained with contrastive objective to extract rich semantic infor- mation from visual inputs. To enhance the ability of processing low-level, tinegrained information, encoders such as SAM, Pix2Struct are utilized to enhance the capability of handling detection and OCR related tasks.
Table 6: Fusion strategy and vision encoders used by MLLMs with multi encoders. The adopted vision encoders and fusion strategies presented.
Cambiran-1 [Tong et al. (2024a)
Mini Mini-Gemi Gemini (Li Li 2024
2024a
Ferret-v2 Zhang et 2024a
2024
LEO-mini (Wang ct 2025)
Eagle-X4 (Shi et al. [2024]
Eagle X5- Shi et al. 2024
I-MoF Tong et al. 2024b
2023
MOME (Shen et al 2024
2024
CoMM (Jiang et al. 2024
LEO Azadani et al al_]] 2025
Eagle2 et al. 20256)
Mousi Fan et al. 2024
Brave Kar Ktreeta et al. 2024
LLaVA HR Luo et al.
deepseek-vl [Lu al.
Lin et al.
MoVA Zong al.
SPHINX
Model
SVA
ATT
MLP
I-MoF
MoLE
MoV
MRA
MLP
COTR
CC
CC
CC
SA
CC
SA
SA
Fusion
CLIP
SigLIP
ConvNext
/
d
DINOv2
Pix2Struct
d
EVA-02
d
d
SAM
(
VIT
v
Co-DETR
Deplot
Vary
BiomedCLIP
Intern ViT
18

---

Published as a conference paper at ICLR 2026
C
EXPERIMENT RESULT DETAILS
Table 7 Table 8 Table Table TO Table Table 12 Table 13 and Table [4 shows the detailed
performance of each combination of different multi-encoder MLLMs, we can see, the best-case performance is attained with few combinations of encoders, for example, in Eagle-X5 7B, with ConvNext and EVA-02, the performance achieves 96% of the baseline (with no encoders are masked), indicating that other encoders contributes little to final performance.
Table 15 gives detailed performance different LLM sizes against the number of masked en- coders. As we can see, encoder redundancy does not reduce as we increasing the size of LLM,, meaning that though larger LLM shows better performance, still faces digesting redundant or conflicting visual signals.
Table T6 is an extended version of Table 2 which reveal the difference between two evaluation configuration under different metrics, results are consistent for most metrics, so only report the KL divergence this paper.
Table 17 shows the training time and corresponding performance of different variants of Eagle- X5 7B, as the table shows, a dual-encoder variant hits a good trade-off between efficiency and performance, where it achieves 94% performance of Eagle-X5 7B while reducing the total training time by 34.4%.
Table 18 shows the inference times with respect different masking operations, as the result shows, by masking redundant encoders, the performance is largely retained and the efficiency is improved. Table 19 shows computation
of each vision encoder of Eagle-X5 7B, the FLOPSs of vision encoders is PL.OPsencoderr _encoders Rs 6.6174 TFLOPs, the FLOPS of projection layer is FLOOPsprroj R 0.07 TFLOPs and the FLOPs of LLM is FLOPs. LLM 21.3 TFLOPs, for a full yes-or-no QA task (64 prompt tokens + answer token, excluding special tokens), visual encoder processing accounts for approximately 30.6% of the total compute. Retaining only 2 encoders reduces total compute by - 14.1% 47. .3% ) 30.6% 11.8% and inference latency by 19.5%, while limiting performance degradation no more than 4%.
Table 20 detailed the performance of trained ones compared with the full model. Results show that the masked one performs slightly better than the trained one, revealing that incorporating more encoders may help on gaining more knowledge, however the improvement is marginal and at cost of consuming more training and inference resources.
19

---

Published as a conference paper at ICLR 2026
Table 7: Benchmark details for Eagle-X5 7B. Encoders: 1.CLIP; 2.ConvNext; 3.SAM;
4.EVA- 02; 5. ruct. v means the corresponding encoder is masked. Best scores within each subset of encoders have been bolded.
Performance
#Masked
Encoders
General
Knowledge
OCR & Chart
Vision-Centric
XXXXX x
70.77
54.79
66.60
67.54
X V
70.64
54.18
66.54
67.38
X X X x
63.63
51.07
57.71
55.83
X
70.36
53.67
63.80
66.84
X X
69.40
52.07
46.44
65.82
69.79
53.79
65.92
67.44
X V X
70.21
53.78
63.60
66.51
X X
63.07
50.73
57.46
54.82
X XXXVV
61.65
49.62
44.58
52.92
XXXX V
68.97
52.40
62.05
65.97
XXXX 65.69
51.35
40.81
64.53
XXXX
59.24
51.14
55.92
52.68
X X
69.39
52.51
46.22
65.59
XV
53.88
47.76
24.97
47.56
V X
69.86
53.64
66.01
67.29
X
68.22
50.62
16.53
65.29
X X
69.04
52.77
62.04
66.05
V X
64.97
47.96
10.84
62.91
V X X
33.39
45.85
27.33
43.71
/
65.32
51.16
40.26
64.06
W 53.07
47.50
24.73
46.84
XXVE
61.13
49.26
44.19
51.84
XXXVV
58.08
48.88
44.20
50.76
XXVVX
58.45
50.63
55.57
51.92
XV X XVV
53.27
47.25
10.07
47.59
XV
68.17
50.63
16.25
65.39
XXVEV
56.91
48.95
44.18
49.99
XVVVV
51.94
47.47
10.13
46.87
NVVVX
28.07
34.78
15.30
43,21
XV
31.03
45.91
7.653
43.64
VVVXV
64.60
47.70
10.68
62.83
5
VVVVV
31.93
43.69
7.521
46.70
20

---

Published as a conference paper at ICLR 2026
Table 8: Benchmark details for Eagle-X4 8B Plus. Encoders: 1.CIP;; 2.ConvNext;
3.PIX2STRUCT; 422. v means the corresponding encoder masked. Best scores within each subset of encoders have been bolded.
Performance
#Masked
Encoders
General
Knowledge
OCR & Chart
Vision-Centric
X X x
66.48
61.88
71.92
70.62
X x
65.68
61.57
71.97
70.50
X X x
65.60
57.46
52.89
68.38
X X X
10.99
27.03
5.165
35.09
X x
67.77
0.64
70.98
69.58
X X
7.813
23.93
1.180
32.85
V x
62.86
56.00
52.10
68.01
V X X
6.905
33.60
0.175
33.78
V X X
67.28
59.83
70.57
69.60
X
65.85
51.60
10.17
66.93
XXV 6.501
24.27
0.980
34.47
VVXX
64.22
51.21
9.141
66.68
X
6.840
26.74
1.000
28.26
3
XVVV
6.604
26.67
0.125
38.66
X XXV
6.860
28.16
1.970
39.23
VVV
25.60
44.54
5,316
40.00
Table 9: Benchmark details for Cambrian-1 3B. Encoders: 1.CLIP; 2.SigLIP; 3.DINO; 4.ConvNext; means the corresponding encoder masked. Best scores within each subset of encoders have been bolded.
Performance
#Masked
Encoders
General
Knowledge
OCR & Chart
Vision-Centrie
x xx
67.81
59.58
60.60
64.57
V X x x
64.65
58.54
58.44
62,93
x
66.34
58.31
58.36
62.88
X X X
61.39
53.72
18.16
57.48
X X
65.70
58.15
57.61
60.57
x V
$1.45
$1.22
10.52
$2.08
x
$5.02
55.57
53.35
57.77
/ X
52.14
51.14
12.01
53.87
VX
62.29
56.67
54.92
57.21
X
63.28
56.88
53.80
58.24
XXVV
55.54
52.18
16.83
52.65
1448
41.03
52.84
47.01
49.01
11%V
31.48
47.62
6.896
48.52
XV47
44.84
50.34
9,510
48.96
V LXVV
44.35
49.97
11.14
48.27
29.26
47.81
6.586
45.88
21

---

Published as a conference paper at ICLR 2026
Table 10: Benchmark details for Cambrian-1 8B. Encoders: 1.CLIP; 2.si 3.DINO; 4.ConVNext; v means the corresponding encoder is masked. Best scores within each subset of encoders have been bolded.
Performance
#Masked
Encoders
General
Knowledge
OCR Chart
Vision-Centric
x x xx
67.47
57.87
70.08
56.65
X
66.57
56.
68.19
53.47
X
69.29
58.05
67.96
65.80
X X X
66.69
51.34
17.53
55.84
X X V X
66.68
55.83
68.74
57.43
X X V
60.05
48.98
10.70
57.53
V X
59.77
52.64
63.91
50.31
VXX 59.35
50.89
10.62
55.16
X 68.63
58.08
66.40
63.24
X 64.91
53,92
66.75
55.61
XXVV
64.28
51.61
16.71
56.95
57.04
53.72
60.57
55.93
XV
26.52
45.81
5,486
38.30
XVVV
51.75
47.78
10.14
49.39
VXVV
52.58
50.10
10.13
47.38
23.33
45.06
5.914
35,83
Table 11: Benchmark details for Cambrian-1 13B. Encoders: 1.CLIP: 2.SigLIP; 3.DINO; 4.ConvNext; / means the corresponding encoder is masked. Best scores within each subset of encoders have been bolded.
Performance
#Masked
Encoders
General
Knowledge
OCR Chart
Vision-Centric
x xx
71.96
60.76
69.99
65.21
V X X
69.91
57.28
67.93
64.96
X
71.24
60.03
68.68
64.21
X X X
64.32
50.99
15.72
56.95
X X V x
71.20
59.54
69.07
62.70
x x d
56.75
49.39
9,838
51.76
// x
63.02
56.45
63.47
61.86
X
51.99
47.60
9.222
50.61
Vxvx 68.71
56.44
67.19
62.89
x 69.18
57.58
67.34
62.07
xXVV
58.89
50.10
14.83
52.27
144x
54.16
52.77
60.18
55.21
30.67
46.50
5.781
42.68
XV44
50.62
48.02
9.286
48.20
VXVV
43.57
47.64
8.686
45.13
27.92
45.96
5.806
41.06
22

---

Published as a conference paper at ICLR 2026
Table 12: Benchmark details for DeepSeek- VI, 7B. Encoders: 1.SAM; 2.SigLIP; means the corresponding encoder is masked. Best scores within each subset of encoders have been bolded.
Performance
#Masked
Encoders
General
Knowledge
OCR & Chart
Vision-Centric
X X
69.84
52.37
53.96
61.83
X
61.42
46.64
27.80 27.53
57.40
X V
60.60
46.38
55.89
V
31.09
42.71
6.771
46.69
Benchmark details for Eagle2 9B. Encoders: 1.SAM; 2.SigLIP; / means the corre- sponding encoder is masked. Best scores within each subset of encoders have been bolded.
Table 13:
Performance
#Masked
Encoders
General
Knowledge
OCR & Chart
Vision-Centric
X X
75.61
75.02
86.53
74.95
67.36
70.39
83.37
67.75
X 75.30
70.22
59.12
74.34
30.99
51.57
6.542
48.31
Table 14: Benchmark details for I-MoF 13B. Encoders: 1.CLIP; 2.DINOv2; means the corre- sponding encoder is masked. Best scores within each subset of encoders have been bolded.
Performance
#Masked
Encoders
General
Knowledge
OCR & Chart
Vision-Centric
X
47.89
48.99
4.325
59.70
22.41
46.27
0.800
43.53
X
47.14
49.55
4.300
57.44
24.17
45.47
0.725
49.26
Table 15: Impact of the size of LLM on encoder redundancy. The min, and average perfor- mance of Cambrian-1 3B, 8B, 13B with different number masked encoders are reported.
LLM
#masked
0
2
max
63.14
61.47
58.04
47.47
32.38
3B
min
63.14
47.68
41.31
33.62
32.38
avg 63.14
57.70
49.86
39.48
32.38
max 63.02
65.28
64.09
56.81
27.53
8B
min
63.02
47.85
44.00
29.03
27.53
avg 63.02
59.10
52.79
41.41
27.53
max 66.98
66.04
64.04
55.58
30.18
13B
min 66.98
46.99
39.85
31.40
30.18
avg 66.98
60.92
52.47
40.56
30.18
23

---

Published as a conference paper at ICLR 2026
Table 16: Attention Analysis on different encoders. We recore the value of last-layer attention scores related to visual tokens during full MME evaluation, and we imply Pearson Correlation coefficient (PC), Spearman's 's Correlation (SC), Mean Squared Error (MSE). Mean Absolute Error (MAE), Trnsen-Shanner divergence (JS) and Kullback-Leible divergence (KL) for visual attention analysis. Higher PC, SC and lower MSE, MAE, JS, KL values indicates higher similarity. Eagle-X5 7B, Eagle-X4 X4 8B Plus and Cambrian-1 1 7B are used for analysis.
Model
Encoder
PC 1
SC 1
MSE
MAE
JS 4
KL
CLIP
.0277
.3878
.00007
.0013
.5229
2.658
ConvNext
.0279
.3726
.00007
.0013
.5700
3.004
Eagle-X5 7B
SAM
.0578
.3712
.00007
.0012
.5496
2.537
EVA
.7446
.7432
.00003
.0006
.3699
0.982
Pix2Struct
.1934
.3512
.00015
.0013
.6082
2.959
CLIP
.2742
.4180
.00003
.0010
.4353
1.007
Eagle-X4 8B Plus
ConvNext
.5854
.2674
.00002
.0010
.4146
inf
Pix2Struct
.8281
.4377
.00001
.0088
.3553
inf
EVA
.7695
.7545
.00001
.0006
.2760
.3921
SigLIP
.9419
.5468
.00005
.0009
.1927
.0951
Cambrian-1 8B
CLIP
.9444
.5453
.00004
.0009
.1906
.1017
DINOv2
.9900
.4956
.00004
.0009
.1988
.1279
ConvNext
.9678
.6157
.00003
.0006
.1436
.0804
Table 17: Training time with specific vision encoders. Training time Eagle-X5 7B (EVA-02g ConvNext; Pix2Struct, + CLIP: SAM4 in hours, including pre-train stage and fine-tune stage. All experiments are conducted on a single-node with 8 NVIDIA A100 GPUs. The subscript such 0123 refers retained encoder index.
Model
Pre-train
Fine-tune
Overall
Performance
Eagle-X5 7B
12.12
94.57
104.72
64.9
-40123
10.90
110.0%
81.92
113.4%
92.82 111.4%
62.06
J4.38%
-X201
2
6.888 J42.8%
61.83 J34.6%
68.71 J34.4%
60.95 J6.09%
-Xlo
6.042
[49.8%
58.20
[38.5%
64.24
[38.7%
51.57 120.5%
24

---

Published as a conference paper at ICLR 2026
Table 18: Inference time of specific vision encoders with respect to masking operation. Infer- ence time comparison of Eagle-X5 7B (EVA-02, + ConvNext, Pix2struct, CLIP3 SAM,) and E4 8B Plus (EVA-020 ConvNext; + Pix2Structs + CLIPs), evaluated by calculating the average inference time for MME benchmark evaluation with 2374 samples ms, where the max number of generated tokens has ben set to 1. The subscript such 0123 refers to retained encoder index.
Model
Time
Performance
Eagle-X5 7B
749.656
64.93
-X4 0123
4
730.761 12.52%
64.69 J0.37%
-x3 012
643.989 114.1%
64.20
11.12%
X2 01
2
603.153 119.5%
62.48
13.78%
-X1
260. y. 22.5%
46.45 (28.5%
Eagle- X4 8B Plus
4
681.679
67.73
-X3 012
3
665.946
2.31%
67.43
J0.44%
X2
2
600.126
&. 12.0%
66.82 $1 11.34%
-X1
549.621 19.4%
47.81 429.4%
Table 19: Per-encoder vision FLOPs for Eagle-X5 7B. Including input-image resolution, vision FLOPs and contribution Eagle-X5 7B vision computing.
Encoder
Input Resolution
Vision FLOPs (G)
Contribution
CLIP-VIT-L/14
448
214.0
3.23%
ConyNext-Large
1024
3133.
47.3%
EVA-02-L
1024
933.7
14.1%
Pix2Struct-0024
1024
1026.
15.5%
SAM-ViT-ION
1024
1311.
19.8%
Table 20: Performance for masked Eagle models with 1, 2, 4 and 5 encoders correspond to trained ones. Performance subsets based on Eagle-X5
(EVA-02g + ConvNexti Pix2Structz CLIP; SAM),, where all training strategies strictly follows the official instruc- tion of training Eagle-XC X5 7B. The subscript such as 0123 refers to retained encoder index.
Model
General
Knowledge
OCR & Chart Vision-Centric
Overall
X5 (Masked) 0123
70.64
54.19
66.55
67.39
64.69
X5 (Trained) 0125
69.72 J 1.30
49.25 L. 9.11
63.44 %
65.85 J 2.29 %
62.06 ( 4.07
XS (Masked)
2
69.04
52.77
62.04
66.05
62.48
X5 (Trained) o
2
67.22 J. 2.64 %
52.37 L. L.7.6 %
60,17 J 3.01 %
64..05 3.03 %
60.95 J. 2.45
25

---

Published as a conference paper at ICLR 2026
D CASE STUDY
We provide case study in this section. Results in Fig D and Fig show that once a specific encoder masked, the answer generated by the model changes dramatically, which reveals the fact that multi-encoder MLLM depends on specific encoder to complete specific tasks.
Most proferred onjects different
categories
Ground Caption:
Percent of Prople
preferred by than 90 percant of people in any gory. Therefore, the rinal value is Il.
Eagle-X4-Plus 8B:
There
Mask Encoder There are no objects preferred by more than 90 percent of people in any calog ory, highest percentage shown 90 Therefore, final value is O.
20
Mask Encoder 01: To determine number of abjects prefered ople one cstegory, need count number bars exceed 90 percent k on the vertical axis. Upon examining the chart, we can see that there are no bans that rench the percent Therefore, the valuoe
tham percent
mare
detemnine
objects preferzed more percent of
cople in al least one calegory, we need to count the number of burs that reach the 90% mark on k. Therefore, the final value is O.
By examining chart, can
Question: How many objects prefetred more percent of people least one categary?
Mask Encoder 3: Answer:
Figure 4: show case of how Eagle-X4 8B Plus would behave with/without specific encoders masked.
Ground Caption:
Eagle-X4-Ples
Mask Encoder C.2.0
Mask Encoder 01: C.2.0
Mask Encoder 012: There chest visible the which chest the foul bed. Therefore, correct answer
many image? Select from the following choices.
C2
DO
Figure 5: show case of how Eagle-X4 8B Plus would behave with/without specific encoders masked,
26


Reviews:
[{"review_id": "zBpiYXwy47", "reviewer": "Reviewer_rQtg", "summary": "This is paper examines whether adding more vision encoders truly improves multimodal large language model (MLLM) performance.\nIt finds that multi-encoder MLLMs often exhibit substantial redundancy—removing some vision encoders either barely affects or even improves accuracy. To quantify this, the authors propose two new metrics: Conditional Utilization Rate (CUR): measures an encoder’s marginal contribution to overall performance. Information Gap (IG): measures imbalance in encoder usefulness within a model.\nThrough systematic encoder masking experiments on models like Eagle and Cambrian-1, they show: Many encoders are redundant or even detrimental. Some tasks (e.g., OCR & Chart understanding) rely heavily on one specialized encoder (CUR > 90%), while others (e.g., general VQA) show high redundancy. Two-encoder variants recover > 90 % of full performance with much lower computational cost.\nOverall, the study challenges the “more encoders = better performance” assumption and provides a diagnostic framework (via CUR and IG) for designing more efficient, balanced multimodal architectures.", "questions": "Questions are mentioned in the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles an underexplored but highly relevant question in multimodal LLM design — encoder redundancy. While prior works focused on adding more encoders or improving fusion, this paper challenges the “more is better” assumption and provides a new analytical perspective.\n2. The introduction of Conditional Utilization Rate (CUR) and Information Gap (IG) offers principled and interpretable metrics for quantifying encoder contribution and redundancy, enabling future researchers to diagnose model efficiency systematically.\n3. The authors conduct extensive masking experiments across multiple representative multi-encoder MLLMs (e.g., Eagle, Cambrian-1), covering diverse benchmark categories (OCR, Chart, General, Knowledge, and Vision-Centric). The analyses are thorough and reproducible.\n4. The finding that two carefully selected encoders (e.g., EVA-02 + ConvNeXt or ConvNeXt + CLIP) can retain over 90% of full-model performance provides actionable guidance for designing more efficient multimodal architectures.", "weaknesses": "1. While the paper introduces CUR and IG as empirical metrics, it does not offer a strong theoretical framework explaining why redundancy arises or how encoder representations overlap in feature space.\n2. The study primarily measures the effect of removing one encoder at a time (via single-encoder masking). This ignores higher-order interactions — for example, two encoders might each seem redundant individually but provide complementary information together.\n3. Can the author provide the analysis of which fusion method is most effective and efficient? How can we decide which encoders are most important before training? Does the training strategy also influence the encoder selection? These are the follow-up questions. I do appreciate what the authors did in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u201LMVKS5", "reviewer": "Reviewer_vJKH", "summary": "This paper mainly investigates the impact of multiple vision encoders in multimodal llms. By isolating individual encoders, the paper finds that not all vision encoders are needed in downstream tasks, some of which are even detrimental to performance. To quantify their findings, the paper introduces two metrics, CUR and IG, which jointly describe the contribution of different encoders to performance. Evaluations on popular multi-encoder mllms demonstrate that not all vision encoders are necessary during inference, as they barely drop performance.", "questions": "1. Table 7 in the appendix is quite interesting. When masking all encoders, tasks that claim to be vision-centric, still receive relatively high scores. Why do you think that is?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and metrics are well-defined. Easy to follow.\n2. This paper challenges the view on vision encoder selection for open mllms, and provides a fresher perspective on inference selection strategy.\n3. The evaluation setups are extensive, which back up their claim about inference not needing all encoders.", "weaknesses": "1. This paper feels also related to visual token selection strategy, I think the paper should include relevant references in related work section.\n2. In the final paragraph of the introduction, the paper also mentions \"in our setup, fine-tuning a dualencoder variant is 1.69× faster than its five-encoder counterpart.\", however, I did not find experiments about finetuning in the paper.\n3. To further show the merit of reducing vision encoders during inference, I believe it's better to also include the compution reduction (i.e. FLOPs) during evaluation, for example in Table 3.\n4. The paper positions that not all evaluations are needed to mllms. According to the experiment, I agree that when inferencing it is true. However, I believe to validate this claim completely, I believe the paper also needs to demonstrate that not all vision encoders are needed for training. For example, some vision encoders may contribute less during inference, but they might help other encoders learn better during training. The training dynamic here is still unclear.\n\nI will be happy to raise my score if the authors can address the above concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O4XOhjVXch", "reviewer": "Reviewer_yWcu", "summary": "The paper examines whether adding multiple visual encoders improves multimodal large language models. It introduces two metrics—Conditional Utilization Rate (CUR) and Information Gap (IG)—to quantify encoder contribution and redundancy. Experiments across several benchmarks show that additional encoders often bring limited or inconsistent gains, and sometimes degrade performance. The study concludes that simply increasing the number of encoders does not guarantee better results, highlighting redundancy and imbalance in current multi-encoder designs.", "questions": "1. Does a more balanced utilization of visual encoders (i.e., lower IG) really lead to better practical results?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. Problem framing and metrics. Treats redundancy in multi-encoder MLLMs as a first-class research object and introduces two reusable, precisely defined measures  (CUR) and (IG) that turn a vague intuition (“more encoders help”) into testable quantities.", "weaknesses": "1. Lack of Constructive Improvements. While the proposed CUR and IG metrics help quantify encoder redundancy, the contribution appears incremental. Prior works such as Eagle have already observed and empirically analyzed similar redundancy phenomena (Fig. 4 in their paper). This paper mainly reformulates those observations into statistical metrics based on existing accuracy measures, without offering causal insights into why redundancy arises, e.g., whether it stems from overlapping feature spaces, pretraining objectives, or fusion mechanisms.\nTo strengthen the work, the authors could do further experiments and implement learnable gating or dynamic routing based on CUR (refer to MoVA), rather than only suggesting them as future work.\n\n2. Experimental Design Issues. The experimental coverage is incomplete. For models with fewer visual encoders, such as DeepSeek-VL 7B, Table 1 reports only IG, while Table 3 omits corresponding accuracy metrics. \n\n3. Insufficient Efficiency Evaluation. The paper asserts that removing redundant encoders improves efficiency but provides no quantitative evidence, such as GPU hours, FLOPs, or latency metrics. In contrast, Eagle reports detailed efficiency tables and trade-offs between accuracy and computation.\n\nThe paper offers a useful quantitative perspective on multi-encoder redundancy but lacks theoretical depth, complete experimental validation, and concrete improvement mechanisms. By integrating causal or information-theoretic reasoning, implementing dynamic encoder selection (like MoVA), and reporting efficiency metrics, the study could become substantially more constructive and impactful for the multimodal LLM community.\n\n1. MoVA: Adapting Mixture of Vision Experts to Multimodal Context\n2. Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xhho0JOUak", "reviewer": "Reviewer_VGcs", "summary": "This paper presents an investigation into the phenomenon of encoder redundancy in Multi-encoder Multimodal Large Language Models (MLLMs). The authors challenge the prevailing assumption that integrating more vision encoders, pre-trained with diverse objectives, necessarily leads to better performance. Through extensive ablation studies on models like Eagle and Cambrian-1, they demonstrate that performance often degrades gracefully—and sometimes even improves—when one or more encoders are masked. To quantify this redundancy, they introduce two metrics: the Conditional Utilization Rate (CUR), which measures an encoder's marginal contribution given the presence of others, and the Information Gap (IG), which captures the disparity in utility across the encoder set.", "questions": "1. How would you suggest extending the CUR framework to quantify the complementary or synergistic effects between pairs or groups of encoders, to address the limitation you noted?\n2. What is the interplay between fusion mechanism complexity and redundancy? Could a better fusion strategy (e.g., a gating network) dynamically suppress redundant or conflicting encoders on a per-input basis, thereby making a larger ensemble more justifiable?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses a critical, underexplored, and timely issue in MLLM design. As models grow more complex, understanding and mitigating redundancy is essential for improving efficiency and performance.\n2. The proposed metrics, CUR and IG, are intuitive, easy to compute, and provide a rigorous, quantitative framework for analyzing encoder contributions. They effectively capture both individual utility and overall system imbalance.\n3. The empirical evaluation is thorough. The authors test their hypothesis across multiple state-of-the-art model families (Eagle, Cambrian-1), various model sizes (3B to 13B), and a comprehensive set of benchmarks categorized by task type. The results are robust and clearly support the claims.", "weaknesses": "1. The study is conducted on pre-trained, static models. It does not explore whether redundancy emerges during training or if dynamic, input-dependent routing of encoders (a potential solution) could better leverage the available encoders.\n2. **Narrow Scope of \"Performance\"**: Performance is primarily measured as accuracy on standardized benchmarks. Other critical dimensions like robustness, out-of-distribution generalization, calibration, and fairness are not considered. A redundant encoder might contribute to robustness even if it hurts average accuracy.\n3. **Higher-Order Interactions**: The authors correctly identify this as a limitation. CUR only measures the effect of ablating a single encoder. It cannot detect complementary pairs or higher-order synergies where two seemingly redundant encoders (low individual CUR) are both essential for performance (e.g., if removed together, performance collapses). A metric capturing these interactions would be a valuable extension.\n4. **Limited Exploration of Fusion Mechanisms**: While the paper notes that more sophisticated fusion (like Cambrian-1's SVA) does not fully mitigate redundancy, a deeper analysis is lacking. It remains unclear how different fusion strategies (e.g., MoE, cross-attention) might amplify or suppress redundancy, which is a crucial aspect of multi-encoder architecture design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)