Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMs
Authors: Mozhgan Nasr Azadani
James Riddell
Sean Sedwards
Krzysztof Czarnecki
Venue: Accepted by TMLR ()

Abstract: Mixture of Vision Encoders (MoVE) has emerged as a powerful approach to enhance the fine-grained visual understanding of multimodal large language models (MLLMs), improving their ability to handle tasks such as complex optical character recognition and scene understanding. Despite these advances, effectively combining diverse encoders and their visual tokens, while also scaling to high-resolution inputs, remains an open challenge. In this work, we conduct a systematic study of fusion designs for MoVE-based MLLMs, highlighting principles for token-level integration across complementary encoders. Our study shows that a lightweight recipe consisting of post-adaptation fusion with independent projectors, tile-level sequence interleaving, and dynamic tiling with global context delivers strong performance on diverse benchmarks. We integrate these principles into a simple and effective architecture that we call LEO. Extensive evaluation on 11 vision–language benchmarks demonstrates that LEO achieves better results on the majority of tasks compared to existing MoVE-based approaches. Furthermore, LEO adapts effectively to the specialized domain of autonomous driving without altering its architecture or training recipe, achieving competitive performance against established baselines and thereby highlighting its ability to generalize. The code is available at https://github.com/Mozhgan91/LEO.

Published in Transactions on Machine Learning Research (02/2026)
Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMs
Mozhgan Nasr Azadani
mmssrca@@uuaterrooocoo
University of Waterioo
James Riddell University of Waterloo
jumeesriddelll@unnaterroo
Sean Sedwards University of Waterloo
cen..edduords@uuuaterroo...
Krzysztof Czarnecki
keccarnne@uusterrooo.c
University of Waterloo
Reviewed on OpenReview: https: //
// openreview. net/ forum? id- EgnTVmRybs
Abstract
Mixture of Vision Encoders (MoVE) has emerged powerful approach to enhance the fine-grained visual underslanding of multimodal large language models (MLLMs), improving their ability to handle tasks such as complex optical character recognition and scene under- standing. Despite these advances, effectively combining diverse encoders and their visual tokens, while also scaling to high-resolution inputs, remains an open challenge. In this work, we conduct a systematic study of fusion designs for MoVE.-based MI.LMs,, highlighting prin- ciples for token-level integration across complementary encoders. Our study shows that lightweight recipe consisting of post-adapiation fusion with independent, projectors, tile-level sequence interleaving, and dynamic tiling with global context delivers strong performance diverse benchmarks. We integrate these principles into simple and effective architecture that we call LEO. Extensive evaluation on 11 vision language benchmarks demonstrates that LEO achieves better results on the majority of tasks compared to existing MoVE-based approaches. Furthermore, LEO adapts effectively to the specialized domain of autonomous driving without altering its architecture or training recipe, achieving competitive perfor- mance against established baselines and thereby highlighting its ability to generalize. The code is available at https:/ https://githhb...ee /ttg:///bb.oo////cggggg///////// Moon//goggoaagnL.....
1 Introduction
Multimodal large language models (MLLMs) Li et al. 2023a Alayrac et al. 2022 Liu et al. 2024c Gao
ct al. 2023 have recently achieved strong performance by aligning vision encoders with large language models (LLMs) through multi-stage training on large-scale image text datasets. This alignment allows visual tokens from pretrained vision foundation models such as CLIP Radford et al. (2021) be mappod into the latent space of LLMs, enabling progress in variely of vision language reasoning tasks (Liu al.) 2024cc Driess et al. 2023). However, these models still face challenges in tasks that require fine-grained perception, such complex optical character recognition or chart understanding, where the ability to process high-resolution inputs is critical for preserving detailed visual information. Enhancing visual understanding has therefore become key priority, not only for improved performance in high-resolution tasks, but also for reducing hallucinations (Shi et al.) 2024).
Recent studies Lin et 2024h: Chen al. 22025 Vasu al. 2025 Shen al. 2025) have explored
different strategies to enhance the visual understanding capabilities of M.. One line of work strengthens

---

Published in Transactions on Machine Learning Research (02/2026)
LLM
Empirical design insights:
Port
78.0
Tile-level sequence interleaving
02.4
642
Post adaptation token fusion
Champs
46.1
Projector
Independent projectors for MoVE
67.0
Tiled MoVE
Tiling with global context
Tile segmentation
20.1
Ceston
66.4
Is safe for you inurease your speed in the current driving conditions? Provide rationale.
46.
48.5
50.3
57
72.9
LEO: No, not safe for me to incresse my speed in the there eyclist ahead of me, and need to maintaäa safe distance from them.
driving conditions
193
Figure Overview of LEO and its performance. LEO integrates key design principles into lightweight MoVE-based architecture for high-resolution visual reasoning.
vision encoders, cither by scaling model paramcters and pretraining data to match those of LLMs Zhai et al. (2023) Chen et al. (2024e;; Sun et al. (2023) or by enabling fine-grained perception through tiling Liu
et al. 2024b;; Chen et al. (2024b); Wu el al. (2025) and high-resolution inputs Beyer et al. (2024) Zhang
et al.
2024). Another promising direction is the Mixture of Vision Encoders (MoVE) paradigm, which integrates multiple pretrained experts to employ their complementary strengths. These models maintain the
standard MLLM architecture while enriching visual understanding through specialized encoders. Existing MoVE-basee models employ fusion strategies ranging from straightforward approaches such as sequence Kar
et al. 2024); Lin et al. (2023) channel (Shi et al. (2025) Lu et al. (2024a) concatenation, and sequence
interleaving Tong et al. [2024). to more advanced methods like of adaptation Luo et al.
2025 cross-attention Li et al. 2024b) or routing mechanisms Zong et al. (2024). While these methods have demonstrated gains the visual reasoning capacity of MLLMs, most have been studied in isolation, leaving their broader interactions underexplored. As a pioncer empirical work, Eagle (Shi et al.) (2025) investigated encoder selection and training strategies for scaling mixtures. However, key questions remain, including how strategies for enhancing visual capacity interact, the granularity at which fusion is most effective, and whether scaling the nunber of experts is necessary, or if effective fusion designs yield competitive gains.
Addressing these questions requires moving beyond isolated design choices toward principled understanding of how MoVE models function. To this end, we conduct a systematic study to determine which strategies are most effective for integrating multiple vision encoders. Specifically, we examine three core aspects of MoVE design: how dynamic tiling interacts with MoVE; which token merging strategies are most effective, from simple concatenation structured interleaving or cross-attention; and when fusion should be applied, either before or after adaptation into the multimodal space.
Our empirical study yields several insights into how MoVE models can be most effectively designed. We find that simple but crucial choices consistently improve multimodal reasoning. Our three key findings are as follows. (1) Combining the mixture of vision encoders with dynamic tiling and global context (Tiled MoVE) enables models to process high-resolution inputs without exceeding context length. This design preserves fino-grained dctails while strengthening overall visual understanding. strategies often outperform more complex designs such cross-attention Li et. al. (2024b). Among these, tile-level interleaving consistently achieves the best results, surpassing both sequence append and channel concatenation. (3) Aligning each encoder's tokens independently with dedicated projectors before merging preserves encoder-specific features and consistently outperforms pre-adaptation fusion.
(2) Straightforward token merging
Building on these insights, we propose LEO, a lightweight and effective MoVE-based MLLM. As illustrated in Fig. LEO integrates the key principles identified in our empirical study: (1) dynamic tiling with global context to preserve fine-grained details, (2) tile-level sequence interleaving ensure efficient and balanced token integration, and (3) post-adaptation fusion with independent, projectors to retain encoder-specific
2

---

Published in Transactions on Machine Learning Research (02/2026)
strengths. Despite its simplicity, this design yields a powerful architecture that achieves strong performance across wide range of vision-language tasks.
Our main contributions are summarized as follows:
We conduct a systematic study of design choices in MoVE-based MLLMs, examining the interac- lion between visual reasoning enhancements, token-level merging strategies, and fusion timing, and identify key findings (1) (3).
We integrate these insights into LEO, a lightweight MoVE-based MLLM that provides an efficient recipe for high-resolution visual reasoning.
We conduct extensive experiments across multiple vision- language benchmarks, demonstrating LEO'S effectiveness on the majority of tasks compared to existing MoVE models.
We demonstrate that LEO can be applied to the specialized domain of autonomous driving with- out modifying its architecture or training recipe, achieving competitive results and highlighting its generalizability.
2
Related work
2.1
Multimodal large language models
With the rapid advancement of large language models (Touvron et al. 2023 Achiam et al. 2023 Team
et al.) 2023: Chiang et al. 2023), there has been considerable interest in multimodal extensions that enhance understanding and reasoning capabilities. Pioneering works such as BLIP-2 Li et al. (2023a ) Dai et al. (2023) introduce the Q-Former to bridge the modality gap between images and text, while Flamingo Alayrac et al. (0022) enables flexible processing of mixed visual textual sequences through a resampler for in-context few-shot learning. The LLaVA family Liu et al. (2024c a) simplifies this design by adopting lightweight projection modules, such as linear layers or MLPs, to align vision encoder outputs with the LLM token space. Our work follows this general framework of vision encoder - aligmment module + LLM, building on its simplicity and elfectiveness. Rather than revisiting the alignment. mechanism itself, we adopt it as a foundation and shift our focus to the integration of multiple vision encoders. In particular, we investigate principles that guide the effective fusion of complementary encoders in MoVE-basod MLLMs.
2.2 Enhanced Visual Understanding for MLLMs
To address the constraints of lower input resolutions, recent MLLMs have concentrated on enhancing their vision encoder module. From a vision-focused slandpoint, these approaches can broadly categorized inlo four main strategies: (1) robust vision encoders, which design stronger backbones by scaling model size and pretraining data to better capture complex features Zhai et al. (2023): Chen al. (2024c). (2) tile segmentation, which processes high-resolution inputs by dividing images into smaller tiles Shi et al. (0224]
Liu et al. 2024ct Chen et al.
(2024b): ; Li et al. (2024c), (3) knowledge-distilled vision encoders, where large pretrained experts are distilled into smaller encoders for efficiency while retaining fine-grained perception, as
in Radio Ranzinger et al. (2024); Heinrich et al. (2025), and (4) mixture vision encoders (MoVE), which
integrates multiple vision experts into a unified backbone Lin al. (0023); Luo et al. (2025 Tong et al.
(0025 Li ct al. (2024b); Shen et al. (2025 Shi
(202 Wei et al. (2024).
Focusing MoVE approaches, some models suggest merging high-resolution visual details with low- resolution tokens to enhance visual representation Luo et al. (2025): Li et al. (2024b). LLaVA-HR Luo ct al. 2025g proposes a dual-pathway vision model that integrates fcatures from high-resolution convolu- tional blocks with those from low-resolution ViT blocks. These pretrained vision experts can nevertheless lack key capabilities, such as text understanding and object localization. To broaden encoder capacity, sev- eral studics integrate multiple vision experts trained on diverse tasks. Brave Kar al. (2024) and MouSi Fan let al. (2024) perform sequence appending, combining vision tokens from multiple experts into longer se- quence. Tong et al. (2024) identify distinct differences in the visual features captured by CLIIP Radford et al.
3

---

Published in Transactions on Machine Learning Research (02/2026)
Projector
Fosion Block
Proj
Proj 2
VE
VE
VE
(a) Pre-adaption Tiled MoVE
(b) Post-adaptation Tiled MoVE
(I) Sequence
(2) Sequence
(3) Channel
(4) Cross-atiention
Tile-Jevel sequence interleaving
appending
interleaving
concatenatice
0000
O visual tokenss tile
Attention
visual tokets: tile
(c) Existing token merging strategies for MoVE-based MLLMs
(d) Tiled MoVE
Figure 2: Illstraaionn of fusion strategics studied in our empirical analysis. @a Pre-adaptation fusion. (b
Post-adaplation fusion. (0) The most common fusion blocks in the literature: (1) Sequence appending Kar
et al. 2024), (2) Sequence interleaving Tong et al. (2024), (3) Channel concatenation [hhi et al. (025
catss-ntt Li et al. (2024b A sample tile-level token merging strategy in Tiled MoVE.
2024 leading to the design of an image-level mixture-of-feature.. strategy that employs sequence interleaving. Other models apply channel concatenation to preserve sequence length, as in DeepSeck- VL Lu et al. (2024a) and Eagle Shi et al. 2025)) or adopt more advanced fusion and routing mechanisms
2021 and DINOv2 Oquab et al.
Lee et al. (2024]:
b: Zong et al. (2024).
Overall, these methods differ not only in the choice of vision encoders but also in their fusion mechanisms. Despite demonstrated gains, most have been studied isolation with limited systematic comparisons. Our work is most related to Eagle Shi al. (2025), which provides empirical insights into vision encoder selec- tion and explores training stratogies for scaling MoVE. Yet important questions remain: How do different euhancement. strategies, such tiling and MoVE, interact when combined? At what granularity is fusion most effective? And does stronger visual reasoning truly require scaling the number of experts? Our work is complementary. We conduct a systematic empirical study of fusion strategies for MoVE-based MLLMs. Specifically, we investigate post-adaptation fusion, sequence interleaving, and dynamic tiling with global context, offering practical guidance on which design choices are most effective.
3 Rethinking mixtures of vision encoders
In this section, we take a fresh look at MoVE by systematically examining the factors that govern how multiple vision encoders can be combined most effectively. In contrast to prior studies that aim to propose new merging architectures
routing mechanisms, we instead focus on deriving clear, practical insights through extensive ablations. To this end, we organize our study around three investigative directions: D1 how the integration of visual reasoning enhancement techniques operates; D2 token merging strategies, contrasting straightforward concatenation or sequence intericaving with more advanced approaches such as cross allention; and D3 the timing of loken merging, asking whether encoder oulputs are better merged prior to or following alignment. Through these investigations, we establish guiding principles that inform the design of LEO, our simple and effective Mose-based MLLM.
3.1 Tiled MoVE (D1)
Recent MLLMs enhance visual understanding either by employing tile segmentation to support high- resolution inputs with single vision encoder (Li et al. 2024cc Chen et al.] (2024b). or by using mixture of vision encoders exploit complementary expertise (Shi et al. 2025 Kar et al.J 2024 Shen et al. 2025 Lin
et al.
2023 Lau et. al. 2024ab. While both directions have shown benefits independently, their combination
4

---

Published in Transactions on Machine Learning Research (02/2026)
Table Tiled MoVE:
Investigating the impact of tiling within MoVE. 1, S, VQA",an and MMB denote
InternViT, SigLIP, VQX, and MMBench, respectively.
MoVE
Tiling Method
VQA"
GQA
VizWiz
MMB
POPE
SEED
SQA
MMVeT
Avg.
no-tiling
62.3
63.1
52.9
65.1
86.7
68.8
68.3
34.1
62.7
1+ SAM
fixed-grid
62.9
63.0
53.8
67.5
87.0
69.4
71.2
34.2
63.6
overlapping
62.6
63.3
53.4
66.3
69.8
67.3
33.9
62.8
dynamic
63.4
63.9
54.2
67.5
87.1
70.8
71.2
34.6
64.1
no-tiling
61.9
63.4
52.7
64.9
86.8
66.3
65.5
33.7
61.9
I+ ConyNeXt
fixed-grid
62.5
63.5
53.9
66.0
86.9
68.1
69.5
34.3
63.1
overlapping
61.9
62.9
53.6
65.5
86.9
66.9
68.1
33.9
62.5
dynamic
62.6
63.9
54.0
66.4
87.0
68.5
71.1
34.6
63.5
no-tiling
62.2
63.5
52.7
65.0
86.8
68.5
67.3
34.0
62.5
++ DINOv2
fixed-grid
63.7
63.8
54.0
66.5
87.1
68.5
70.1
34.2
63.5
overlapping
62.5
63.6
53.4
65.1
86.8
68.2
68.7
33.8
62.8
dynamic
63.3
64.0
53.9
67.1
68.4
71.0
34.5
63.7
no-tiling
62.1
62.9
53.2
64.8
86.3
67.6
66.5
34.2
62.2
SAM
fixed-grid
62.7
62.9
53.7
65.5
86.7
68.8
68.4
34.7
62.9
overlapping
63.1
63.2
53.6
66.7
86.5
69.2
68.8
34.7
63.2
dynamic
63.4
63.2
54.0
67.3
86.9
69.8
70.4
34.8
63.7
no-tiling
60.8
63.2
52.8
64.77
86.4
66.9
66.0
33.4
61.8
S ConvNeXt
fixed-grid
61.9
63.7
53.4
66.1
86.7
68.1
68.9
33.9
62.8
overlapping
61.5
63.4
53.1
65.7
67.5
68.3
33.6
62.5
dynamic
62.3
63.8
53.6
66.5
86.9
68.4
70.8
33.9
63.3
no-tiling
60.4
63.2
52.8
65.2
86.2
68.6
67.2
33.9
62.2
S + DINOv2
fixed-grid
62.8
63.9
53.3
66.0
86.9
70.1
68.4
34.7
63.3
overlapping
61.7
63.7
53.1
65.7
86.9
69.6
68.6
34.0
62.9
dynamic
63.2
61.3
53.9
66.9
87.2
70.5
70.1
31.7
63.9
has not been systematically studiod. Here, we investigate how different tiling methods and MoVE interact when applied together.
To explore this interaction, evaluate four tiling strategics across multiple encoder combinations prepro- cessing for high-resolution inputs: (1) No-liling, where the full image is treated as single tile, (2) Fized-grid tiling, which partitions the image into a uniform grid of equally sized tiles, (3) Overlapping tiling, which uses tiles of the same size but allows adjacent tiles to partially overlap for denser spatial coverage, and (4) Dynamic tiling, which adapts the number and arrangement tiles based on the image aspect ratio while keeping the tile size fixed. These strategies differ how they partition a high-resolution image, and therefore vary in the spatial detail they preserve and the redundancy they introduce. Given an image Ling [ each method partitions it into set of tiles Iites = I2. IN, where cach tile I, R (Huue XWne) has dimension of Hale x Weile-
As an illustrative example, we describe the dynamic tiling procedure in detail. Dynamic tiling Chen et al. (2024b) is aspect-ratio-aver aware adapting the number and shape of tiles to the image geometry. The input image Jmng resized to closest available aspect ratio that dividable into square patches of size 448 448. For example, for common aspect ratios such as 3:2, the procedure produces up six tiles, ensuring coverage of informative regions (see Fig. 3). In parallel, we generate thumbnail representation I, of the full image, which preserves global structure and provides complementary context to the localized tile features. Figure illustrates this tiling process with example driving scene image, where the tiles are shown after normalization by the SAM preprocessor Kirillov et al. 2023
Each tile is then processed independently by two pretrained vision encoders, yielding embeddings I and 1 Bocause these encoders are trained on different domain-specific vision tasks and objectives, they emphasize distinct aspects of the visual signal: one may specialize in semantic categorization, while another better captures fine-grained textures geometric cues. This diversity allows the downstream fusion module to
5

---

Published in Transactions on Machine Learning Research (02/2026)
benefit from complementary perspectives of the same visual input. To reduce the number of visual tokens and ensure that both vision encoders generate the same number of tokens, we apply pixel unshuffling SShi et al. (2016) if necessary. This technique rearranges the spatial layout of pixels, reducing the number of visual tokens while preserving important visual features. Given an imput visual embedding I, E R (V1x((1mha x and a downscaling factor the module outputs T E ((uxx()))
Finally, each segmented tile is represented by 256 visual tokens per encoder. Experimental settings are provided in Appendix A.2
Table I reports the effect incorporating different tiling strategies into MoVE across a variety of vision encoders, in- cluding InternViTI Chen al. (2024c), SAM Kirillov et al. 2023), SigLIP Zhai et al. (2023), ConyNeXt [Woo al.
(203 and DINOv2 Oquab et al. (2024 Across all con-
figurations, we observe three consistent patterns. First, dy- namic uiling achieves the strongest performance, improving the average score in encoder combinations. For example, pairing InternViT with SAM, ConyNeXt, or DINOv2 yields clear gains of 2.3%, 2.6%, and 1.9%, respectively, compared to the no-tiling baseline. This illustrates the benefit of adapt- ing the number of tiles to the image aspect ratio, allowing the model to preserve spatial structure while maintaining a stable token budget. Second, fixed-grid tiling typically ranks second, outperforming both the no-tiling and overlapping settings. Its uniform grid increases spatial coverage, although the inability to adapt to diverse image gcometrics may limit its effectiveness rel- ative to dynamic tiling. Third, overlapping tiling provides only modest gains and often remains close to the no-tiling baseline, suggesting that the redundant coverage introduced by overlapping regions does not yield proportionally more informative tokens for the LLM. Overall, these results demonstrate that tiling and MoVE are complementary: while MoVE uses the diversity of multiple vision encoders, tiling strengthens cach encoder's ability lo process high-resolution visual content, leading to more robust multinodal understanding.
Figure 3: Dynamic tiling with global context. The tiles are shown after preprocessing with the SAM preprocessor Kirillov el. al. (2023)
3.2 Token merging strategies for Tiled MoVE (D2)
Existing MLLM frameworks have explored different ways of merging tokens from multiple vision encoders, aiming ex- ploit their complementary strengths. While improvements in multimodal performance have been consistently reported, the contribution of the merging strategy itself has rarely been iso- lated. In most cases, token merging is entangled with other architectural innovations, making it difficult to assess whether gains arise from the fusion design or from stronger encoder representations. Shi et al. (2025) investigated MoVE fusion strategics with respect to the trade-off between accuracy and efficiency, but their analysis focused image-level fusion in general. In contrast, our work examines token merging strate- gies at the tile level (see Fig 2d): within Tiled MoVE, we per- form a controlled comparison of four representative methods under a consistent tiled MoVE setting.
Table 2: Comparison of various token merg-
ing strategies within Tiled MoVE acroos 8
benchmarks (sce Appendix A.3.
MoVE
Merging
Avg. (8 benchmarks)
SA
64.3
T SAM
1
SI
64.9
CC
64.1
CA
63.5
SA
63.7
T ConvNeXt
SI
63.7
CC
63.5
CA
62.5
SA
65.
SI
66.7
I DINOv2
CC
63.7
CA
62.9
SA
64.2
S + SAM
SI
64.6
CC
63.7
Figure 2c illustrates the four most common token merging strategies used in recent MoVe-based MLLMs. In this work, explore these strategies to combine token sequences from two vision encoders. (1) Sequence Appending (SA) Kar et al. (2024), where tokens from each encoder are concalenated along the sequence dimension. a tile produces D. tokens per en- coder, the combined sequence has length 2D,- (2) Sequence Interleaving (SI) Tong et al. (2024), where tokens are inter- leaved position by position, e.g., 11 122 3...,,...,.. .....,,p.. [..,, which
CA
63.0
SA
63.5
SI
$ + ConyNeXe
GC
63.3
CA
62.5
SA
64.8
SI
S + DINOv2
CC
63.9
CA

---

Published in Transactions on Machine Learning Research (02/2026)
preserves the order while mixing encoder streams. As above, the combined sequence has length 2Dc- (3) Channel Concateraation (CC) [Shi et al. 2025), 1, where instead of extending the sequence length, the features from both encoders at each token position are concatenated along the channel dimension, resulting in vectors of size C1 + C2 per token. (4) Gross-Attention (CA) Li et al. (2024b), where tokens from one encoder act as queries attending to the other encoder's s keys and values, enabling adaptive integration of complementary information through cross-attention.
Table 2 summarizes the effect of different token merging strategies within Tiled MoVE across six encoder pairs. We observe that sequence interleaving achieves the best performance in five out of six combinations, with the only exception being I + ComuNeXt, where it ties with soquence append. We hypothesize that in- terleaving the visual tokens at the tile level helps preserve spatial relationships while improving information integration, leading to stronger overall performance. In contrast, channel concatenation and cross-attention consistently underperform relative to interleaving and appending. Overall, these results highlight the impor- tance of the token merging strategy, with interleaving emerging as the most effective design choice for Tiled MoVE. An efficiency analysis of the four token-merging stralegies revealed distinct accuracy accuracy-throughpae trade-offs (See Appendix A.55.
3.3 Pre- Pre-Aaaptation versus Post Dost-Adaptation Fusion Strategy (D3)
An important design consideration in MoVE models is when fusion should occur: should visual tokens be mergod immediately after feature extraction, or only after cach encoder has been adapted to the multimodal backbone? Most existing MoVE MLLMs Luo et al. (2025): Li al. (2024b); Kar al. (2024); [Shi et al.) (2025); Fan et al. (2024); Lu et al.] (2024a) adopt the former, reffered to pre-adaptation fusion, where tokens from different encoders are merged before the vision-text aligument stage. In this setup, visual tokens from different encoders are first merged (by standard token merging method), and the fused tokens are then mapped into the multimodal space using a single shared projector module.
In contrast, post-adaptation fusion equips each encoder with its own dedicated projector, such that visual embeddings are first aligned independently before being merged. For instance, the outputs I and I from two encoders are mapped through their respective projectors, FI-T, (I + T. and Fi-T, (T;²² producing normalized token sequences prior to fusion. This design allows the integration step to operate representations that are already aligned with the multinodal backbone while preserving encoder-specific characteristics. The relative effectiveness of these two strategies has not yet been systematically investigated in the literature. Accordingly, we design a controlled empirical study within Tiled MoVE to isolate the effect of fusion timing by contrasting pre- and post-adaptation approaches. We adopt a simple and effective two-layer ML. as the projector design. Each vision encoder outpul, T:: and T is independently mapped into aligned token sequences 7 TT2 DDD.D, where D. is the per-tile token length and D. matches the hidden dimension of the LLM. This ensures compatibility with the multimodal backbonn while maintaining encoder-specific characteristics. Experimental settings are provided in Appendix A.2
T
Table 3 presents the comparison between pre- and post-adaptation fusion across several encoder combina- tions. In all cases, post-adaptation yields consistent improvements over pre-adaplation, highlighting the benefits of aligning encoder outputs independently before fusion. On average, aost-adaptation delivers gains of around 2.9% across the evaluated backbones. We hypothesize that these improvements stem from allowing each encoder to independently align its features before fusion, which both preserves encoder-speciffe infor- mation and ensures that the subsequent fusion operates on already normalized representations. Moreover, the consistency of these gains across five different encoder pairs suggests that preserving encoder-specific features during alignment facilitates stronger integration between visual and language representations, ulti- mately leading to better multimodal reasoning and overall performance. We also compare the computational efficiency of pre- and adsttadapt fusion for the I + SAM encoder pair. The latency difference between the two designs minimal, with post-adaptation showing slight advantage (3.19s vs. 3.25s). This in- dicates that both approaches are computationally comparable, with adsttadaptation providing marginally faster generation while achieving higher accuracy.
7

---

Published in Transactions on Machine Learning Research (02/2026)
Table 3: Comparison of pre-adaptation vs, post-adaptation fusion strategies within Tiled MoVE. I, VQA", and MMB denote InternViT, TextVQA, and MMBench, respectively.
MoVE
Fusion Method
VQA"
GQA
VizWiz
MMB
POPE
SEED
SQA
MMVeT
Avg.
pre-adaplation
65.2
64.2
54.7
68.7
87.8
69.1
74.6
35.2
64.9
I 1+ SAM
post-adaplation
68.8
64.8
57.9
72.9
88.0
72.2
78.5
37.2
67.5
pre-adaplation
69.2
62.4
53.6
67.8
86.5
71.2
72.3
33.0
64.5
+ SigLIP
post-adaptation
70.4
64.8
56.5
70.5
88.2
73.0
74.5
36.9
66.8
pre-adaptation
64.2
63.3
54.7
64.7
86.4
68.3
73.4
34.6
63.7
+ ConvNeXt
post-adaptation
67.3
63.7
56.7
66.8
87.4
72.0
75.4
35.2
65.6
pre-adaptation
65.0
62.5
48,4
68.9
86,4
71.6
70.4
31.3
63.1
1+ CLIP
post-adaptation
67.0
63.8
48.4
72.7
87.8
73.0
75.7
34.3
65.3
pre-adaptation
68.5
64.3
56.8
71.3
88.0
72.9
76.8
35.2
66.7
1+ DINOv2
post-adaptation
70.6
64.8
57.3
72.7
87.8
73.2
74.3
35.4
67.0
4 LEO
Having systematically analyzed tiling (D1), token merging strategies (D2), and the timing of fusion (D3), combine these insights into unified multimodal large language model, which call LEO. This section introduces the overall design of LEO, constructed with the best-performing settings identified in our empirical study: dynamic tiling for high-resolution processing, sequence interleaving for effective token merging, and post-adaptation fusion for stronger alignment across encoders. In this Section, we describe the model's architecture in detail and further discuss how LEO can be adapted to autonomous driving scenarios, where fine- areiggaanod perception and robust reasoning are both critical.
LEO begins by dividing input image Fing xxx into set of tiles Ttils (1, 12. ........ [N), along the lines of Chen et al. (0024)), where each tile In E Ha captures high-resolution patch of the scene. This tiling step preserves fine-grained details without exceeding the token budget.
Each tile is processed by two complementary vision encoders, VE1 and VÆ2. In practice, our implementation follows a VIT projector LLM pipeline, using internViTT300M Chen et al. (2024c) as the first encoder, selected for strong vision language alignment and SAM-L Kirillov al. (2023) as the second ecnoder, chosen for its ability to capture segmentation- based, region-level features (see Section 3 Architecturally, LEO does not impose any explicit weighting priority tokens from either encoder.
The two encoders extract embeddings E" RixV and 1th E R2XV,, which are compressed through pixel unshuffling to yield compact representations Int and I These are projected into the multimodal space using two independent MI.P-bseed projectors, ry-ı; (I") > Th and TI-I,, (T*²) + T where Th T E DD.xD. are aligned token sequences. This post-adaptation design ensures that each encoder is independently normalized before fusion, preserving encoder-specific information.
For fusion, LEO employs tilc-level sequence intcrleaving strategy F(T + T. which merges tokens from the two encoders in alternating order within each tile. This design maintains local spatial structure while promoting cross-encoder interaction. The fused visual tokens T, are then combined with text tokens T, and processed by the multimodal backbone LLM (T,, Tī) for joint vision language reasoning:
L II ply T.,T..
p(Y
T., T))
(1)
4.1 Adaption to autonomous driving
Although numerous studics have successfully applied MLLMs autonomous driving (Marcu al. 2024 Cao et al. 2024 Tian al.] 2024]] Wang al.] (202)). straightforward approach that avoids extensive modifications to model architecture, training processes, heavy data collection has yet. be fully explored.
8

---

Published in Transactions on Machine Learning Research (02/2026)
In this work, we investigate the potential of applying LEO the autonomous driving domain without altering its architecture or training recipe, aiming to offer insights into stremmlined transfer learning and facilitate MLLM adaptation to specialized domains. Instruction tuning plays a crucial role in helping models learn to follow user prompts, utilizing training data in visual question answering and conversational formats. For this domain, we design lasks in VQA format, with each frame represented as: <img> <IMG-CONTEXT, </img>> At the prompt level, the temporal aspect of video frames is managed by treating sequential frames as multiple image inputs. A sample prompt is formulated " "<imagel> ... <image N> Is it safe to enter the intersection at this time?".
5 Experiments
5.1 Implementation Details.
The training of LEO is performed in two stages. The first stage serves warm-up phase for the projector layers, where both vision encoders remain frozen and optimization focuses solely on the projector modules to ensure stable and effective aligmment. To mitigate representation inconsistencies, the initialization of the projector MLPs depends on the encoder type: if the encoder has been protrained on a vision language lask, we initialize its projector from pretrained weights; otherwise, we initialize it. randomly. The second stage involves supervised instruction tuning, where the projector modules are unfrozen together with the language model. Both stages are trained with context length of 8196 tokens using the AdamW optimizer and a cosine learning rate schedule. In the first stage, apply learning rate of 4 x 10-4 with a weight decay of 0.01, while in the second stage the learning rate is reduced to 4 x 10-5 with the same weight decay. Each stage is trained for one epoch.
Training Procedure.
Training Datasets. For the first (alignment) stage, we use the LLaVA-595K dataset Liu et al. (024c)) which contains 595K vision-language instruction pairs. For the second (supervised fine-tuning) stage, we follow the same dataset setup as the baseline model instruction tuning samples, all publicly available.
modellChen Chen al. (2024c), comprising approximately 1M visual
Training Infrastructure. Training was performed on 8 NVIDIA A100 GPUs (80 GB GB each) using Deep- Speed's ZeRO-2 optimization strategy. The complete training process took approximately 72 hours.
Benchmarks. We evaluate LEO on a broad suite multimodal benchmarks, grouped into three cate- gories. (1) OCR and chart understanding: DocVQA MMthhw et al. (2021), Text (Singh et al. (2019). ChartQA Masry et al. (2022). and AI2D Kembhavi al. (2016). (2) General VQA: GQA Hudson Manning (2019) VizWiz Gurari et al. (2018), and VQA v2 Goyal al. (2017). : (3) General mullimodal evaluation:
MMMU Yue et al. (2024), MMBench Liu et al. (2025), SEED Li et al. (2024a). POPE Li et al. (2023b), 2024 and ScienceQA Lu et al. (2222) MathVista Lu ct al. (2024b). In addition, we assess LEO in the autonomous driving domain using LingoQA Marcu et al. (2024) which requires reasoning
MM-Vet Yu et al.
about road scenes and language-based driving queries. Further details are provided in Appendix A.1
5.2 Main Results
Table presents a comprehensive comparison of LEO with existing MoVE-based MLLMs across 11 bench- marks. Overall, LEO delivers strong results in out of 11 tasks, demonstrating the effectiveness of the design principles established through our empirical analysis. In particular, LEO achieves strong performance DQA (80.1) and ScienceQA (78.5), with margins of improvement that substantially greater than those observed in prior models, indicating robust capabilities in both text-heavy OCR settings and reasoning- oriented benchmarks. LEO also records clear gains on ChartQA (+3.2%) and VizWiz (+3.7%), showing that the proposed architecture can generalize effectively to both structured visual data and challenging real-world images. On MMBench (72.9), LEO remains competitive, ranking just below the top-performing system. Im- portantly, these improvements are achieved with substantially less pretraining and instruction-tuningg data than models such as SPHINX Lin et al. (2023) and DeepSeck-VV VL Fan al. (2024), demonstrating that Leo's performance gains stem from architectural design rather than data scale.
9

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: Comparison with MoVE MLLMs on 11 evaluation benchmarks. All models use a 7B language
model. The best and second best values are shown in bold and underlined, respectively.
TexsVQ
ChartQA
MMBench
DoEVQA
SciencoRQA
MathVista
Vie
MMVes
POPE
SPT
GQA
SEEED
Model
Brave-X5 X55
Kar ol.
22024
100 M
NA
52.7
54.2
87.6
Deepsoole
VT Pan ct nl
2124
3.7 M
3 M
73.2
41.5
88.1
70.4
36.1
Eagle-X2
Shi et al.
2026
595 K
1.8 M
63.2
67.0
48
77.7
88.3
73.5
70.7
Eagle-X3
S00 Bl.
2026
67.8
51.7
77.7
89
73.9
69.4
LLaVA-HR
Luo
2025
1.2
18.7
31.2
87.6
61.2
65.1
Mini-Gemial
21246
1.2 M
1.5 M
65.2
69.3
40.8
71.1
31.4
MoME- X3
Shen el al
2022
NA
NA
53.2
59.7
57.2
50.8
MouSi.
am el al.
2724
1.2 M
1.6 M
53.4
60.5
65.4
29.1
85.4
62.0
71.6
Xo
21/24
32.2
87.3
70.2
SPHINX
2023
NA
51.6
62.6
39.9
66.9
36.0
80.7
69.1
69.3
LBO
1 M
68.8
64.8
71.0
57.9
72.9
37.2
80.1
88.0
72.2
78.5
37.2
1.7
10.6
13.2
13.7
10.3
14.3
12.4
10.9
16.9
11.1
A closer comparison with strong baselines further highlights the efficiency of our approach. Models such LLaVA-HR Luo et al. (2025) and Mini-Gemini Li et al. (2024b), which employ more complex fusion mech- anisms, still lag behind on reasoning-intensivee tasks including SEED and ScienceQA. Likewise, Brave Kar et al. 2024), despite integrating as many as five vision encoders through pre-adaptation fusion, achieves weaker or only comparable results. LEO attains competitive or superior performance with just two encoders. These findings suggest that the consistent improvements of LEO arise from the design principles identified our empirical study, including tiling for high-resolution processing, sequence interleaving for balanced token integration, and post-adaptation fusion for preserving encoder-specifie strengths. Taken together, these re- sults establish LEO as both data- and computation-effeient multimodal model, capable of outperforming or matching more heavily engineered MoVE MLLMs while using a straightforward and effective architecture.
5.3 Ablation studies
Analysis of training strategies. To more effectively analyze the impact of training strategies for the vision encoders, we conduct ablation studies to test whether unfreczing the vision backbones improves performance (Table 5). We first observe that using single encoder alone results in reduced performance: InternViT by itself achieves reasonable average score of 60.8, while SAM alone performs much worse (51.4), reflecting its specialization in segmentation rather than general visnal-language understanding. Freezing SAM slightly improves stability and yiclds small gain (53.7), but it still lags far behind InternViT. Combining InternViT and SAM without freezing substantially boosts performance to 65.6, confirming that their complementary strengths InternViT's strong vision language alignment and SAM's fine-grane ane-grained region cues synergize when trained together.
Interestingly, the results in Table show that the best performance is achieved when both vision encoders are frozen at SFT, reaching an average of 67.5. This finding stands in sharp contrast to Eagle Shi et al. (2025), where unfreezing the encoders was found to necessary for competitive results. We hypothe- size that this difference stems from the tiling setup in Tiled MoVE: since each encoder already processes high-resolution patches, their pretrained representa- tions remain highly effective without further adapta- tion. Freezing prevents catastrophic forgetting of these pretrained priors and shifts the burden of adaptation to the lightweight projector modules, which can more efficiently normalize encoder-specific embeddings into the multimodal space. This design leads to more stable training and better overall reasoning performance.
Table 5: Ablation study on training settings for vi- sion backbones (full results in Appendix A.3)
InternViT
SAM
Freeze
Avg. (8 benchmarks)
X
60.8
X
51.4
53.7
65.6
67.5
10

---

Published in Transactions on Machine Learning Research (02/2026)
Analysis of fusion strategies. To further isolate the contribution of individual design choices in LEO,
Table 6: Ablation study on various types of fu- sion strategies and SFT data (full results in Ap- pendix A.3
we conduct an ablation across eight benchmarks, with
average results summarized in Table 0 While Sec- tions 3.2 and 3.3 examined loken merging and fusion strategies more broadly, here we focus on their im- pact within the finalized LEO setting. We find that tile-level sequence interleaving yields the best perfor- mance (avg. 67.5), surpassing both sequence append- ing (avg. 56.5) and channel concatenation (avg. 66.1). This confirms the advantages of interleaving in bal- ancing token integration across encoders. In addition, removing dynamic tiling leads to clear drop in per-
Model
Avg. (8 benchmarks)
LEO
67.5
w/ pre-adaptation
64.9
w/ sequence appending
66.5
w/ channel concatenation
66.1
w/o tile segmentation
64.6
w/ 1.8M SFT data
67.3
formance (avg. 64.6), underscoring the role of high-resolution tiling in preserving fine-grained delails for visual understanding. Finally, post-adaptation fusion again outperforms pre-adaptation, reinforcing that aligning encoder outputs independently before fusion facilitates stronger integration with the multimodal backbone. Together, these results echo the broader analyses from Sections 3.2 and 3.3 while providing a controlled validation of LEO'S design principles.
Effect of training data scale. We evaluate the impact of increasing SFT data by training LEO with Eagle- 1.8M SFT data Shi al. (2025). While this leads to improvements on three benchmarks (see Appendix(A.. AA.3). the overall average remains stable (67.3 vs. 67.5), suggesting that LEO achieves strong performance even with limited data, demonstrating its training efficiency and robustnoss.
5.4 How Does LEO Compare to
Models with Knowledge Distillation?
To further contextualize LEO is performance, Table 7 compares it against models that em- ploy single vision encoder obtained through multi-teached knowledge distillation Ranzinger
Table 7: Comparison with knowledge distillation (multiple vision encoder teachers) approaches.
GQA
ChartQA
VQA"
DocVQA
POPE
RADIO Ranzinger es al.
2024
63.01
56.32
86.20
et al.)) 2024) ;; Heinrich el al. (2025 Cao et al.
RADIO25 Hemrich at.
21725
30.40
69.74
52.33
(2025). These approaches compress informa-
MoVE-KD
2025
63.9
59.6
86.3
Leo
64.8
71.0
68.8
80.1
88.0
tion from several teacher encoders into a single
backbone during training. For example, RADIO uses per-teacher adapters only during the distillation stage to better align teacher features; for the VQA task at inference, the trained student model operates standard single vision encoder followed by two-layer MLP adaptor. Consistent with their original design, these models are evaluated using their default single-tile input resolution. Despite relying on substantially greater pretraining (RADIO is trained on 614M samples with 64 GPUs, and RADIO-2.5 on 9.8M samples), LEO achieves competitive or stronger performance across most benchmarks using only 1M training samples and GPUs It worth mentioning that knowledge-distilled eucoders such as RADIO or MoVE-KD's encoder have lower inference-tine time latency than MoVE encoders, since they require only a single backbone forward pass, whereas MoVE-bbsed approaches perform one forward pass per encoder. Furthermore, unlike knowledge distillation-based models, which blur the representational roles of individual vision experts, LEO preserves modularity and interpretability by explicitly integrating multiple encoders. Moreover, LEO shows equally strong performance against general MLLMs with similar resource constraints (See Appendix A.4)
5.5 Efficiency analysis
Table compares the efficiency of LEO and Eagle Shi et al. (0255) in terms of vision encoder parameters, latency, FLOPs, and generation time, using 1024x 1024 input image with the prompt "describe this image in ".tail". LEO demonstrates significant efficiency improvements with only 612M vision encoder parameters just over half the parameters of Eaglo-X2 and less than half those of Eaglo-X3. Despite higher latency due to SAM's lack of flash attention support Dao (0024), LEO achieves a notable 61.6% reduction in vision encoder FLOPSs and a 19.6% decrease in generation time compared to Fagle-X3, highlighting its efficiency.
11

---

Published in Transactions on Machine Learning Research (02/2026)
5.6 Results in the autonomous driving domain
We evaluate LEO the LingoQA validation set Marcu et al. (2024), with results presented in Table Against the closed-source Lin- goQA baseline Marcu et al. (2024), which pretrained on over 22M data samples, LEO demonstrates competitive performance on the Lingo-J and BLUE metries, and significantly outperforms the baseline the METEOR. and CIDEr metries. Without modifying its archi- tecture or training recipe, LEO also surpasses all existing open-source baselines across all four metrics. Notably, LEO achieves higher scores than the top- performing model of Chen et al. (2024c). These results indicate that LEO can better capture fine-grained multimodal cues, critical property for tasks such as scene understanding and instruction following in dynamic driving environments.
Table 8: Results on the LingoQ.A benchmark Marcu et al.
(2024). N denotes the number of frames used during train-
ing. Lingo-J represents the Lingo-Judge metric.
Model
N Lingo-J
BLUE
METEOR
CIDEr
BLIP 2 et al 20236
52.20
13:00
17.40
60.10
LLaVA
51.00
29.44
48.18
InternV.
Chen
58.00
13.53
34.27
67.17
LingQA
Marcu at al.
2024
3
50.80
14.61
18.44
62.61
LiugoQA
Maren ct a.
2112
5
60.80
15.00
18.36
65.62
LEO (ours)
2
61.00
35.44
69.72
5.7 Visualization
To highlight the visual understanding capabilities of LEO, we present a qualitative analysis in Fig. + Our model is applied to a variety of vision-language tasks, including complex reasoning, detailed count- ing, OCR, spatial and mathematical reasoning, accounting analysis, and multi-image and multi- frame reasoning. With an efficient tile-level post- adaptation fusion strategy, LEO exhibits impressive performance across these challenging lasks. For example, our model can perform attribute-based counting, such as identifying the absence of parked cars while there are several moving vehicles the driving scene. Beyond simple recognition, LEO demonstrates spatial awareness, enabling to answer OCR-related ques- tions like " What is located to the right of the shampoo?" In multi-image reasoning, LEO accurately identifies detail differences between images, such as the dog's head being in different positions. LEO also demonstrates strong capabilities in multi-frame reasoning in the autonomous driving domain, including recognizing safe actions in dynamic scenes, such as stopping to allow a pedestrian to cross. Finally, LEO excels in OCR tasks, effectively interpreting dense texl, and also handles complex mathematical and accounting problems, showcasing its strong reasoning abilities. See Appendix A.7
Table 9: Efficiency analysis of LEO. 'GT" and 'lat"
denote generation time and latency, respectively.
Model
VE Param.
VE lat.
VE FLOOPs
GT
Eagle-X3
1460M
0.07s
4281
3.97s
Eagle-X2
1155M
0.04s
3347
3.92s
LEO
612M
0.788
1642
3.19s
5.8 Limitations
The processing capacity of our model is limited to maximum of six image patches (excluding the global context), constrained by the LLM's context length and available computational resources. This restriction hinders support for higher-resolutee over, due to computational constraints, our setup relics on a fixed set of vision encoders and a 7B LLM, which, while efficient, may limit the diversity of visual features and the breadth and depth of the system's reasoning abilities.
images or tasks requiring larger number of multi-image inputs. More-
6 Conclusion
Through systematic empirical study, we have investigated how different design choices within MoVE influence the elfectiveness of multimodal large language models. We organized our study around three investigative directions: visual reasoning enhancement methodologies, token merging strategies, and the timing of fusion. From our experiments, we have derived clear principles for effectively combining multiple vision encoders. We have demonstrated that possible to consistently strengthen multinodal reasoning in computationally efficient. way by means of judicious lightweight. design choices. Specifically, by integrating
12

---

Published in Transactions on Machine Learning Research (02/2026)
Reasoning:
Complex counting:
Is summenine scnn! Atewer give the nitooal::
LBO: No. Paspkin paiclss are typscally asecisted will crumn as they ase havvaard darieg the scwon preaene pumpkine it the srrne seggort der image was taken daring scace, armmer.
OCR:
How many parked cars can you spot!
LEO: I can't see any parked cans
Accounting and math:
Unit
Direct manerials
$10
Direct labor
manufactuuing ovartead
Ver able manulaa
manuuaduunnn overneed
75g
sales
expenses
Wha oprir A. Y- C. V wnteed
What located the right the shampoo?
What first website name on the page?
How many grams are the white chocolate?
Border Company hos relevart nage production betwom 15,000 and 30,000 unis. The following cost data represents average variblle costs per unit fir 25,000 unias of prsdtacionn 15 0000 unils are produced, what are the perunt manufictrning overhead costs incurred! A. $10.
LBO: london- undegroundbboggoot.ccm blogspot.com
Hair conditioner
underground
LED:
B,511. C.212 D.513.
LED:C
ontario library
Multi-image reasoning:
What does
for'!
Tell me the difference between imagel and image
LEO: Ontario
The difference between the two images the dog's havd position, image the dog head is turnal to the left, whille in image 2, the dog theing dinetly the comera.
Multi-frame cullti-rame reasoning:
What is the curentt action and its justifiction?? Answer in the form "action, justification.
How many cars are driving your direction?
L.FO: am stopping let polestriun safely cross on the zebm or podestria crossing abead.
1.FO: There are cam driving my direction
Figure 4: Qualitative results of LEO'S enhanced visual understanding on various vision-language lasks. Images are taken from the following benchamrks: MMVet Yu et al. (2024). MMMU Yue et al. (2024) TextVQA VQA Singh et al. (2019), and LingoQA Marcu et al. (2024)
dynamic tiling with MoVE, by employing tile-level sequence interleaving, and by adopting post-ndaptation fusion. Using these principles, we designed a compact and powerful architecture that call LEO. LEO achieves better results than prior MoVE models on the majority of benchmarks and is competitive with the best models on benchmarks that it does not lead. Leo further demonstrates its generalizability by transferring effectively to the specialized domain of autonomous driving without requiring extensive domain- specific adjust We envision LEO as a foundation for advancing MoVe-based MLLMS and as a practical guide for adapting them efficiently to domain-specife applications.
Broader Impact Statement
The proposed LEO framework extends the capabilities of existing vasion-language models and therefore shares many of their broader social impacts. As with other multimodal systems, potential risks include biased or misleading outputs, privacy concerns, and the propagation of harmful content. These risks are mitigated in part by thoughtful data selection and the application of safeguards designed to encourage safe and responsible use. We present. LEO primarily as a research contribution, and recommend that its outputs be applied with caution in sensitive contexts, with human oversight to mitigate possible misuse.
Acknowledgement
This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC)) 's Postdoctoral Fellowship and NSERC-CSE Research Community project entitled "An End-to-End Approach to Safe and Secure AI Systems". Researchers funded through the NSERC-CSE Research Communities Grants do not represent the Communications Security Fstablishment. Canada the Government of Canada. Any
13

---

Published in Transactions on Machine Learning Research (02/2026)
rescarch, opinions, positions they produce as part of this initiative do not represent the official views of the Government of Canada.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Nlge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv: 2303.08774, 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurPPS), pp- 23716-23736.
Curran Associates, 2022.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Danicl Salz, Maxim Neumann, Tbrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, el al. PaliGemma: A versatile 3B VLM for transfer. arXiv: 2407.07726, 2024.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. InternLM2 technical report. arXiv: 2403.17297, 2024.
Jiajun Cao, Yuan Zhang, Tao Huang, Ming Lu, Qizhe Zhang, Ruichuan An, Ningning Ma, and Shanghang Zhang. MoVE-KD: Knowledge distillation for vlns with mixture of visual encoders. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 19846-188 1946-19856, June 2025.
Xu Cao, Tong Zhou, Yunsheng Ma, Wenqian Ye, Can Cui, Kun Tang, Zhipeng Cao, Kaizhao Liang, Ziran Wang, James M Rehg, et al. MAPLM: A real-world large-scale vision-language benchmark for map and traffic scene understanding. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition (CVPR), pp. 21819 21830, 2024.
Jiuhai Chen, Jianwei Yang, Haiping Wu, Dianqi Li, Jianfeng Gao, Tianyi Zhou, and Bin Xiao. Florence- VL.: Enhancing vision-language models with generative vision encoder and depth-breadie fusion. In Proceedings of the Computer Vision and Pottern Recognition Conference (CVPR), pp. 24928 24988, 2025.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv: 2412.05271, 2024a.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are models with open-source suites. Science China Information Sciences, 67(12):220101, 2024b.
GPT-4V? Closing the gap to commercial multimodal
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu,, et al. InternVI.: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24185-24199 2024c.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open- chatbot impressing GPT-4 with 90%* ChatGPT quality. Blog, https://lmss:: www.oo/r/////////oooooooo
-30--000000000---- March 30 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In 37th Conference on Newral Information Processing Syslems (NeurIPS). 2023.
Tri Dao. F 2: Faster attention with better parallelism and work partitioning. In 12th Interna- tional Conference on Learning Representations (ICLR), 2024.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Danny Driess, Fei Xia, Mchdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM-E: An embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202, pp. 8469 8488. PMLR, 2023.
Xisoran Fan, Tao Ji, Changhao Jiang, Shuo Li, Senjie Jin, Sirui Song, Junke Wang, Boyang Hong. Lu Chen, Guodong Zheng, et al. MouSi: Poly-visual-epert vision-lagguage models. arXiv: 2401.17221, 2024.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijic Geng, Aojun Zhou, Wei Zhang. Pan Lu, Conghui He, Xiangyu Yue, et al. LLaMA-Adapler v2: Parameter-elfiiint visual instruction model. arXiv: 2304,15010, 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognation (CVPR), pp. 6904-6913, 2017.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. VizWiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 3608-3617, 2018.
Greg Heinrich, Mike Ranzinger, Yin Hongxu, Yao Lu, Jan Kautz, Andrew Tao, Bryan Catanzaro, and Pavlo Molchanov. RADIOv2.5: Improved basclincs for agglomerative vision foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognilion (CVPR), pp. 22487-222 22497,, June 2025.
Drew A Hudson and Christopher D Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEFE/CV Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6700 6709, 2019.
Oguzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. BRAVE: Broadening the visual encoding of vision-language models. In European Conference Computer Vision (ECCV), pp. 132. Springer, 2024.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Soo, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In Compuler Vision - ECCV 2016, pp. 235 251. Springer, 2016.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tote Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceediiggs of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4015 4026, 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vision, 123 (1):32-73, May 2017.
Byung- Kwan Lee, Beomchan Park, Chae Won Kim, and Yong Man Ro. MoAl: Mixture of all intelligence for large language and vision models. In European Conference on Computer Vision (ECCV), pp. 273-302. Springer, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-OneVision. Easy visual task transfer. Transactions on Machine Learning Research, 2025.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. SEED- Bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), pp. 13299 13308, 2024a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning (ICML), pp. 19730-19742. PMI.R, 2023a.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Yanwei Li, Yuechen Zhang, Chengyao Wang. Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-Gemini: Mining the potential of multi-modality vision language models. arXiv: 2403.18814, 2024b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large lisionllanguage models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023b.
Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-moda models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 2024c.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. VILA: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26689 26699, 2024.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp. 740 755, Cham, 2014. Springer International Publishing.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. SPHINX: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arviv: 2311.07575, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26296 26306, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaVA- NeXT: Improved reasoning, OCR, and world knowledge. Blog, https://Alava-vl gith.g. /llg//////// 2024-01-30-11av--nes January 30 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NewrIPS), 2024c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. MMBench: Is your multi-modal model an all-around player? In European Conference on Computer Vision (ECCV), pp. 216 233. Springer, 2025.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. DeepSeek-VIL VL: towards real-world vision- lssion-language understanding, arXiv: 2403.05525, 2024a.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Wei Chang, Song-Chus Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Newral Information Processing Systems (NeurIPS), pp. 2507 -2221, 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In 12th International Conference on Learning Representations (ICLR), 2024b.
Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: Mixturro-ofrreslutiio adaptation for multimodal large language models. In 13th International Conference on Learning Representations (TCLR), 2025.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. LingoQA: Visual question answering for autonomous driving. In European Conference on Computer Vision (ECCV), pp. 252 269, 2024.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-2279. ACL, May 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for VQA on document images. In Proceedings of the IEEE/CVE winter conference on applications of computer vision, pp. 2209,
2021.
Meta AL. LLaMA 3.2: Bringing vision and intelligence to edge and mobile devices. Blog, https: meta. 2024. Accessed July 17, 2025.
com/blog/ 1lama http//oog//mmmm-22-2---e--e-0--- htt.a:/ccccccc000aaaaaaaaanaeeee
OpenAL. Hello GPT-40.
2024. Accessed July 17, 2025.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pp. 8748-8763. PMLR, 2021.
Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. AM-RADIO: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVP Conference Computer Vision and Paltern Recognition (CVPR), pp. 12490 12500, June 2024.
Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. MoME: Mixture of multimodal experts for generalist multimodal large language models. In Advances in Neural Information Processing Systems (NeurlPS), pp. 42048 42070, 2025.
Baiseng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do we not need larger vision models? In European Conference on Computer Vision (ECCV), pp. 444 462. Springer, 2024.
Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, Yilin Zhao, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, et al. Eagle: Exploring the design space for multimodal LLMs with mixture of encoders. In 13th International Conference on Learning Representations (ICLR), 2025.
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueck- ert, and Zehan Wang. Real-time single image and video super-resolution using efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1874-1883, 2016.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8317-8326, 2019.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv: 200023333110 15389, 2023.
Gemini Team, Rohan Anil, Scbastian Borgcaud, Yonghui Wu, Sean-Baptiste Alayrac, Jiahui Yu, Radu Sori- cut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv: 2312.111. 2023.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Thomas Tian, Boyi Li, Xinshuo Weng, Yuxiao Chen, Edward Schmerling, Yue Wang, Boris Ivanovic, and Marco Pavone. Tokenize the world into object-level knowledge to address long-tail events in autonomous driving. In 8th Annual Conference on Robot Learning, 2024.
Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri TYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. In Advances in Neural Information Processing Systems (NewrIPS), pp. 87310 87356, 2025.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal LLMs. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognétion (CVPR), pp. 9568 00089958, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Amm Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv: 2302.13971, 2023.
Pavan Kumar Anasosalu Vasu, Fartash Faghri, Chun-Liang Li, Cem Koc, Nate True, Albert Antony, Gokula Santhanam, James Gabriel, Peter Grasch, Oncel Tuzel, et al. Vas Efficient vision encoding for vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pp. 19769 19780, 2025.
Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose Alvarez. OmniDrive: A holistic LLM-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv: 2405.01533 2024.
Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language model. In European Conference on Computer Vision (ECCV), pp. 408 424. Springer, 2024.
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. ConvNeXt V2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.
2023.
Ziheng Wu, Zhenghao Chen, Ruipu Luo, Can Zhang, Yuan Gao, Zhentao He, Xian Wang, Haoran Lin, and Minghui Qiu. Valloy2: Exploring multimodal models with scalable vision-language design.
arXiv: 2501.05901, 2025.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Li- juan Wang. MM-Vet: Evaluating large multinodal models for integrated capabilities. In International Conference on Machine Learning (ICML), 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition (CVPR), pp. 9556 (5669,56, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pro-training. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 11975 11986, 2023.
Yipeng Zhang, Yifan Liu, Zonghao Guo, Yidan Zhang, Xuesong Yang, Chi Chen, Jun Song, Bo Zheng, Yuan Yao, Zhiyuan Liu, et al. LLAVA-UHD v2: an MLLM integrating high-resolution feature pyramid via hierarchical window transformer. arXiv: 2412.13871, 2024.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, al. InternVL3, Exploring advanced training and test-time recipes for open-source multimodal models. arXiv: 1044, 2025.
18

---

Published in Transactions on Machine Learning Research (02/2026)
Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and Yu Liu. MoVA: Adapting mixture of vision experts to multimodal context. In 38th Annual Conference Newral Information Processing Systems (NeurIPS), 2024.
A
Appendix
Benchmark Datasets
Experimental Scttings
Benchmark Details
A
LEO performance beyond the MoVE setting
Additional efficiency analysis
A
6
Does Scaling the Training Data Improve Cross-Attents Merging Performance?
7
Additional Visualization Results
A.1
Benchmark Datasets
This section provides a more detailed description of the evaluation benchmarks, highlighting their key fea- tures. Unless specified otherwise, we report scores based on the performance models evaluated on the provided test splits. We consider the following benchmarks:
OCR and chart understanding benchmarks:
Doc Mathew et al. (2021). . Visual Question Answering (VQA) benchmark for document images, featuring 50K questions on more than 12K images sourced from the UCSF Industry Docu- ments Library. It contains diverse set of document types (letters, forms, tables, reports), assessing the capabilities of multimodal models in text recognition, document understanding (structure and context), and figure interpretation. The dataset is composed of questions categorized into 9 types, including form-based, table/ table/list-base blis-bbase,,, sysout-based. running text, handwritten text, figure-based, sastograp--based. Ycs/No, and other.
TextVQA (VQAT) Singh et al. (2019). A VQA benchmark designed to evaluate the ability of MLLMs to read and interpret text. within images. consists of 45,336 questions paired with 28,408 images with a focus on categorics that frequently contain text, such as billboards, street signs, and product packaging. Each question-image pair includes len human-provided ground truth answers. The dataset is designed necessitate OCR for accurate question answering, many responses rely on text embedded within the visual scene.
ChartQA Masry et al. (2022). VQA benchmark for charts, designed evaluate MLLMs' ability to perform visual and logical reasoning. It consists of 9,608 auman-authored questions and 23,111 machine-generated questions based 20,882 real-world charts collected from diverse sources. The dataset covers bar, line, and pie charts, ensuring a variety of styles and topics. The questions are categorized into data retrieval, visual reasoning, compositional reasoning, and combined visual- compositional reasoning, requiring models to extract data, interpret visual attributes, and perform arithmetic/logical operations.
AI2D Kembhavi et al. (2016). A VQA benchmark for diagram understanding and reasoning, for scientific diagrams. It comprises 5,000 grade-school science diagrams, annolated with 118,000+ labelled constituents and 53,000+ relationships, along with 15,000 multiple-choice questions that test comprchension and reasoning. The datasot covers diverse set of scientific concepts such as food webs, life cycles, and planetary systems. The dataset divided into a training of 4,000 diagram images and blind test set consisting of 1,000 images.
19

---

Published in Transactions on Machine Learning Research (02/2026)
2. General visual question answering benchmarks:
VQA" 12 Goyal et al. (2017). A VQA benchmark, designed to reduce language biases and assess MLLMs ability to rely on image understanding. It consists of approximately 1.1 million question- image pairs, with 13 million answers sourced from ~ 200, 000 images in the COCO datasct. A key feature of VQA 22 is its balanced question-image pairs, where each question is linked to two similar images that yield different answers, ensuring that models cannot rely solely on language priors. The dataset covers diverse question types, including ycs/no, number, and open-ended questions.
GQA Hudson & Manning (2019). A VQA benchmark designed to evaluate visual reasoning capabilities by incorporating scene graph representations of images. It consists of 22 million ques- tions derived from 113,000 real-world images sourced from the Visual Genome dataset KKishha et al. (017)), with each question grounded in a structured scene graph that captures object relationships, aurributes, and spatial arrangements. This dataset emphasizes compositional reasoning, linguistic clarity, and reduced bias, ensuring questions are well-formed, diverse, and require multi-step infer- ence. The datasct includes detailed annotations specifying the reasoning steps involved, supporting fine-grained evaluation of model interprelability and robustness. Also, GQA provides balanced question-answer pairs to mitigate dataset biases and includes multiple difficulty levels.
Viz Gurari et al. (2018). A VQA benchmark designed to address challenges faced by blind individuals in obtaining visual information. It comprises over 31,000 visual questions, where each consists of an image captured by blind user via a mobile phone and spoken question that was later transcribed. Each question is paired with 10 crowd-sourced answers, allowing for a robust evaluation of answer variability. Unlike cxisting VQA datascts, VizWiz introduces real-world complexities such as poor image quality, conversational question phrasing, and a high rate of unanswerable questions (~ 28%), reflecting the natural uncertainties blind users face.
3. General multimodal benchmarks:
MMMU Yue et al. (2024). The Massive Multi-discipline Multimodal Understanding and Rea- soning benchmark is dataset designed to evaluate MLLMs on expert-level perception and reasoning across a diverse set of academic disciplines. It contains 11.5K multimodal questions sourced from college exams, quizzes, and textbooks, covering six broad disciplines including Art & Design, Busi- ness, Science, Health & Medicine, Humanitics & Social Sciences, and Technology & Engincering, spanning 30 subjects and 183 subfields. The dataset includes 30 heterogeneous image lypes, such as charts, diagrams, medical scans, sheet music, and chemical structures, to test models' ability to integrate textual and visual information. Unlike existing benchmarks, MMMU emphasizes complex domain-specific reasoning, requiring models to apply advanced subject knowledge to solve problems.
MMBench Liu et al. (2025). A bilingual, multimodal benchmark designed to rigorously evaluate MLLMs across 20 fine- -ue-gra ability dimensions. It consists of over 3,000 multiple-choice questions, covering a diverse range of tasks including object localization, social reasoning, structuralized image- lext. understanding, and spatial relationships. MMBBnnc introduces novel CircularEval strategy to ensure robust evaluation by testing models multiple times with shuffled answer choices. Additionally, it incorporates GPT-4-based choice extraction to handle free-form model predictions, improving evaluation accuracy. We report results on the English subset of MMBench.
SEED-Bench Li et al. (2024a). . A multimodal benchmark designed to evaluate generative com- prehension in MLLMs. It consists of 19,000 multiple-choice questions, making it six times larger than existing benchmarks, with numan-annotated ground-truth answers to ensure evaluation accuracy. SEED-Bench spans 12 evaluation dimensions covering both spatial and temporal understanding, incorporating image and video modalities. These dimensions are as follows: scene understand- ing, instance identity, instance attributes, instance localization, instance counting, spatial relations, instance interaction, visual reasoning, text recognition, action recognition, action prediction, and procedure understanding. We report results on the image set of this dataset.
POPE Li et al. (2023b). Polling-based Object Probing Evaluation is an evaluation pipeline designed to systematically evaluate object hallucination in MLLMs. Following the approach of Li et. al. 2023b), we evaluate MLLLMs with POPE built on the validation set of MSCOCO Lin et al.
20

---

Published in Transactions on Machine Learning Research (02/2026)
2014). Unlike traditional evaluation methods, POPE formulates object hallucination detection as binary classification task, prompting ML.LMs with Yes/No questions about the presence of specific objects in images. It introduces three distinct negative sampling strategies, including random, popular, and adversarial, to assess the tendency of MLLMs to hallucinate objects based on frequency and co-occurrence patterns in training data. POPE provides a stable, scalable, and fair evaluation framework, making use of both human annotations and automated segmentation tools to extend assessments unannotated datasets.
MMVet Yu et al. 2024): A multimodal benchmark designed to systematically evaluate MLLMs on complex vision-langune tasks by assessing their ability to integrate multiple core capabilities. It defines six fundamental vision-language capabilities, including recognition, OCR, knowledge, lan- guage generation, spatial awareness, and math, and evaluates models across 16 integrated lask categories requiring different capability combinations. MM-Vet contains 200 images and 218 open- ended questions, sourced from various datasets and human annotations. To evaluate open-ended responses, MetVeet employs a GPT-4-based evaluator, to provide unified and scalable scoring metric across different answer styles and question types.
ScienceQA Lu et al. (2022). multimodal datasct designed to evaluate multi-step reasoning and interpretability Al systems through science-based question answering. comprises 21,208 multiple-choice questions soureed from elementary and high school science curricula, covering nat- ural science, social science, and language science across 26 topics, 127 categorics, and 379 skills. This dataset includes rich multimodal contexts, with 48.7% of questions containing images (both diagrams and natural images) and 48.2% containing textual contexts, as well annotated lectures and explanations to support Mhain-of-thougll (CoT) reasoning. The benchmark assesses whether models can generate explanations alongside answers.
MathVista Lu et al. (2024b). A benchmark designed to evaluate the mathematical reasoning capabilitios of foundation models in visually complex contexts. It contains 6,141 examples, drawn from 31 existing multimodal datasets and three newly created ones. MathVista spans five primary tasks, figure question answering, geometry problem solving, math word problems, textbook question answering, and visual question answering, covering seven reasoning types including algebraic, arith- metic, geometry, logical, numeric commonsense, scientific, and statistical reasoning. The dataset features diverse visual contexis, such as natural images, geometry diagrams, scientific figures, charts, tables, and function plots.
4. Autonomous Driving benchmark:
LingoQA Marcu et al. (2024). A large-scale vision-language dataset designed to evaluate visual question answering in autonomous driving. It consists of 28,000 unique short video scenarios, video captioning, and 419.9K question-answer pairs, covering broad range of perception and reasoning based tasks. The dataset features free-form questions and answers, extending beyond object de- tection to include driving behaviour and scenery based questions, while allowing for more robust evaluation with greater answer variability. Question types include a variety of categories that en- compass different aspects of understanding the driving task. Below, we present these categories along with an example of each type.
(a) Action identification: What action are you taking as the current driver?
(b) Action justification: Why are you taking this action?
(c) Object/Scenery identification: What type of structures do you see on the right side of the road?
(d) Object/Scenery description: Can you describe the SUV you see? or How the weather today?
(c) Attention: What are you paying attention to?
(f) Anticipation: What are you adjusting your position in anticipation of?
(g) Object localization: Is the pedestrian crossing the road on a zebra crossing?
(h) Counting: How many parked cars can you spot?
Counterfactual: How should you react if there are pedestrians still crossing the zebra crossing?
21

---

Published in Transactions on Machine Learning Research (02/2026)
Table 10: Comparison various token merging strategies within Tiled MoVE. SA, SI, CC, and CA denote sequence append, sequence interleaving, channel concatenation, and cross-attention, respectively.
MoVE
Merging
VQAT
GQA
VizWiz
MMB
POPE
SEED
SQA
MMVeT
Avg.
SA
63.8
64.0
54.4
68.0
87.5
69.5
72.4
34.7
64.3
SI
65.2
64.2
54.7
68.7
87.8
69.1
74.6
35.2
64.9
I+ SAM
CC
63.4
63.9
54.2
67.5
87.1
70.8
71.2
34.6
64.1
CA
62.6
63.5
53.4
66.9
86.9
68.3
71.4
34.7
63.5
SA
63.5
63.7
53.9
66.5
86.7
68.9
71.5
34.7
63.7
SI
64.2
63.3
54.7
64.7
86.4
68.3
73.4
34.6
63.7
1+ Con
CC
62.6
63.9
54.0
66.4
87.0
68.5
71.1
34.6
63.5
CA
62.3
63.1
53.5
63.5
85.9
67.3
70.7
33.8
62.5
SA
65.4
64.2
55.3
69.2
87.4
70.6
73.9
34.7
65.1
ST
68.5
64.3
56.8
71.3
88.0
72.9
76.8
35.2
66.7
1+ DINOv2
CC
63.3
64.0
53.9
67.1
87.2
68.4
71.0
34.5
63.7
CA
63.4
63.6
54.2
66.2
86.8
64.5
70.3
33.8
62.9
SA
63.6
63.5
54.3
68.0
87.4
69.1
72.5
34.8
64.2
SI
65.0
64.3
54.5
68.3
87.4
68.4
73.9
35.1
64.6
+ SAM
CC
63.4
63.2
54.0
67.3
86.9
69.8
70.4
34.8
63.7
CA
62.6
63.4
53.2
66.3
86.4
68.1
69.8
34.5
63.0
SA
63.4
63.8
53.5
66.3
86.5
68.9
70.9
34.6
63.5
SI
64.5
63.6
54.7
64.6
86.8
69.0
74.1
34.6
64.0
ConvNeXt
CC
62.3
63.8
53.6
66.5
86.9
68.4
70.8
33.9
63.3
CA
62.1
62.8
53.2
64.6
86.1
67.2
70.2
34.0
62.5
SA
65.3
64.0
55.6
68.7
87.0
70.1
73.9
34.1
64.8
SI
68.1
64.2
56.0
70.9
88.2
71.8
76.2
35.9
66.4
S DINOv2
CC
63.2
64.3
53.9
66.9
87.2
70.5
70.1
34.7
63.9
CA
62.7
63.9
53.7
64.5
86.0
66.3
69.6
33.4
62.5
To further assess model performance, LingoQA introduces Lingo-Judge, a learned text classifier that achieves 0.95 Spearman correlation with human ratings of model performance. We calculate Lingo-Judge (Lingo-J), BLUE, METEOR, and CIDEr scores the evaluation suite of the LingoQA benchmark, which consists of 1,000 additional examples.
A.2 Experimental Settings
In this section, describe the detailed experimental settings for empirical analysis. Starting with Tiled MoVE in Section 3.1 (D1), we build on the baseline introduced by InternViT Chen et al. (2024c) We employ
two vision foundation models as first encoders, InternViT Chen et al. (2024c) and SigLIP Zhai et al. (2023)
and combine them with three additional encoders selected for their diverse and complementary strengths:
SAM Kirillov et al. (2023) for segmentation, ConvNeXt Woo et al. (2023) for OCR-related performance, and DINOv2 Oquab et al. (2024) for representation learning. As the baseline, we adopt InternL.M2-BB
Chat CCai al. (2024) as the LLM backbone, with two-layer MLP as the projector. Token merging is performed via channel concatenation with pre-adaptation, ie., fusion occurs before alignment, and a shared projector is used throughout.
For the next set of experiments (Section 3.2 D2), we retain the same settings as before, including the LLM, projector, and related con configurations. We adopt pre-adaptation as the fusion strategy and apply tiling, while varying the token merging strategies.
For the experiments comparing pre- and post-adaptation (Section 3.3 D3), and ensure a fair comparison, fix the best-performing settings from the previous stages, namely sequence interleaving and tiling. More-
22

---

Published in Transactions on Machine Learning Research (02/2026)
Table 11: Ablation study on training settings for vision backbones.
InternViT
SAM
Freeze
VQAT
GQA
VizWiz
MMB
POPE
SEED
SQA
MMVet
Avg.
x
x
57.0
62.9
52.5
64.6
86.4
65.4
66.2
31.2
60.8
X
45.2
56.4
47.5
44.7
84.2
51.3
64.0
18.2
51.4
V
V
49.5
58.2
50.6
48.3
85.4
54.7
65.2
19.8
53.7
V
67.2
63.1
55.7
71.0
87.6
69.6
75.8
35.0
65.6
V
V
68.8
64.8
57.9
72.9
88.0
72.2
78.5
37.2
67.5
Table 12: Ablation study on various types of fusion strategies and SFT data.
Model
VQAT
GQA
VizWiz
MMB
POPE
SEED
SQA
ММVeт
Avg.
LEO
68.8
64.8
57.9
72.9
88.0
72.2
78.5
37.2
67.5
w/ adaptaatnn
65.2
64.2
54.7
68.7
87.8
69.1
74.6
35.2
64.9
w/ sequence appending
67.9
63.1
56.3
71.2
87.3
71.8
78.4
36.3
66.5
W channel concatenation
67.3
62.8
54.3
70.9
87.6
72.0
78.4
35.7
66.1
W/o tile segmentation
64.2
63.6
54.4
67.3
86.9
70.4
74.8
35.4
64.6
1.8M SFT data
68.9
64.3
55.4
72.7
89.8
73.7
76.7
36.7
67.3
over, as reported in Tables and combinations with InternViT consistently outperform those with SigLIP. Therefore, we focus on InternViT as the first vision encoder, while adding two additional combinations for comparison: SigLIP Zhai et al. (0233 and CLIP Radford et al. (2021).
A.3
Benchmark Details
This section provides additional details on results reported in the main text.
Table 10 presents the complete results across all cight benchmarks for the comparison of token merging strategies within Tiled MoVE, as summarized in Table 2 Although sequence interleaving not always optimal for every vision-language task and MoVE combination, achieves the highest average score five out of six combinations.
Table 1 provides detailed results of the ablation study different training strategies for vision encoders. Unfreezing both vision encoders does not improve performance. This is likely because the pretrained encoders already provide strong visual representations and fine-tuning them may disrupt their learned features rather than enhancing them.
A.4 LEO performance beyond the MoVE setting
In addition to the main evaluation on MoVE MLLMs, compare LEO against. representative non-MoVE (general) models with similar resource constraints, i.e., those trained with comparable amounts of data and using a 7B language model (Table 13 For reference only, we also include larger-resouree models. The goal here is not to compete with frontier-scale systems Tong et al. (2025 OpenAl (022)) Li et al. (2025 ; Meta AT (2024): Zhu et al. (2025), which operate with orders of magnitude more resources, but rather to place our approach into perspective against fair baselines. Despite being optimized for the MoVE setting, LEO demonstrates strong multimodal reasoning and understanding, achieving high scores ChartQA (71.0), DocVQA (80.1), GQA (64.8), POPE (88.0), SQA (78.5), MMVet (37.2), and MMBench (72.9). It also secures the second-best performance on VizWiz, MMMU, AI2D, and SEED. Compared to other general MLLMs that. support high-resolution inputs, LEO performs competitively; for instance, it surpasses the LLaVA-NeX: Liu et al. (2024c) model on all available benchmarks and achieves superior results in six out of seven benchmarks when compared to Monkey Li et al. (2024c), underscoring robust visual reasoning capabilities. Models highlighted in grey are included for reference only, as they use significantly more training data larger I.Mss.
23

---

Published in Transactions on Machine Learning Research (02/2026)
Table 13: Comparison with general MLLMs with similar resource constraints. All models in the lower section
use a 7B language model. Models in grey use significantly more training data or larger-scale LLMs.
Model
ChariQA
DocVQA
VQA
GQA
VQA*2
ViaWiz
MMB
MMMU
POPE
A12D
SEED
SQA
AlM
GPT-4V
OpenA]
2024
78.5
98.4
78.0
75.8
06.8
78-2
69.1
75.7
Cambriiar---3 11
Long
al
2026
75.6
75.5
76.7
65.8
81.4
19.7
79.7
70.3
85.6
3.2-11B
73.2
50.7
BLII
Dai
2125
50.1
19.2
31.5
36.0
60.5
26.2
LLaVA.1.3 Liu et
21224
58.2
62.0
78.5
30.0
64.3
68.8
31.1
InternVI
C
57.0
62.9
79.3
52.5
64.6
86.4
65.4
66.2
31.2
LLAVA- NoXT
Li et
23224
67.4
35.8
N6.2
70.2
VILA Lin
79.9
57.8
68.2
85.5
VILA-Ta To
21:44
52.7
10.6
68.5
83.0
38.6
76.6
73.8
Monkey
LEO
71.0
80.1
68.8
64.8
78.3
57.9
72.9
36.4
88.0
69.6
72.2
78.5
37.2
A.5 Additional efficiency analysis
We report the computational characteristics of the four loken-merging strategies used in Tiled MoVE. For each method, measure the number of visual tokens per tile, the inference throughput, and the parameter count excluding the LLM. Table 14 sum- marizes the results. The four strategies exhibit clear efficiency accuracy trade-offs. Sequence interleav- ing achieves the highest average score, indicating that lightly mixing visual lokens produces richer joint representations, though with slight reduction in throughput. Sequence appending offers balanced middle ground, providing strong accuracy with marginally higher speed. Channel concatenation is the most efficient option, but consistently yields lower accuracy. In contrast, cross-allention introduces the highest parameter overhead and the slowest inference speed, while also delivering the weakest accuracy among the methods.
Table 14: Efficiency analysis of merging strategies.
Merging
# #Tokens/tti
# #Tookes//s
Params.
Avg.
SA
512
41.14
679M
64.3
ST
512
40.31
679M
64.9
CC
256
42.49
670M
64.1
CA
256
38.74
680M
63.5
A.6 Does Scaling the Training Data Improve
Merging Performance?
To evaluate whether incrcasing the amount of supervised training data benefits the cross-attention merging stralegy, we train cross-atlention variants using the larger Eagle-1.8M dataset [hi et al. (2025). We conduct experiments on two encoder configurations, and report results across eight standard benchmarks in Table [5] Training the cross-attention merging strategy with the larger Eagle-1.8M datasct yiclds only marginal im- provements in both encoder configurations. The average gains range from 0.2-0.3 points, indicating that increasing the supervised training scale has limited impact on the effectiveness of cross-attention fusion.
A.7 Additional Visualization Results
Figure provides additional visualizations show- casing LEO's capabilities across OCR-related tasks, chart interpretation, and document understanding. In the OCR setting, LEO is able to localize and read fine- theeggained textual elements embedded within complex scenes. For example, not only identifies the player wearing green jersey the right but also correctly recognizes the number printed on the jersey, demonstrating robust text recognition under challenging visual conditions. Beyond natural im- ages, LEO exhibits strong performance on structured documents: can navigate dense layouts, focus on the appropriate textual regions, and extract seman- tically relevant information. In one example, LEO determines the number of meetings attended by "B. Sen" by precisely locating the corresponding entry and
Aliah vinde
Becol
What does tho wnihaded her regoneee??
Pro-cap
many mestings Sen staa??
for pe eapeereeeeeeeeee tical voluer
1,00 0.15
Figure 5: More qualitative results of OCR and chart and document understanding.
24

---

Published in Transactions on Machine Learning Research (02/2026)
Is it safe for you to increase speed in the current driving conditions? Provide rationale.
LEO: No, is not safe for me increase my speed in current driving conditions because there is cyclist ahead me, and need to maintain safe distance from them.
which direction is car driving? Answer question using one or two sentences.
LEO: The red car is driving in the opposite direction.
Can you spot any immediate hazard on the road ahead? Answer the question using one or two sentences.
LEO: Yes, there is zebra crossing ahead with pedestrian crossing.
Figure 7: Additional qualitative results of scene understanding. The images are taken from LingoQA Marcu ct al. 2024
interpreting the surrounding context. These cases highlight LEO'S ability to integrate visual perception and text reasoning, enabling accurate understanding across both scene-text scenarios and complex document analysis.
Figure Y gives additional qualitative examples of the scene understanding capabilities of LEO in the au- tonomous driving domain. As shown, LEO can effectively reason about the driving scenario, identify the appropriate safe action, interpret dynamic scenes, and detect any immediate hazards.
Finally, Fig. G highlights two representative failure cases of LEO, covering both document understanding and scene-level traffic sign recognition. In the driving scenario, the model cor- rectly detects the circular road marking that indicates speed limit of 20 mph, yet it misinterprets the question "Are there any traffic signs?" by treating the painted road symbol as equivalent to a physical signpost. Since no actual traffic sign is visible the scene, the model's answer is strictly incorrect, revealing a limitation in distinguishing between road markings and standard signboards. In the second example, which in- volves document understanding, the model is asked to identify specific individual associated with a query. LEO incorrectly outputs AA Boijal" after detecting the nunber four that corre- sponds to the count of other directorships, rather than linking the correct entity "H. Khan." This error illustrates how LEo can become distracted by salient numerical patterns while fail- ing to ground them in the appropriate semantic context. To-
Can you trafiic What do they indiced!
traffic indicating speed linit of 20
Who is the ron executive director who bulds 4 shsimandipps other compusies?
LBO.A. Buijal
Figure 6: Two failure cases of LEO's scene and document understanding.
25

---

Published in Transactions on Machine Learning Research (02/2026)
Table 15: Impact of training data scale on cross-attention merging.
MoVE
Dataset
VQAT
GQA
VizWiz
MMB
POPE
SEED
SQA
MMVet
Avg.
1M
62.6
63.5
53.4
66.9
86.9
68.3
71.4
34.7
63.5
+ SAM
1.8M
62.5
63.6
53.6
66.9
87.1
68.5
71.2
35.9
63.7
1M
62.6
63.4
53.2
66.3
86.4
68.1
69.8
34.5
63.0
S + SAM
1.8M
62.7
63.7
53.0
66.5
86.9
68.6
70.4
34.8
63.3
gether, these examples underscore that while LEO demonstrates strong capabilities, it can still conflate visually similar cues or overlook fine-grained semantic relations, leading to incorrect predictions.
26


Reviews:
[{"review_id": "LWKNIXZ550", "reviewer": "Reviewer_1wnh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Paper 6111 conducts a systematic study of design choices in Mixture of Vision Encoder (MoVE) based models.\nThe paper's focus lies on the interaction between visual reasoning, token-level merging and fusion timing. The papers findings are used to devise Leo, a MoVE inspired multi-modal large language model for visual reasoning.\nThe experimental section presents an extensive set of experiments on OCR and chart understanding, general VQA as well as general multi-modal evaluation benchmarks.  The authors observe competitive or improved results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, very much so! The paper carefully investigates tiling (Table 1), token merging (Table 2) and feature fusion ( Table 3 ). \nBased on their analysis the authors follow Chen et al. (2024a) for their tiling approach, choose to work with two independent MLP projections for token merging and finally a tile-level sequence interleaving strategy for feature fusion. \n\nFurthermore Table 4 presents convincing benchmark results for the LEO architecture which is built based on the lesions learned from tables one two and three. The numbers look extremely promising, which further validates the paper's approach. \n\nLast but not least the papers includes an ablation study, efficiency analysis, an application to autonomous driving application as well as visualizations of network outputs.", "audience_yn": "Yes", "audience_explanation": "Yes, the paper is well written, its experiments are well connected and the paper focuses on a relevant vision topic.", "requested_changes": "None, this paper is ready for publication.\n\nEdit: It would be neat if supplementary code could be added for reproducibility purposes.", "broader_impact_concerns": "The Broader impact statement covers potential concerns very well.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZvdUXiH4Bh", "reviewer": "Reviewer_Xzjk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates Mixture of Vision Encoders for Multimodal Large Language Models (MLLMs) and proposes LEO as a simple yet effective framework for high-resolution visual reasoning. The authors justify their design choices through multiple ablation studies, and benchmark LEO against recent MLLMs and show state-of-the-art performance on multiple VQA and automatic driving datasets.\n\nOverall, this is an excellent paper. The literature review is thorough, and the research question is interesting and practical. The exploration of the design space for Mixture of Vision Encoders for MLLM is well-motivated and well-executed, and the proposed method is simple yet effective. The empirical results are favorable.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors include thorough investigations on multiple key aspects of MoVE design space in Section 3, including dynamic tiling, token merging, and fusion timing. They provide strong evidence for the benefits of tiling, sequence interleaving, and post-adaptation fusion, which they built their framework upon.", "audience_yn": "Yes", "audience_explanation": "This paper will be of interest to multiple communities within TMLR’s audience, including but not limited to researchers in Multimodal Large Language Models (MLLM), Vision Foundation Models (VFMs), Autonomous Driving, and Multi-teacher Knowledge Distillation.", "requested_changes": "Questions and suggestions:\n1. Can Leo be easily scaled up into more than two vision encoders? Currently, LEO takes InternViT-300M and a SAM-L as encoders. Is extensive modification to the current framework needed if you were to scale this up to e.g. InternVL with SAM, DINO V3, and ConvNeXt V2? \n2. Can Leo fuse (and benefit from) two MLLMs’ vision encoders, or is it restricted to having only one MLLM within its suite of vision encoders? The paper mentioned \"primary encoder\" and \"secondary expert\", but currently it isn’t clear if and how the priority is enforced in practice.\n3. The comparison between Leo and RADIO family could be described in more details, for example, RADIO models have adaptors for each teacher on top of the backbone, but it wasn’t clear from 5.4 if RADIO was evaluated with only its backbone. Similarly, was tiling also used for RADIO? At the very least, RADIO is cheaper than LEO at inference time (albeit more expensive to train), which is also unmentioned in the current section.\n4. I find Tiling in D1 (subsection 3.1) to be the weakest in terms of empirical support among the three investigation directions. Currently the section reads \"tiling is better than no tiling\", which is not exactly a new finding. It could be beneficial to compare dynamic tiling to other tiling methods and perhaps even direct representation upsampling methods (AnyUp by Wimmer et al. 2025 might be a good baseline for this).\n5. Subsection 3.2 on Token merging introduced four main token merging methods in current literature, however, the computational cost for each was not discussed. Similar question goes for post-adaptation fusion in Subsection 3.3.\n\n\n**Clarifying 1 and 2 is critical to secure my recommendation, addressing 3-5 will further strengthen this work.**", "broader_impact_concerns": "There is no substantial ethical implications beyond average research in MLLMs.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "a98PKxSRMk", "reviewer": "Reviewer_Q4Tp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Initial vision-language models (VLMs) underperformed on visual tasks relative to our expectations. Recent work has improved the ability of VLMs to see via strategies like mixture of vision encoders (MoVE), image tiling, fusion networks, etc. This paper investigates three design choices of VLMs: (1) combining MoVE and image tiling, (2) visual-token merging, and (3) fusion timing. First, they find that image tiling consistently helps MoVE. Second, they find that sequence appending and sequence interleaving outperform cross-attention and channel concatenation strategies. Third, they find that placing encoder-specific projections before the fusion network outperforms earlier fusion.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The submission makes claims based on controlled experiments across 8 vision-language benchmarks. And the findings appear to be consistent across datasets.", "audience_yn": "Yes", "audience_explanation": "VLMs, and multimodal transformers generally, are powerful and popular. The paper investigates subtle strategies to improve performance. The findings are interesting to me, and I suspect many others.", "requested_changes": "I recommend the paper is accepted regardless, although I have one request:\n\nPlease experiment with varying the number of output tokens from the cross-attention tile-merger. If it gives fewer visual tokens to the LLM than the other methods, then maybe this explains the result alone? This can be done, for example, by using single self-attention layer so that the number of input visual tokens matches the output.", "broader_impact_concerns": "no concerns", "additional_comments": "Q1: I find the difference between sequence appending and interleaving interesting. The paper hypotheses that interleaving outperforms appending because it \"preserves spatial relationships\" via position encoding (since the LLM is causal it can internally encode positions even without position embeddings). Can we add position embeddings to visual tokens to further improve performance or even match the performance of interleaving without interleaving?\n\nQ2: I see that cross-attention tiling performs worse. Since it is the only parametric tile-merging approach, maybe it performs better with more data? Do you see any data-dependent effects?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)