NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints
Authors: Changyao Tian
Hao Li
Gen Luo
Xizhou Zhu
Weijie Su
Hanming Deng
Jinguo Zhu
Jie Shao
Ziran Zhu
Yunpeng Liu
Lewei Lu
Wenhai Wang
Hongsheng Li
Jifeng Dai
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Large Language Models
Scaling Properties
Primary area: deep_learning
TL;DR: We systematically investigate the design space and scaling property of native Multimodal Large Language Models and introduce a novel MLLM that achieves competitive performance against existing MLLMs.

Abstract: Compositional training has been the de-facto paradigm in existing Multimodal Large Language Models (MLLMs), where pre-trained vision encoders are connected with pre-trained  LLMs through continuous multimodal pre-training. However,  the multimodal scaling property of this paradigm remains difficult to explore due to the separated training. In this paper, we focus on the native training of MLLMs in an end-to-end manner  and systematically study its design space and scaling property under a practical setting, i.e., data constraint. Through careful study of various choices in MLLM, we obtain the optimal meta-architecture that best balances performance and training cost. After that, we further explore the scaling properties of the native MLLM and indicate the positively correlated scaling relationship between visual encoders and LLMs.  Based on these findings, we propose a native MLLM called NaViL, combined with a simple and cost-effective recipe.  Experimental results on 14 multimodal benchmarks confirm the competitive performance of NaViL against existing  MLLMs. Besides that, our findings and results provide in-depth insights for the future study of native MLLMs.

Va Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints
Changyao Tian
Hao Li
Gen Luo?* Xizhou Zhu Weijie Su² Jie Shao Ziran Zhu Yunpeng Liu
Hanming Deng"
Jinguo Zhu'
Lewei Lu Wenhai Wang2.1 Hongsheng Li Jifeng Dai
Shanghai Al Laboratory The Chinese University of Hong Kong Tsinghua University Sensetime Research 5 Nanjing University Code:
Abstract
Compositional training has been the de-facto paradigm in existing Multimodal Large Language Models (MLLMs), where pre-trained visual encoders are con- nected with pre-trained LLMs through continuous multimodal pre-training. How- ever, the multimodal scaling property of this paradigm remains difficult to explore due to the separated training. In this paper, focus on the native training of MLLMs in an end-to-end manner and systematically study its design space and scaling property under practical setting, i.e., data constraint. Through careful study of various choices in MILM, we obtain the optimal meta-architecture that best balances performance and training cost. After that, further explore the scaling properties of the native MLLM and indicate the positively correlated scal- ing relationship between visual encoders and LLMs. Based on these findings, we propose native MLLM called Na ViL, combined with a simple and cost-effective recipe. Experimental results on 14 multimodal benchmarks confirm the competitive performance of ViL against existing MLLMs. Besides that, our findings and results provide in de-thpth insights for the future study of native MLLMs. ||
Introduction
Multimodal Large Language Models (MLLMs) have demonstrated remarkable progress in computer
vision [12 43 63
50
54]. continuously breaking through the upper limits of various multimodal
tasks 47 68
38
45).
The great success of MLLM inseparable from its compositional training
paradigm, which independently pre-trains visual encoders (28 and LLMs [61], and then integrates them through additional multimodal training. Due to the engineering simplicity and effectiveness, this paradigm has dominated MLLM area over the past few years. However, the shortcomings of compositional training have been gradually recognized by the community recently, e,g., unclear multimodal scaling property [19 (6))
Therefore, increasing attention has been directed toward the development of more native MLLMs. As illustrated in Fig. [ native MLLMs aim to jointly optimize both visual and language spaces in an end-to-end manner, thereby maximizing vision- languaee alignment. Compared the compositional paradigm, existing native MLLM methods demonstrate a promising scaling law and a significantly simplified training process
9 56 Despite these advancements, the primary benefits of native MILMs are often evaluated under the assumption of infinite training resources, overlooking the
* Equal contribution. az Corresponding to Jifeng Dai <daijifeng @ tsinghua.eduu edu.en>>ddd......... Work was done when Changyao Tian, Hao Li, and Jie Shao were interns at Shanghai Al Laboratory.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Design Choices of Native MLLMs
Scaling Properties Native MLLMs
ChargA
743
XMB
MMVVet
AI2D
MMMM
2.25
742
43.
2.0
2.00
1.8
GQ4 123
204 OCRBench
1.75
Walderion
1.8
Valceton
1.50
91.
MatVViea
2.4
SQA-
1.25
1.2
TeetVQA
CCBa
1.0
of MoE
basc
0.58 parameters parameters
InfoVQ4
DoVVQA
0.8
Emu3
EVEv2
10
10' Trannnn Data Size
10
Mono-IntemVI.
Chamokoon-7B 7B
Traninng Data
Intem\V1-2.5228
Na (Ours)
Figure 1: Comparison of design choices, scaling properties, and performance of our native MILMs. We systematically investigate the designs and the scaling properties of native MLLMs under data constraints and yield valuable findings for building native MLLMs. After adopting these findings, our native MLLMs achieve competitive performance with top-tier MLLMs. V2 (-) denotes the visual encoder with optimal parameter size.
substantial challenges posed by limited data and large-scale training. Consequently, a critical practical question remains: whether and how native MLLMs can feasibly achieve or even surpass the performance upper bound of top-tier MLLMs at acceptable cost.
To answer this question, in this paper, we aim systematically investigate the designs and the scaling properties of native MLLMs under data constraint. Specifically, we first explore the choices of key components in the native architecture including the mixture-of-experts, the visual encoder and the initialization of the LLM. Our findings can be summarized in two folds. Firstly, an appropriate pre-training initialization (e.g., the hase L.LM) of the L.LM greatly benefits the training convergence on multimodal data. Secondly, combining visual encoder architectures and MoEs results in obvious gains against the vanilla decoder-only LLM. Following these findings, we build a meta architecture that optimally balances performance and training cost.
Based on the optimal meta architecture, we further explore the scaling properties of the visual encoder, the LLM and the entire native MLLM. Specifically, we first scale up the LLM and the visual encoder independently and observe different scaling properties: while scaling LLM exhibits similar patterns as the conventional language scaling laws, scaling visual encoder shows an upper bound in return due to the limitation of the LLM's capacity, suggesting that the optimal encoder size varies with the LLM size. Further analysis reveals that the optimal encoder size increases approximately proportionally with the LLM size log scale. This observation yields a different guidance against compositional paradigm, which employs visual encoder of one size across all LLM scales.
Based on above principles, we propose native MLLM called NaViL, combined with a simple and cost-effective recipe. To validate our approach, we conduct extensive experiments across diverse benchmarks evaluate its multimodal capabilities including image captioning [T0] 67 2 optical character recognition (OCR) [5 17 39, etc. Experimental results reveal that with -600M pre- training image-text pairs, NaV achieves competitive performance compared to current top-tier compositional MLLMs, highlighting the great practicality and capabilities of NaViL. In summary, our contributions are as follows:
We systematically explore the design space and the optimal choice in native MLLMS under data constraint, including the LLM initialization, the visual encoder and the MoEs, and draw three critical findings that greatly benefit the training of native MLLMs.
Based on above findings, we construct a novel native MLLM called NaViL In NaViL, we explore the scaling properties of the visual encoder and the LLM and indicate their positively correlated scaling relationship.
We conduct large-scale pre-training and fine-tuning experiments on Na ViL. Experimental results show that NaViL can achieve top-tier performance with nearly 600M pre-training data. Our findings and results will encourage future work for native MLLMs in the community.

---

2 Related Work
Multimodal Large Language Models. Recent years have witnessed the significant progresses of Multimodal Large Language Models (MLLMs) [44] 36 downstream tasks (24 26 [37 BO]. Starting from LLaVA [6]. most existing MLLMs adopt the compositional paradigm, which connects the pre-trained visual encoder [53] and LLM K through a projector and finetune them on for alignment. Then, the whole structure will be further fine-tuned on multimodal data for alignment. Based on this paradigm, existing works mainly focus on the
35
63 12 which have dominated various
improvement of visual encoders [63 64 44 and the design of connectors [33 36]. Despite the
progress, such paradigm struggles to explore the joint scaling properties of vision and language. Their
potential limitations training pipeline [56] and vision- Ssiounlanguage alignment 19 are also gradually
recognized by the community.
Native Multimodal Large Language Models. To overcome the limitations of compositional
paradigm, native MLLMs have emerged as another candidate solution [20 19 43
32
62
56
9.
Compared compositional paradigm, native MILMs aim pre-train both vision and language parameters in an end-to-end manner, thus achieving better alignment. The most representative methodology [56] g is directly pre-train the LLM from scratch on large-scale multimodal corpora, which typically requires expensive training costs. To address this issue, recent attempt initialize the LLM with pre-trained checkpoint to facilitate training convergence 20 19
43
32
62
Nevertheless, current research still lacks systematic investigation into the architectural design and scaling characteristics of native MLLMs, limiting their performance.
3 Visual Design Principles for native-MLLM
3.1 Problem Setup
We define native MLLMs as models that jointly optimize vision and language capabilities in an end-to-end manner. Dispite recent progress that shows promising scaling law and potential better performance compard with their compositional counterparts, how to build competitive native MLLMs compare to the state-of-the-er MLLMs with a practical data scale remains underexplored. In particular, there are two problems requiring to be investigated:
(Sec. 3.2)
How to choose the optimal architectures of the visual and linguistic components?
(Sec. 3.3
How to optimally scale the visual and linguistic components?
Meta Architecture. To study these two questions, we first define general meta architecture of native MILMs consisting of visual encoder, an LLM, and mixture-of-exper architecture injected to the LLM. The visual encoder V consists of series transformer layers and can be defined as
Va (T) = C Fi O .... C F ④ F P(T) C
F P(T),
i=L...
where Ft denotes the i-th transformer layer (out of layers) with hidden dimension w, denotes the Patch Embedding Layer, lf denotes the input image. Note that the visual encoder degenerate to a simple patch embedding layer when d = O. For simplicity. we use the same architectures as the LLM for the visual encoder layers JF but with bi-directional attention and vary the hyperparameters d and w. Here C is the connector which downsamples the encoded image embeddings through pixel shuffle [5] and projects them the LLM's feature space by MLP.
Experiment Settings. All the models are trained on web-scale, noisy image-caption pair data [55] with Next-Token- Poker-rttionnnniniiiiii (NTP) and an image captioning task. We use held-out subset of the multimodal dataset to calculate the validation teacher-forcing loss for measuring and comparing different design choices. Models with LLM initializations are initialize from InternI.M2-Base B
3.2
Exploring the Optimal Design of Architecture Components
In this section, we explore the design choices of three key components: 1) the initialization of the LLM; 2) the effectiveness of MoEs; 3) the optimal architecture of the visual encoder.
3.2.1
Initialization of LLM
straightforward way to construct native MI.LMs is to train modalities from scratch with mixed corpora, as shown in prior work (56). While this approach theoretically offers the highest performance
3

---

ceiling given ample data and computational resources, practical limitations such as data scarcity and large-scale optimization challenges hinder its feasibility. Alternatively, initializing the model from pre-trained L.L effectively leverages linguistic prior knowledge, significantly reducing data and computational demands.
To evaluate the effectiveness of LLM initialization, we compare model per- formance in terms of loss and im- age captioning. As shown in Fig. 2 (left), the model trained from scratch performs significantly worse than the initialized model, requiring over 10x more data to reach comparable loss.
Validation Loss
COCO Caption
Valldattoon Loss
30
102
110
"50"
Further analysis of zero-shot image
Training Data Size
Training Data Size
captioning (Fig. [ (right)) reveals substantial performance gap favor- ing the initialized model, even with significantly more data for the non- initialized model. This is likely due to the lower textual quality and diversity of multimodal training data compared to the LLM pre-training corpus, lim- iting the textual capability of models trained from scratch. These findings highlight the practical advantage of using LLM initialization in multimodal pre-training.
w/ LLM Init
w/o LLM inkt
Figure 2: Effectiveness of LLM initialization. Left: The validation loss. The LLM initialized one converges much faster. Right: The zero-shot caption performance. Due to the lack of textual knowledge, the uninitialized model continues to lag behind.
Observation 1: Initializing from pre-trained LLM greatly benefits the convergence on mul- timodal data, and in most cases delivers better performance even with large amount of multimodal data.
3.2.2 Effectiveness of MoEs
cetture-of-Experte (MoEs) effective for handling het- erogeneous data and are widely used native MILMs.
Validation Loss
Validation Loss
We evaluate the MoE architecture within our meta architec- ture by comparing two configurations: one with visual encoder and vanilla LLM, and another with visual encoder and an MoE-extended LLM. We follow Mono- InternVL 43 adopt the modality-specifft MoEs and training settings. However, we empirically found that us- ing only the feed-forward network (FFN) expert would lead to significant difference in feature scale between visual and language modalities. To mitigate this issue, we further introduced modality-speciffe attention experts, that is, using different projection layers (i.e. qkvo) the self-attention layer to process visual and text features respectively, and then perform unified global attention calculation. Specifically, the output a! r/.m E Rd of the i-th token with modality m E (visual, linguistic} at the /-th layer of the MoE-extended LLM can be defined as
2.0
w/a MoE w/ MoE
107
10"
Training Data Size
Figure 3: The validation loss of adding MoE or not. Using MoE extension will cause the loss to decrease more quickly.
MHA MHA-MMoo,RMMSNom (2( 1)
(2)
FRN-MMOO, (c(
where RMSNorm(-) is the layer normalization operation, and MHA-MMoE(-) and FFN-MMoE() are the modality-speciff attention and FFN expert, respectively, formulated by
MHA-MMofE (HAAMMME((IT..)) = (softmax QKT
Vd WWW
(3)
Qi.m Ti W Ks,m = Ii.m Wm Ki Vim Zi,m Wm
FFN-MMOE(I,,)) -SiLUU(I; Wm gmee O Zi.m up down
Wm Wm

---

Here W W.. WV, W and W ()) denotes the activation function, @ denotes the element-wise product operation. The number of activated experts is set to one to maintain consistent inference costs.
gate Wm W down are all modality-peecific projection matrices, and
As shown in Fig. 3 the MoE architecture significantly accelerates model convergence compared to the vanilla LLM, achieving the same validation loss with only 1/10 of the data without increasing training or inference cost. This demonstrates that MoE enhances model capacity and effectively handles heterogeneous data, making it suitable for native MLLMs.
Observation 2: MoEs significantly improve model performance without increasing the num- ber of activated parameters.
3.2.3
Optimizing the Visual Encoder Architecture
Valida
Caption
Flickr Caption
NoCaps Caption
Validalion Less
60-
2.4
50
30-
30
2.2
40-
40
2.0
30
30-
0.8
20
05
25
012
024
.46
03
05
012
0.4
645
06
012
026
dCc
23
26
012
424
045
Training Data Size
30M
120M
Figure 4: The validation loss and zero-shot caption performance of different visual encoders. The loss and performance only differ when the visual encoder is extremely wide or shallow.
The visual encoder precedes the LLM to perform preliminary extraction of visual information, converting raw pixels into semantic visual features aligned with the textual embedding space. Due to its bidirectional attention mechanism and the increased capacity introduced by additional parameters, the visual encoder has the potential to enhance the model's ability to represent visual information.
In this section, we investigate the optimal architecture of the visual encoder under given parameter budget. The total parameter count C can be approximately calculated [29] as / = 12 x d x w². Given fixed N, the structure of the visual encoder is mainly determined by its width w and depth
Depth (d): Typically, deeper models can capture richer and more complex features, while also being more prone to gradient vanishing problems [58). When it comes to MLLM, visual encoder that is too shallow may not be able to extract enough high-level semantics, while visual encoder that too deep may cause low-level features to be lost, thus limiting the capture of fine-grained details.
Width (w): Compared depth, width has relatively little impact visual transformer perfor- mance 21., as long as it does not cause additional information bottlenecks. That is, it cannot be lower than the total number of channels within single image patch. Under this premise, the width of the visual encoder does not have to be the same as the hidden size of the LLM.
We train various MLLMs with different Va.w configurations (combinations of depth and width) while keeping the pre-trained LLM and visual encoder parameter count fixed at 600M. The depth d ranges from {3, 6, 12, 24, 48), and the width is adjusted {4096, 2880, 2048, 1472, 1024] to maintain a consistent parameter count. Fig. D shows the validation loss for different depth and width combinations as training data size varies. Models with extremely high low depths perform worse than those with moderate configurations. Among reasonably configured models, shallower ones converge faster in the early phase (less than 30M data), but this advantage diminishes with more data. In zero-shot image captioning benchmarks, deeper visual encoders show slightly better performance, consistent with prior research on compute-optimal LLM architectures [29]. which suggests wide range of optimal width and depth combinations.
Observation 3: Visual encoders achieve near-optimal performance across wide range of depth and width configurations. Shallower encoders converge faster in early training, while deeper encoders perform slightly better with larger datasets.
5

---

3.3 Scaling Up Native MLLMs
In this section, we consider the scaling properties of our meta architecture. Specifically, we investigate: 1) the impact of scaling up visual encoder and the LLM independently; 2) the optimal way of scaling the visual encoder and the L.M simultaneously. All models follow the optimal architecture discovered in Sec. 3.2 visual encoders.
3.2 i.e., with LLM initialization, MoEs, and optimal depth-to-widd ratios of the
3.3.1 Scaling up Visual Encoder and LLM Independently
We first investigate the scaling properties of the visual encoder and the LLM independently, i.e., scaling up one component while keeping the other fixed. Specifically, we evaluate series of LLMs with parameter sizes {0.5B, 1.8,7BB and visual encoders with sizes (75M, 150M, 300M, 600M, 2.4B..
Scaling up LLMs. The results are shown in Fig. 5 Scal- ing up the LLM parameters in native MILMS exhibits a pattern consistent with the conventional L.M scaling law, where the loss decreases linearly the parameter size increases exponentially.
Validation Loss vs LLM Size
1.20
Validation Loss
Visual Encoder Size: 600M
1.00
Scaling up Visual Encoder. The results are shown in
0.80
Fig. 6 In contrast to the LLM scaling law, increasing the visual encoder size does not consistently enhance multimodal performance. Instead, with a fixed LLM, the performance gains achieved by enlarging the visual encoder diminish progressively. Beyond a certain en- coder size, further scaling results in only marginal loss reduction, indicating that the performance upper limit of the MLLM is constrained by the LLM's capacity.
0.60
LLM Size (B)
Figure 5: The validation loss when seal- ing up LLMs. With the same visual en- coder (i.e. 600M), the validation loss de- creases log-linearly with the LLM size.
Validation Loss (log scale)
LLM-0.58
LLM-1.88
LLM-78
30M
120M
15M
30M
60M
6OM
120M
1.0
120M
75
150
300
600
150
300
600
1200
300 600
1200
2400
Visual Encoder Size
Visual Encoder Size
Visual Encoder Size
Figure 6: The validation loss curves of different LLMs with different training data sizes. As the training data size increases, the loss gap narrows near zero when the visual encoder size reaches certain threshold.
Scaling the LLM consistently improves multimodal performance, following the typical LLM scaling law. However, increasing the visual encoder size shows diminishing returns, suggesting that the MLLM's performance limited by the LLM's capacity.
Observation 4:
3.3.2 Scaling up Visual Encoder and LLM Together
Visual Encoder Size (B)
Visual Encoder Size vs LLM Size
1.20
The diminishing returns from increasing the visual en- coder size suggest the existence of an optimal encoder size for a given LLM. We define this optimal size as the smallest encoder whose loss difference compared to an encoder twice its size is less than = 1% of the loss with the 75M encoder (the smallest used in exper- iments). Fig. 7 shows the relationship between visual encoder size and LLM size.
0.60
0.30
0.5
LLM Size Figure 7: Relationship of visual encoder size and LLM size. The optimal visual encoder size increases log-linearly with the LLM size.
The logarithm of the optimal visual encoder size scales linearly with the logarithm of the LLM size, indicating

---

that both components should be scaled jointly for balanced performance. This highlights the subop- timality of compositional MLLMs, which typically use fixed visual encoder size across varying LLM scales.
Observation 5: The optimal size of the visual encoder scales proportionally with the LLM size in log scale, indicating that both components should be scaled jointly. This further implies that the pre-trained visual encoders using a single pre-trained visual encoder across a wide range of LLM scales like existing compositional MLLMs is suboptimal.
NaViL: A Novel Native MLLM with Strong Capabilities
4.1
Architecture
text token
The cat wears a yellow flower on its head, golden neeklace around neck, and pink blushes its checks.
image token
PEN-MMaE
MLP Connector
Multimodal Large Language Models
Transformer Layer
RAISNom
Visual Encoder
Layer
MILA-MMoEE MM.E
MILA
RMSSNom
Putch Emnhed
Please provide detailed description of the the picture.
XN
Floual Mattiicale Parking
Figure 8: Architecture of NaViL. As native MoB-extended MILM, NaViL be trained end-to-end and supports input images of any resolution.
Based on above studies, we construct NaViL. with the optimal settings in Sec. 3.1 The architecture is shown in Fig. X NaViL inherently supports input images of any resolution. These images first encoded into visual tokens by the visual encoder and the MLP projector, and then concatenated with the textual tokens to formulate the multimodal token sequence and fed into the LLM. Special tokens <begin_ of image> and <end_oo of image> are inserted before and after each image token subse- quence indicate the beginning and end of the image, respectively. Special token <end_ of _ine>> is inserted at the end of each row of image tokens to indicate the corresponding spatial position information.
Visual Multi-scale Packing is further introduced to improve the model performance during inference. Specifically, given input image To R and downsampling rate multi-scale image sequence [ E TR H, xW,33 fi=0 is obtained by continuously downsampling the original image (i.e. H, = r* Ho, W. r'o)) until its area is smaller than given threshold. These images in the sequence are processed separately by the visual encoder. The obtained visual token embeddings 1n { are then concatenated and fed the LLM. Special token <end_of of _ed___f__ale>>>> is inserted after each scale image to indicate the end of different scales.
4.2 Training
Stage 1: Multi-modal Generative Pre-training. In this stage, the model is initially trained on 500 million image-text pairs to develop comprehensive multimodal representations. Of these training samples, 300 million are directly sampled from web-scale datasets (i.e. Laion-2B [55]. Coyo- 700M 77. Wukong 25 and SA-1B (31) while the remaining 200 million consist of images from these datasets paired with captions synthesized by existing MLLMs (i.e. Intern VL-8B (1]). During this process, the textual parameters of the model remain frozen, with only the newly-added vision- specific parameters (i.e., the visual encoder, MLP projector, and MoE visual experts) being trainable.
To enhance the alignment between visual and textual features in more complex multimodal contexts, the model is subsequently trained on 185 million high-quality data consisting of both multimodal
7

---

Table I: Comparison with existing MILMS on general MILM benchmarks. "#A-Param" denotes the number of activated parameters. 'Intern VL-2.5-B adopts the same LLM and high-quality data with VaViL, so we mark the compositional counterpart. Note that its 300M visual encoder is distilled from another 6B large encoder. Bold and underline indicate the best and the second-best performance among native MLLMs, respectively. denotes our reproduced results. For MME, sum the perception and cognition scores. Average scores are computed by normalizing each metric to a range between 0 and 100.
Model
#A-Puram
Avg
MMVet
MMMU
MMB
MME
Math
OCRBench
CCB
Compositional Ms.
MobileLLM-V2-L.7B
[6
1.7B
57.7
MobileV..M-V2338 [6
3.0B
63.2
Mini-Gemini-2B
3.5B
59.8
1653
29.4
MMJ-3B-MoB-Chat
48
42.2
70.8
1772
32.6
DeepSeck-VL--1 VL.-1333
10
2.0B
42.3
34.8
32.2
64.6
1532
31.1
409
37.6
PaliGemma-3B
2.9B
45.6
33.1
34,9
71.0
1686
28.7
614
29.6
MiniCPM-X-2 66
2.8B
51.1
41.0
38.2
69.1
1809
38.7
605
45.3
Intem\L.-115528 [4
2.2B
54.7
39.3
34.6
70.9
1902
41,1
634
63.5
Qwen2VL-2B [3
2.1B
58.6
49.5
41.1
74.9
1872
43.0
809
53.7
VnteemV1-2.522
2.2B
67.0
60.8
43.6
74.7
2138
51.3
804
81.7
Native MLLMs:
Fuyu-8B (HD) S
21.4
10.7
SOLO
1260
34.4
Chameleon-701
170
22.3
EVE-7B
33.0
25.6
49.5
1483
25.2
327
EVE (HD) 19
7B
37.0
25.7
32.6
52.3
1628
34.2
398
16.3
Emu3 35
8B
37.2
31.6
58.5
687
VoRA 62
7B
33.7
32.2
64.2
1674
VoRA-AnyRes 62
7B
33.7
32.0
61.3
1655
EVEv2 20
7B
53.2
45.0
39.3
66.3
1709
60.0
702
30.8
SAIL
53.7
46.3
38.6
1719
57.0
783
24.3*
Mono- InterVLL
1875
45.7
NaViL-2B (ours)
2.4B
67.1
71.2
1822
50.0
796
83,9
alignment samples and pure language data. In this phase, the textual parameters within the self- attention layers are also unfrozen, enabling more refined cross-modal integration.
Stage 2: Supervised Fine-tuning, Following common practice in developing MLLM, an additional supervised fine-tuning stage is adopted. In this stage, all parameters are unfrozen and trained using relatively smaller (i.e. 68 million) but higher quality multimodal dataset.
Experiment
5.1 Experimental Setups
Evaluation Benchmarks. We evaluate NaViL and existing MLLMs on a broad range of multimodal
benchmarks. Specifically, MLLM benchmarks encompass MMVet 66), MMMU val 770, MMBench-
EN test (37, MME [22] MathVista MINI [41, OCRBench 39 and CCBench [37 Visual question
answering benchmarks include TextVQA val [57] ScienceQA- Q-MIM test 442, GQA test dev [27).
DocVQA test 47), A12D test (30], ChariQA test 45). and InfographieVQA test [6). These benchmarks cover various domains, such as optical character recognition (OCR), chart and document understanding, multi-image understanding, real-world comprehension, etc.
Implementation Details. By default, Na ViV-LBB implemented upon InterLM22 (5 using its weights initialization for the text part parameters. The text tokenizer and conversation format are also the same. The total number of parameters is 4.2B, of which the number of activation parameters is 2.4B (including 0.6B of visual encoder). The input images are first padded to ensure its length and width are multiples of 32. The stride Patch Embedding layer set to 16. The visual encoder adopts bidirectional attention and 2D-RoPE to capture global spatial relationships, while the LLM adopts causal attention and ID-RoPE to better inherit its capabilities. the pre-training phase, the global batch size is 7000 for stage and 4614 for stage 2, respectively. The downsampling rate of

---

Table 2: Comparison with existing MLLMs visual question answering benchmarks. "IntemVV-2255-2 adopts the same LLM and compositional counterpart. Note that its 300M visual encoder is distilled from another 6B large encoder. denotes our reproduced results. Bold and underline indicate the best and the second-best performance among native MILMM,, respectively.
data with NaViL, so we mark it as the
Model
MN-Param
Avg.
TexVVQ
SQA-T
GQA
DoeVQA
A12D
ChariQ
InfoVQA
Compositional MLL.Ms:
Mobile
[6
Gemini
MM1-3B
76.1
DeepSeck-VL-H
VL-133
PaliGemma-38
MiniCPM-V-2
IntenVL-11.5228
2.2B
70.5
84,9
61.6
74,8
Qwen2VL-2B
2.1B
78.2*
60.3*
73.5
'IneemVL-2.5-28
Native MLLMMs:
Fuyu-8B (HD)
8B
64.5
SOLO |
7B
73.3
61.4
Chameleon-77
7B
17.9
4,8
47.2
1.5
46.0
2.9
5.0
EVE-78 19
7B
40.8
$1.9
63.0
60.8
22.0
48.5
19.5
20.0
EVE-7B (HD)
10
7B
54,6
56.8
64.9
62.6
53.0
61.0
59.1
25.0
Emu3 65
8B
67,6
64,7
89.2
60.3
76.3
70.0
68.6
43.8
VoRA 0H
7B
56.3
75.9
65.6
VoyReAnyRes
02
7B
58.7
72.0
61.1
EVEv2 20
7B
71.7
71.1
96,2
62.9
77.4
74.8
73.9
45.8*
SAIL 32
7B
71.5
77.1
93.3
58.0%
78.4
76.7
69.74
47.3*
Mono IntemVL.
1.8B
70.1
72.6
93.6
59.5
80.0
68.6
73.7
43.0
NaVit. 2B (ours)
2.4B
75.1
76.9
95.0
59.8
85.4
74.6
78.0
56.0
visual multi-scale packing is set to v2/2. To demonstrate the scaling capability of our approach, we also trained Na VaViL-9B based on Qwen3-8B 60. More details are given the appendix.
5.2 Main Results
In Tab. we compare the performance of our model with existing MLLMs across multimodal benchmarks. Compared to native MLLMs, compositional MLLMs demonstrate superior overall performance. For example, InternVL-2.5--DD outperforms existing native MLLMs on MLLM benchmarks. This indicates that current native MILMs still have significant room for performance improvement. contrast, our proposed Na ViL achieves overall performance exceeding all existing native MLLMs with relatively small paramter size. Compared to the compositional baseline model Intern VL-2.5-BB that uses the same LLM. NaViL also achieves comparable performance on most benchmarks. It is worth noting that the 300M visual encoder used by Intern V1.-2.5-2B is distilled from another pre-trained encoder Intern VTT-6H demonstrates the superiority of our visual design methods and visual parameter scaling strategies.
[S] with significantly larger parameter size. This
In Tab. 2 further compare the performance of model with existing MLLMs mainstream visual question answering tasks. NaViL's average performance still leads previous state-of-the- native MILMS and is roughly on par with compositional baselines that require pre-trained encoders. Specifically. in tests such as Doc VQA 49). ChariQA 45 and Info VQA 46), NaViL significantly outperforms the previous sfate-of-the-s art native MILM, demonstrating the superiority of using an optimal size visual encoder in processing high-resolution images. However, NaViL's performance still has some gap compared to the best compositional MIL.Ms. We believe that higher-quality instruction data and more powerful LLMss will further narrow this gap-
'The performance of Chameleon-75 7B from 43

---

150M
1.2B
Layer 1
Layer 5
Layer 14
Layer 23
Provide a one-sentence caption for the image:
troffic light showing red lights against a blue sky with clouds.
Figure 9: Visualization of attention maps in LLM-1.8B with different encoder sizes (i.e. 150M and 1.2B). Text and image tokens are in blue and green, respectively. Larger encoder allows LLMs attend global patterns at shallow layers while maintaining higher attention textual tokens.
5.3 Qualitative Experiments
To further analyze the characteristics of native MLLM, we visualized the attention maps of different LLM layers when using encoders of 150M and 1.2B sizes, shown in Fig. [9] Two findings can be drawn from the figure. First, similar to previous native-MLLMs [43] despite having an encoder, the attention patterns in shallow layers still exhibit obvious locality, gradually shifting toward global information as the depth increases. For example, when using a 150M encoder, image tokens in the first layer tend to attend spatially adjacent tokens. However, encoder is scaled up to 1.2B, visual tokens in shallow layers already begin to attend more to global information. This indicates that a sufficiently large visual encoder can better pre-extract high-level semantic information from the entire image.
observe that when the visual
Secondly, from cross modal interaction perspective, larger visual encoder also facilitates carlier interaction between visual and language features. When using a 1.2B visual encoder, the attention weights between visual tokens and text tokens in the first layer significantly higher than those in the 150M counterpart. Earlier interaction is more beneficial for feature alignment between modalities, thus providing explanatory perspective for the improved performance achieved when using larger encoder sizes. We believe these findings will provide beneficial insights for developing native MLLMs. More visualizations can be found in the supplementary materials.
Conclusion
This paper systematically investigates native end-to-end training for MLLMs, examining its design space and scaling properties under data constraints. Our study reveals three key insights: 1) Initial- ization with pre-trained LLMs, combined with visual encoders and MoE architecture, significantly improves performance; 2) Visual encoder scaling is limited by the LLM''s capacity, unlike traditional LLM scaling; 3) The optimal encoder size scales log- proportionally with the LLM size. Based on these findings, we propose NaViL, a native MLLM that achieves competitive performance on diverse multimodal benchmarks, outperforming existing compositional MLLMs. We hope these insights will inspire future research on next-generation MILMs.
Limitations and Broader Impacts. Due to limited computation resources, this paper only in- vestigates the scaling properties of native MLLMs up to 9B parameters. Subsequent experiments with larger scales (e.g., 30 billion, 70 billion, 100 billion, etc.) can be conducted to further validate this scaling trend. addition, this paper focuses only on visual and linguistic modalities. Future research may explore broader modalities and provide more in-depth insights beyond the current lingustiig paradigm.
The work is supported by the National Key R&D Program of China (NO. 20227D0161 3000, and NO. 20222D0160102), by the National Natural Science Foundation of China (U24A20325, 62321005, 62376134), and by the China Postdoctoral Science Foundation (No. BX20250384).
Acknowledgments
10

---

References
[1] Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed- modal language models. In International Conference on Machine Learning, pages 265-279. PMLR, 2023.
[2] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning scale. In ICCV, pages 8948 8857-899772227 2019.
[3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mci Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma. Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang. Jian Yang. Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang. Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint orXiv:2309 2309.1669,,, 2023.
[4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenhin Ge, Sibo Song. Kai Dang. Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-v1 technical report. arXiv preprint arXiv:2502.13923, 2025.
[5] Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Introducing our multimodal models, 2023.
[6] Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang. Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:240.007726 2024.
[7] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee. Woonhyuk Back, and Saehoon Kim. Coyo-700m: Image-texi pair dataset. lattps://githiii com http://www...oooeaaaaaaaaa/aooo/
2022.
[8] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internim2 technical report. arXiv preprint arXiv:2403.17297, 2024.
[9] Chameleon Chameleon: fixed-modal early-fusion foundation models.
arXiv preprint
arXiv:2405.098/1 2024.
[10] Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.0000 2015.
[11] Yangyi Chen, Xingyao Wang, Hao Peng, and Heng Ji. A single transformer for scalable vision-language modeling, arXiv preprint mxXiv:2077.044355 2024.
[12] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries open-soueee multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05277, 2024.
[13] Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXhv:2412.0527/ 2024.
[14] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv:2404. 16821, 2024.
[15] Zhe Chen, Jiannan WU, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual- iisuallinguistic tasks. arXiv: 2312. 14238, 2023.
[16] Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang. Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevim v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv: 240220330606 2024.
[17] Christopher Clark and Matt Gardner. Simple and effective multi paragraph reading comprehension. In ACL, pages 845 855, 2018.
11

---

[18] Contributors. Opencompass: A universal evaluation platform for foundation models. https://github com/ httpp///ccooppppp/encaapapasssss 2023.
[19] Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, and Xinlong Wang. Unveiling encoder-free vision-language models, arXiv preprint arXiv:2406. 1832, 2024.
[20] Haiwen Diao, Xiaotong Li, Yufeng Cui, Yueze Wang, Haoge Deng, Ting Pan, Wenxuan Wang, Huchuan Lu, and Xinlong Wang. Evev2: Improved baselines for encoder-free vision-language models. arXiv preprint ar0::: 067888 2025.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2020.
[22] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models, arXiv: 2306. 13394, 2023,
[23] Behrooz Chorbani. Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. arXiv preprint arXiv:2109.0774) 2021.
[24] Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, pages 6904 6913, 2017.
[25] Jiaxi Gu, Xiaojun Meng, Guansong Lu, Hou, Niu Minzhe, Xiaodan Liang, Lewei Yao, Runhui Huang, Wei Zhang, Xin Jiang, et al. Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark. NeurlPS, 35:26418-26433 2022.
[26] Drew A Hudson and Christopher D Manning. Gqa: A new datasct for real-world visual reasoning and compositional question answering. In CVPR, pages 5700-6709 2019.
[27] Drew A. Hudson and Christopher D. Manning, GQA: new dataset for real- real-world visual reasoning and compositional question answering. In CVPR, pages 6700-6709, 2019.
[28] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi. Ali Farhadi, and Ludwig Schmidt, Openclip. Zenodo. Version 0.1. htttps://doi
5281///e00222/ 5143773
2021. DOL:
10.5281/2enodo5544577
[29] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.088 003661..088111111101111111111111 2020.
[30] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. diagram is worth a dozen images. In ECCV, pages 235-251, 2016.
[31] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloć Rolland, Laura Gustafson, Tote Xiao, Speneer Whitehcad, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. arXiv: 2304.02643, 2023.
[32] Weixian Lei, Jiacong Wang, Haochen Wang, Xiangtai Li, Jun Hao Liew, Jiashi Feng, and Zilong Huang. The scalability simplicity: Empirical analysis of laioun-lan learning with a single transformer. arXiv preprint ar5iv:2504. 10462, 2025.
[33] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-ing image pre- ge-training for unified laniun-language understanding and generation. In ICML, pages
2022.
[34] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv: 2400.18814, 2024.
[35] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arxiv: 2310.03744, 2023.
[36] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[37] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao Yike Yuan, Jiaqi Wang. Conghui He. Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi multi-modal model an around player? arXiv: 06280, 2023.
12

---

[38] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui IIe, Ziwei Liu, al. Mmbench: Is your multi-modal model all-around player? arXiv preprint Xiv:2307.0662// 2023.
[39] Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng. Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895, 2023.
[40] Haoyu La, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepscek-vl: Towards real-world vision-language understanding. arXiv 2024.
preprint or arii::4403.05555
[41] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv: 2310.02255, 2023.
[42] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Rei Chang, Song-Chun Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, In NewrlPS, 2022.
[43] Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jiawen Liu, Jifeng Dai, Yu Qiao, and Xizhou Zhu. Mono-intemwy:: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training. In CVPR, 2025.
[44] Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models. arXiv preprint 2403v:4003005003 2024.
[45] Ahmed Masry, Xuan Long Do. Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In ACL, pages 2263-2279, 2022.
[46] Minesh Mathew, Viraj Bagal, Rubên Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. WACV, pages 1697-1706, 2022.
[47] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV, pages
2021.
[48] Brandon McKinzie, Zhe Gan, Jean- -han-philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hc, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lci, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, and Yinfei Yang. MMI: methods, analysis & insights from multimodal LLM pre-training. arXiv: $403.09614 1, 2024.
answering by reading text in images. In ICDAR, pages 947 957,99522277 2019. [49] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question
[50] OpenAL. Gpt-4v(ision) system card, stttps//ccde opw.aieena........ cww/ 2023.
http:///ccc..ppppaaaoooooppppppp Saappeerr//GPVSSSyytee Card. pdf
[51] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaacidin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2023.
[52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML., pages 8748-8763, 2021.
[53] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In JCML, volume 139, pages 8748-8763, 2021.
[54] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. Jean baptise Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schritwieserr et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint 0555v:2403.05530, 2024.
13

---

[55] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-Sb: An open large-scale dataset for training next generation image-text models. NeurIPS, 15522228--2294 2022.
[56] Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind. and Alaaeldin El-Nouby. Scaling laws for native multimodal models. arXiv preprint arXiv:2504.09551 2025.
[57] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang. Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In CVPR, 2019.
[58] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114, 14. PMLR, 2019.
[59] InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. http://ittuu..coomeaeoooeoe httpps///ggguub..mmm/mmem/////// /otte//ooo//mtttm/////nttemmlll 2023.
[60] Qwen Team. Qwen3 blog.
http://wwwn..gttuubooooo/ooooo// http//loogq/wennn 2025.
[61] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Erie Hambro, Faisal Azhar, et al. I. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[62] Han Wang, Yongjie Ye, Bingru Li. Yuxiang Nie, Jinghui Lu, Jingqun Tang, Yanjie Wang, and Can Huang. Vision as lora, arXiv preprint (eX.Xii25333.660 2025.
[63] Peng Wang. Shuai Bai, Sinan Tan, Shijic Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision- saiion-anguage model's perception the world any resolution. arXiv preprint 12XVv:2409.22191, 2024.
[64] Wenhai Wang. Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In CVPR, pages 1408-114499 2023.
[65] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo,, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang. Jingjing Liu, Yonghua Lin, Tiejun Huang, and Zhongyuan Wang. Emu3: Next-token prediction is all you need. arXiv: 2409. 18869 2024.
[66] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang. Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li. Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408.0/ 2024.
[67] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metries for semantic inference over event descriptions. TACL, 2:67-78, 2014.
[68] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint MxXiv2230..02490 2023.
[69] Weihao Yu, Zhengyuan Yang, Linjie Li. Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv: 2308.02490. 2023.
[70] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv: 1.6522 2023.
[71] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lueas Beyer Scaling vision transformers. In CVPR, pages 12104 12113, 2022.
[72] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, pages 11975 1 11986, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Sec.
Sec.
Sec.
and the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [No]
Justification: Due to company policy, the training data cannot be fully released. However, will specify all training and testing details in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
publicc/uuddes/Codeeuubutt Soaiiddes/oodeSuubbsssssooo oolloo for more details.
While we encourage the release code and data, we understand that this might not be possible, so 'No' is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
/mips/ccc/pblic//uuudee hetpolliicggiddeeCooooooo http://dddd//oeuuubinnn ssionlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the training and test details. Please refer to Sec. and the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due limited resources, do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Due to company policy, we cannot provide too many details about the compute resources. But we will elaborate on the training details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited and respected them in the paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

Technical Appendices and Supplementary Material
A Va Scaling up to 9B parameters
To further demonstrate the scaling capability of our method, we trained NaViL-99 based on Qwen3- 8B [60]. The total number of activation parameters is 9.2B, of which 1.2B belongs the visual encoder. The training recipe is similar to NaViL-2B, as shown in Tab. 8 except the visual multi- scaling packing disabled in the first sub-stage of pre-training for acceleration.
Tab. 3 presents a comparison of the total training tokens required by our method versus two composi- tional counterparts. Notably, our approach achieves comparable performance while using substantially fewer training tokens, demonstrating improved training efficiency.
Table 3: Comparison between VaL and existing MLLMs on the number of training tokens.
Models
Train VIT
Train MLLM
Total
Qwen2.5VI VIL
unknown
4.17
>4.1T
VL2.588 [2]
>3.3T
140B
>3.5T
NaViL-2B (ours)
0
800B
800B
NaViL-9B (ours)
0
450B²
450B
The performance results on multimodal and visual question answering benchmarks shown in Tab. 4 With a similar parameter size, our NaViL-99 outperforms all existing native MLLMs by large margin on almost all benchmarks. Besides that, compared the compositional baseline model VL-255 with a similar parameter size, NaViL-99 also achieves competitive performance. Such results show that our proposed native MLLM can be scaled up to larger parameter sizes and achieve consistent performance gains.
B More discussions on Compositional MLLMs and Native MLLMs
Visual Encoder
Contrsstve
Visual / Texl Encoder
Loss
Copy Weights
Randomly Initialized
Visual
Visual Encoder
Encoder
Visual Expert
Multimodal Large Language Models
Multimodal Large Language Models
Next Token
Next Token Prediction
Prediction
(a) Compositional MLLMs without MoE
(b) Native MLLMs with MoE
Figure 10: Paradigm Comparison between Compositional MLLMs and Native MILMs. Compositional MLLMs adopt different training objectives and strategies (e.g. Contrastive Loss or Next-Token- )reniken-PPeiictioon) to pre-train the visual encoder and LLM separately, while native MLLMs optimize both image and text components in end-to-end manner using a unified training objective (f.e. Toke--Toen-Boo Preenn--rdittion.
Fig. 10 further illustrates the difference between compositional MILMS and native MLLMs. Compo- sitional MILMs typically have different components initialized by separate unimodal pre-training, where different training objectives and strategies are employed to train the LLM and visual encoder. For example, the visual encoder can be trained using an image-text contrastive learning objective (e.g.. CLIP [52]. SigLIP (72) or a self-supervised learning objective (e.g.. DINOv2 (51)). The complexity
Due to limited computational resource and time, current version of NaViL-9B in this paper is only trained with 450B tokens.
22

---

of such training process increases the difficulty of scalability. On the other hand, discussed in [56]. native MLLM optimizes both image and text modalities end-to-end using unified training objective (i.e., next-token prediction (NTP)). This avoids introducing additional bias and significantly simplifies the scaling effort.
C More Related Works
Research on Neural Scaling Laws. The foundational work Neural Scaling Laws began in the Natural Language Processing (NLP) domain, where [29] established predictable power-law relationships demonstrating that performance loss (I) scales reliably with model size (N) and data size (D), and that larger, decoder-only Transformer models are more compute-
Following works [23] further extended such research to encoder-decoder architectures, observing consistency in scaling exponents on Neural Machine Translation (NMT) tasks. Driven by these successes, in the vision domain,
71 confirmed the applicability of scaling laws Vision Transformers (ViT), systematically demonstrating continuous performance improvement by scaling both model size (up billion parameters) and training data. Most recently, these principles have been generalized to Large Multimodal Models, where [T] developed scaling laws that unify the contributions of text, image, and speech modalities by explicitly modeling synergy and competition as additive term. Furthering this, [56] explored Native Multimodal Models (NMMs) using Mixture of Experts (MoEs), finding an unbalanced scaling law that suggests scaling training tokens (D) is more critical than scaling active parameters (N) as the compute budget grows.
D Implementation Details
The hyperparameters of model architecture for NaViL-2B and NaViL-9B are listed Tab. 6 while the hyperparameters of training recipe for NaViL-2B and NaViL-9B are provided in Tab, and Tab. g respectively. The high-quality multimodal data used in Pre-training and Supervised tuneittning is from Intern VL-2.5 [12], which is sourced from various domains, such image captioning, general question answering, multi-tur turn dialogue, charts, OCR, documents, and knowledge, etc.; while the pure language data is primarily from InternI.22.5 g
E
The NLP capability
We also evaluate the NLP capability of our model on three popular NLP tasks, as shown in Tab. [ Thanks the modality-speeifie MoE architecture. NaViL maintains the NLP capabilities of its initialization LLM (Qwen3-8B) Despite not using large amount of high-quality text data, NaViL performs well on the common NLP tasks and show much stronger NL.P capabilities compared to other native MLLMs, showing its data efficiency.
F
More Qualitative Results
More visualization results of multimodal understanding are provided below.
"The performance of Chameleon-7B from
43
23

---

Table 4: Comparison between NaViL-99 and existing MLLMs on multimodal benchmarks. "#A-Param" denotes the number of activated parameters. IInter L--2.58B- adopts the same high- quality data with NaViL-9B, so we mark it as the compositional counterpart. Note that its 300M visual encoder distilled from another 6B large encoder. denotes our reproduced results. Bold and underline indicate the best and the second-bes performance among native MLLMs, respectively. For MME, we sum the perception and cognition scores. Average scores are computed by normalizing each metric to range between O and 100.
Model
#A-Param Avg MMVet MMMU MMB MME Math OCR-B TVQA DocVQA AI2D ChartQA InfoVQA
Composttonal MLLMs;
MobileVLM-V2
16
1.7B
57.7
MobileVLMM-V2
16
57.5
Mini-Gemini
34
3.5B
31.1
31.7
$9.8
1653
29.4
56.2
34.2
MMIIMoE- Chat
48
3.5B
42.2
38.6
70.8
1772
32.6
72.9
DeepSeck-VI VL
20
2.0B
34.8
32.2
64.6
1532
31.1
409
57.8
51.5
PaliGemma 0
2.9B
33.1
34,9
71.0
1686
28.7
614
68.1
68.3
MiniCPM V-2 66
2.8B
41.0
38.2
69.1
1809
38.7
605
74.1
71.9
62.9
VL-1
61.3
39.3
34.6
70.9
1902
70.5
85.0
55.4
Qwen2VI.
49.5
41.1
InterVV1-2.5
2138
74.9
Qwen2VL 65
8.2B
77.1
62.0
54.1
83.0
2327
58.2
866
84.3
94.5
83.0
76.5
Qwen2.5VVL 4
8.2B
80.2
67.1
58.6
83.5
2347
68.2
864
84.9
95.7
83.9
87.3
82.6
InnemmLL22.5 13
8.1B
77.3
62.8
56.0
84.6
2344
64.4
822
79.1
91.9
84.5
84.8
75.7
Notive MLLMs:
Fayu-8B (HD) 5
8B
21.4
10.7
64.5
SOLO 11
7B
126()
34.4
61.4
Chaneleon-7B-
7B
14,0
8.3
25.4
31.1
170
22.3
7
4.8
1.5
46.0
2.9
5.0
EVE-7B 19
7B
34,6
25.6
32.3
49.5
1483
25.2
327
51.9
22.0
48.5
19.5
20.0
EVE-7B (HD)
45.2
25.7
32.6
52.3
1628
34.2
56.8
53.0
61.0
59.1
25.0
Emu3
31.6
76.3
43.8
VoRA 62
33.7
32.2
64.2
1674
56.3
VoRA-AnyRes
33.7
32.0
1655
58.7
EVEv2 20
62.3
45.0
39.3
66.3
1709
60.00
702
71.1
77.4*
74.8
73.9
45.8*
SAIL 32
7B
63.7
46.3
38.6*
70.1
1719
$7.0
783
77.1
78.4*
76.7
69.7*
47,3"
Mono-IntenVL.
43
1.8B
60.6
40.1
33.7
65.5
1875
45.7
767
72.6
80.0
68.6
73.7
43.0
NaViL-2B (ours)
2.4B
68.8
78.3
41.8
71.2
1822
50.0
796
76.9
85.4
74.6
78.0
56.0
NaViL.9B (ours)
9.2B
77.0
79.6
54.7
76.5
2225
66.7
837
77.2
90.6
82.4
85.4
70.2
Table 5: Comparison of ViL. and existing native MLLMSs three common NLP tasks. Except for Chameleon, models are evaluated using OpenCompass toolkit (8].
Models
#A-Param
MMLU
CMMLU
MATH
MnternLM2-Chen (59
1.8B
47.1
46.1
13.9
Qwen3-8B (non-thinking) 60
8B
76.5
76.8
71.1
EVE [99
7B
43.9
33.4
0.7
Chameleon
7B
52.1
11.5
Mono-InterVIL.
4
2B
45.1
44.0
12.3
NaViL-99B (ours)
9.2B
74.9
75.1
66.2

---

Table 6: Klppe-Parrametess of Model Architecture.
Component
NaViL-9B NaViL-2B
visual encoder
# Params depth width
0.6B
1.2B
24
32
1472
1792
MLP width
5888
7168
# attention heads
23
28
LLM (w/ MoE)
# experts A-Params
2
2
1.8B
8.0B
depth width
24
36
2048
4096
MLP width
8192
12288
# attention heads
16
32
Table 7: Hyper- parameters for training Na ViL-2B.
Multi-modal Generative Pre-training (S1)
Supervised Fine-tuning (S2)
Configuration
S1.1
S1.2
Maximum number of image patches
1096
12188
21576
Training steps
70k
40k
30k
Global batch size
7, 000
4, 614
2, 234
Weight decay
0.05
0.1
0.01
Learning rate schedule
constant with warm-up
cosine decay 2e 5
Peak learning rate
5e
Visual Multi-scale Packing
LLM max sequence length
16, 384
Warm-up steps
200
Optimizer
Adam W
Optimizer hyperparameters
81 0.9, 82 0.95, eps le
Gradient accumulation
Numerical precision
bfloat16
Table 8: Hyper-parameters for training ViL.-9B.
Multi-modal Generative Pre-training (S1)
Supervised Fine-tuning (S2)
Configuration
S1.1
S1.2
Maximum number of image patches
4096
12188
24576
Training steps
50k
33k
6k
Weight decay
0.05
0.01
Global batch size
10, 300
1, 792
3, 520
Visual Multi-scale Packing
X
Learning rate schedule
constant with warm-up 5
cosine decay -5
Peak learning rate
LLM max sequence length
16, 384
Warm-up steps
200
Optimizer
Adam
Optimizer hyperparameters
81 0.9, 82 0.95, eps 1e-8
Gradient accumulation
Numerical precision
bfloat16
25

---

Image Captioning and Visual Question Answering
Prompt: Describe this image in detail.
The image shows acrial view of modern architectural complex situated near coastal area. The building has distinetive design with series of curved, wave-like structures its roof, giving it a futuristic appearance. The exterior primarily composed of light-colored materials, possihly concrete or stone, with large glass windows.
Response:
In front of the building, there wide, multi-lane road lined with palm trees, suggesting warm, coastal climate. The road appears pathways. There are also train tracks running parallel to the road, indicating the presence of public transportation system.
major thoroughfare, with designated bike lanes and pedestrian
The background features large body of water, likely bay ocean, with several boats and yachts docked a marina. The coastline is lined with more buildings, and there are hills or mountains visible in the distance, adding to the scenic view. The overall setting suggests a bustling urban area with a focus modern infrastructure and strong connection to the waterfront.
Prompt: Which city is this building most likely
Response: This building is most likely in San Diego, California. The architectural style and the surrounding landscape, including the water and palm are characteristic of San Diego
Prompt: Who is the person in black in the middle?
The person in black in the middle is Lionel Messi.
Response:
Prompt:
What they celebrating about?
The image shows a group of soccer players celebrating with a trophy, wearing medals and jerseys. They are likely celebrating victory in soccer tournament championship.
Response:
Prompt: What is the content in this image?
The image appears to be a screenshot from video game, likely Grand Theft Auto V (GTA V), given the visual style and interface elements. It shows character standing sidewalk in small town suburban
Response:
The environment includes a street with tram tracks, few buildings with signs, and some cars parked along the road, The buildings have signs for businesses like "Wine" and "On The Bone." The scene has somewhat deserted look, with no other characters significant activity visible.
26

---

OCR
Prompt: Extract all the text in this image.
NUTRITIONAL INFORMATION
Serving size: 10
Servings per package:
Response: NUTRITIONAL INFORMATION
Average Quantity Per Serving
Average
Serving size: 10
Quantity
Servings package: 25
Per 100g
Average Quantity Per Serving Average Quantity Per 100g
Energy
143kJ
Energy 143KJ 1430
Fat, total
less than 0.1g
than 0.1g
0.16g
0.37g
Protein less than 0.1g 0.16g
-aturated
less than
0.26g
Fat, total | less than 0.1g g 0.37g
Carbahydrates
8.31g
83.1g
saturated less than 0. 0.26g
Sugars
8.11g
81.1g
Carbohydrates 8.31g 18.31g
Sodium
41m
4 (44g
Sugars 8.11g 81.1g
INGREDIENTS: 100% Pure New Zealand Honey
Sodium 0.41mg 4.04mg
Keep Stored Below deg' C, BFA Contamtr
INGREDIENTS: 100% Pure New Zealand Honey
Not suttalle for children under year age
Keep Stored Below 20 deg"C, BPA Freee Container.
Not suitable for children under year of age.
Food - Lunch 5 people
Prompt: Describe the image.
CHAPELBANK HOTEL 69 EAST HIGH STREET FORFAR 0D8 2EP TEL NO 01307 483151
The image is receipt from Chapelbank Hotel for lunch order for five people. The items listed the receipt include 3x soup at f3.95 each, totaling L1188 lunch main course at f6.95; lasagne (V) at E6.95; mise food at 26.95; and 2x chips/omion rings f1.75 each, totaling 53. The subtotal for the food is #29.25. The receipt also includes a previous balance of #29.25, making the total amount due $29.25. The hotel's address is 69 East High Street, Forfar DD8 2EP, and the telephone number is 01307 4631511 151,
Response:
*
TABLE
5
RESTAURANT
3x SOUP
3.95
11.85
Lunch Main
Lasagne (V)
6.95
MISC FOOD
6.95
Prompt: How many people are attending this lunch?
2x CHIPS / ONION RINGS 1.75
3.50
29.25--
There are 5 people attending this lunch. This is indicated at the top of the receipt where says "Food Lunch x 5 people".
Response:
Prev Bal:
29.25
SUBTOTAL
29.25
CHI
Prompt: Where is the left exit going to?
WEST
South
NORTH
ReStOR Merndon
Chain Br
Response: The left going to Reston, Herndon, and Dulles Airport.
Dulles Airport
MCLeOn
Prompt: Which road lcads to the EXTT 46 B?
Response: The road that leads to EXIT 46 is Chain Br Rd North.
27

---

Multi-lingual OCR
AF ((60))
AB
4 AL
12
M.
45
-bs
7
4K
8
$
3
R
4/
13
& #
82
K44
1
#
Prompt: - X
Response: X
FXX (60
: # *
1I A JORERA FRTH RARITAX М RANAEE ク A T EH # А -
E ビT
- 2
A. L6
BS
Essay (60 points)
Title Focus on your goal, enjoy life.
The light of the goal is like the clear glow of moonlight; the brilliance of life is like the dazzling stars. Without the stars, the moonlight becomes cold and desolate; without the moon, the stars will inevitably become dull. Only by focusing on moonlight can you find direction; only by appreciating the stars can you enjoy life. The stars and moon shine together, making the night sky enchanting; focusing on your goal, enjoying life, allows you to savor the quiet joy of life.
Document Understanding
What the duration of this venue sponsorship agreement?
Prompt:
HRVRMMATING CHITP DOLOE
Response: January 2001 December 31, 2001
Prompt: Where is Vitucci's located (complete address)?
SHELANL RESUCTOD
Response: 1832 East North Avenue, Milwaukee WI 53202
Prompt: What cigarette brand mentioned in the agreement for delivery and distribution?
Response: Camel cigarctte brand styles purchased by Venue Owner for resale in Venue
Prompt: Who has discretion over the distribution of flyers for special events?
COPY
ORIGINAL
Response: KBA Marketing Group ("KBA") discretion over the distrihution flyers for special events.
28

---

LaTeX and Math
(o)zo(0) - Elq 111g-P(00 P(()) ~ AR (ola)
min
motouile o(eer)
Ais clip
naw (ouly 91,er) 1- e, 1 Aig
matoule oorr)
DDD (sollxell]
(3)
Aaaa (oi (i,er)
Prompt:
Convert the formula in the figure to latex format.
Response:
<<taat
\pegtnfequation]
(beginfsplit)
(ratheal() (1tt)) à- (q Aein P(Q), (i_iV_(i--1"a
Aein (pl_(taaata_told))(aaa) (0l4)) V
(G)) (i-1)"d Vfrac((Ko__il) (t-1")(oi)))) Wefft vain (ts e_(i,<t))>
Aleftl
Aefft( \text(clip)
(hat(A)_(t,tt)
Hipi (hthti/tihtttttaaaaaaaaaaa (Athetta (olt)) (.),)))),, Warrpeslon, 1 Warepeilon \right) (hat()_((, \right]
Ibeta \natheal(D)_(KL) Weftt (Vthett) II (ri_(ref) \right] \right\), lend(eplit)
(equuaittio
Prompt: Find x. = 148 m Please think step by step.
To find the value of in the parallelogram, can use the formula for the area parallelogram:
Response:
A = base x height
Given: Area = 148 m² Base 8m Height Substitute the given values into the formula:
x xm
148 x
8 m
Solve for ar:
148
18.5
So, æ 18.5 m.
Prompt: A dog show enthusiast recorded the weight the winning dog at recent dog shows. According to the table, what was the rate of change between 2016 and 2017? (Unit: kilograms per year)
Weight of winning dog at dog show
Response: To determine the rate change between 2016 and 2017, we use the formula for rate of change:
Year
Weight (kg)
2013
27
Weight in 2017 Weight in 2016
Rate of Change
Year 2017 Year 2016
2014
From table: Weight 2016 29 kg Weight 2017
2015
34
40 kg
Substitute these values into the formula:
2016
29
40 kg 29 kg
kg 2017 2016
2017
40
Rate of Change
I
Rate of Change = 11 kg/year
Thus. the rate of change between 2016 and 2017 is 11 kg/year
29


Reviews:
[{"review_id": "xRkVDMMixV", "reviewer": "Reviewer_kjaE", "summary": "This paper investigates the design space and scaling properties of native multimodal large language models (MLLMs) under data-constrained conditions. The authors distinguish between former compositional approaches, which separately pretrained vision encoders and language models before combining them, and native approaches that jointly optimize both components end-to-end. Through experimentation across three architectural components, the authors propose NaViL, a native MLLM that achieves competitive performance with approximately 600 million pre-training image-text pairs. The key claimed contribution is the discovery that optimal visual encoder size scales log-linearly with language model size.", "questions": "1.\tSection 3.2.2 lacks crucial implementation details about the MoE architecture. What is the exact MoE configuration used in the comparison? How many experts are employed, and what is the expert selection mechanism? Most critically, what are the precise parameter counts for both vanilla transformer and MoE configurations to ensure fair comparison? Without parameter equivalence, the observed performance improvements in Figure 3 may simply reflect increased model capacity rather than architectural advantages.\n2.\tSection 3.2.3 explores visual encoder architecture only through width-depth variations. Have the authors considered more sophisticated architectural explorations such as attention mechanism variations, different normalization strategies, or alternative activation functions that might yield more novel insights for native MLLM design?\n3.\tWhile the log-linear relationship in Figure 7 is empirically observed, what theoretical principles could explain this phenomenon? Besides, the only three pairs (0.5B, 2B, 7B) of datapoints are quite limited for such a claim of scaling law. Can the authors make sure this trend can generalize to larger scales?\n4.\tBaseline Evaluation. Tables 1 and 2 show incomplete results for baselines evaluation across multiple benchmarks. Would complete baseline evaluation change the relative performance rankings?\n5.\tAs mentioned in Section 5.1, NaViL-2B is built upon InternLM2-1.8B, initializing its text part parameters with the weights of InternLM2-1.8B. I assume this means that NaViL-2B is, to some extent, further trained based on InternLM2-1.8B. While this approach may facilitate training convergence, is it really fair for the performance comparisons? If NaViL-2B were trained from scratch, would it still achieve similar performance results?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1.\tThis paper is very well written and easy to understand.\n2.\tThe research question that this paper wants to explore is significant. The native MLLM training approach should be more explored and compared against the compositional approaches.\n3.\tThe log-linear scaling relationship between optimal visual encoder size and language model size represents a potentially significant discovery. This finding challenges compositional paradigms that employ fixed visual encoder sizes across varying LLM scales and could provide quantitative guidance for future model design.\n4.\tThe research addresses real-world constraints by demonstrating competitive performance using approximately 600 million pre-training image-text pairs.\nWeaknesses\n1.\tSection 3.2 presents limited novel insights across its three subsections. The LLM initialization exploration (Section 3.2.1) yields unsurprising conclusions that initialization of pre-trained weights help. The visual encoder architecture study (Section 3.2.3) only examines width-depth trade-offs and reaches conventional conclusions that shallow models converge faster while deeper models achieve better performance, consistent with established deep learning principles.\n2.\tSection 3.2.2 (lines 137-151) provides inadequate description of the MoE architecture implementation and fails to ensure fair experimental comparison. The paper does not specify the exact parameter counts for vanilla versus MoE configurations, the number of experts used, or the expert selection mechanism. \n3.\tThe downsampling rate mentioned in line 232 is represented by the symbol \\tau, which is also used to represent another quantity used in line 211, potentially causing ambiguity.\n4.\tTables 1 and 2 contain substantial missing experimental results across multiple baseline models and benchmarks and without further explanation. This incomplete evaluation results undermines the paper's claims about competitive performance against existing methods and makes it difficult to assess the true relative standing of NaViL.", "was_revised": true, "final_justification": "The rebuttal has addressed most of my concerns.", "rebuttal": "Thank you for your valuable and detailed comments. We address your questions as follows.\n\n> **W1.1: Section 3.2 presents limited novel insights across its three subsections. The LLM initialization exploration (Section 3.2.1) yields unsurprising conclusions that initialization of pre-trained weights help.**\n\nA1: We understand your concerns about LLM initialization benefits seeming intuitive. However, this intuition doesn't always hold even in computer vision; for example, [1] shows ImageNet pretraining doesn't improve object detection and instance segmentation performance.\n\nIn the MLLM community, while LLM initialization has become the default practice, this fundamental question remains unanswered. Therefore, we quantitatively demonstrate LLM initialization's impact in Fig. 2, breaking this black box and addressing this fundamental question for the entire community. From this perspective, our conclusions and results provide valuable insights to the community.\n\n> **W1.2: (1) Section 3.2.3 only examines width-depth trade-offs and reaches conventional conclusions that shallow models converge faster while deeper models achieve better performance. (2) Have the authors considered more sophisticated architectural explorations such as attention mechanism variations and different normalization strategies?**\n\nA2: (1) Similar to W1.1, visual encoder width-depth design for MLLMs remains controversial. Previous studies [2,3] focus on single visual encoders for pure vision tasks, but these principles may not transfer to MLLMs with large-scale LLMs. Most works use fixed-size visual encoders (e.g., SigLIP) or discard them entirely (Mono-InternVL), making it difficult to draw clear conclusions about width-depth design for native MLLMs.\n\nWe systematically ablated visual encoder width-depth design and found **a different conclusion** that visual encoders achieve near-optimal performance across wide depth/width ranges. Except for extremely shallow/deep configurations, relatively shallow encoders converge faster while deeper ones perform better in later training stages. These findings provide valuable guidance for future native MLLM research.\n\n(2) We focus on encoder size scaling properties, keeping meta-architecture (attention, normalization, activation functions) the same while varying only depth and width, following common practices for scaling law research [4,7]. Better architectural components will be explored in future work.\n\n> **W2: Section 3.2.2 (lines 137-151) provides inadequate description of the MoE architecture implementation and fails to ensure fair experimental comparison. The paper does not specify the exact parameter counts for vanilla versus MoE configurations, the number of experts used, or the expert selection mechanism.**\n\nA3: The number of activated experts is set to one (L145-L146) and static modality routing is used (L143-145). Therefore, **the activation parameter count and training FLOPs are the same for both vanilla and MoE configurations**, which ensures fair comparison and is consistent with the experimental configurations in many prior works [4].\nAppendix C has introduced the details of MoE implementation. We thank the suggestion and will revise by discussing more implementation details of our MoE in the main paper.\n\n> **W3: The downsampling rate mentioned in line 232 is represented by the symbol \\tau, which is also used to represent another quantity used in line 211, potentially causing ambiguity.**\n\nA4: Thanks for the suggestion. Different symbols will be used in L232 and L211 for the camera-ready version to avoid ambiguity.\n\n> **W4: Tables 1 and 2 contain substantial missing experimental results across multiple baseline models and benchmarks and without further explanation. This incomplete evaluation results undermines the paper's claims about competitive performance against existing methods and makes it difficult to assess the true relative standing of NaViL.**\n\nA5: In Tab. 1 and 2, we report only the official experimental results of other baseline models, sourced from their respective papers. Since different papers evaluate their models on different benchmarks, some results are missing. To further support our claims about NaViL's performance, we report the full results of Qwen2VL-2B, SAIL, and EVEv2, as shown in the following tables (* denotes our reproduced results).\n\n| Model | #A-Param | Avg | Avg (w/o CCB) | MMVet | MMMU | MMB | MME | MathVista | OCRBench | CCB |\n|-------|----------|-----|---------------|-------|------|-----|-----|-----------|----------|-----|\n| EVEv2 | 7B | 53.2 | 57 | 45 | 39.3 | 66.3 | 1709 | 60* | 702 | 30.8* |\n| SAIL | 7B | 53.7 | 58.6 | 46.3 | 38.6* | 70.1 | 1719 | 57 | 783 | 24.3* |\n| NaViL-2B | 2.4B | 67.1 | 64.3 | 78.3 | 41.8 | 71.2 | 1822 | 50 | 796 | 83.9 |\n| NaViL-9B | 9.2B | 73.2 | 73.4 | 79.6 | 54.7 | 76.5 | 2225 | 66.7 | 837 | 71.6 |\n\n\n| Model | #A-Param | Avg | TextVQA | SQA-I | GQA | DocVQA | AI2D | ChartQA | InfoVQA |\n|-------|----------|-----|---------|-------|-----|--------|------|---------|---------|\n| Qwen2VL-2B | 2.1B | 73.1 | 79.7 | 78.2* | 60.3* | 79.7 | 74.7 | 73.5 | 65.5 |\n| EVEv2 | 7B | 71.7 | 71.1 | 96.2 | 62.9 | 77.4* | 74.8 | 73.9 | 45.8* |\n| SAIL | 7B | 71.5 | 77.1 | 93.3 | 58.0* | 78.4* | 76.7 | 69.7* | 47.3* |\n| NaViL-2B | 2.4B | 75.1 | 76.9 | 95 | 59.8 | 85.4 | 74.6 | 78 | 56 |\n| NaViL-9B | 9.2B | 80.4 | 76.4 | 98.1 | 59.5 | 90.6 | 82.4 | 85.4 | 70.2 |\n\nThe conclusion remains consistent: NaViL's average performance is still ahead of previous native MLLMs and is competitive with the compositional baselines.\n\n> **Q1: Section 3.2.2 lacks crucial implementation details about the MoE architecture. What is the exact MoE configuration used in the comparison? How many experts are employed, and what is the expert selection mechanism? Most critically, what are the precise parameter counts for both vanilla transformer and MoE configurations to ensure fair comparison? Without parameter equivalence, the observed performance improvements in Figure 3 may simply reflect increased model capacity rather than architectural advantages.**\n\nA6: Please refer to A3.\n\n> **Q2: Section 3.2.3 explores visual encoder architecture only through width-depth variations. Have the authors considered more sophisticated architectural explorations such as attention mechanism variations, different normalization strategies, or alternative activation functions that might yield more novel insights for native MLLM design?**\n\nA7: Please refer to A2.\n\n> **Q3: While the log-linear relationship in Figure 7 is empirically observed, what theoretical principles could explain this phenomenon? Besides, the only three pairs (0.5B, 2B, 7B) of datapoints are quite limited for such a claim of scaling law. Can the authors make sure this trend can generalize to larger scales?**\n\nA8: Several theories explain the log-linear relationship in Fig. 7, including encoder-decoder scaling laws in neural machine translation [5] and multimodal scaling laws for text, speech, and images [6]. Similar theories could apply to native MLLMs, but our paper focuses on design space and scaling properties under data constraints, leaving theoretical analysis for future work.\n\nScaling laws predict larger-scale model performance from small-scale experiments. Previous works [7] don't show all datapoints for every scale. Since native MLLM scaling research typically uses models below 8B parameters [4], we follow this convention while planning verification on larger MLLMs. We will further verify the results of the scaling law on a larger MLLM.\n\n> **Q4: Baseline Evaluation. Tables 1 and 2 show incomplete results for baselines evaluation across multiple benchmarks. Would complete baseline evaluation change the relative performance rankings?**\n\nA9: Please refer to A5.\n\n> **Q5: As mentioned in Section 5.1, NaViL-2B is built upon InternLM2-1.8B, initializing its text part parameters with the weights of InternLM2-1.8B. I assume this means that NaViL-2B is, to some extent, further trained based on InternLM2-1.8B. While this approach may facilitate training convergence, is it really fair for the performance comparisons? If NaViL-2B were trained from scratch, would it still achieve similar performance results?**\n\nA10: NaViL-2B's textual parameters are initialized from InternLM2-1.8B. Since this paper focuses on native MLLM scaling properties under data constraints, convergence speed is crucial. All MLLMs in Tab. 1-2 use pre-trained LLM initialization for practical high performance, ensuring fair comparison. As shown in L113-115 and Fig. 2, while non-LLM initialization can achieve similar final performance, it requires substantial high-quality data, violating our data-constrained setting.\n\n[1] He, Kaiming, Ross Girshick, and Piotr Dollár. \"Rethinking imagenet pre-training.\" In ICCV. 2019.\n\n[2] Alabdulmohsin, Ibrahim M., Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. \"Getting vit in shape: Scaling laws for compute-optimal model design.\" In NeurIPS. 2023.\n\n[3] Zhai, Xiaohua, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. \"Scaling vision transformers.\" In CVPR. 2022.\n\n[4] Shukor, Mustafa, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind, and Alaaeldin El-Nouby. \"Scaling laws for native multimodal models.\" arXiv preprint arXiv:2504.07951. 2025.\n\n[5] Ghorbani, Behrooz, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. \"Scaling Laws for Neural Machine Translation.\" In ICLR. 2022.\n\n[6] Aghajanyan, Armen, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. \"Scaling laws for generative mixed-modal language models.\" In ICML. 2023.\n\n[7] Kaplan, Jared, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. \"Scaling laws for neural language models.\" arXiv preprint arXiv:2001.08361. 2020."}, {"review_id": "IkG6MitijC", "reviewer": "Reviewer_Y9RL", "summary": "This paper presents a systematic study on training native Multimodal Large Language Models (MLLMs) in an end-to-end manner, with particular focus on design space exploration and scaling properties under data constraints. The authors derive several valuable insights, including: (1) the necessity of initializing from pretrained LLMs, (2) the benefits of adopting Mixture-of-Experts (MoE) architecture, (3) comprehensive analysis of visual encoder design choices, and (4) the finding that scaling visual encoders logarithmically proportional to LLMs outperforms fixed-size visual encoders. The extensive experimental validation leads to the development of NaViL, a native MLLM that achieves competitive performance across various multimodal benchmarks while outperforming existing compositional MLLMs. The paper is well-written and provides clear experimental insights.", "questions": "The related work section should include more comprehensive discussion of visual scaling paradigms, particularly foundational work such as Zhai et al. (2022) \"Scaling Vision Transformers\" (CVPR), which established important principles for visual model scaling that are relevant to this work.", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strengths\n\n1. The motivation is compelling and addresses a practically important problem. Investigating the scaling properties of native MLLMs represents a valuable research direction that could significantly simplify the training pipeline for building robust MLLMs.  \n\n2. The analytical experiments are extensive and yield actionable insights for the community. The investigation follows a logical progression: first providing detailed quantitative analyses of individual design choices like  LLM initialization strategies and MoE architecture adoption. This systematic approach effectively demonstrates the scaling characteristics of native MLLMs.\n\n3. The experimental results are convincing and demonstrate clear advantages.   NaViL  achieves competitive performance compared to compositional counterparts while requiring significantly less multimodal training data. The preserved NLP capabilities further validate the approach's effectiveness compared to previous native MLLM methods.\n\n4. The paper is well-structured and accessible. The introduction clearly establishes motivation, the analysis proceeds logically, and comprehensive figures and tables support the findings effectively.\n\n## Weaknesses\n\n1. The final result of the article only provides a 2B-sized model. It is better to conduct experiments on a larger model to verify the effectiveness of the scaling law.\n\n2. The paper lacks detailed discussion of computational overhead and training efficiency comparisons between native and compositional approaches, which is crucial for practical adoption.", "was_revised": true, "final_justification": "I appreciate the author's response, which truly addressed all my questions. I highly appreciate the author's 7B model in the appendix, which demonstrates the generalization of the scaling property. I believe this paper will bring new insights to the field, so I decided to increase my score.", "rebuttal": "Thank you for your valuable and detailed comments. We address your questions as follows.\n\n> **W1: The final result of the article only provides a 2B-sized model. It is better to conduct experiments on a larger model to verify the effectiveness of the scaling law.**\n\nA1: We also report the results of NaViL-9B in Appendix A. With a similar parameter size, our NaViL-9B outperforms all existing native MLLMs by a large margin on almost all benchmarks. Besides that, compared to the compositional baseline model InternVL-2.5-8B with a similar parameter size, NaViL-9B also achieves competitive performance.  Such results demonstrate that NaViL can be scaled up to larger parameter sizes and achieve consistent performance gains.\n\n> **W2: The paper lacks detailed discussion of computational overhead and training efficiency comparisons between native and compositional approaches, which is crucial for practical adoption.**\n\nA2: As described in L143-146 and L262-271, NaViL adopts a modality-specific MoE architecture, resulting in the same number of parameters and inference FLOPs as the compositional counterpart. The implementation details of the modality-specific MoE are provided in Appendix C. A comparison of total training tokens is shown in Table 1 in the Appendix. Our approach achieves comparable performance while using significantly fewer training tokens, resulting in improved training efficiency.\n\n> **Q1: The related work section should include more comprehensive discussion of visual scaling paradigms, particularly foundational work such as Zhai et al. (2022) \"Scaling Vision Transformers\" (CVPR), which established important principles for visual model scaling that are relevant to this work.**\n\nA3: Thank you for your suggestion. More discussion on the visual scaling paradigm will be included in the related work section of the final version. “Scaling Vision Transformers” (Zhai et al., 2022) is an important and valuable work related to our research, which will also be discussed in the final version."}, {"review_id": "TJqjcCdWqd", "reviewer": "Reviewer_KNVW", "summary": "* This paper presents a systematic study on how to best design and scale “native” Multimodal Large Language Models (MLLMs), which are trained end-to-end.  \n* The authors empirically investigate several key design choices under a “data constraint”: the LLM initialization, the use of *modality-specific* MoEs, and visual encoder architecture.  \n* The primary contribution is the derivation of a scaling law for these native MLLMs: the optimal visual encoder size is limited by the LLM's capacity and should scale log-proportionally with the LLM's size.   \n* Based on these principles, the authors propose NaViL, a model that achieves competitive performance on 14 benchmarks, often with significantly higher training efficiency than “compositional” counterparts.", "questions": "The core scientific contribution of this paper is strong, but several issues hold back its impact in the presentation. Addressing the following points would greatly improve the paper:\n\n1. **Architectural Diagram:** As noted in W1, the paper is difficult to understand without an architectural diagram. To address this, could you please add a figure that clearly illustrates the NaViL architecture? It would be very helpful to see the data flow from the trainable vision encoder through the connector to the LLM, and to visualize the differences between the MoE and non-MoE variants you tested.  \n2. **Clarification of Terminology:** To resolve the ambiguity mentioned in W2, could you more precisely define \"native\" vs “compositional” MLLMs?   \n   1. “Early-fusion” a la Chameleon had a more clear-cut definition. What is the defining characteristic of a “native” model? That the *vision model* is trained from scratch?  \n   2. What if the vision model were initialized with pretrained vision-model weights (just as the paper advocates for initializing the LLM component with pretrained LLM weights)—would this still be a *native* MLLM?  \n3. **Minor Presentation Issues:** Please address the minor issues listed in W3.", "limitations": "No.\n\nThese scaling laws are validated only at a small scale, though there are “final results” in the appendix up to 9B.\n\nIt is unrealistic to run these experiments, so perhaps a limitation should be listed that it is unknown if the observed scaling trends would hold at drastically larger scale (\\~30B, \\~70B, 100+B, etc).... ie, caveat the observed scaling laws by emphasizing they are observed at a small scale.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper's core strength is its systematic and well-designed empirical study in Section 3\\. The experiments are clean, the visualizations are clear and informative (e.g., Figs. 2–6), and the derived \"Observations\" provide valuable design principles for the community.  \n2. The main finding—that the optimal visual encoder size should scale with the LLM size—provides a clear, data-driven “recipe” that challenges the common practice of using fixed-size vision encoders in compositional MLLMs.  \n3. The resulting NaViL models demonstrate impressive performance, outperforming prior native MLLMs and competing with SOTA compositional models of comparable scale. The supplement highlights the model's training efficiency, achieving these results with a fraction of the training tokens used by other large models.\n\n**Weaknesses:**\n\n1. **Clarity / Self-Containedness:** There is a lack of clarity regarding the model's architecture. The paper is not well self-contained in this regard, and requires the reader to reference external work (e.g., Mono-InternVL) to understand crucial architectural details. A clear architectural diagram that clearly displays the exact model setup (and possibly contrasts it with the “compositional” counterpart) would be a significant improvement.  \n2. **Vague Terminology:** The terms “native” and “compositional” MLLMs are not clearly defined or well-contrasted with existing community terms like \"early-fusion\" (most standard) or \"monolithic\".  \n   1. A diagram could help clear this up additionally (see W1)  \n3. **Minor Writing and Presentation Issues:**  \n   1. The paper has several minor but frequent English writing errors (e.g., \"with Data Constraint,\" \"Besides,\") that reduce its overall polish. Sending it through a grammar/spell checker would help.  \n   2. Some mathematical notation is overloaded (e.g., the symbol $\\\\tau$ is used for two different concepts in §3.3.2 and §4.1), which could be easily rectified.  \n   3. Citation \\[24\\] referencing pretraining vision encoders on L20 seems like a mis-citation? This citation appears to be about weight interpolation", "was_revised": true, "final_justification": "all major concerns addressed. I think this is a good paper worthy of acceptance", "rebuttal": "Thank you for your valuable and detailed comments. We address your questions as follows.\n\n> **W1: Clarity / Self-Containedness: There is a lack of clarity regarding the model's architecture. The paper is not well self-contained in this regard, and requires the reader to reference external work (e.g., Mono-InternVL) to understand crucial architectural details. A clear architectural diagram that clearly displays the exact model setup (and possibly contrasts it with the “compositional” counterpart) would be a significant improvement.**\n\nA1: The model architecture is introduced in text form in the paper. L95-103 first defines the meta architecture of MLLM, consisting of a visual encoder (with an MLP connector), an LLM, and a MoE architecture injected into the LLM. L262-270 further describes the detailed architecture of NaViL-2B, including model parameter size, attention mechanism, positional embeddings, and more. Appendix C explains the detailed implementation of the modality-specific MoE adopted by our approach. \n\nDue to the policy of NeurIPS 2025, we were unable to provide the architectural diagram during the rebuttal process. However, to further clarify the model configuration, the following table lists the key hyperparameters of NaViL-2B. Both the table and the figure will be updated in the final version. The code for the model structure and inference will also be open-sourced.\n\n| Component | Hyper-Parameter | Value |\n|------|------|------|\n| **visual encoder** | # Params | 0.6B |\n| | depth | 24 |\n| | width | 1472 |\n| | MLP width | 5888 |\n| | attention heads | 23 |\n| **LLM (w/ MoE)** | # experts | 2 |\n| | # A-Params | 1.8B |\n| | depth | 24 |\n| | width | 2048 |\n| | MLP width | 8192 |\n| | attention heads | 16 |\n\n> **W2: Vague Terminology: The terms “native” and “compositional” MLLMs are not clearly defined or well-contrasted with existing community terms like \"early-fusion\" (most standard) or \"monolithic\".**\n>\n> **A diagram could help clear this up additionally (see W1)**\n\nA2: Please refer to A7. In addition, a figure for comparing “native” and “compositional” MLLMs will be included in the final version.\n\n> **W3.1: The paper has several minor but frequent English writing errors (e.g., \"with Data Constraint,\" \"Besides,\") that reduce its overall polish. Sending it through a grammar/spell checker would help.**\n\nA3: Thanks for the suggestion. The paper will be polished to eliminate such errors in the final version.\n\n> **W3.2: Some mathematical notation is overloaded (e.g., the symbol  is used for two different concepts in §3.3.2 and §4.1), which could be easily rectified.**\n\nA4: Thanks for the suggestion. Different notations will be used in the final version.\n\n> **W3.3: Citation [24] referencing pretraining vision encoders on L20 seems like a mis-citation? This citation appears to be about weight interpolation**\n\nA5: Thanks for the suggestion. However, citation [24] is not a mis-citation, as it refers to the repository page for the OpenCLIP software project on Zenodo (one of the pre-trained visual encoders widely adopted by MLLMs), not to weight interpolation.\n\n> **Q1: Architectural Diagram: As noted in W1, the paper is difficult to understand without an architectural diagram. To address this, could you please add a figure that clearly illustrates the NaViL architecture? It would be very helpful to see the data flow from the trainable vision encoder through the connector to the LLM, and to visualize the differences between the MoE and non-MoE variants you tested.**\n\nA6: Please refer to A1. An additional figure will be included in the final version, illustrating both the data flow and the differences between MoE and non-MoE variants.\n\n> **Q2: Clarification of Terminology: To resolve the ambiguity mentioned in W2, could you more precisely define \"native\" vs “compositional” MLLMs?**\n>\n> **1. “Early-fusion” a la Chameleon had a more clear-cut definition. What is the defining characteristic of a “native” model? That the vision model is trained from scratch?**\n>\n> **2. What if the vision model were initialized with pretrained vision-model weights (just as the paper advocates for initializing the LLM component with pretrained LLM weights)—would this still be a native MLLM?**\n\nA7: (1) The distinction between native MLLMs and compositional MLLMs primarily lies in whether the training process and training objectives can be unified. \n\nAs discussed in L69-71, compositional MLLMs have different components initialized by separate unimodal pre-training, where different training objectives and strategies are employed to train the LLM and visual encoder. For example, the visual encoder can be trained using an image-text contrastive learning objective (e.g., CLIP, SigLIP) or a self-supervised learning objective (e.g., DINOv2). The complexity of such training process increases the difficulty of scalability.\n\nAs shown in L79-83, the generally accepted definition of a native MLLM [1] is that it optimizes both image and text modalities end-to-end using a unified training objective (i.e., next-token prediction (NTP)). This avoids introducing additional bias and significantly simplify the scaling effort.\n\nIn contrast, **early-fusion (and monolithic) models** define MLLMs from a model architecture perspective, meaning that visual and text modality inputs are fused in the early stage and no modality-specific parameters are used. For example, Chameleon is both an early-fusion model and a native MLLM because it integrates visual and textual inputs from the outset and trains the entire model end-to-end using a unified training objective.\n\n(2) Initializing the vision encoder with a contrastive learnining pre-trained model introduces different training objectives and thus violates the definition of native MLLM in our paper.\n\n> **Q3: These scaling laws are validated only at a small scale, though there are “final results” in the appendix up to 9B.\nIt is unrealistic to run these experiments, so perhaps a limitation should be listed that it is unknown if the observed scaling trends would hold at drastically larger scale (~30B, ~70B, 100+B, etc).... ie, caveat the observed scaling laws by emphasizing they are observed at a small scale.**\n\nA8: Thanks for the suggestion. We will mention this as our limitation and potential future work in the final version.\n\n[1] Shukor, Mustafa, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind, and Alaaeldin El-Nouby. \"Scaling laws for native multimodal models.\" arXiv preprint arXiv:2504.07951. 2025."}]

Author rebuttal:

Meta-review: This paper presents a systematic study of end-to-end training for native Multimodal Large Language Models (MLLMs) under data constraints.
The authors analyze key design choices, including LLM initialization, modality-specific Mixture-of-Experts (MoE), and visual encoder architectures. A central contribution is the derivation of a scaling law, showing that visual encoders should scale log-proportionally to LLM size rather than remain fixed. Building on these findings, the authors introduce NaViL, a native MLLM that matches or surpasses compositional approaches, which separately pretrained vision encoders and language models before combining them. Extensive experiments validate these insights and demonstrate their impact on multimodal performance and training efficiency.  

All reviewers were positive and ask for clarification about implementation details, architecture types, scaling laws explanation, computational overhead and experiments.   The reviewers appreciated the authors' answers. Most of their concerns have been addressed. All remain positive or very positive.  The AC proposes to follow their proposal for acceptance.
Decision: Accept (poster)