Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training

## Abstract

Abstract The rapid advancement of Large Language Models (LLMs) has led to an influx of efforts to extend their capabilities to multimodal tasks. Among them, growing attention has been focused on monolithic Multimodal Large Language Models (MLLMs) that integrate visual encoding and language decoding into a single LLM.
Despite the structural simplicity and deployment-friendliness, training a monolithic MLLM with promising performance still remains challenging. In particular, the popular approaches adopt continuous pre-training to extend a pre-trained LLM to a monolithic MLLM, which suffers from catastrophic forgetting and leads to performance degeneration. In this paper, we aim to overcome this limitation from the perspective of delta tuning. Specifically, our core idea is to embed visual parameters into a pre-trained LLM, thereby incrementally learning visual knowledge from massive data via delta tuning, i.e., freezing the LLM when optimizing the visual parameters. Based on this principle, we present Mono-InternVL, a novel monolithic MLLM that seamlessly integrates a set of visual experts via a multimodal mixture-of-experts structure. Moreover, we propose an innovative pre-training strategy to maximize the visual capability of Mono-InternVL, namely Endogenous Visual Pre-training (EViP). In particular, EViP is designed as a progressive learning process for visual experts, which aims to fully exploit the visual knowledge from noisy data to high-quality data. To validate our approach, we conduct extensive experiments on 16 benchmarks. Experimental results not only validate the superior performance of Mono-InternVL compared to the state-of-the-art MLLM on 6 multimodal benchmarks, e.g., +113 points over InternVL-1.5 on OCRBench, but also confirm its better deployment efficiency, with first token latency reduced by up to 67%.

## 1 Introduction

In recent years, the rapid development of Large Language Models (LLMs) (OpenAI, ; Bai et al., ; Cai et al., ) has spurred increasing efforts to extend their multimodal capabilities (Chen et al., ; Liu et al., ). As shown in Fig. 1 (a), most existing Multimodal Large Language Models (MLLMs) adopt a modular architecture, where visual encoding and language decoding are processed separately. This is typically achieved by combining a pre-trained visual encoder like CLIP-ViT (Radford et al., ) with an LLM (Liu et al., ; Chen et al., ; Li et al., ). Recent research has also started exploring monolithic MLLMs (Bavishi et al., ; Diao et al., ; Chen et al., ), as shown in Fig. 1 (b), which integrate visual perception and multimodal understanding directly into a single LLM. Due to their simplicity and unity, monolithic MLLMs can be more easily deployed using existing LLM inference libraries (LMDeployContributors, ) and show superior efficiency than modular MLLMs (Diao et al., ; Chen et al., ).

Figure: Figure 1: Comparison of Mono-InternVL and existing MLLMs. Compared with modular MLLMs, Mono-InternVL embeds visual experts into the pre-trained LLM and integrates visual encoding and language decoding into a single LLM. Through endogenous visual pre-training, Mono-InternVL significantly pushes the performance boundaries of monolithic MLLMs.
Refer to caption: x1.png

Despite recent progress, training a monolithic MLLM with promising performance still remains challenging. In particular, monolithic MLLMs struggle to replicate the successful usage of pre-trained visual encoders in modular MLLMs for visual perception. Therefore, researchers often rely on additional pre-training to compensate for the shortcomings in visual perception in monolithic MLLMs (ChameleonTeam, ; Diao et al., ).
A straightforward approach is the native pre-training (ChameleonTeam, ), which pre-trains a monolithic MLLM from scratch using a mixture of text-only and multimodal data. However, it requires prohibitively high training costs and often suffers from challenges of unstable optimization (ChameleonTeam, ).
Another common solution is the continuous pre-training (Diao et al., ), which extends the pre-trained LLM to multimodality via additional visual pre-training. Benefiting from the knowledge in the pre-trained LLM, the cost of continuous pre-training becomes much more affordable. Nevertheless, due to the catastrophic forgetting issue (Zhai et al., ), the pre-trained language knowledge is inevitably undermined during continuous pre-training, thereby weakening the multimodal capabilities.

In this paper, we aim to address the forgetting issue of continuous pre-training through delta tuning (Ding et al., ).
In particular, we argue that such issue arises from the shared architecture for joint vision and language modeling, where optimizations for vision can negatively impact language capabilities.
Therefore, it is a natural thought to introduce an independent visual parameter set into the pre-trained LLM, which allows formulating visual pre-training with partial parameter tuning.
This can help retain the language knowledge by freezing the entire LLM during continuous pre-training, while improving visual learning.
This principle is also aligned with previous endeavors in modular MLLMs, *e.g.,* QwenVL (Bai et al., ) and InternVL-1.5 (Chen et al., ), where the visual parameters are placed outside the LLM.

Based on the above principle, we propose a novel monolithic MLLM, namely Mono-InternVL. As shown in Fig. 2, the visual parameters in Mono-InternVL are designed as a set of expert networks that are seamlessly integrated into the LLM via the mixture-of-experts mechanism. To better capture the visual knowledge, these experts are initialized from the Feed-Forward Networks (FFNs) in the pre-trained LLM. Based on this architecture, we present an innovative visual pre-training method called Endogenous Visual Pre-training (EViP). Specifically, EViP is formulated as a progressive learning process of three stages: 1) concept learning to grasp basic visual concepts, 2) semantic learning to capture high-level semantics, *e.g.,* world knowledge, and 3) alignment learning to align knowledge with downstream tasks. Benefiting from the architecture and the pre-training strategy, the visual scalability of Mono-InternVL is fully unleashed, where the downstream performance consistently improves as the scale of the pre-training data increases. After visual pre-training, Mono-InternVL accommodates complex multimodal tasks via supervised instruction tuning.

To validate our method, we develop Mono-InternVL using the pre-trained LLM InternLM2-1.8B (Cai et al., ), and conduct extensive experiments on 16 multimodal benchmarks. Experimental results not only demonstrate the significant performance improvements of Mono-InternVL against previous monolithic MLLMs, but also validate its superior efficiency compared to existing modular MLLMs. For instance, Mono-InternVL with 1.8 billion activated parameters can outperform existing monolithic MLLMs with 7 billion parameters by a significant margin, *e.g.,* +15.5% over EVE on average. Compared to the state-of-the-art modular MLLM InternVL-1.5 (Chen et al., ), Mono-InternVL shows superior performance on 6 multimodal benchmarks while reducing first token latency by 67%. In conclusion, our contributions can be summarized in threefold:

- •
We present Mono-InternVL, a novel monolithic MLLM that seamlessly integrates a set of visual experts via a multimodal mixture-of-experts architecture. This architecture can effectively extend the pre-trained LLM to a monolithic MLLM while retaining the pre-trained knowledge.
- •
We propose a novel visual pre-training approach for Mono-InternVL called endogenous visual pre-training (EViP). EViP adopts a progressive learning strategy to encourage visual experts of Mono-InternVL to continuously grasp visual knowledge from noisy data to high-quality data.
- •
Mono-InternVL demonstrates for the first time that the leading performance of MLLM no longer depends on the pre-trained visual encoder, thereby opening new avenues for designing future MLLMs. In particular, Mono-InternVL achieves the state-of-the-art (SoTA) results compared to existing MLLMs on 6 multimodal benchmarks.

## 2 Related Work

Modular multimodal large language models.
Recent progress in large language models (LLMs) has catalyzed the integration of vision and language modalities, giving rise to multimodal large language models (MLLMs). Both commercial entities, such as GPT-4V (Yang et al., ) and Gemini series (Team et al., ), and other open-source initiatives, *e.g.* BLIP series (Li et al., ; ; Dai et al., ), LLaVA series (Liu et al., ; ; ), InternVL (Chen et al., ; ), have pursued this integration, often linking LLMs (Touvron et al., ; ; Team, ; Cai et al., ) with large vision models (LVMs) (Radford et al., ; Dosovitskiy et al., ; Dehghani et al., ; Chen et al., ) via intermediate layers.
Specifically, there are lightweight MLLMs (i.e., $\leq$ 4B parameters), such as MobileVLM-V2 (Chu et al., ), Mini-Gemini (Li et al., ), MM1 (McKinzie et al., ), DeepSeek-VL (Lu et al., ), PaliGemma (Beyer et al., ) and MiniCPM-V (Yao et al., ).
However, such encoder-based vision-language models (modular MLLMs) encounter challenges like limitations in input processing due to pre-trained visual encoders, deployment inefficiencies, and complexities in balancing the capacities of LLMs and LVMs, as also pointed out by (Diao et al., ).

Monolithic multimodal large language models.
The issues related to modular MLLMs have steered research toward encoder-free architectures, also known as monolithic MLLMs, which can be summarized into two categories. The first category obtains continuous visual tokens through a lightweight structure before feeding into MLLMs. For instance, Fuyu-8B (Bavishi et al., ) processes images directly through a simple linear projection, adeptly handling high-resolution images without using a dedicated visual encoder. EVE-7B (Diao et al., ) prioritizes vision-language pre-alignment from an LLM-centric perspective and enhances image recognition through visual representation supervision. SOLO (Chen et al., ) introduces an open-source training recipe for developing monolithic MLLMs. In contrast, the second category introduces VQ tokenizer-based models to generate discrete visual tokens to support image generation tasks, with representative works such as Chameleon (ChameleonTeam, ), Show-o (Xie et al., ), Transfusion (Zhou et al., ), and Emu3 (Wang et al., ).

Multimodal mixture-of-experts.
VLMo (Bao et al., ) and BEiT-3 (Wang et al., ) employ a pool of modality experts to replace the feed-forward network in standard Transformer to capture modality-specific information by switching to different modality experts, and use the shared self-attention across modalities to align visual and linguistic information.
VL-MoE (Shen et al., ) introduces mixture-of-experts (MoE) (Yuksel et al., ) based on the above works for better training and deploying.
MoMa (Lin et al., ) also uses multimodal mixture-of-experts for pre-training of MLLMs (ChameleonTeam, ) and cooperates with sparse components, *e.g.* MoE and mixture-of-depths (MoD) (Raposo et al., ) to improve the efficiency of pre-training from scratch with trillions of mixed-modal tokens. Inspired by the above literature, we introduce multimodal mixture-of-experts (*i.e.,* a visual expert and a language expert) for pre-training monolithic MLLMs and use a novel progressive learning strategy, named endogenous visual pre-training (EViP), to address the unique challenges faced by monolithic MLLMs.

Figure: Figure 2: Monolithic architecture of Mono-InternVL. Mono-InternVL is designed as a multimodal MoE structure, where visual and textual tokens are processed by the corresponding experts. Such a design greatly facilitates the visual pre-training while retaining the model efficiency.
Refer to caption: x2.png

## 3 Mono-InternVL

### 3.1 The Monolithic Architecture

As shown in Fig. 2, we first outline the architecture of Mono-InternVL, which consists of tokenizers and a multimodal mixture-of-experts structure.

Visual and textual embeddings. Compared to modular MLLMs, Mono-InternVL directly patchifies images to input visual sequences using a lightweight module. Specifically, given the input image $I\in\mathbb{R}^{H\times W\times 3}$, the input visual embedding $x_{v}\in\mathbb{R}^{(h\times w)\times d}$ is obtained by

$$ $\displaystyle x_{v}=\text{MLP}(\text{PatchEmbed}(I)+\text{PE}).$ (1) $$

Here, $\text{PatchEmbed}(\cdot)$ denotes a patch embedding layer with a stride of 28, meaning each visual token represents a $28\times 28$ image patch. $\text{PE}\in\mathbb{R}^{(h\times w)\times d}$ is the learnable positional embedding as similar to InternVL-1.5 (Chen et al., ). Besides, we also add an additional thumbnail to provide global visual information.
After that, an MLP layer, *i.e.,* $\text{MLP}(\cdot)$, is used to project visual patches into the $d$-dimensional embedding space of the LLM. This simple visual tokenizer allows Mono-InternVL to process arbitrary-resolution images with up to 8 millions of pixels, *i.e.,* $10,240$ image patches, which can cover most high-resolution scenarios.

In Mono-InternVL, the textual tokenizer remains the same as the original one in the LLM. In particular, given the input text $T\in\mathbb{Z}^{n}$, we obtain textual embedding $x_{t}\in\mathbb{R}^{n\times d}$ by

$$ $\displaystyle x_{t}=\text{Tokenizer}(T).$ (2) $$

Afterwards, the multimodal embedding is constructed as the concatenation of visual and textual embeddings, denoted as $x_{m}\in\mathbb{R}^{n^{\prime}\times d}$.

Multimodal mixture-of-experts structure.
The key principle of Mono-InternVL is to embed visual experts into a pre-trained LLM. In this case, Mono-InternVL can not only facilitate the visual pre-training with the pre-trained LLM knowledge, but also significantly mitigates the catastrophic forgetting issue during pre-training.
Specifically, given the multimodal input $x_{m}\in\mathbb{R}^{n^{\prime}\times d}$, a decoder-only LLM with a set of visual experts is used to generate the textual tokens step by step, which can be formulated by

$$ $\displaystyle p_{s}=\mathcal{F_{\text{llm}}}(y_{s}|x_{m},y_{0:s-1};\theta, \theta_{v}).$ (3) $$

Here, $y\in\mathbb{R}^{S}$ and $S$ denote the word length and its length, respectively. $p_{s}\in\mathbb{R}^{m}$ is the next-token probability and $m$ is the size of word vocabulary. $\mathcal{F}_{\text{llm}}$ and $\theta$ denote the LLM and its pre-trained parameters, respectively. $\theta_{v}$ is the parameters of patch embedding layer and visual experts.

As shown in Fig. 2, $\mathcal{F}_{\text{llm}}$ is designed as a multimodal mixture-of-experts structure. In particular, we adopt the static routing strategy that assigns visual and textual experts to the corresponding tokens. Therefore, the l-th LLM layer can be defined by

$$ $\displaystyle x_{m}^{l^{\prime}}$ $\displaystyle=x_{m}^{l-1}+\text{MHA}(\text{RMSNorm}(x_{m}^{l-1})),$ (4) $\displaystyle x_{m}^{l}$ $\displaystyle=x_{m}^{l^{\prime}}+\text{MMoE}(\text{RMSNorm}(x_{m}^{l^{\prime}} )).$ $$

Here, $\text{MHA}(\cdot)$ and $\text{RMSNorm}(\cdot)$ denote the multi-head attention (Vaswani et al., ) and the layer normalization (Zhang & Sennrich, ), respectively. $\text{MMoE}(\cdot)$ is the proposed multimodal mixture-of-experts, formulated by

$$ $\displaystyle\text{MMoE}(x)=\begin{cases}\text{FFN}_{v}(x)\quad\text{if }x\in x _{v},\\ \text{FFN}_{t}(x)\quad\text{if }x\in x_{t}.\end{cases}$ (5) $$

Here, $x\in\mathbb{R}^{d}$ is the element of $x_{m}$. $\text{FFN}_{v}$ and $\text{FFN}_{t}$ denote the visual and textual experts, respectively. In practice, $\text{FFN}_{v}$ is initialized from the $\text{FFN}_{t}$ to leverage the pre-trained knowledge.

As defined in Eq. 4 and 5, the MMoE structure has two distinct advantages over the existing monolithic MLLM. Firstly, the visual learning of Mono-InternVL can greatly benefit from the pre-trained language knowledge, while the language ability can still be preserved by freezing $\text{FFN}_{t}$. Secondly, the MMoE structure significantly enhances the model’s capacity for vision-and-language modeling, while the additional inference cost is almost negligible due to the MoE mechanism.

Figure: Figure 3: The training recipe of Mono-InternVL. In the first stage, Mono-Internvl is progressively pre-trained on massive data via three sub-stages (S1.1, S1.2, S1.3), where most parameters of LLM are frozen to preserve the pre-trained knowledge. In the second stage (S2), the entire model is optimized to accommodate various instructions.
Refer to caption: x3.png

### 3.2 Endogenous Visual Pre-training

**Table 1: Summary of datasets used in the endogenous visual pre-training. In S1.2, caption for each image is synthetically produced by the pre-trained InternVL-8B (Chen et al., ).**
| Stage | #Samples | Datasets |
| --- | --- | --- |
| S1.1 | 922M | Laion-EN (en) (Schuhmann et al., ), COYO (en) (Byeon et al., ) |
| S1.2 | 258M | Laion-EN (en) (Schuhmann et al., ), COYO (en) (Byeon et al., ), SAM (en) (Kirillov et al., ) |
|  |  | Captioning: Laion-EN (en) (Schuhmann et al., ), Laion-ZH (zh) (Schuhmann et al., ), |
|  |  | COYO (zh) (Byeon et al., ), GRIT (zh) (Peng et al., ), COCO (en) (Chen et al., ), |
|  |  | TextCaps (en) (Sidorov et al., ) |
|  |  | Detection: Objects365 (en&zh) (Shao et al., ), GRIT (en&zh) (Peng et al., ), |
|  |  | All-Seeing (en&zh) (Wang et al., ) |
|  |  | OCR (large): Wukong-OCR (zh) (Gu et al., ), LaionCOCO-OCR (en) (Schuhmann et al., ), |
|  |  | Common Crawl PDF (en&zh) |
|  |  | OCR (small): MMC-Inst (en) (Liu et al., ), LSVT (zh) (Sun et al., ), |
|  |  | ST-VQA (en) (Biten et al., ), RCTW-17 (zh) (Shi et al., ), ReCTs (zh) (Zhang et al., ), |
|  |  | ArT (en&zh) (Chng et al., ), SynthDoG (en&zh) (Kim et al., ), COCO-Text (en) (Veit et al., ), |
|  |  | ChartQA (en) (Masry et al., ), CTW (zh) (Yuan et al., ), DocVQA (en) (Clark & Gardner, ), |
| S1.3 | 143M | TextOCR (en) (Singh et al., ), PlotQA (en) (Methani et al., ), InfoVQA (en) (Mathew et al., ) |

Endogenous Visual Pre-training (EViP) aims to maximize the benefits of Mono-InternVL from visual experts through pre-training on massive noisy and synthetic data. Unlike existing methods (Diao et al., ; ChameleonTeam, ), we formulate EViP from the perspective of delta tuning (Ding et al., ), in which most of the LLM parameters are frozen to preserve its pre-trained knowledge. Therefore, the objective of EViP can be defined by

$$ $\displaystyle\arg\min_{\Delta\theta}\mathcal{L}(\mathcal{F}_{\text{llm}}(x_{m} ;\theta,\theta_{v}),\hat{y}),$ (6) $$

where $\mathcal{L}(\cdot)$ and $\hat{y}$ denote the auto-regressive loss and the ground-truth, respectively. As shown in Fig. 3, $\Delta\theta$ denotes parameters of patch embedding and visual experts in the concept and semantic learning, *i.e.,* $\theta_{v}$, while in the alignment learning stage $\Delta\theta$ also includes the parameters of multi-head attentions.
Based on Eq. 6, EViP is designed as a progressive learning process. As shown in Fig. 3 and Tab. 1, EViP consists of three sub-stages, namely concept learning (S1.1), semantic learning (S1.2) and alignment learning (S1.3). For different sub-stages, we use carefully partitioned data to achieve coarse-to-fine visual learning.

Concept learning. Concept learning aims to encourage the model to learn fundamental visual concepts, such as object categories or basic shapes. Therefore, we first pre-train Mono-InternVL with about 922 million noisy samples, which are sampled from Laion-2B (Schuhmann et al., ) and Coyo-700M (Byeon et al., ). In this sub-stage, Mono-InternVL employs a simple prompt to perform generative learning, *i.e.,* “provide a one-sentence caption for the image”. Meanwhile, we constrain the maximum number of image patches of the visual tokenizer to 1,280 for training efficiency. To ensure that the foundational language capabilities are preserved while enabling visual specialization, the entire LLM is kept frozen during concept learning, and only the patch embedding and visual experts are optimized.

Semantic learning. After concept learning, Mono-InternVL is able to understand basic concepts in the image, but organizing this information to produce reasonable descriptions remains challenging. To achieve a higher-level visual understanding, we utilize the pre-trained InternVL-8B (Chen et al., ) to produce short captions for 258 million images. Compared to the original noisy captions, synthetic captions typically depict complex visual knowledge, such as relationship and world knowledge, etc., while containing less noisy information unrelated to the image, *e.g.,* time of shooting, and the photographer. In this sub-stage, we adopt the same optimization strategy as concept learning, except that the maximum number of image patches is increased to 1,792.

Alignment learning. To meet the visual requirements of downstream tasks, we further perform alignment learning on Mono-InternVL. As shown in Tab. 1, our alignment data is sampled from the pre-training data of InternVL-1.5 (Chen et al., ), including 143 million samples of image captioning, detection and optical character recognition (OCR). In particular, captioning data, detection data and OCR data account for about 53.9%, 5.2% and 40.9% of the total, respectively. In this sub-stage, we utilize the task-specific prompts from InternVL-1.5 for the generative learning, and increase the maximum number of image patches to 3,328. Compared to previous sub-stages, the multi-head attention layers are additionally optimized to achieve better vision-language alignment.

### 3.3 Instruction Tuning

In this stage, we follow InternVL (Chen et al., ) to use around 5 million bilingual instructions for supervised learning, covering various tasks such as visual question answering, multimodal dialogue, mathematics, knowledge, etc. In addition to this, we further include additional instruction data for video understanding and handwritten text recognition. In this stage, the entire models are optimized and the maximum number of image patches is increased to 6,400 to accommodate high-resolution images. Details of instruction data can be found in Appendix §A.1.

## 4 Experiments

**Table 2: Comparison with existing MLLMs on general MLLM benchmarks. “#A-Param” denotes the number of activated parameters. For MME, we sum the perception and cognition scores. Average scores are computed by normalizing each metric to a range between 0 and 100.**
| Model | #A-Param | MMB | MMVet | MMMU | MME | MathVista | SEED-I | OCRBench | HallB | CCB | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| *Modular MLLMs:* |  |  |  |  |  |  |  |  |  |  |  |
| MobileVLM-V2-1.7B | 1.7B | 57.7 | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ |
| MobileVLM-V2-3B | 3.0B | 63.2 | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ |
| Mini-Gemini-2B | 3.5B | 59.8 | 31.1 | 31.7 | 1653 | 29.4 | $-$ | $-$ | $-$ | $-$ | $-$ |
| MM1-3B-MoE-Chat | 3.5B | 70.8 | 42.2 | 38.6 | 1772 | 32.6 | 69.4 | $-$ | $-$ | $-$ | $-$ |
| DeepSeek-VL-1.3B | 2.0B | 64.6 | 34.8 | 32.2 | 1532 | 31.1 | 66.7 | 409 | 27.6 | 37.6 | 43.4 |
| PaliGemma-3B | 2.9B | 71.0 | 33.1 | 34.9 | 1686 | 28.7 | 69.6 | 614 | 32.2 | 29.6 | 46.7 |
| MiniCPM-V | 2.8B | 64.1 | 31.1 | 38.3 | 1650 | 28.9 | 65.6 | 366 | 36.2 | 41.4 | 44.6 |
| MiniCPM-V-2 | 2.8B | 69.1 | 41.0 | 38.2 | 1809 | 38.7 | 67.1 | 605 | 36.1 | 45.3 | 51.2 |
| InternVL-1.5-2B | 2.2B | 70.9 | 39.3 | 34.6 | 1902 | 41.1 | 69.8 | 654 | 37.5 | 63.5 | 54.4 |
| *Monolithic MLLMs:* |  |  |  |  |  |  |  |  |  |  |  |
| Fuyu-8B (HD) | 8B | 10.7 | 21.4 | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ |
| SOLO | 7B | $-$ | $-$ | $-$ | 1260 | 34.4 | 64.4 | $-$ | $-$ | $-$ | $-$ |
| Chameleon-7B(^1^11Chameleon-7B frequently rejects to perform the task with a response of “I can’t help you with this”, thus resulting in poor performance.) | 7B | 31.1 | 8.3 | 25.4 | 170 | 22.3 | 30.6 | 7 | 17.1 | 3.5 | 16.1 |
| EVE-7B | 7B | 49.5 | 25.6 | 32.3 | 1483 | 25.2 | 61.3 | 327 | 21.1 | 12.4 | 34.8 |
| EVE-7B (HD) | 7B | 52.3 | 25.7 | 32.6 | 1628 | 34.2 | 64.6 | 398 | 26.4 | 16.3 | 38.9 |
| Emu3 | 8B | 58.5 | 37.2 | 31.6 | $-$ | $-$ | 68.2 | 687 | $-$ | $-$ | $-$ |
| Mono-InternVL-2B | 1.8B | 65.5 | 40.1 | 33.7 | 1875 | 45.7 | 67.4 | 767 | 34.8 | 66.3 | 55.2 |

### 4.1 Evaluation Benchmarks

We evaluate Mono-InternVL and existing MLLMs on 16 comprehensive multimodal benchmarks.
Specifically, general MLLM benchmarks encompass MMBench-EN test (Liu et al., ), MMVet (Yu et al., ), MMMU val (Yue et al., ), MME (Fu et al., ), MathVista testmini (Lu et al., ), SEED Image (Ge et al., ), OCRBench (Liu et al., ), HallusionBench (Guan et al., ), and CCBench dev (Liu et al., ).
Visual question answering benchmarks include TextVQA val (Singh et al., ), SQA test (Lu et al., ), GQA test-dev (Hudson & Manning, ), DocVQA test (Clark & Gardner, ), AI2D test (Kembhavi et al., ), ChartQA test (Masry et al., ), and InfographicVQA test (Mathew et al., ).
The evaluation metrics follow the existing methods (Chen et al., ; Diao et al., ).
Part of the results of Chameleon and EVE are evaluated with VLMEvalKit (Duan et al., ) or from the OpenCompass leaderboard (Contributors, ).

**Table 3: Comparison with existing MLLMs on visual question answering benchmarks.**
| Model | #A-Param | TextVQA | SQA-I | GQA | DocVQA | AI2D | ChartQA | InfoVQA | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| *Modular MLLMs:* |  |  |  |  |  |  |  |  |  |
| MobileVLM-V2-1.7B | 1.7B | 52.1 | 66.7 | 59.3 | $-$ | $-$ | $-$ | $-$ | $-$ |
| MobileVLM-V2-3B | 3.0B | 57.5 | 70.0 | 66.1 | $-$ | $-$ | $-$ | $-$ | $-$ |
| Mini-Gemini-2B | 3.5B | 56.2 | $-$ | $-$ | 34.2 | $-$ | $-$ | $-$ | $-$ |
| MM1-3B-MoE-Chat | 3.5B | 72.9 | 76.1 | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ |
| DeepSeek-VL-1.3B | 2.0B | 57.8 | $-$ | $-$ | $-$ | 51.5 | $-$ | $-$ | $-$ |
| PaliGemma-3B | 2.9B | 68.1 | $-$ | $-$ | $-$ | 68.3 | $-$ | $-$ | $-$ |
| MiniCPM-V | 2.8B | 60.6 | $-$ | $-$ | 38.2 | 56.3 | $-$ | $-$ | $-$ |
| MiniCPM-V-2 | 2.8B | 74.1 | $-$ | $-$ | 71.9 | 62.9 | $-$ | $-$ | $-$ |
| InternVL-1.5-2B | 2.2B | 70.5 | 84.9 | 61.6 | 85.0 | 69.8 | 74.8 | 55.4 | 71.7 |
| *Monolithic MLLMs:* |  |  |  |  |  |  |  |  |  |
| Fuyu-8B (HD) | 8B | $-$ | $-$ | $-$ | $-$ | 64.5 | $-$ | $-$ | $-$ |
| SOLO | 7B | $-$ | 73.3 | $-$ | $-$ | 61.4 | $-$ | $-$ | $-$ |
| Chameleon-7B(^1^1footnotemark: 1) | 7B | 4.8 | 47.2 | $-$ | 1.5 | 46.0 | 2.9 | 5.0 | 17.9 |
| EVE-7B | 7B | 51.9 | 63.0 | 60.8 | 22.0 | 48.5 | 19.5 | 20.0 | 40.8 |
| EVE-7B (HD) | 7B | 56.8 | 64.9 | 62.6 | 53.0 | 61.0 | 59.1 | 25.0 | 54.6 |
| Emu3 | 8B | 64.7 | 89.2 | 60.3 | 76.3 | 70.0 | 68.6 | 43.8 | 67.6 |
| Mono-InternVL-2B | 1.8B | 72.6 | 93.6 | 59.5 | 80.0 | 68.6 | 73.7 | 43.0 | 70.1 |

**Table 4: Zero-shot pre-training performance of Mono-InternVL and existing MLLMs. “S1.2” and “S1.3” denote pre-training stages of semantic learning and alignment learning, respectively. Images of COCO have been seen in Mono-InternVL-S1.3, so we mark its performance in gray.**
| Model | #A-Param | Data | Shots | COCO Caps | Flickr30k | NoCaps | VQAv2 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Flamingo | 3B | >2.1B | 0 | 73.0 | $-$ | $-$ | 49.2 |
| MM1 | 3.5B | >2.3B | 0 | 73.5 | $-$ | 55.6 | 46.2 |
| Chameleon | 34B | >1.4B | 2 | 120.2 | 74.7 | $-$ | 66.0 |
| Mono-InternVL-S1.2 | 1.8B | 0.9B | 0 | 87.3 | 72.7 | 54.1 | $-$ |
| Mono-InternVL-S1.3 | 1.8B | 1.1B | 0 | 135.6 | 77.3 | 116.5 | 71.1 |

### 4.2 Implementation Details

Mono-InternVL is implemented based on InternLM2-1.8B (Cai et al., ) with newly added visual tokenizer and visual experts. The visual experts are initialized from pre-trained MLPs in InternLM2-1.8B to leverage existing learned representations for improved visual feature extraction, which accounts for 1.2 billion parameters. We adopt a similar dynamic high-resolution strategy from InternVL-1.5 (Chen et al., ) to align an optimal resolution for input image, which is then patchfied to visual tokens. The remaining configurations are kept identical to InternLM2-1.8B. The endogenous visual pre-training and instruction tuning take approximately 16 days (646k iterations) and 1 day (14k iterations) on 256 NVIDIA A100 GPUs, respectively. More detailed training configurations are given in Appendix §A.2.

### 4.3 Quantitative Experiments

**Table 5: Ablation of different strategies for visual pre-training. All models are pre-trained on 61 million image-text pairs from Laion-2B (Schuhmann et al., ) and fine-tuned on instruction data from LLaVA-665k. (Liu et al., ). “Full” and “Delta” denote full tuning and delta tuning, respectively. “T-Param” refers to trainable parameters.**
| Model | #T-Param | Strategy | MME-P | DocVQA | InfoVQA | SQA-I | GQA | ChartQA | AI2D |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| InternLM2 | 1.8B | Full | 753 | 16.1 | 11.6 | 36.7 | 51.4 | 10.8 | 27.7 |
| + V-Expert | 3.0B | Full | 948 | 18.6 | 11.9 | 37.7 | 53.0 | 11.1 | 26.6 |
| + V-Expert | 1.2B | Delta | 995 | 18.9 | 14.6 | 56.5 | 53.4 | 13.5 | 42.7 |

Comparison with existing MLLMs. In Tab. 2 and 3, we compare Mono-InternVL and existing MLLMs on 16 multimodal benchmarks. From Tab. 2, the first observation is that most modular MLLMs outperform existing monolithic MLLMs by significant margins. For example, the average performance of InternVL-1.5-2B (Chen et al., ) on 9 MLLM benchmarks greatly exceeds the SoTA monolithic MLLM, *i.e.,* + 15.5% over EVE-7B (HD) (Diao et al., ). These results strongly suggest the challenges in existing monolithic MLLMs. In contrast, Mono-InternVL-2B with a slightly smaller model size can even outperform the SoTA modular MLLM, *i.e.,* + 0.8% against InternVL-1.5-2B on average. Notably, Mono-InternVL-2B demonstrates distinct advantages on MathVista and OCRBench, suggesting its seamless text recognition and reasoning capabilities. Moreover, superior bilingual ability of Mono-InternVL-2B is also validated on CCBench, which contains a large amount of questions related to Chinese culture. Compared to existing monolithic MLLMs, performance gains of Mono-InternVL are more distinct, *e.g.,* +15.4% over EVE-7B (HD) (Diao et al., ) on MMVet and +7.9% over Emu3 (Wang et al., ) on TextVQA, while using a much smaller parameter scale. Similar advantages of Mono-InternVL can also been witnessed in Tab. 3, *e.g.,* +2.1% on TextVQA. Nevertheless, we also observe that Mono-InternVL is still inferior to InternVL-1.5 on high-resolution benchmarks, *e.g.,* -12.4% on InfoVQA. This is because specific optimizations for high-resolution encoding are not the focus of the paper, *e.g.,* positional embedding and high-resolution training data, which we plan to explore in future research. Overall, these comparisons significantly validate the architecture and training strategy of Mono-InternVL.

In Tab. 4, we further compare the pre-training performance of Mono-InternVL and existing MLLMs. From this table, we can observe that with concept and semantic learning, Mono-InternVL-S1.2 already exceeds existing modular MLLMs, *e.g.,* +13.8 CIDEr over MM1 (McKinzie et al., ) on COCO Captions, demonstrating that Mono-InternVL-S1.2 is effective in capturing fundamental multimodal relationships. It is worth noting that pre-training in Mono-InternVL-S1.2 only consumes about 0.9B image-text pairs, but the cost in MM1 and Flamingo is much more expensive, *e.g.,* more than 2B data. Compared to monolithic MLLMs, Mono-InternVL also demonstrates superior performance. For instance, even though Chameleon has a much larger model size, it is still inferior to Mono-InternVL-S1.3 by -2.6 CIDEr on Flickr30k (Young et al., ). These results further confirm the effectiveness of EViP for Mono-InternVL.

Figure: Figure 4: Downstream performance breakdown with the increase of pre-training data size across three sub-stages: (S1.1) Concept learning; (S1.2) Semantic learning; (S1.3) Alignment learning. For each data point, we fine-tune the corresponding pre-trained model on the instruction data of LLaVA-665k and obtain the downstream performance. Results of captioning and VQA are averaged from 3 and 8 tasks, respectively. See Appendix §A.3 for complete results.
Refer to caption: x4.png

Ablation studies. To validate the design of Mono-InternVL, we conduct extensive ablation studies in Tab. 5 and Fig. 4. Specifically, Tab. 5 compares different strategies for visual pre-training. The first row is the common strategy used in existing monolithic MLLMs, *i.e.,* full tuning of the LLM, which yields the worst downstream performance in the table. After employing visual experts (the second row), such a full-tuning strategy becomes more effective, *e.g.,* +1.6% on GQA. These comparisons well validate that the shared architecture for joint vision and language modeling is sub-optimal in monolithic MLLMs. Besides, we also observe that the delta tuning strategy greatly benefits the visual pre-training, providing +18.8% and 16.1% gains on SQA-I and AI2D, respectively. Compared to full tuning, delta tuning can effectively preserve the knowledge of the pre-trained LLM, which is also crucial for maintaining the language understanding capabilities required for effective multimodal interactions.

Fig. 4 further demonstrates the relationship between downstream performance and pre-training data size. From it we can observe that performance of Mono-InternVL will gradually reach an upper bound in the concept learning. Through additional semantic learning and alignment learning, capabilities of Mono-InternVL consistently boost as the data size increases. It is important to note that that the alignment learning plays a significant role for VQA and MME, which can provide sufficient task-related knowledge, *e.g.,* OCR knowledge. These results not only demonstrate the data scalability of Mono-InternVL, but also confirm the advantages of coarse-to-fine learning in EViP.

Comparison of inference efficiency. In Tab. 6, we compare the inference speed of Mono-InternVL and InternVL-1.5 using the popular deployment library LMDeploy (LMDeployContributors, ). From this table, we can find that due to the elimination of visual encoder, Mono-InternVL demonstrates superior efficiency under different number of input tokens. In particular, the first-token time is greatly reduced in Mono-InternVL, *e.g.,* up to -67% against InternVL-1.5. Benefiting from this, the overall throughput is correspondingly increased by around 31%. These results greatly validate the efficiency of Mono-InternVL. We note that this is only an initial attempt, and using Turbomind backend or further optimization techniques may yield better performance.

**Table 6: Inference speed comparison of Mono-InternVL and InternVL-1.5. Models are deployed on an NVIDIA A100 using LMDeploy with Pytorch backend (LMDeployContributors, ), with a concurrency of 16 and the number of output tokens fixed as 120. “TTFT” and “TPS” denotes the time to first token in seconds and throughput in tokens per second, respectively.**
| Model | #Image | #Text | #Total Input | TTFT | TPS |
| --- | --- | --- | --- | --- | --- |
| Tokens | Tokens | Tokens |  |  |  |
| InternVL-1.5-2B | 768 | 256 | 1024 | 0.24 | 382 |
| Mono-InternVL-2B | 768 | 256 | 1024 | 0.09 (-63%) | 436 (+14%) |
| InternVL-1.5-2B | 1792 | 256 | 2048 | 0.45 | 183 |
| Mono-InternVL-2B | 1792 | 256 | 2048 | 0.15 (-67%) | 232 (+27%) |
| InternVL-1.5-2B | 3840 | 256 | 4096 | 1.93 | 52 |
| Mono-InternVL-2B | 3840 | 256 | 4096 | 0.79 (-59%) | 68 (+31%) |

Figure: Figure 5: Visualization of attention maps in Mono-InternVL. The first blue segment, green segment and the second green segment in the axes represent the system prompt tokens (text), image tokens (visual) and user prompt tokens (text), respectively. The numbers on the left side of attention maps indicate the number of tokens.
Refer to caption: x5.png

### 4.4 Qualitative Experiments

To gain in-depth insights into Mono-InternVL, we visualize its attention maps of different layers in Fig. 5. From this figure, we can draw two noteworthy findings. Firstly, despite the global connectivity of the Transformer architecture, locality still exists in the visual encoding of shallow layers. As shown in Fig. 5, in the first layer, visual tokens only interact with their nearby content, yielding patterns that closely resemble those produced by convolutional neural networks (He et al., ). Secondly, modalities are barely interactive at shallow layers but gradually fused as the layers deepen. As illustrated in Fig. 5, the attention weights between visual and textual tokens are extremely small in the first layer and become larger in deeper layers. We believe these examples will provide useful hints for the design of monolithic MLLMs. More examples are given in Appendix §A.4.

## 5 Conclusion

In this paper, we propose Mono-InternVL, a monolithic multimodal large language model (MLLM) that integrates visual encoding and textual decoding into a single LLM. In Mono-InternVL, a set of visual experts is embedded into the pre-trained LLM via a mixture-of-experts mechanism. By freezing the LLM, Mono-InternVL ensures that visual capabilities are optimized without compromising the pre-trained language knowledge. Based on this structure, an innovative Endogenous Visual Pre-training (EViP) is introduced to realize coarse-to-fine visual learning. Extensive experiments demonstrate the effectiveness and efficiency of Mono-InternVL compared to existing MLLMs. Our work greatly pushes the boundaries of monolithic MLLMs, providing new possibilities for the development of MLLMs.

## Appendix A Appendix

### A.1 More Dataset Details

**Table 7: Summary of datasets used in instruction tuning.**
| Task | Dataset |
| --- | --- |
| Captioning | TextCaps (en) (Sidorov et al., ), ShareGPT-4o (en&zh) (Chen et al., ) |
|  | VQAv2 (en) (Goyal et al., ), GQA (en) (Hudson & Manning, ), OKVQA (en) (Marino et al., ), |
| General QA | VSR (en) (Liu et al., ), VisualDialog (en) (Das et al., ) |
| Science | AI2D (en) (Kembhavi et al., ), ScienceQA (en) (Lu et al., ), TQA (en) (Kembhavi et al., ) |
|  | ChartQA (en) (Masry et al., ), MMC-Inst (en) (Liu et al., ), DVQA (en) (Kafle et al., ), |
| Chart | PlotQA (en) (Methani et al., ), LRV-Instruction (en) (Liu et al., ) |
|  | GeoQA+ (en) (Cao & Xiao, ), TabMWP (en) (Lu et al., ), MathQA (en) (Yu et al., ), |
| Mathematics | CLEVR-Math/Super (en) (Lindström & Abraham, ; Li et al., ), Geometry3K (en) (Lu et al., ) |
|  | KVQA (en) (Shah et al., ), A-OKVQA (en) (Schwenk et al., ), ViQuAE (en) (Lerner et al., ), |
| Knowledge | Wikipedia (en&zh) (He et al., ) |
|  | OCRVQA (en) (Mishra et al., ), InfoVQA (en) (Mathew et al., ), TextVQA (en) (Singh et al., ), |
|  | ArT (en&zh) (Chng et al., ), COCO-Text (en) (Veit et al., ), CTW (zh) (Yuan et al., ), |
|  | LSVT (zh) (Sun et al., ), RCTW-17 (zh) (Shi et al., ), ReCTs (zh) (Zhang et al., ), |
| OCR | SynthDoG (en&zh) (Kim et al., ), ST-VQA (en) (Biten et al., ) |
| Document | DocVQA (en) (Clark & Gardner, ), Common Crawl PDF (en&zh) |
| Grounding | RefCOCO/+/g (en) (Yu et al., ; Mao et al., ), Visual Genome (en) (Krishna et al., ) |
|  | LLaVA-150K (en&zh) (Liu et al., ), LVIS-Instruct4V (en) (Wang et al., ), |
|  | ALLaVA (en&zh) (Chen et al., ), Laion-GPT4V (en) (LAION, ), |
| Conversation | ShareGPT (en&zh) (Zheng et al., ), SVIT (en&zh) (Zhao et al., ) |
|  | OpenHermes2.5 (en) (Teknium, ), Alpaca-GPT4 (en) (Taori et al., ), COIG-CQIA (zh) (Bai et al., ), |
| Text-only | ShareGPT (en&zh) (Zheng et al., ) |
|  | EgoTaskQA (en) (Jia et al., ), Mementos (en) (Wang et al., ), STAR (en) (Wu et al., ), |
|  | NTU RGB+D (en) (Shahroudy et al., ), VideoChat2IT (en&zh) (Li et al., ), |
| Video | LSMDC-QA (en) (Rohrbach et al., ), ShareGPT-4o (en&zh) (Chen et al., ) |
| Handwritten | SROIE (en) (Huang et al., ), FUNSD (en) (Guillaume Jaume, ), POIE (en) (Kuang et al., ) |

The datasets used in the instruction fine-tuning stage are listed in Tab. 7.

### A.2 More Training Details

**Table 8: Hyper-parameters used in the pre-training and instruction tuning of Mono-InternVL.**
| Configuration | Concept | Semantic | Alignment | Instruction |
| --- | --- | --- | --- | --- |
| Learning (S1.1) | Learning (S1.2) | Learning (S1.3) | Tuning (S2) |  |
| Maximum numer of patches | $1,280$ | $1,792$ | $3,328$ | $6,400$ |
| LLM sequence length | $1,425$ | $1,925$ | $4,096$ | $8,192$ |
| Use thumbnail | ✓ |  |  |  |
| Optimizer | AdamW |  |  |  |
| Optimizer hyperparameters | $\beta_{1}=0.9,\beta_{2}=0.999,eps=1e^{-8}$ |  |  |  |
| Peak learning rate | $1e^{-4}$ | $1e^{-4}$ | $5e^{-5}$ | $2e^{-5}$ |
| Learning rate schedule | constant with warm-up | constant with warm-up | cosine decay | cosine decay |
| Drop path rate | $0.1$ |  |  |  |
| Weight decay | $0.01$ |  |  |  |
| Training steps | $450$k | $126$k | $70$k | $14$k |
| Warm-up steps | $100$ | $100$ | $100$ | $420$ |
| Global batch size | $2,048$ | $2,048$ | $2,048$ | $1,024$ |
| Gradient accumulation | $1$ | $1$ | $1$ | $4$ |
| Numerical precision | $\mathtt{bfloat16}$ |  |  |  |

Hyper-parameters used in the training stages are listed in Tab. 8.

### A.3 More Ablation Studies

In Fig. 6, we provide the full results of Fig. 4 with all the downstream tasks.

Figure: Figure 6: Complete results of downstream performance breakdown with the increase of pre-training data size.
Refer to caption: x6.png

In Tab. 9, we examine the effects of freezing and unfreezing attention layers in alignment learning. We observe that unfreezing attention results in consistent improvements across all metrics, suggesting that it is crucial to optimize the multi-head attentions in this sub-stage for better vision-language alignment.

**Table 9: Results of freezing and unfreezing attention in alignment learning. “T-Param” refers to trainable parameters. All models are pre-trained on 20 millions of data in alignment learning and fine-tuned on instruction data from LLaVA-665k (Liu et al., ).**
| Methods | #T-Param | MME-P | DocVQA | InfoVQA | SQA-I | GQA | ChartQA | AI2D |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Freeze attention | 1.8B | 1136 | 39.5 | 19.7 | 56.5 | 59.1 | 27.2 | 44.1 |
| Unfreeze attention | 3.0B | 1153 | 49.3 | 22.7 | 61.8 | 59.9 | 49.5 | 46.4 |

### A.4 Visualizations

Image captioning and OCR

Visual grounding

VQA

Chinese OCR

Code generation

Document understanding

Math

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: See Weaknesses
soundness: 3
strengths: 1. The monolithic architecture presented in this paper is highly innovative and relatively easy to reproduce. The MMoE framework not only allows the visual learning components to benefit from the pretrained language sections but also maintains linguistic capabilities. This dual advantage significantly accelerates the monolithic pretraining process.
2. The EViP pretraining strategy is well-motivated, enabling smoother pretraining through progressive learning, which addresses catastrophic forgetting effectively.
3. The paper includes extensive experiments and ablation studies, providing compelling evidence for the effectiveness of the proposed method.
summary: This paper introduces a novel monolithic Multimodal Large Language Model (MLLM) architecture, which integrates visual parameters directly into a pretrained Language Model through a multimodal mixture-of-experts (MoE) framework, eliminating the need for additional visual feature extractors and projectors. Additionally, a delta tuning pretraining strategy, named EViP, is proposed to address the issue of catastrophic forgetting. Experiments conducted on 16 benchmarks demonstrate the superiority of both the proposed model and the pretraining strategy.
weaknesses: 1. While the paper introduces several key contributions, such as the multimodal MoE architecture and the progressive training strategy, it does not clearly delineate the performance gains attributable to each component.
2. The concept of a multimodal MoE structure has been explored in prior works such as MoE-LLaVA, MARS, and Uni-MoE. However, the paper lacks a detailed discussion on how this model differs from these existing approaches.
3. In Section 3.3, the authors mention the inclusion of additional data sets for video understanding and handwritten text recognition. However, the paper does not clarify the impact of these data sources on the model's performance.

[1] MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

[2] MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesis

[3] Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Refer to the weakness.
soundness: 3
strengths: - The paper tackles the significant challenge of catastrophic forgetting in MLLM foundation model training. By utilizing delta tuning approach within monolithic MLLMs, it mitigates this problem, ensuring the retention of core language capabilities while learning new visual parameters.
- Given the large-scale nature and critical importance of data quality in pre-training and instruction tuning, the authors' introduction of a progressive learning strategy is commendable. 
-  The paper is supported by extensive experimental validation, demonstrating the framework’s performance across 16 benchmarks.
summary: The paper enhances monolithic MLLMs by employing delta tuning, which prevents the model from suffering catastrophic forgetting. Building on this, it integrates a set of visual experts using a MoE structure and introduces a novel pre-training strategy EViP for progressive learning of visual knowledge. Experimental results in the paper show that the proposed framework achieves superior performance across various benchmarks.
weaknesses: - A key weakness of the paper lies in the lack of clear motivation for adopting monolithic MLLMs over modular MLLMs. Understanding the advantages and rationale behind choosing monolithic structures is crucial for assessing the significance and justification of the work. The authors need to provide a detailed explanation of why this approach is preferable and what specific benefits it offers compared to modular MLLMs.
- The method proposed in the paper lacks sufficient novelty. To address catastrophic forgetting, the paper introduces independent visual parameters, an approach that is already widely used in modular MLLMs such as QwenVL and InternVL-1.5. The authors need to clearly explain how their approach differs from these existing methods. Simply transferring these techniques from modular to monolithic MLLMs does not constitute a significant innovation. 
- The paper's approach to processing visual input in monolithic MLLMs employs PatchEmbed and MLP, while modular MLLMs typically use a visual encoder and MLP. PatchEmbed, as a learnable module, effectively encodes visual input into tokens that can be fed to the LLM, similar to the role of a visual encoder. This raises the question of how the proposed method truly differentiates itself from modular approaches, as both share the fundamental idea of transforming visual input into LLM-readable tokens. The authors need to clarify how this design embodies the monolithic concept and what the essential differences are compared to modular MLLM structures.
 - The paper introduces a progressive learning approach that divides the pre-training process into three distinct stages. To substantiate the effectiveness of this method, the authors should provide ablation studies demonstrating its impact. Additionally, it would be beneficial to explain why the pre-training data is segmented into these three categories, potentially from the perspective of data distribution. Moreover, it raises the question of whether instruction-tuning data would also benefit from a similar progressive approach.
- The benchmarks used for comparison in the paper are incomplete, as they do not include prominent MLLMs such as LLaVA-1.5 and Qwen2-VL. The absence of these mainstream models weakens the claim of superiority for the proposed method.
 - The authors need to compare MLLMs of different sizes to demonstrate the scalability of their approach. Currently, the models evaluated in the paper are all under 2B parameters. To validate the proposed method's applicability and performance at scale, it is essential to include experiments with larger models, such as 7B and 13B.
- The paper lacks detailed implementation specifics for FFN_v and FFN_t. It states that FFN_v is initialized from FFN_t to leverage pre-trained knowledge. The author should provide a comprehensive explanation of the implementation process. This detail is crucial for understanding the decoupling and interaction between visual and textual knowledge.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: Please refer to Weaknesses.
soundness: 3
strengths: I have a very positive and encouraging attitude toward the construction of native MLLM models. I believe this work is a very meaningful exploration of Monolithic MLLM. The model structure is relatively simple, yet the performance achieved is quite significant. The writing of the paper is also clear and easy to follow.
summary: This paper present Mono-InternVL, a novel monolithic MLLM that seamlessly integrates a set of visual experts via a multimodal mixture-of-experts structure. Besides, they leverage a innovative pre-training strategy to maximize the visual capability.  Extensive experiments on 16 benchmarks validate its superior performance.
weaknesses: * **Confusion in Structural Design:** In L314-315 of the manuscript, it is mentioned that unfreezing the parameters of the MHA component during the **Alignment Learning** stage helps achieve better vision-language alignment. However, this improvement does not seem intuitive. Furthermore, will unfreezing the MHA while freezing the T-Ext lead to potential losses in language capability?

* **Scalability of the Method:**
    * Compared to mainstream paradigms based on ViTs, the monolithic architecture requires a substantial amount of image-text data to facilitate endogenous visual pre-training. My concern is that, unlike ViTs which can adapt to various LLMs through lightweight image-text alignment (*e.g.*, LLaVA 558K), the visual expert trained within Mono-InternVL is specific to a particular LLM. Specifically, **The semantic space and model structure of the visual expert are both specialized for InternLM**, making it difficult to reuse them with other LLMs (e.g., Qwen, LLaMA3). It significantly limits the scalability of the approach, especially considering the high training costs involved.
    * Additionally, performance validation was only conducted on a 1.8B parameter scale. While the performance is notable compared to other ViT-based models of similar size, scaling up to 7B or even 72B while continuing to utilize a visual expert reveals a clear disadvantage in training costs compared to ViT-based models.

* **Fairness in Performance Comparison:** Another concern arises from Table 6. In the comparison of computational overhead, is it truly necessary to only compare with InternVL-1.5-2B? For instance, as the authors demonstrate in Table 3, several ViT-based models perform comparably to Mono-InternVL under similar activation parameters (e.g., MiniCPM-V-2). To my knowledge, these models utilize only SigLiP or CLIP-L as their image encoder, resulting in a minimal number of image tokens. In contrast, Mono-InternVL employs a significantly larger number of patches. Therefore, I believe a comprehensive comparison of performance and training costs between these models and the proposed Mono-InternVL is necessary to accurately reflect the advantages of the monolithic model.

I hope the author can provide a more detailed explanation and proof of the proposed method. I will consider changing my perspective if the author's response is valuable.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The Avg score used in Table 2 is very strange. Most of the models in the table have never seen Chinese multimodal data, or even Chinese text data. In contrast, Mono-InernVL uses Chinese multimodal data in both pre-training and fine-tuning stages. In this case, introducing a Chinese multimodal evaluation dataset into the calculation of the Avg score is a very unfair and strange behavior.
2. Mono-InternVL's training data includes InternVL-1.5's training data, and the high-resolution image processing strategy is also inherited from InternVL-1.5. However, there is a significant gap in performance on high-resolution benchmarks (DocVQA, InfoVQA in Table 3), even InternVL-1.5 uses a weaker LLM backbone. Could you provide more dicussion and analysis on this issue?
3. Considering authors claim about catastrophic forgetting, it is very interesting to compare the text-only performance of the LLM backbone and Mono-InternVL, to verify the claim.
4. In Table 6, the authors compare the decoding speed of Mono-InternVL with InternVL-1.5 via TTFT and TPS. However, the introduction of MMoE may slow down the decoding speed LLM part in the multimodal input, especially image-text interleaved settings. When the model scale and output token number are small, the speed advantage of Mono-InternVL without visual encoder can be highlighted. However, when the model scale and output token number are large, the speed advantage of Mono-InternVL may be weakened and the slower decoding speed of MMoE may become an important issue that deserves attention. Could you provide more discussion and analysis on this issue? For example, analyzing the decoding speed (i.e., for each sample, TTFT + Decoding Time = Processing Time of each sample) of Mono-InternVL and InternVL-1.5 in both simple image-text setting and complex image-text interleaved setting when parallel generating answers.
5. In Sec. 3.1, what's the meaning of "add a thumbnail to provide global visual information" when transforming the image to image patches?
soundness: 3
strengths: 1. This paper conducts large-scale multimodal pretraining and SFT to train a monolithic MLLM with multimodal MoE. The well-designed model architecture and multi-stage training strategy help Mono-InternVL achieve strong performance on various benchmarks, compared with both monolithic and modular MLLMs.
2. This paper provides the scaling curve of multi-stage pre-training data and the performance of Mono-InternVL on downstream tasks, which is helpful for understanding the impact of data scale on different training stages.
3. This paper provides a visualization of attention maps to help understand the attention mechanism of such a monolithic MLLM between text and image modalities.
summary: This paper presents Mono-InternVL, a strong monolithic MLLM with multimodal MoE that directly embeds images and texts into patches and tokens without widely-used visual encoders in MLLMs.
Large-scale multi-stage multimodal pretraining and SFT helps Mono-InternVL achieve strong performance on various benchmarks, compared with both monolithic and 
modular MLLMs.
Stands on the shoulder of previous works, Mono-InternVL provides a feasible solution to extend 
pre-trained LLMs to monolithic MLLMs.
weaknesses: 1. The main contribution of this paper is to extend pre-trained LLMs to monolithic MLLMs via multimodal MoE and multi-stage pre-training. However, the novelty of this work is limited, as it mainly relies on previous works, especially VLMo, which proposed multimodal MoE and stagewise pre-training in the VLMs era. With full respect to the author's contribution as wrote in the strengths, but Mono-InternVL is very much a VLMo that combines the experience of data and training engineering tricks widely used in existing MLLMs.
2. This paper lacks fair comparison and in-depth analysis on modular and monolithic MLLMs. As a counterfact to Mono-InternVL, the authors should provide a modular version (Modular-InternVL) to fairly compare the performance (Table 2,3,4). Take Table 2 as an example, compared with other modular MLLMs, too many factors that have a significant impact on performance make such comparisons unfair and unconvincing (LLM backbone, training data, evaluation setting like image resolution).
3. This paper seems to avoid comparing with strong modular and monolithic MLLMs, such as InternVL2, Qwen2-VL, Phi-3-Vision, and Reka (a very strong monolithic MLLM). These comparisons are meaningful and important. It is very strange that the authors ignore these comparisons, especially when InternVL2 shares the same LLM backbone with Mono-InternVL, and the LLM backbone has a significant impact on performance.