Improving Multi-modal Large Language Model through Boosting Vision Capabilities

## Abstract

Abstract We focus on improving the visual understanding capability for boosting the vision-language models. We propose Arcana , a multiModal language model, which introduces two crucial techniques. First, we present Multimodal LoRA (MM-LoRA), a module designed to enhance the decoder. Unlike traditional language-driven decoders, MM-LoRA consists of two parallel LoRAs – one for vision and one for language – each with its own parameters. This disentangled parameters design allows for more specialized learning in each modality and better integration of multimodal information. Second, we introduce the Query Ladder adapter (QLadder) to improve the visual encoder. QLadder employs a learnable “ ladder ” structure to deeply aggregates the intermediate representations from the frozen pretrained visual encoder (e.g., CLIP image encoder). This enables the model to learn new and informative visual features, as well as remaining the powerful capabilities of the pretrained visual encoder. These techniques collectively enhance Arcana’s visual perception power, enabling it to leverage improved visual information for more accurate and contextually relevant outputs across various multimodal scenarios. Extensive experiments and ablation studies demonstrate the effectiveness and generalization capability of our Arcana. The code and re-annotated data are available at https://arcana-project-page.github.io .

## 1 Introduction

In recent years, multimodal large language models (MLLMs) Wang et al. (); Bai et al. (); Liu et al. (); Ye et al. () have made significant advancements.
These models amalgamate image representations into large language models (LLMs) through an adaptor Touvron et al. (); Zheng et al. ().
Various methods Dai et al. (); Liu et al. (); Wang et al. (); Dong et al. ()
leverage the capabilities of the powerful LLM to excel in various multimodal tasks.

While existing MLLMs showcase remarkable proficiency in multimodal tasks,
they still face challenges in visual perception
that is crucial for further tasks, such as reasoning or creation Chen et al. (); Liu et al. ().
Fig. 1 (a) presents several examples that clearly highlight this issue.
We observe deficiencies in current MLLMs regarding low-level visual perception, such as color and quantity, as well as high-level visual perception, such as small object detection and localization. Consequently, there is a pressing necessity to bolster the comprehension capabilities of existing MLLMs, specially for *vision*.

The insufficient visual perception capabilities of MLLMs can mainly be attributed to two factors: decoder and visual encoder.
As depicted in Fig. 1(b), existing language driven decoder structures directly couple visual and language modalities.
Such design not only disregards their unique characteristics but also may lead to information confusion, thus impairing the accurate understanding and processing of visual information.
On the other hand, freezing visual encoder directly limits the ability to learn and represent visual information.
Therefore, improving the visual perception requires rethinking the decoder design and optimizing the use of the visual encoder to better capture and process visual features.

Figure: Figure 1: (a) Sampled some VQA examples involving color, quantity, small objects, and localization tasks, showcasing the importance of visual recognition capabilities for multimodal language models (MLLMs). (b) Contrasting Arcana’s multimodal decoder with mainstream methods’ language driven decoder. The language-driven decoder employs a language decoder (LLMs) directly to handle tokens from different modalities, which may lead to modality interference and performance degradation. In contrast, the multimodal decoder independently processes different token types to avoid modality interference. (c) illustrates the structures of different visual encoders and the resulting number of visual tokens obtained. The bar chart displays the model’s performance across various architectures.
Refer to caption: x1.png

As shown in Fig. 1(c), previous multimodal large language models (MLLMs) typically relied on CLIP as the visual encoder. However, research Tong et al. (); Xu et al. () has revealed limitations in CLIP’s ability to capture complex visual features. To address this, recent methods have incorporated self-supervised learning (SSL) pretrained models, such as DINOv2 Oquab et al. (), and fused their outputs with CLIP’s features to enhance the visual encoder’s representation capacity. While this fusion approach improves feature representation, it also introduces significant computational overhead. The use of two visual encoders doubles the number of visual tokens, leading to a substantial increase in training costs, particularly when handling large-scale datasets and complex models.

Toward this end, we propose a new multimodal large language model Arcana that aims to enhance visual perception capabilities from both visual encoder and decoder. Specifically, we design a multimodal LoRA (MM-LoRA) to construct a multimodal decoder as show in Fig. 1(b). This decoder provides independent learning spaces for each modality, ensuring the decoupling of different modalities, avoiding information confusion, and preserving the uniqueness of each modality. Additionally, we propose a novel design, the Query Ladder Adapter (QLadder), as shown in Fig. 1(c). Unlike methods that significantly increase the number of visual tokens, our approach introduces only a small set of visual tokens (set to $64$, where $64<<N$). Despite the limited number of tokens, QLadder effectively enhances the model’s ability to learn and represent visual information by progressively refining and integrating visual features through its "ladder" structure. Notably, even with the introduction of only a small number of visual tokens, QLadder achieves performance comparable to DINOv2-based MOF Tong et al. () methods on the MMVP benchmark, which demands strong visual representations. Furthermore, our approach demonstrates performance improvements on traditional multimodal benchmarks, such as MMbench Liu et al. () and TextVQA Singh et al. (), highlighting its versatility and effectiveness across various tasks.

Finally, we conducted an extensive series of experiments to thoroughly evaluate the performance and effectiveness of Arcana. These experiments were designed to assess various aspects, including the robustness of MM-LoRA and QLadder across different benchmarks, its ability to generalize in diverse scenarios, and its performance in comparison to state-of-the-art methods.

## 2 Related Work

Multi-Modal Large Language Models.
Fueled by the tremendous success of large language models (LLMs) Achiam et al. (); Touvron et al. (); Jiang et al. (), there is growing interest in developing end-to-end multi-modal large language models (MLLMs) Dai et al. (); Ye et al. (); Dong et al. (). These models aim to enhance the visual perceptual capabilities of LLMs by integrating additional modalities, allowing for unified handling of multi-modal tasks. Currently, there are three primary approaches to building Multi-Modal foundational models, each demonstrating strong potential for zero-shot generalization in the visual-language domain.

The first approach, exemplified by Flamingo Alayrac et al. (), uses cross-attention to align visual models with large language models across modalities. The second approach, used by models like PaLM-E Driess et al. (), directly integrates extracted visual features into a pre-trained PaLM Anil et al. () model via a linear layer, achieving robust performance. This method is widely adopted by mainstream models such as LLaVA Liu et al. (), CogVLM Wang et al. () and Internlm-Xcomposer Zhang et al. () but incurs high inference costs due to the lengthy visual tokens. To address this, the third approach, inspired by DETR Meng et al. (); Zhu et al. () and represented by BLIP-2 Li et al. (), employs a Q-former to effectively reduce the sequence length of visual features. Similar designs are used by mPLUG-OWL2 Ye et al. (), and MiniGPT-4 Zhu et al. (). However, these methods Anil et al. (); Bai et al. (); Chen et al. () couple visual and language modalities in the same space using language-guided decoders, overlooking the uniqueness of different modalities. This oversight may result in interference between modalities, potentially affecting performance. To this end, we employ MM-LoRA to implement a multimodal decoder, aiming to mitigate the impact of modality interference on the model.

Improve visual perception for MLLMs. Currently, MLLMs are the most popular approach in VL community Alayrac et al. (); Li et al. (), and enhancing their visual recognition capabilities has become a prominent research trend. Integrating visual features into large language models (LLMs) via a linear layer has become the mainstream approach Liu et al. (); Wang et al. (). However, this approach often relies on frozen vision encoders to provide visual features, which limits the visual recognition capabilities of multimodal large language models (MLLMs). To address this issue, existing methods enhance visual recognition in two ways. The first method Luo et al. (); Tong et al. (); Xu et al. () introduces new high-resolution vision encoders, significantly improving visual recognition by increasing the number of visual tokens. For example, LLaVA-HR Luo et al. () achieves this by incorporating ConvNeXt Liu et al. () to handle high-resolution images. However, these methods significantly increases the number of visual tokens. Therefore, we propose QLadder, which can significantly enhance the model’s visual perception capability with the introduction of a small number of visual tokens. The second method Wang et al. (); Dong et al. (); Ye et al. () expands the learning space for visual tokens within the large language model to accelerate visual-language alignment, thereby enhancing visual perception. For instance, Internlm-Xcomposer2 Dong et al. () introduces Partial-LoRA, adding a LoRA to visual tokens to strengthen their representation. However, experiments with MM-LoRA have shown that directly increasing the learning space for visual tokens in the decoder does not improve the model’s performance.

## 3 Method

Figure: Figure 2: (a) The architecture of the Arcana. (b) The training pipeline of Arcana. MM-LoRA is optional during the pre-training phase.
Refer to caption: x2.png

### 3.1 Overview

We propose a new model, named Arcana as shown in Fig 2, designed to enhance visual perception in multimodal language models. Like most existing models Liu et al. (); Chen et al. (), it includes a visual encoder, a vision-language adapter, and a large language model.
The key difference is that we use MM-LoRA to implement a multimodal decoder. Unlike traditional fine-tuning where visual and language modalities share LoRA parameters, MM-Lora assigns different LoRA parameters to each modality. Additionally, we introduce QLadder in the visual encoder, which significantly enhances the model’s ability to learn and represent visual information with the introduction of a small number of visual tokens. We first briefly introduce Arcana’s architecture in Section 3.2. Additionally, in Section 3.3, we detail MM-LoRA, which aims to decouple the learning spaces of different modalities to achieve a multimodal decoder. Lastly, we introduce the training paradigm of Arcana in Section 3.4.

### 3.2 Architecture

Our approach Arcana
(illustrated in Fig. 2(a) consists of three main components:
visual encoder,
vision-language adapter and
large language model.
Each component is described in the following.

Visual Encoder. The primary objective is to extract visual features from the image.
The encoder learned with language-supervision,
e.g., the CLIP Radford et al. () visual model,
is widely adopted.
The CLIP encoder is often fixed during fine-tuning,
e.g., in LLaVA,
for keeping the representation capability
of the original CLIP encoder.
We propose to
improve the visual encoder
through a query ladder adaptor (QLadder)
from the fine-tuning data
that may contain new visual semantics.
The structure is shown in Fig. 3(b). This adapter enhances the visual feature representation of the visual encoder by adding a small number of query visual tokens while retaining the pretrained image encoder. It improves Arcana’s visual perception capability.

We extract visual features $\mathbf{F}_{c}\in\mathbb{R}^{N_{I}\times C_{v}}$
through the CLIP encoder, where $C_{v}$ represents the channel of visual feature, $N_{I}$ indicates the number of image patch.
A set of learnable vectors $\mathbf{x}_{q}$ is fed into QLadder to
acquire additional visual features $\mathbf{F}_{q}\in\mathbb{R}^{N_{q}\times C_{v}}$, where $N_{q}<<N_{I}$.
The two kinds of visual features are concatenated: $\mathbf{F}_{v}=\operatorname{concat}(\mathbf{F}_{c},\mathbf{F}_{q})$. As shown in Fig 2(b), QLadder comprises multiple layers composed of cross-attention and feed-forward networks (FFNs).

Vision-Language Adapter.
We map the output of visual encoder to the same space as the language embedding space
through an Vision-Language adapter.
The adapter $\operatorname{g}$ consists of two MLP layers.
The output visual features are denoted as
$\mathbf{F}^{I}=\operatorname{g}(\mathbf{F}_{v})$.

Large Language Model. For multimodal tasks Goyal et al. (); Hudson & Manning (), leveraging pre-trained large language models (LLMs) can provide valuable linguistic priors. Through multimodal instruction tuning, LLMs learn to comprehend visual features within images, enabling comprehensive understanding and processing of multimodal data. Typically, this process is accomplished through full fine-tuning or LoRA Hu et al. (). However, these methods overlook the unique characteristics of modalities, leading to modality confusion. This not only damages MLLMs’ accurate understanding and processing of visual information but also affects natural language understanding. Therefore, a multimodal decoder that provides separate learning spaces for each modality is a better choice for MLLMs.

Figure: Figure 3: (a) The farmework of MM-LoRA *vs.* LoRA. MM-LoRA introduces two new hyperparameters, $\beta$ and $\gamma$, to control the ranks of the visual and language LoRAs, respectively. Notably, we set $\beta+\gamma=1$ to ensure that MM-LoRA has the same number of parameters as LoRA. (b) The architecture of the visual encoder includes the QLadder adapter and CLIP. The QLadder adapter consists of cross-attention and FFN layers, with weights initialized from those of CLIP.
Refer to caption: x3.png

### 3.3 Multimodal LoRA

To implement a multimodal decoder based on a large language model, we propose a multimodal LoRA. This approach projects visual and language features into separate semantic spaces to decouple their representations, thereby avoiding modality interference. This allows Arcana to retain the unique characteristics of each modality, enhancing its visual perception without compromising natural language understanding. Next, we detail the MM-LoRA process.

MM-LoRA, as illustrated in Fig. 3, consists of visual LoRA and language LoRA. In comparison to LoRA, we introduces two parameters, $\beta$ and $\gamma$, to control the rank size of ($R$) visual LoRA and language LoRA. It’s worth noting that $\beta+\gamma=1$ to ensure that no additional parameters are introduced compared to LoRA. Specifically, given a sequence of visual-language features $F\in\mathbb{R}^{(N_{v}+N_{t})\times C}$ and a multimodal mask $M\in\{0,1\}^{(N_{v}+N_{t})}$, where $C$ represents the hidden dimension in LLMs, $N_{v}$ and $N_{t}$ indicates the number of visual and language tokens, respectively. We define a modality separation function $\Theta$ to separate the tokens of different modalities within $F$.

$$ $\Theta(F,M,m)=F\odot(M==m)\,,$ (1) $$

where $m\in\{0,1\}$ is used to select between visual tokens ($m=0$) and language tokens ($m=1$). Therefore, based on multimodal mask $M$ , we can obtain $F^{I}$ and $F^{T}$.

$$ $F^{I}=\Theta(F,M,0)\qquad F^{T}=\Theta(F,M,1)$ (2) $$

Then, $F^{I}$ and $F^{T}$ are separately inputted into the visual part and language part of MM-LoRA. In Visual LoRA, the weights are denoted as $W_{A}^{I}\in\mathbb{R}^{C\times\beta R}$ and $W_{B}^{I}\in\mathbb{R}^{\beta R\times C}$, while in Language LoRA, the weights are denoted as $W_{A}^{T}\in\mathbb{R}^{C\times\gamma R}$ and $W_{B}^{T}\in\mathbb{R}^{\gamma R\times C}$.

Similarly to LoRA Hu et al. (), $F$ is inserted into the LLM layer to obtain $\hat{F}$. Finally, the output results of MM-LoRA are added to the output of LLM according to the mask $M_{I}$.

$$ $\begin{split}\hat{F}&=W_{o}\times F\\ \Theta(\hat{F},M,0)+=W_{B}^{I}\times W_{A}^{I}\times F^{I}&\qquad\Theta(\hat{F },M,1)+=W_{B}^{T}\times W_{A}^{T}\times F^{T}\\ \end{split}$ (3) $$

In Arcana, MM-LoRA is applied to all linear layers of the large language model, thereby achieving an optimal multimodal decoder.

### 3.4 Training Paradigm

Following prior work Liu et al. (); Wang et al. (), we adopt a two-stage approach involving pretraining and multimodal instruction fine-tuning to train Arcana, as illustrated in Fig. 2(b). The purpose of the pretraining stage is to align the visual encoder with the language model, while multimodal instruction fine-tuning aims to adapt the model better to specific tasks through fine-tuning. We found that freezing the visual encoder limits the MLLM’s ability to capture complex visual information, such as scene text and visual knowledge. To address this issue, we introduce Qladder and enable it to be trained in both the pretraining and instruction fine-tuning stages. This strategy allows the model to more effectively capture both low-level and high-level semantic visual information. Additionally, we introduce MM-LoRA fine-tuning as an alternative to full fine-tuning and LoRA fine-tuning, enabling a multimodal decoder that minimizes modality interference. Specifically, in the pretraining stage, we train Qladder and the vision-language adapter, while in the instruction fine-tuning stage, we train Qladder, the vision-language adapter, and MM-LoRA. Furthermore, to ensure the linguistic capabilities of Arcana, we employ joint training, adjusting the entire model during instruction fine-tuning, integrating textual and multimodal instructions.

## 4 Experiments

### 4.1 Implementation Details

Model. In the visual encoder, we utilize the CLIP-L Radford et al. () model with an input resolution of 336 and a patch size of $14\times 14$. Furthermore, the QLadder adapter adopts the same structure as CLIP-L, replacing self-attention with cross-attention. Notably, QLadder utilizes pre-trained CLIP weights as its initial weights. For the LLMs, we employ the pre-trained Vicuna-7B Chiang et al. () model. The Vision-Language adapter comprises two layer MLP. MM-LoRA, used for fully supervised multimodal instruction tuning, consists of a visual LoRA with a rank of $\beta\times R$ and a language LoRA with a rank of $\gamma\times R$.

**Table 1: Performance on six General Visual Question Answering benchmarks. Specialist models, indicated in gray, are fine-tuned on each individual dataset. The red and blue colors respectively represent the optimal and suboptimal results on each benchmark. $*$ indicates that MM-LoRA is trained during the pretrain stage.**
| Type | Model | LLM | In-domain VQA Tasks | Zero-shot VQA Tasks |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| VQAv2 | OKVQA | GQA | TextVQA | ScienceQA | Ai2d |  |  |  |
| Generalists | BLIP2 Li et al. () | Flan-T5 | 65.0 | 45.9 | 41.0 | 42.5 | 61.0 | - |
| InstructBLIP Dai et al. () | Vicuna (7B) | - | - | 49.2 | 50.1 | 60.5 | 40.6 |  |
| InstructBLIP Dai et al. () | Vicuna (13B) | - | - | 49.5 | 50.7 | 63.1 | - |  |
| Shikra Chen et al. () | Vicuna (7B) | 77.4 | 47.2 | - | - | - | - |  |
| IDEFICS-Instruct Laurençon et al. () | LLaMA (65B) | 37.4 | 36.9 | - | 28.3 | 61.8 | 54.8 |  |
| LLaVA-v1.5 Liu et al. () | Vicuna (7B) | 78.5 | - | 62.0 | 58.2 | 66.8 | 55.5 |  |
| Qwen-VL-Chat Bai et al. () | Qwen (7B) | 78.2 | 56.6 | 57.5 | 61.5 | 68.2 | - |  |
| mPLUG-Owl2 Ye et al. () | LLaMA (7B) | 79.4 | 57.7 | 56.1 | 58.2 | 68.7 | 55.7 |  |
| Arcana | Vicuna (7B) | 79.2 | 57.9 | 61.6 | 59.5 | 71.2 | 56.8 |  |
| Arcana^∗ | Vicuna (7B) | 79.5 | 58.9 | 61.8 | 58.7 | 69.5 | 56.9 |  |
| Specialists | GIT2 Wang et al. () | - | 81.7 | - | - | 59.8 | - | - |
| PaLI-17B Chen et al. () | - | 84.3 | 64.5 | - | 58.8 | - | - |  |

Data Sets. During pre-training, we used approximately 1.2M image-text pairs from ShareGPT4V Chen et al. (). In the multimodal instruction tuning stage, we utilize six types of supervised data totaling 934k, namely: (1) text-only instruction data (ShareGPT ShareGPT ()); (2) vision question-answering data (VQAv2 Goyal et al. (), GQA Hudson & Manning (), A-OKVQA Schwenk et al. (), OK-VQA Marino et al. ()); (3) OCR QA (OCRVQA Mishra et al. (), TextCaps Sidorov et al. ()); (4) Region-aware QA (RefCOCO Kazemzadeh et al. (); Mao et al. (), VG Krishna et al. ()); (5) multi-modal instruction data (LLaVA-instruct Liu et al. ()); and (6) image captions (VG-COCO Hao et al. (), shareGPT4V Chen et al. ()). In the Ablation study, we only use the multimodal instruction data from LLaVA-v1.5.

Training Setting. During the pretraining step, we use language modeling loss with a batch size of $256$ for $1$ epoch. The learning rates are set to $1e-3$ for the vision-language adapter and $2e-5$ for Qladder. In the multimodal instruction tuning step, we integrated MM-LoRA into the LLM to create a multimodal decoder, thus preventing information interference between modalities. We set the learning rate for MM-LoRA to $1e-4$, and for both Qladder and the vision-language adapter, to $2e-5$. MM-LoRA is configured with a default rank $R$ of 256, $\beta$ set to 0.25, and $\gamma$ set to 0.75. All experiments are conducted on 8 NVIDIA A100 GPUs.

### 4.2 Main Results

**Table 2: Performance on five Large Vision-Language Models (LVLM) benchmarks.The red and blue colors respectively represent the optimal and suboptimal results on each benchmark. $*$ indicates that MM LoRA is trained during the pretrain stage.**
| Method | Vision Encoder | Language Model | MME | MMBench | MM-Vet | SEED-Bench | LLaVA^W | POPE |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BLIP-2 Li et al. () | ViT-g (1.3B) | Vicuna (7B) | 1293.84 | - | 22.4 | 46.4 | 38.1 | 85.3 |
| MiniGPT-4 Zhu et al. () | ViT-g (1.3B) | Vicuna (7B) | 581.67 | 23.0 | 22.1 | 42.8 | 45.1 | - |
| LLaVA Liu et al. () | ViT-L (0.3B) | Vicuna (7B) | 502.82 | 36.2 | 28.1 | 33.5 | 63.0 | 80.2 |
| mPLUG-Owl Ye et al. () | ViT-L (0.3B) | LLaMA (7B) | 967.34 | 46.6 | - | 34.0 | - | - |
| InstructBLIP Dai et al. () | ViT-g (1.3B) | Vicuna (7B) | 1212.82 | 36.0 | 26.2 | 53.4 | 60.9 | 78.9 |
| LLaMA-Adapter-v2 Gao et al. () | ViT-L (0.3B) | LLaMA (7B) | 1328.40 | 39.5 | 31.4 | 32.7 | - | - |
| Otter Li et al. () | ViT-L (0.3B) | LLaMA (7B) | 1292.26 | 48.3 | 24.6 | 32.9 | - | - |
| Qwen-VL-Chat Bai et al. () | ViT-G (1.9B) | Qwen (7B) | 1487.58 | 60.6 | - | 58.2 | - | - |
| LLaVA-v1.5 Liu et al. () | ViT-L (0.3B) | Vicuna (7B) | 1510.70 | 64.3 | 30.5 | 58.6 | 63.4 | 85.9 |
| mPLUG-Owl2 Ye et al. () | ViT-L (0.3B) | LLaMA (7B) | 1450.19 | 64.5 | 36.2 | 57.8 | - | 86.2 |
| Arcana | ViT-L (0.3B) | Vicuna (7B) | 1476.48 | 66.9 | 34.8 | 62.6 | 67.3 | 86.5 |
| Arcana^∗ | ViT-L (0.3B) | Vicuna (7B) | 1520.93 | 67.4 | 34.4 | 63.2 | 72.7 | 87.1 |

General Visual Question Answering Benchmarks. In Table 1, we compare with both SOTA MLLMs model on six General VQA benchmarks, including VQAv2 Goyal et al. (), OKVQA Schwenk et al. (), GQA Hudson & Manning (), TextVQA Singh et al. (), ScienceQA Lu et al. () and Ai2d Kembhavi et al. (). We found that Arcana achieved competitive results on six VQA benchmarks. Notably, it achieved accuracies of 57.9 on OKVQA, 71.2 on ScienceQA, and 56.8 on Ai2d , surpassing most recently proposed MLLMs methods. Additionally, Arcana^∗ with MM-LoRA used during the pre-training stage achieved better performance, indicating the importance of preserving the uniqueness of different modalities during pre-training. The superior performance on zero-shot VQA tasks particularly highlights strong generalization ability and potential across different domains of our model.

Large Vision-Language Model Benchmarks. Table 2 presents our comparative results on five different LVLM benchmarks: MMBench Liu et al. (), MM-Vet Yu et al. (), SEED-Bench Li et al. (), LLava^W Liu et al. (), and POPE Li et al. (). It is evident that Arcana achieves highly competitive performance across these benchmarks. Compared to mPLUG-OWL2 Ye et al. (), Arcana scores 2.4 and 4.8 points higher on MMBench and SEED-Bench, respectively. Additionally, Arcana achieves a score of 86.5 on the hallucination evaluation dataset POPE, indicating significant advancements in visual recognition capabilities. These impressive results not only demonstrate its strong reasoning and multi-task generalization abilities but also clearly show that Arcana significantly outperforms others in these areas. Notably, we achieved this using a 0.3B visual encoder, with MM-LoRA and QLadder significantly enhancing the model’s visual perception and generalization.

Figure: Table 3: Performance on language benchmarks of our model compared to LLaMA-2 0-shot for BBH, AGIEval, ARC.

Natural Language Understanding. Although MLLMs excel in various multimodal downstream tasks, existing work Liu et al. (); Dong et al. () often overlooks their natural language understanding capabilities. To address this, we also evaluated our model’s language understanding performance on BIG-Bench Hard (BBH) Suzgun et al. (), AGIEval Zhong et al. (), and ARC Clark et al. (), as shown in Table 3. Compared to LLaMA-like Touvron et al. () language models, Arcana achieved competitive results across multiple benchmarks. This demonstrates that our model not only performs well in multimodal tasks but also excels in language understanding, further highlighting the superiority of our approach.

### 4.3 Ablation Study

To validate the effectiveness of QLadder and MM-LoRA, we designed a series of experiments. Additionally, to ensure fairness, we used only LLaVA-v1.5 Liu et al. () data for these experiments.

Multimodal LoRA (MM-LoRA). To validate the effectiveness of the multimodal decoder, we compared the performance of MM-LoRA and LoRA. Additionally, to investigate the importance of visual tokens and language tokens in the multimodal instruction tuning process within the decoder, we compared different ratios of $\beta$ and $\gamma$ parameters. In all experiments, the RANK of MM-LoRA and LoRA was set to 256. The results are shown in Table 4.3. It clearly indicate that MM-LoRA achieves optimal performance when $\beta=0.25$ and $\gamma=0.75$. When $\beta$ is set to 1, performance significantly drops, indicating that aligning language distribution using only visual tokens is challenging for

Figure: Table 4: Ablation of $\beta$ and $\gamma$ in MM-LoRA. The default rank is set to 256, while $\beta$ and $\gamma$ are used to control the rank values in visual and language LoRA components, respectively.

Figure: Table 5: Ablation of query number in QLadder. Nq represents the number of learnable query.

MLLMs. However, introducing $\gamma$ greatly improves performance, demonstrating that learning both vision and language simultaneously accelerates modality alignment. When $\gamma$ is set to 1, there is a slight performance decline, but MM-LoRA still matches LoRA’s performance, suggesting that visual token learning is less critical than language token learning in LLMs. This indicates that during the instruction tuning phase of MLLM training, more emphasis should be placed on learning language tokens. Furthermore, when both $\beta$ and $\gamma$ are set to 0.5, the performance of MM-LoRA significantly outperforms LoRA. This intuitively demonstrates that the multimodal decoder can avoid interference between modalities by separating them, thus significantly enhancing the performance of MLLMs.

Figure: Figure 4: Visualization of attention maps. We compare the attention maps in different layer of LLM between different composition, include (a) Baseline, (b)Baseline+MM-LoRA, and (c) Baseline+MM-LoRA+QLadder. Higher brightness indicates higher attention values, with the x-axis representing all tokens, and the y-axis containing only the generated text tokens.
Refer to caption: x4.png

QLadder in Vision Encoder. To validate the effectiveness of QLadder and determine the optimal number of queries, we conducted experiments with QLadder. The results, shown in Table 4.3, indicate that the inclusion of QLadder significantly enhances our model’s performance. This demonstrates that even with a slight increase in visual tokens, without introducing a new visual encoder, the model’s visual recognition capabilities can be improved. As the number of queries increased, our model’s performance gradually improved, reaching its best performance with 64 queries. However, further increasing the number of queries led to a performance decline, indicating that too many queries can negatively impact the model’s performance.
To explicitly demonstrate the computational costs and efficiency of MLLMs with and without QLadder, we tested the memory usage and inference speed under both setting. As shown in Table 4.3, even with QLadder, MLLMs only increase memory usage by 0.582G, and the inference speed decreases by just 0.11 tokens/s.
This shows that the additional computational costs and efficiency impacts of QLadder are minimal and acceptable given the improvements it brings.

QLadder tuning v.s. Visual Encoder tuning.
To explore the impact of fine-tuning QLadder versus directly fine-tuning the Visual Encoder, we conducted comparative experiments to evaluate the effects of tuning the vision encoder, freezing the vision encoder, and adding Q-Ladder. The results are shown in Table 4.3.
Tuning the Vision Encoder often leads to the loss of pre-trained knowledge and does not significantly enhance MLLM’s performance. In some benchmark tests, it may even have a negative impact.
Freezing the Vision Encoder preserves pre-trained knowledge but lacks further optimization potential.
Adding Q-Ladder significantly improves MLLM’s performance by enhancing visual feature representation with a small number of additional visual tokens, while retaining pre-trained knowledge.
These results demonstrate that Q-Ladder effectively strengthens visual feature representation and avoids the negative effects associated with tuning the vision encoder.

**Table 6: Comparision with QLadder and additional Visual Encoder. To explore the performance in visual grounding ability, we selected MMVP, POPE, MMBench, and TextVQA for experiments. The data used in the experiments is consistent with that of LLaVA-v1.5.**
| Method | Size | add visual tokens | MMVP | POPE | MMBench | TextVQA |
| --- | --- | --- | --- | --- | --- | --- |
| LLaVA-v1.5 | 7B | - | 24.0 | 85.9 | 64.3 | 58.2 |
| LLaVA-v1.5 + MOF Tong et al. () | 7B | 256 | 27.1(+3.1) | 86.2(+0.3) | 60.1(-4.2) | 56.5(-1.7) |
| LLaVA-v1.5 + QLadder | 7B | 64 | 27.6(+3.6) | 86.5(+0.6) | 66.3(+2.0) | 58.8(+0.6) |
| LLaVA-v1.5 | 13B | - | 24.7 | 85.9 | 67.7 | 61.3 |
| LLaVA-v1.5 + MOF Tong et al. () | 13B | 256 | 28.0(+3.3) | 86.3(+0.4) | 61.6(-6.1) | 55.3(-6.0) |
| LLaVA-v1.5 + MOF Tong et al. () | 13B | 576 | 31.2(+6.5) | 86.7(+0.8) | 65.4(-2.3) | 58.7(-2.6) |

Figure: Table 7: Comparing different tuning strategies for visual encoders.

Figure: Table 8: Comparison of computational load and resource utilization during inference.

QLadder v.s. additional Visual Encoder.
Recently, there has been works exploring the addition of extra visual encoders to achieve better visual representations, e.g., MOF Tong et al. (), which uses Dinov2 Oquab et al. () as a second visual encoder to enhance the grounding ability of MLLMs.
To explore the impact of adding QLadder and adding extra visual encoder, we conducted detailed experiments to directly compare Q-Ladder with the MoF method, which integrates DINOv2, under the LLaVA-v1.5 setting. The results are shown in Table 6.
Our experimental results show that both Q-Ladder and MoF performed well in visual grounding, achieving significant improvements on the MMVP and POPE benchmarks. However, MoF’s performance declined on more comprehensive benchmarks like MMbench and OCR benchmarks like TextVQA. This decline is primarily due to MoF’s reliance on DINOv2 for visual grounding, which, while enhancing grounding capabilities, weakened visual understanding, leading to poorer results on MMbench and TextVQA. Additionally, the integration of DINOv2 significantly increased the model’s training time.
In contrast, Q-Ladder enhances both visual grounding and visual understanding through adaptive learning of distinguishing features. This dual improvement allows Q-Ladder to maintain or boost performance across a wide range of benchmarks, even when using a smaller dataset (over 2 million samples from Arcana). This is why Q-Ladder continues to achieve performance gains across various benchmarks, including comprehensive and OCR benchmarks.

Impact of MM-LoRA and QLadder in MLLMs.
To investigate the impact of MM-LoRA and QLadder in multimodal scenarios, we visualized the attention maps of Arcana with and without these modules in MM-Vet benchmark Yu et al. (). The visualization results, shown in Fig. 4, display the attention scores of generated tokens over the input sequence during the generation process. It can be seen that MLLM decoder initially focuses more on text tokens and gradually increases attention to visual tokens in the middle and subsequent layers. This indicates that visual and language information play different roles in MLLMs. The discussion about shallow-level attention maps, which also reflects this point, is provided in the Appendix.
Additionally, with MM-LoRA, we observe a significant increase in attention to visual tokens in the middle and subsequent layers, indicating that MM-LoRA helps prevent information confusion and promotes cooperation between different modalities. With the introduction of QLadder, the MLLM decoder shows increased attention to visual tokens across all layers. The highlighted regions of visual tokens further indicate that QLadder not only enhances the model’s focus on visual tokens but also enriches the visual information, achieving optimal performance in multimodal tasks.

Visualization results.
To showcase Arcana’s outstanding performance in visual perception, we visualized its performance across various types of multimodal tasks. As illustrated in Fig. 5, visual perception information is highlighted in orange. In detailed description tasks, our model not only accurately identifies and describes low-level visual information such as colors and textures for each object in the image but also precisely recognizes and describes high-level visual information such as positions and relationships of each object. Moreover, detection tasks further demonstrate our model’s effectiveness in visual recognition and localization. OCR-Free inference and chart-based question answering tasks not only exhibit our model’s OCR recognition capabilities but also demonstrate its reasoning prowess. Visual question answering tasks showcase our model’s excellent multi-turn dialogue capabilities on the foundation of precise identification.

Figure: Figure 5: Examples of results generated by Arcana were sampled, focusing on tasks that test visual perception capabilities, such as detailed captions, detection, and OCR-reasoning. In the answers, all visual recognition-related responses are highlighted in orange.
Refer to caption: x5.png

In summary, Arcana utilizes a multimodal decoder to avoid information interference between different modalities. QLadder offers an innovative strategy for enhancing visual representations with limited data. By adding a small number of visual tokens, it significantly improves the performance of large multimodal language models. This finding is significant for the future of multimodal model, as it presents an effective approach to achieving notable performance improvements even with limited data resources. By combining these techniques, future multimodal models will handle complex tasks with greater flexibility and efficiency.

## 5 Conclusion

In this paper, we introduce a new multimodal large language model, Arcana, which incorporates two novel techniques. Unlike current mainstream methods, Arcana employs MM-LoRA for a multimodal decoder, enabling more efficient information processing and integration across different modalities. MM-LoRA effectively combines data from various modalities without significantly increasing computational complexity, reducing information interference between modalities. Secondly, we present the QLadder structure, which demonstrates for the first time that with limited multimodal training data, retaining the capabilities of a pre-trained model and adding a small number of visual encoders can still enhance the performance of multimodal language models. This hierarchical structure progressively refines and enhances the expression of visual information, resulting in improved adaptability and generalization in multimodal tasks. With these two key techniques, Arcana not only excels in multimodal tasks but also shows potential for performance improvement even in data-constrained environments. Additionally, the severe lack of visual information in the image captions of open-source data limits the visual perception capabilities of multimodal large language models. To address this, we designed a data engine that uses diverse visual annotation models and large language models to generate captions rich in visual information.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: In terms of motivation, the paper aims to resolve MLLM visual perception issues such as color recognition, object counting, small object understanding, and spatial location. However, the structural designs of QLadder and MM-LoRA do not seem specifically tailored to address these problems, leading to the impression that performance improvements may stem from data rather than a well-targeted structural design, which appears somewhat forced into explaining the results.
soundness: 3
strengths: The paper is well written and structured
summary: The paper proposes Arcana, a multi-modal large language model (MLLM) designed to improve visual perception capabilities. Arcana introduces two key techniques: MM-LoRA and QLadder. MM-LoRA enables separate vision and language pathways, reducing modality interference, while QLadder enhances the visual encoder's ability to capture fine-grained visual details. Extensive experimentation across benchmarks like VQAv2 and TextVQA demonstrates Arcana’s  improvement over existing MLLMs in both zero-shot and fine-tuning scenarios, highlighting its capacity for accurate visual reasoning and multi-modal alignment
weaknesses: The structural innovations of MM-LoRA and QLadder are not sufficiently solid, as the design does not appear to specifically address identified issues such as color recognition, object counting, small object understanding, and spatial location.

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: What was the main problem that was being addressed? Was it limited data adaptation, was it visual capabilities? If it was just visual capabilities, how does LORA or a few-learnable tokens based adaptation compare against scaling up?
soundness: 2
strengths: - The identified problem of the lack of strong visual capabilities (e.g. detection, localization, color-comprehension etc.) in current vision language models is interesting and worth studying
- It's also interesting to see the need for modality specific adaptation 
- The paper is easy to comprehend and well supported by various block diagrams
summary: This paper seeks to improve visual understanding capability of vision language model. It introduces two components to enhance the capacity of the VLM: Multimodal (MM) -LoRA and Query Ladder. The MM-LoRA increases the capacity of the decoder by introducing low-rank adaptable matrices separately for the vision and language modalities. The QLadder increases the capacity of the encoder by incorporating learnable tokens at the input. Overall, this approach shows benefits of individual components and also competitive performance across MM/Language benchmarks.
weaknesses: * The summary section (line 519-520) mentions "achieving notable performance improvements even with limited data resources". However, the problem of limited data sources is not convincing. For instance, given that LLM’s and Visual-Encoders are trained with web-scale data, it’s not clear how and why would data be limited. Perhaps, the authors want to focus on specific domains (say, medical) where curating data might be difficult due to privacy concerns. But, for the kind of problems mentioned in the paper (detection, localization, color-comprehension), its not clear why data is limited.
* The paper lacks explanations for why components like LORA and QLadder should improve visual capabilities like  detection, localization, color-comprehension. While the attention visualization (line 469-482) demonstrates the effect of these components to visual-token-attention, it’s not clear why that itself should improve performance. Further, some of the statements like “promotes cooperation between different modalities” (line 478) and “enriches the visual information” (line 481) are not corroborated with any intuition or experiments. 
* The contributions of the proposed components are unclear. For instance, the benefit of LORA for limited-data-adaptation has been well studied in the past (e.g. [1]). The importance of introducing additional visual tokens to visual encoders has also been shown in [2]. In the light of the prior works, the paper should more clearly distinguish it’s technical contributions. 
* Are the benefits of Qladder/MM-LORA consistent across scales? If we increase the scale of LLM and Visual Encoder, will Qladder/MM-LORA still show benefits?
* Miscellaneous
    * Is the beta gamma ration study consistent across a range of LORA ranks (say 64 - 1024)? here it was set to 256
    * Why was LORA applied only to linear layers?
    * In qualitative evaluations (Fig. 5), comparisons should be made with other models to clearly show qualitative gains from using Qladder/MM-LORA

[1] https://arxiv.org/abs/2106.09685
[2] https://arxiv.org/abs/2309.16588

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: N/A
soundness: 2
strengths: It is well known that MLLMs often exhibit limitations in their visual capabilities, and this work addresses this important issue. Additionally, the paper is well-written and easy to follow.
summary: This work aims to enhance the visual capabilities of MLLMs through two main contributions: (1) the introduction of a MM-LoRA for the LLM decoder, and (2) the development of a Query module for visual encoders.
weaknesses: - The proposed method leverages additional learning parameters to enhance the visual capabilities of MLLMs. Recent studies (e.g., LLaVA-Next, VILA-1.5, Qwen-VL-2) have shown that simply improving image resolution using various (*any resolution*) techniques is a straightforward and effective way to address this issue. I am skeptical that the proposed method will achieve performance comparable to these AnyRes approaches, particularly on tasks requiring high resolution. The proposed method appears limited by the visual encoder, despite the incorporation of additional LoRA modules.
- The focus of this study is on the visual capability of MLLMs. However, only one ViT is examined, and there are no ablations on different ViTs. This raises doubts about the generalizability of the proposed approach.
- The improvements from the proposed method should be evaluated based on the ablation studies, rather than relying on Table 1 and 2, as the model Arcana reported in Table 1 and 2 is trained on a combination of large datasets (comparing to LLaVA-1.5 presented in Table 1 and 2). However, it is important to note that only a limited selection of four benchmarks is presented in ablations.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: 1.Was the visual encoder tuning in Table 7 conducted at the pre-training or instruction fine-tuning stage?

2.Have you tried adding LoRA to the visual encoder as well?
soundness: 3
strengths: 1. The presentation and writing are clear and easy to follow. Figure 1 in the introduction effectively illustrates the background, motivation, and main results of this paper.

2. Tables 1 and 2 show that Arcana achieves better performance than previous MLLM baselines (e.g., LLaVA-1.5, mPLUG-Owl2, etc.) on visual question answering and multi-modal conversation benchmarks.

3. The ablation studies in Tables 4 and 5 clearly validate the effectiveness of MM-LoRA and QLadder.

4. The ablation study demonstrates that QLadder significantly improves MMVP performance, which requires robust visual capabilities. In Table 6, adding QLadder boosts MMVP performance by 3.6%.
summary: This paper proposes a novel Multi-modal Large Language Model (MLLM) called Arcana, designed to enhance visual understanding capabilities. It introduces two key components: Multimodal LoRA (MM-LoRA) and the Query Ladder adapter (QLadder). MM-LoRA consists of two parallel LoRAs (one for vision and one for language) to disentangle the modalities and enhance their specialized capabilities. QLadder aggregates intermediate representations from the visual encoder, further boosting the visual abilities of the MLLM. Experimental results demonstrate that Arcana outperforms previous MLLM baselines (e.g., LLaVA-1.5, mPLUG-Owl2, etc.) on visual question answering and multi-modal conversation benchmarks. Notably, the ablation study shows that QLadder significantly improves MMVP performance, which requires strong vision capabilities.
weaknesses: 1. There is a lack of comparison with the latest open-source VLMs: LLaVA-OneVision, Qwen2-VL, InternVL2, etc. While these methods may use higher-quality training data and achieve stronger results, it is essential for readers to be aware of the current SoTAs. You may also explain why direct comparisons may not be feasible. It is acceptable for a research paper to fall short of SoTA results due to data quality differences, but these results should still be presented for context.

2. MMVP is crucial for demonstrating visual capability, but only QLadder is ablated on the MMVP benchmark. Why not conduct an ablation of MM-LoRA on MMVP as well? This would provide stronger support for the claims.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 4
questions: 1. Compared with Q-Former, why does the proposed Q-Ladder not require an additional stage for alignment with the vision encoder?

2. Is X_q in Q-Ladder a set of learnable tokens? Why not use instruction tokens for initialization, as done in Q-Former?

3. In the visualizations, it’s difficult to conclude that (b) demonstrates more attention on vision tokens compared to (a). But interestingly, It mainly appears that (b) has more sink tokens [1]. 

4. In Table 4, why the Q-Ladder results on 13B model are absent?


[1] Xiao et al. Efficient Streaming Language Models with Attention Sinks. ICLR, 2024.
soundness: 3
strengths: 1. The paper is quite easy to follow. People can quickly grasp the core design and the underlying motivation of the proposed two improvements. 
2. The presentation is quite ok for me.
3. The proposed method has little impact on the efficiency and the memory cost.
summary: This paper proposes a new MLLM named Arcana, mainly offering two improvements for boosting model comprehension on vision information. The first one is MM-LoRA, which learns two separate sets of LoRA parameters for vision and text tokens respectively, aiming to decouple the learning spaces of different modalities and better integrate the multi-modal knowledge. The other one is Q-Ladder, compared with Q-Former, it selects the vision features of different layers in ViT as the key/value vectors for different layers of Q-Ladder, instead of only using the last-layer vision token features. The experiments include the evaluation on VQA benchmarks, multi-modal benchmarks, and language benchmarks, with some ablation studies and further explorations.
weaknesses: 1. I think the proposed MM-LoRA is greatly inspired by some previous works like P-LoRA [1] in InternLM-XComposer2, visual-only modules in mPLUG-Owl2 [2] and CogVLM [3], which somehow reduces the novelty of MM-LoRA. The authors should tell the differences between MM-LoRA and these methods, along with some experiments on effectiveness and efficiency to prove the necessity of MM-LoRA. 

2. The baselines listed in Table 1, 2 are relatively old. I notice Arcana adopts ShareGPT4V data for training, but its benchmark performance seems not good as ShareGPT4V 7B model. So it is recommended to include some more advanced baseline MLLMs. 

3. It seems that the hyper-parameters introduced by MM-LoRA and Q-Ladder are not so robust and can easily affect the model performance. The authors choose the best hyper-parameters according to the ablation results. So does these hyper-parameters still work for different base LLM or architectures?


[1] Dong et al. InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Models. Arxiv preprint 2401.16420, 2024.

[2] Ye et al. mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration. CVPR, 2024.

[3] Wang et al. CogVLM: Visual Expert for Pretrained Language Models. Arxiv preprint 2311.03079, 2024.