MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning

## Abstract

Abstract Multi-modal large language models (MLLMs) have made significant strides in various visual understanding tasks.
However, the majority of these models are constrained
to process low-resolution images,
which limits their effectiveness in perception tasks that necessitate detailed visual information.
In our study, we present MG-LLaVA,
an innovative MLLM that enhances the model’s visual processing capabilities by incorporating a multi-granularity vision flow,
which includes low-resolution, high-resolution, and object-centric features.
We propose the integration of an additional high-resolution visual encoder to capture fine-grained details,
which are then fused with base visual features through a Conv-Gate fusion network.
To further refine the model’s object recognition abilities,
we incorporate object-level features derived from bounding boxes identified by offline detectors.
Being trained solely on publicly available multimodal data through instruction tuning,
MG-LLaVA demonstrates exceptional perception skills.
We instantiate MG-LLaVA with a wide variety of language encoders,
ranging from 3.8B to 34B,
to evaluate the model’s performance comprehensively.
Extensive evaluations across multiple benchmarks demonstrate that
MG-LLaVA outperforms existing MLLMs of comparable parameter sizes,
showcasing its remarkable efficacy.
The code will be available at https://github.com/PhoenixZ810/MG-LLaVA .

## 1 Introduction

Recent works on Multimodal Large Language Models (MLLMs) [, , , , , ] have achieved
rapid development in vision language understanding, visual reasoning, visual interaction, and localization.
Most MLLMs adopt pre-trained Large Language Models (LLMs)
as the base architecture to process concatenated visual and language embeddings.
As one representative work,
LLaVA [] adopts low-resolution (224^2, 336^2, etc.) images as inputs and aligns visual embeddings with the text modality via an MLP projector and then performs instruction tuning.
The architecture of LLaVA has been widely adopted by
subsequent works [, , , ], and has been applied to various vision tasks,
including detection, segmentation, and video understanding.

Figure: Figure 1: MG-LLaVA outperforms LLaVA across various vision-language tasks, particularly on tasks involving object recognition.
Refer to caption: extracted/5694511/latex/images/teaser9.png

Real-world images exhibit a wide range of resolutions, scales, and aspect ratios,
posing significant challenges for MLLMs with low-resolution inputs in robustly processing them.
To tackle this problem, recent works [, , , , , , ]
have proposed various strategies to augment the capabilities of visual encoders in MLLMs,
including training on diverse datasets, utilizing high-resolution image inputs, and employing dynamic aspect ratios.
Most of these approaches involve the integration of additional visual tokens through various techniques.
Despite these advancements, two critical issues persist:
(1) Although object-level features are crucial in nearly all visual understanding tasks,
they are currently absent in existing vision encoders;
(2) None of the existing MLLMs have integrated multi-granularity features, a classic concept in computer vision, into their frameworks.

Motivated by the aforementioned analysis, we introduce MG-LLaVA,
a novel MLLM designed to effectively process multi-granularity visual inputs,
including object-level, origin images, and high-resolution inputs.
Our framework builds upon LLaVA [] and is specifically tailored to incorporate and manage multi-granularity inputs.
For object-level inputs,
we employ a pre-trained open-vocabulary detector to identify object bounding boxes and execute RoI alignment to acquire region visual tokens.
In contrast to close-set detectors,
open-vocabulary detectors offer enhanced generalizability and robustness across diverse scenes.
To handle fine-grained visual inputs,
we utilize a convolution-based backbone [] to extract richer visual features.
Subsequently, we propose a straightforward yet effective fusion strategy to integrate these inputs into the original visual tokens in LLaVA.
Specifically, we initially merge the fine-grained visual tokens with the original visual tokens using a simple Conv-Gate convolution,
and then we append the object-level tokens to the fused tokens.
Fig. 2 illustrate the difference between MG-LLaVA and existing MLLMs.
Experimental results quantitatively validate the efficacy of the design of MG-LLaVA.

We perform extensive experiments with MG-LLaVA integrated with various language encoders,
ranging from 3.8B to 34B, to substantiate the effectiveness of MG-LLaVA.
Our evaluation encompasses 11 popular multimodal benchmarks for both image and video.
Additionally, we present a comprehensive set of ablation studies that illustrate the impact of different components in MG-LLaVA.
Benefiting from multi-granularity visual features,
MG-LLaVA demonstrates a significantly enhanced capability in perception and visual comprehension,
outperforming established counterparts and notably surpassing GPT-4V [] and GeminiPro-V [] on various multimodal benchmarks,
including MMBench [] and SEEDBench [].

The contribution of this work can be summarized as follows:

- $\bullet$
We introduce MG-LLaVA, an advanced multi-modal model adept at processing visual inputs of multiple granularities,
including object-level features, original-resolution images, and high-resolution data.
This advancement significantly enhances the capabilities of MLLMs in visual perception and understanding.
- $\bullet$
We propose the Multi-Granularity Vision Flow, a straightforward yet effective module designed for the integration of features across various granularities, thereby significantly improving the performance of our model.
The effectiveness of our approach is substantiated through empirical experiments.
- $\bullet$
By employing a range of language models scaling from 3.8B to 34B,
our model exhibits clear scalability and a marked proficiency in visual comprehension,
outperforming established counterparts and notably surpassing GPT-4V and GeminiPro-V on MMBench and SEEDBench.

## 2 Related Work

Figure: Figure 2: Comparing Different MLLM Paradigms. MG-LLaVA effectively perceives multi-granularity visual inputs that include object-level, low, and high-resolution inputs, thereby achieving advanced multi-modal understanding.
Refer to caption: x1.png

Large Language Models. In recent years, private large language models (LLMs) like GPT-4 [] and Llama [] have gained remarkable performance. Concurrently, a multitude of open-source research [, , , ] has embarked on the exploration of LLMs. LLM shows strong performance in various NLP tasks.
However, pure LLMs cannot handle image and video inputs.
Our work focuses on designing new multimodal large language models, which jointly take visual and language tokens as inputs.
In this work, we engaged a range of LLMs [, , , ] scaling from 3.8B to 34B. The observed performance across these models has proved the effectiveness of our design.

Multimodal Large Language Models. Multi-modal Large Language Models (MLLMs) [, , , , , , , , , , ] have recently showcased the potential to endow LLMs with visual conversational abilities. Among these models, LLaVA [] typically built a simple architecture that utilizes a vision-language cross-modal adapter to bridge the gap between vision and language tokens. Some research [, , ] tried to increase performance by utilizing high-resolution inputs. LLaVA-UHD [] cost-effectively increased input resolution by dividing high-resolution images into smaller slices. Subsequently, LLaVA-HR [] and Mini-Gemini [], endeavor to incorporate an additional visual encoder to enhance high-resolution details without increasing the count of visual tokens. However, these works consistently overlook the impact of fine-grained object-level features, which compromises their potential for enhanced perception. In comparison, MG-LLaVA explores the potential of multi-granularity input by simultaneously leveraging high-resolution inputs, low-resolution inputs, and object-level inputs. By flexibly integrating visual tokens of multiple granularity, MG-LLaVA achieves superior performance on several benchmarks with a marginal increase in cost.

Multi-Granularity Modeling in Vision. Inputs of multiple granularity have been incorporated into various downstream vision tasks. In object detection and segmentation, the efficacy of multi-level features has been well-established in detecting objects of different scales [, , , , , , ]. For panoptic segmentation, some methods [, , , , , ] applied a multi-granularity network to train instance, semantic, and part segmentation in parallel, and some studies [, , , , ] have indicated that training on various levels of abstraction can improve the performance of the segmentation network.
For example, SAM [] presents a multi-granularity mask prediction method for handling various level masks, such as things, background stuff, and parts.
Motivated by the above works, we aim to capture input from various levels of perception into MLLM.
In particular, we construct our model by developing multiple visual branches for different granularity, thereby augmenting its perceptual capabilities.

## 3 Method

Figure: Figure 3: The illustration of MG-LLaVA. Top left: The overall framework of MG-LLaVA, which includes the Multi-Granularity Vision Flow module and a LLM. Right: Illustration of Multi-Granularity Vision Flow, which aims to extract multiple visual features and integrate disparate features to ensure seamless interaction. Botttom left: Structure of Conv-Gate Fusion module.
Refer to caption: extracted/5694511/latex/images/framework7.png

In this study, we introduce MG-LLaVA,
which effectively harnesses both the high-resolution and object-level features for the improvement of MLLMs.
The architecture of MG-LLaVA is illustrated in Fig. 3.
The model is composed of two key components:
(1) Multi-Granularity Vision Flow framework for extracting visual features with different resolutions and granularities, while also effectively integrating disparate features to ensure seamless interaction.
(2) A large language model dedicated to generating coherent and contextually relevant responses.

### 3.1 Preliminary

As one of the most extensively adopted multi-modal LLM architectures,
LLaVA consists of a vision encoder $f_{\mathrm{V}}$, an MLP projector $f_{p}$, and a language model $f_{\mathrm{L}}$.
Given a visual input $V$ and a textual input $T$,
LLaVA computes the vision and language embeddings as per Eq. 1,
where $f_{\mathrm{T}}$ represents the input embedding layer of $f_{\mathrm{L}}$.
The resulting embeddings,
$\mathbf{E}_{\mathrm{T}}$ and $\mathbf{E}_{\mathrm{V}}$,
are then concatenated into a single token sequence,
serving as the input to the LLM.
LLaVA utilizes Eq. 2 to calculate the probability of the target answer $\mathbf{X}_{\mathrm{A}}$,
where $\theta$ represents the trainable parameters and $L$ is the length of $\mathbf{X}_{\mathrm{A}}$.
The model is trained on visual instruction tuning data to maximize $p\left(\mathbf{X}_{\mathrm{A}}\mid V,T\right)$.

$$ $\mathbf{E}_{\mathrm{T}}=f_{\mathrm{T}}\left(T\right),\mathbf{E}_{\mathrm{V}}=f _{p}\left(f_{\mathrm{V}}\left(V\right)\right)$ (1) $$

$$ $p\left(\mathbf{X}_{\mathrm{A}}\mid V,T\right)=\prod_{i=1}^{L}p_{\theta}\left( \mathbf{X}_{\mathrm{A}}^{[i]}\mid Concat(\mathbf{E}_{\mathrm{V}},\mathbf{E}_{ \mathrm{T}}^{[1:i-1]}),\mathbf{X}_{\mathrm{A}}^{[i-1]}\right)$ (2) $$

Despite the promising results,
LLaVA still restrains itself in processing images at a low resolution (224^2, 336^2, *etc.*),
which significantly hinders the model’s ability,
particularly in the recognition of small objects.
Scaling to high resolution without adapting the vision encoder directly would dramatically increase the number of visual tokens,
rendering the approach ineffective.
Furthermore, the visual input can also be complex and contain numerous objects within an image or video,
which poses challenges for MLLMs in identifying some critical objects.
Empirically, incorporating object-level features can significantly enhance the model’s perceptual abilities.
Therefore, we introduce MG-LLaVA,
which effectively harnesses both the high-resolution and object-level features for the improvement of MLLMs.

### 3.2 Multi-Granularity Vision Flow

#### Hybrid Vision Encoders

As depicted in Fig. 3(b),
MG-LLaVA initially processes images at two different resolutions: low-resolution $V_{L}$ and high-resolution $V_{H}$.
In the low-resolution branch, we follow the LLaVA-1.5 [] to utilize a CLIP-pretrained ViT [] denoted as $f_{V}^{L}$ to derive low-resolution features $\mathbf{E}_{\mathrm{L}}\in\mathbb{R}^{N\times C}$.
The ViT feature $\mathbf{E}_{\mathrm{L}}$ benefits from an expanded receptive field,
capturing a more comprehensive view of global information.
In the high-resolution branch, we employ a CLIP-pretrained ConvNeXt [] denoted by $f_{V}^{H}$ to obtain high-resolution features $\mathbf{E}_{\mathrm{H}}\in\mathbb{R}^{h\times w\times C}$.
$f_{V}^{H}$ effectively extracts detailed features from high-resolution images,
offering detailed local insights.
$f_{V}^{L}$ and $f_{V}^{H}$ downsample the input resolution with strides of 14 and 32, respectively.
We therefore adjust $V_{L}$ and $V_{H}$ to ensure that the number of tokens in $\mathbf{E}_{\mathrm{L}}$ and $\mathbf{E}_{\mathrm{H}}$ remains the same ($N=h\times w$).

#### Conv-Gate Fusion

Combining both low and high-resolution features as inputs results in a doubling of the visual tokens to be processed,
which is computationally ineffective.
Moreover, the distinct architectures of ViT and ConvNeXt lead to a discrepancy between $\mathbf{E}_{\mathrm{L}}$ and $\mathbf{E}_{\mathrm{H}}$,
requiring a careful fusion process.
Inspired from [],
we implement a lightweight Conv-Gate fusion network that facilitates feature aggregation while maintaining a single resolution’s token count, as shown in Fig. 3(c).
We first employ 1D convolutions to align the channel widths of heterogeneous features and subsequently use a gating layer to modulate the semantic information across low and high resolutions,
as described in Eq. 3.
The fusion module is applied to the output of both vision encoders,
resulting in only a marginal increase in computational cost.

$$ $\mathbf{E}_{\mathrm{F}}=\mathbf{E}_{\mathrm{L}}+G(Conv(\mathbf{E}_{\mathrm{L}} ),Conv(\mathbf{E}_{\mathrm{H}}))\times\mathbf{E}_{\mathrm{H}}$ (3) $$

#### Integration of Object-level Features

Given the set of $k$ object bounding boxes derived from the image,
denoted as $B=\{b_{1},b_{2},\cdots,b_{k}\}$,
we employ the Region of Interest (RoI) Align to extract object-level features from the vision features of the high-resolution encoder $f_{V}^{H}$.
Specifically, we upsample and concatenate features from different convolutional stages to a scale of 1/4 the input size,
resulting in a multi-scale feature representation $f_{V}^{H\prime}$,
which provides a fine-grained perspective.
The object-level features are then aligned from $f_{V}^{H\prime}$.
To maintain computational efficiency,
we apply average pooling to each object feature and subsequently concatenate them into a sequence $\mathbf{E}_{\mathrm{B}}\in\mathbb{R}^{k\times C}$,
as detailed in Eq. 4.

$$ $\mathbf{E}_{\mathrm{B}}=Concat(Avg(RoIAlign(f_{V}^{H\prime},B)))$ (4) $$

After the aggregation and extraction of object-level features,
$\mathbf{E}_{\mathrm{F}}$ and $\mathbf{E}_{\mathrm{B}}$ are processed individually by two separate projectors ($p_{F}$ and $p_{B}$) to align with the text embeddings $\mathbf{E}_{\mathrm{T}}$. The aligned features are then concatenated as input for LLM. We try multiple strategies to merge object-level features into visual embeddings and find the concatenation operation yields the most beneficial results. The experiments are discussed in Sec. 4.3. During training, we optimize Eq. 5 on the visual instruction tuning data to enhance the multi-modal comprehension capabilities of MG-LLaVA.
For video training, we execute the aforementioned operations to each frame and then concatenate the results into an extended sequence.

$$ $p\left(\mathbf{X}_{\mathrm{A}}\mid V_{L},V_{H},B,T\right)=\prod_{i=1}^{L}p_{ \theta}\left(\mathbf{X}_{\mathrm{A}}^{[i]}\mid Concat(p_{F}(\mathbf{E}_{ \mathrm{F}}),p_{B}(\mathbf{E}_{\mathrm{B}}),\mathbf{E}_{\mathrm{T}}^{[1:i-1]}) ,\mathbf{X}_{\mathrm{A}}^{[i-1]}\right)$ (5) $$

### 3.3 Training & Inference

Recently, a variety of powerful tagging models and open-vocabulary detectors have emerged,
demonstrating remarkable efficacy.
By using one specific tagging model to output labels, which are then used by the detector to generate bounding boxes,
we can effectively avoid the generation of numerous irrelevant boxes,
contrasting with the direct use of class-agnostic detectors. The details of the inference pipeline are illustrated in Appx D.
For the acquisition of object bounding boxes,
we employ the well-pretrained RAM [] as the tagging model and OWL-ViT v2 [] as detector. The generated bounding boxes are filtered by NMS and then fed to models for training and inference.
It is important to note that while the RAM model aids in generating tags,
these tags serve solely as inputs for the open-vocabulary detector to determine the bounding boxes and are not integrated into the training phase.

Following LLaVA-1.5 [], we conduct a two-stage training process.
During the pretraining stage,
we freeze all visual encoders and the LLM, and only train the fusion module, visual projector, and box projector.
This aims to refine the fusion module’s capability to aggregate features of low and high resolutions and to enhance the projector’s alignment of visual features with the text embeddings.
During instruction tuning, we keep the visual encoders frozen to maintain the integrity of high-quality image feature extraction and finetune the remaining components to enhance multi-modality comprehension.

## 4 Experiments

### 4.1 Implementation Details

#### Model Settings.

In this work, all experiments are conducted based on Xtuner [].
Specially, we choose CLIP pre-trained ViT-Large-14-336 [] as a low-resolution visual encoder and the LAION pre-trained ConvNext-Large-320 [] for high-resolution vision encoder. For the generation of bounding boxes, we have selected RAM-Plus [] as the tagging model and OWL-ViTv2-large-patch14-ensemble [] as the open-vocabulary detector.

#### Datasets.

During the image-based training stage, our dataset comprises 558K image-caption pairs from LAION-CCSBU [] and 708k image-caption pairs from ALLaVA-4V-Caption dataset [], culminating in a total of 1.26M image-caption pairs for pretraining. The datasets employed for instruction-tuning encompass 665K mixture dataset from LLaVA-Instruct [], 692k instructions from ALLaVA-4V-Instruction dataset [], and an additional 25k instructions derived from a combination of ShareGPT4V [], DocVQA [], DVQA  [] and AI2D  [], with a total number of more than 1.3M image-text conversations. The superior quality of this dataset contributes to a swift enhancement in performance. For video training, following Video-LLaVA [], we combine 558K image-text pairs and 703k video-text pairs for video pertaining. For instruction-finetuning, we utilize a 665k image-text instruction dataset from LLaVA and a 100k video-text instruction dataset from Video-ChatGPT [].

**Table 1: Comparison with leading methods on several popular visual benchmarks that concentrate on perception. Params. denotes the total number of parameters within the model. Res. refers to the resolution of the input image, which is assumed to be square by default unless otherwise indicated. The notation ‘()’ signifies the presence of both low-resolution and high-resolution inputs, with the number inside the parentheses specifying the higher resolution.**
| Method | LLM | Param. | Res. | MMB^D | MMB^T | SEED^I | MMStar |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Private Models |  |  |  |  |  |  |  |
| GPT-4V [] | - | - | - | 75.1 | 77.0 | 72.3 | 49.7 |
| GeminiProVision [] | - | - | - | 75.2 | 73.6 | 70.7 | 38.6 |
| Qwen-VL-Plus [] | - | - | - | 66.2 | 67.0 | 65.7 | 39.7 |
| Open-source Models |  |  |  |  |  |  |  |
| BLIP-2 [] | Vicuna-13B | 14.2B | 224 | - | - | 46.4 | - |
| InstructBLIP [] | Vicuna-7B | 8.2B | 224 | - | 36 | 53.4 | - |
| Shikra [] | Vicuna-13B | 7.3B | 224 | 58.8 | 60.2 | - | - |
| IDEFICS-9B [] | LLaMA-7B | - | 224 | 48.2 | 45.3 | - | - |
| IDEFICS-80B [] | LLaMA-65B | - | 224 | - | 54.6 | - | - |
| Qwen-VL [] | Qwen-7B | 9.6B | 448 | 38.2 | 32.2 | 56.3 | - |
| Qwen-VL-Chat [] | Qwen-7B | 9.6B | 448 | 60.6 | 61.8 | 58.2 | 37.5 |
| LLaVA-1.5 [] | Vicuna-7B | 7.2B | 336 | 65.2 | 66.5 | 66.1 | 30.3 |
| LLaVA-1.5 [] | Vicuna-13B | 13.4B | 336 | 69.2 | 69.2 | 68.2 | 32.8 |
| SPHINX [] | Vicuna-7B | 10B | 224 | 66.9 | - | 69.1 | - |
| SPHINX-1k [] | Vicuna-7B | 10B | 448 | 67.1 | - | 71.6 | - |
| Mini-Gemini [] | Vicuna-7B | 7.4B | 336 (768) | 69.3 | - | - | - |
| MiniCPM-V2 [] | MiniCPM-2.4B | 2.8B | 448 | 69.6 | 69.1 | 67.1 | 39.1 |
| MOVA [] | Vicuna-7B | 10B | 576 | 70.4 | - | - | - |
| LLaVA-HR [] | Vicuna-13B | 13.4B | 448 (1024) | - | - | 64.5 | - |
| LLaVA-UHD [] | Vicuna-13B | 13.4B | 672×1008 | 68.0 | - | - | - |
| Our Models |  |  |  |  |  |  |  |
| MG-LLaVA | Phi3-3.8B | 4.2B | 336 (768) | 73.4 | 72.9 | 70.2 | 35.5 |
| MG-LLaVA | Vicuna-7B | 7.4B | 336 (768) | 72.1 | 71.9 | 69.4 | 35.1 |
| MG-LLaVA | LLaMA3-8B | 8.4B | 336 (768) | 76.5 | 76.6 | 71.5 | 36.9 |
| MG-LLaVA | Vicuna-13B | 13.6B | 336 (768) | 72.2 | 73.5 | 70.8 | 34.1 |
| MG-LLaVA | Yi1.5-34B | 34.4B | 336 (768) | 80.1 | 79.1 | 73.7 | 47.9 |

#### Training Details.

We fix all seeds across the training procedures for fairness, where we adopt the XTuner codebase [].
We established the low-resolution parameter at 336 and the high-resolution parameter at 768.
For video training, we uniformly extract 8 frames from each video. During the pretraining stage, we employ a batch size of 32 per device and an aggregate batch size of 256. In the instruction-tuning phase, we reduce the batch size to 16 per device and an overall batch size of 128. The initial learning rate is set to 1e-3 for the pretraining stage and 2e-5 for the instruction-tuning stage. The number of bounding boxes per image is limited to 100 during training. The entire training process takes approximately 23 hours when using the Vicuna7B [] model using 8×A100 GPUs. For our most extensive model, the Yi1.5-34B [], we utilize 32×A100 GPUs and finalize the optimization process in roughly three days by employing the DeepSpeed Zero3 strategy.

### 4.2 Main Results

Perception Benchmarks. In Tab. 1, we compare our MG-LLaVA with previous leading approaches across several settings on Multi-Modal benchmarks, which mainly concentrate on perception capability, including MMBench-Dev and MMBench-Test [], SEEDBench-Image [], and MMStar []. MMBench is dedicated to advancing the understanding of multi-modal perception and cognition, and SEEDBench provides a comprehensive and objective evaluation of MLLM. MMStar further ensures each selected sample exhibits visual dependency. MG-LLaVA exhibits a significantly enhanced perceptual capability compared to a wide range of MLLMs. Our MG-LLaVA equipped with phi3-3.8B [] show superior performance than MiniCPM V2 [] of +3.8% on both MMBench Dev and Test, and +3.1% on SEEDBench. Utilizing Vicuna-7B [], MG-LLaVA outperforms all models with vicuna-7B and even 13B on MMBench and SEEDBench, surpassing LLaVA-1.5-7B by an average of 5.1% across four benchmarks. Moreover, with Yi1.5-34B [], MG-LLaVA consistently outperforms GPT-4V on MMBench and SEEDBench. Concurrently, it maintains equivalent efficacy to GPT-4V on MMStar. Incorporating multi-granularity visual inputs, MG-LLaVA develops its capability of capturing details within the image. More cases are exhibited in Appx. B.

Visual Question Answering Benchmarks.
In this section, we analyze MLLM’s capability of visual conversation. The benchmarks can be divided into two groups:
(1)Benchmarks require understanding the text within images to provide answers, including TextVQA(VQA^T) [] and DocVQA []. We report the accuracy of both validation sets.
(2)General visual question answering benchmarks such as VQA-V2 [], ScienceQA-Image(SQA^I) [], AI2D []. The evaluation results on VQA benchmarks are shown in Tab. 2. MG-LLaVA also demonstrates considerable proficiency on VQA benchmarks. When equipped with Vicuna-7B and 7.4B parameters, MG-LLaVA surpasses both SPHINX-1k [], which has 10B parameters, and Mini-Gemini with 7.4B parameters on these benchmarks. Operating under identical parameter conditions, MG-LLaVA, with low-resolution input of 336 and high-resolution of 768, outperforms LLaVA-UHD [], which incorporates an input resolution of 672×1008 on VQA^T, SQA^I and AI2D. MG-LLaVA exhibits its potential for expansion when integrated with larger LLM. With Yi1.5-34B [], MG-LLaVA surpasses the majority of established baselines across a wide array of VQA benchmarks.

**Table 2: Comparison with laeding methods on popular VQA visual benchmarks.**
| Method | LLM | Param. | Res. | VQA^T | DocVQA | SQA^I | AI2D | VQAv2 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Private Models |  |  |  |  |  |  |  |  |
| GPT-4V | - | - | - | 78.0 | 42.3 | 82.1 | - | - |
| GeminiProVision | - | - | - | 74.6 | - | 81.4 | - | - |
| Qwen-VL-Plus | - | - | - | 78.9 | 82.2 | 73.4 | - | - |
| Open-source Models |  |  |  |  |  |  |  |  |
| BLIP-2 | Vicuna-13B | 14.2B | 224 | 42.5 | - | 61.0 | - | 41.0 |
| InstructBLIP | Vicuna-7B | 8.2B | 224 | 50.1 | 10.9 | 60.5 | 40.6 | - |
| Shikra | Vicuna-13B | 7.3B | 224 | - | - | - | - | - |
| IDEFICS-9B | LLaMA-7B | - | 224 | 25.9 | - | - | 42.2 | 50.9 |
| IDEFICS-80B | LLaMA-65B | - | 224 | 30.9 | - | - | 54.8 | 60 |
| Qwen-VL | Qwen-7B | 9.6B | 448 | 63.8 | 62.1 | 67.1 | 57.7 | 78.8 |
| Qwen-VL-Chat | Qwen-7B | 9.6B | 448 | 61.5 | 57.1 | 68.2 | 63 | 78.2 |
| LLaVA-1.5 | Vicuna-7B | 7.2B | 336 | 58.2 | 21.5 | 66.8 | 55.5 | 78.5 |
| LLaVA-1.5 | Vicuna-13B | 13.4B | 336 | 61.3 | 24.1 | 71.6 | 61.1 | 80.0 |
| SPHINX | Vicuna-7B | 10B | 224 | 51.6 | - | 69.3 | - | 78.1 |
| SPHINX-1k | Vicuna-7B | 10B | 448 | 58.8 | - | 69.1 | - | 80.2 |
| Mini-Gemini | Vicuna-7B | 7.4B | 336(768) | 65.2 | - | - | - | - |
| LLaVA-UHD | Vicuna-13B | 13.4B | 672×1008 | 67.7 | - | 72.0 | - | 81.7 |
| Our Models |  |  |  |  |  |  |  |  |
| MG-LLaVA | Phi3-3.8B | 4.2B | 336(768) | 65.3 | 49.3 | 73.9 | 75.1 | 77.9 |
| MG-LLaVA | Vicuna-7B | 7.4B | 336(768) | 67.3 | 47.9 | 70.8 | 69.3 | 80.2 |
| MG-LLaVA | LLaMA3-8B | 8.2B | 336(768) | 68.1 | 49.0 | 76.3 | 75.6 | 80.7 |
| MG-LLaVA | Vicuna-13B | 13.6B | 336(768) | 69.6 | 52.1 | 74.7 | 73.4 | 81.2 |
| MG-LLaVA | Yi1.5-34B | 34.4B | 336(768) | 70.0 | 56.1 | 77.0 | 81.1 | 82.0 |

Video Question Answering Benchmarks. To demonstrate the effectiveness of our approach, we have expanded our model to encompass video comprehension. We evaluate our models on MSVD and MSRVTT, and results are shown in Tab. 3. MG-LLaVA outperforms Video-LLaVA [] on both benchmarks, which further proves the efficiency of MG-LLaVA. In video understanding, MG-LLaVA demonstrates proficiency in identifying the critical object in the video. More illustrative instances are depicted in Appx. B.

**Table 3: Comparison with other methods on Video-QA benchmarks.**
| Method | LLM | MSVD-QA | MSRVTT-QA |
| --- | --- | --- | --- |
| FrozenBiLM [] | - | 32.2 | 16.8 |
| VideoChat [] | Vicuna-7B | 56.3 | 45.0 |
| LLaMA-Adapter [] | - | 54.9 | 43.8 |
| Video-LLaMA [] | Vicuna-7B | 51.6 | 29.6 |
| Video-ChatGPT [] | Vicuna-7B | 64.9 | 49.3 |
| Video-LLaVA [] | Vicuna-7B | 70.7 | 59.2 |
| MG-LLaVA | Vicuna-7B | 71.5 | 59.8 |

### 4.3 Ablation Experiments

In this section, we conduct comprehensive ablation studies of our model. The ablation experiments are based on the training data provided by LLaVA-1.5 [], with a fixed seed protocol to ensure the stability and comparability of the experimental conditions.

Effect of Each Component. We first conduct ablation studies on object-level features and the Conv-Gate fusion module across multiple datasets, including MMBench-DEV [], SEEDBench [] and TextVQA []. To validate the effectiveness of our method on different scales of LLM, the baseline is built on Vicuna-7B and Phi3-3.8B. The results are shown in Tab. 4.

**Table 4: Ablation results on MMBench-DEV [], SEEDBench [] and TextVQA []. We execute our experiments based on the LLaVA model with Vicuna-7B and Phi3-3.8B.**
| Object-level | Conv-Gate | Vicuna-7B | Phi3-3.8B |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Features | Fusion | #TFLOPS | Params. | MMB^D | SEED | VQA^T | #TFLOPS | Params. | MMB^D | SEED | VQA^T |
| $\times$ | $\times$ | 5.76 | 7.2B | 68.2 | 64.6 | 56.7 | 3.3 | 4.0B | 68.7 | 65.3 | 54.3 |
| $\checkmark$ | $\times$ | 6.20 | 7.4B | 69.2(+1.0) | 65.0(+0.4) | 56.5(-0.2) | 3.72 | 4.2B | 70.4(+1.7) | 66.2(+0.9) | 54.5(+0.2) |
| $\checkmark$ | $\checkmark$ | 6.21 | 7.4B | 69.8(+1.6) | 65.5(+0.9) | 59.5(+2.8) | 3.73 | 4.2B | 71.0(+2.3) | 66.4(+1.1) | 56.8(+2.5) |

It is clear that the model achieves significant gains with the integration of object-level features and Conv-Gate Fusion module. When adding object-level features, the performance of MMBench-Dev, SEEDBench increases 1.0%, 0.4% separately with Vicuna-7B and 1.7%, 0.9% with Phi3. After utilizing the fusion network, the performance on these two benchmarks further increases by 1.6%, 0.9% with Vicuna-7B and 2.3%, 1.1% with Phi3. For the TextVQA benchmark, the incorporation of object-level features does not markedly enhance performance due to the suboptimal detection of textual content within images by the detector. Nevertheless, the integration of high-resolution features mitigates this limitation, culminating in an accuracy increment of 3.0% on Vicuna-7B and 2.5% on Phi3-3.8B. The integration of both modules incurs a marginal increase in computational expense and parameter count, yet it enhances the efficacy of models across various scales. We further enumerate additional comparative outcomes across various subsets of MMBench-Dev and SEEDBench, the comparative results are shown in Appx. A.

**Table 5: Comparison of different fusion modules, methods of merging object-level features, and tagging models.**
| Method | MMB^D | MMStar |
| --- | --- | --- |
| Baseline | 69.2 | 34.1 |
| w/ Resampler | 55.6 | 30.5 |
| w/ Channel Concat | 68.9 | 32.6 |
| w/ Patch Info Mining | 68.3 | 32.9 |
| w/ Conv-Gate Fusion | 69.8 | 34.5 |

Fusion Network Design. We also explore a diverse design of fusion modules and perform ablation studies on various components: (1)Channel Concat. We simply concat the low and high-resolution features in the channel dimension. (2) Patch Info Mining. We replace the gated-fusion model with Patch Info Mining in []. (3) Resampler. We substitute the gated-fusion model with a resampler in []. The results are shown in Tab. 5(a). We find our Conv-Gated fusion module performs better through these methods, which confirms its efficiency.

Method of Merging Object-level Features. We further explore various methods for incorporating object-level features: (1)F-to-B Cross-Attention. We add a cross-attention block to enhance the fusion features by integrating object-level features after the fusion module, the enhanced fusion features are then fed into LLM. (2)B-to-F Cross-Attention. Following the fusion module, another cross-attention block is employed to enhance the object-level features by integrating fusion features. The fusion features and enhanced object-level features are then concatenated as input for LLM. The frameworks of both are depicted in Appx. C and the results are reported in Tab. 5(c). Our observations indicate that cross-attention does not enhance the integration of object-level features into visual representations. Conversely, concatenating object-level features with visual tokens and deferring the decision-making to the LLM yields more favorable outcomes.

Tagging Model. We investigate the impact of the tagging model within the bounding box generation pipeline. We compare our method with assigning fixed tags based on the 80 categories from the COCO [] dataset to open-vocabulary detectors for producing bounding boxes. The comparative results are presented in Tab. 5(b). Given that the COCO dataset’s 80 categories do not comprehensively cover real-world objects, the generated bounding boxes fail to encompass all objects within an image. This limitation consequently diminishes the impact of object-level features.

## 5 Discussions

Conclusions In this work, we propose MG-LLaVA, an expansive multi-modal model adept at processing visual inputs of multiple granularities, encompassing object-level features, original images, and high-resolution data. To effectively amalgamate features of varying granularities, we propose the Multi-Granularity Vision Flow module, thereby equipping the LLM with the ability to discern multi-modal interactions from a consolidated visual framework. Utilizing a range of LLMs extending from 3.8B to 34B parameters, our model exhibits pronounced scalability and remarkable performance in visual understanding, outperforming established models and significantly outperforming GPT-4V and GeminiPro Vision on benchmarks such as MMBench and SEEDBench. The validity of our methodology is substantiated through rigorous empirical studies.

Future Works. While MG-LLaVA attains remarkable results across various multi-modal benchmarks, it encounters challenges in effectively harnessing object features correlated with textual inputs. Our MG-LLaVA model establishes a foundational baseline for future explorations into more sophisticated techniques of integrating inputs of multiple granularities.

Broader Impacts. As a robust multi-modal language model, MG-LLaVA exhibits considerable prowess in visual perception and comprehension, offering an innovative methodology to further refine MLLMs. However, MG-LLaVA’s potential societal implications merit attention, as it may facilitate the creation of multimodal applications, including those with possible adverse effects. There exists a concern that MG-LLaVA could be utilized in the development of deleterious multimodal AI systems. We encourage users to carefully assess the potential impacts when deploying our model.

## Appendix A Appendix / Detailed results on subsets

In this section, we compare the influence of object-level features on several subsets of MMBench-Dev and Seed-bench, as shown in Fig. 4. It can be observed that the integration of object-level features significantly enhances the model’s capability in multiple aspects of perception including Attribute Reasoning, Fine-grained Perception, Physical Relation Perception, Visual Reasoning, etc.

Figure: Figure 4: Ablation study on several subsets of MMBench-DEV-EN and Seed-bench. Fine-grained Perception(I) denotes Fine-grained Perception(instance-level), Property Reasoning(P) means Property Reasoning Perception and SIT Understanding denotes Structuralized Image-Text Understanding.
Refer to caption: extracted/5694511/latex/images/ablationsubsets.png

## Appendix B Appendix / Additional Showcases

In this section, we present additional instances to substantiate the capability of MG-LLaVA. As presented in Fig. 5 and Fig. 6, MG-LLaVA is proficient in addressing queries that necessitate meticulous attention to specifics and in capturing fine-grained details within image or video. These further instances reinforce the superior performance of our MG-LLaVA in visual comprehension.

Figure: Figure 5: More cases of video understanding.
Refer to caption: extracted/5694511/latex/images/exvideo.png

Figure: Figure 6: More cases of image understanding.
Refer to caption: x2.png

## Appendix C Appendix / Method of Merging Object-level Features

The illustration of F-to-B Cross-Attention and B-to-F Cross-Attention is depicted in Fig. 7.

Figure: Figure 7: Illustration of F-to-B Cross-Attention and B-to-F Cross-Attention.
Refer to caption: extracted/5694511/latex/images/btf.png

## Appendix D Appendix / Inference Pipeline

The inference pipeline of MG-LLaVA is displayed in Fig. 8. The tagging model first processes the input image to provide tags within the image, which are subsequently utilized as the text input of the detector to derive bounding boxes corresponding to the tagged obejcts within the image.

Figure: Figure 8: Inference pipeline of MG-LLaVA.
Refer to caption: extracted/5694511/latex/images/inf.png

## Appendix E Appendix / Comparison of Tagging Models

**Table 6: Ablation results on MMBench-DEV [], SEEDBench [] and TextVQA []. We execute our experiments based on the LLaVA model with Vicuna-7B and Phi3-3.8B.**
| Method | Images | Number of Boxes |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 1-10 | 12-20 | 21-30 | 30-50 | >50 |  |  |
| COCO 80 + OWL-ViT v2 | 389722 | 71118 | 245952 | 44059 | 28593 | 0 | 0 |
| RAM + OWL-ViT v2 | 389722 | 43654 | 184706 | 91245 | 34648 | 22827 | 12645 |

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to my comments above.
soundness: 3
strengths: - The paper is well-organized. The proposed model is evaluated on multiple tasks including general visual understanding benchmarks, VQA, and video datasets. Ablation study and runtime evaluation are also provided.
summary: This paper presents an MLLM architecture to improve the multi-granularity visual understanding abilities of multimodal models. The method follows the idea proposed in Mini-Gemini to fuse high and low-resolution visual encoders and adds object recognition from other foundation models to enhance object-level understanding ability. A series of models ranging from 3.8B to 34B are proposed and the models are evaluated on multiple popular benchmarks.
weaknesses: - The technical contribution of the paper is not very significant. The paper claims the main contribution is combining low, high-resolution, and object-level features. But the design of combining low and high-resolution features mainly comes from mini-Gemini and some modifications on the fusion module are proposed in the paper. The introduction of object-level features requires extra models and makes the base architecture more complex. 

- I am not convinced about the necessity of introducing the extra object-level information. Recent state-of-the-art MLLMs like Qwen2 VL [r1], LLaVA-Onevision [r2] and Pixtral [r3] all adopt a single encoder solution. I think the MLLMs themselves should have the ability to capture object-level information from the images with sufficient data and a proper training strategy. The method proposed in the paper may reduce the requirement for training data but the more complex architecture also makes the solution less general. Besides, the method also didn't show large improvements on SEED (69.4 vs. 68.9) and MMStar (35.1 vs. 37.6) compared to Mini-Gemini with the same base LLM. 

- Many recent MLLMs like [r2, r4] are not compared. Compared to these methods, the proposed solution is not strong enough and imore complex. 

[r1] Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

[r2] LLaVA-OneVision: Easy Visual Task Transfer

[r3] Pixtral 12B

[r4] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: The concerning questions are stated in the weakness section. Based on the weaknesses listed above, I lean to reject this manuscript at its current version.
soundness: 2
strengths: 1. The key claim for the paper that multi-granularity features with low-res, high-res, and object features can improve detailed understanding and object recognition skills is reasonable. The authors design the conv-gated fusing module and demonstrate its effectiveness through complete ablation studies.
2. The series of models and benchmarks are clear and complete. The authors train the variants for MG-LLaVA based on Phi, Vicuna, LLaMA3, and Yi1.5 and conduct experiments on various multi-modal benchmarks. 
3. The overall architecture of the paper is well-structured and easy to follow.
summary: The paper proposes MG-LLaVA to improve the capability for recognizing multi-granularity features for current MLLMs and flatten the restriction of the resolution in visual inputs. MG-LLaVA includes the low-resolution, high-resolution, and object-level features altogether and fuses them with a conv-gate fusion module for the general visual features. The object ROIs are extracted by a pre-trained detection model for better object-level understanding skills. The paper proposes a series of MLLMs ranging from 3.8B to 34B based on various LLMs and shows strong performance across image and video multimodal benchmarks.
weaknesses: 1. The idea of fusing multi-granularity features is not novel, as integrating low-resolution and high-resolution images has been demonstrated effect by a range of works, including LLaVA-NeXt, LLaVA-HR, Mini-Gemini, LLaVA-UHD, etc. The difference in MG-LLaVA lies in the usage of detected objects. However, the detection operation introduces extra computational costs and external models with extra information, which is not an optimal solution. 
2. The performance comparisons against existing MLLMs are relatively weak. For example, the results of MG-LLaVA equipped with Vicuna-7B do not surpass baselines with similar efforts on some benchmarks, including SQA, TextVQA, MMStar, etc. The overall number of training data is significantly heavier than the baselines (2.6M), which makes the comparisons more unfair. 
3. Some key ablations seem lacking. The authors are encouraged to clearly show the contributions for performance with every part of the visual feature to show the difference between previous works.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. The performance improvement on similarly sized LLMs in Table 2 and Table 3 appears modest.
2. The ablation study would benefit from visual comparisons to illustrate the impact of each component, such as case studies or visualizations of feature-level effects.
3. Some failure cases should be shown to provide insights into the method’s limitations.
4. It is unclear if the method can handle larger images, such as 1024p or 2k resolutions.
soundness: 2
strengths: 1. The integration and fusion of multi-granularity features with object-centric features is novel for MLLMs.
2. Experimental results demonstrate the effectiveness of the proposed pipeline.
3. The paper is well-written and clearly presented.
summary: This paper presents a novel Multimodal Large Language Model (MLLM) that improves visual processing capabilities by incorporating a multi-granularity vision flow, which includes low-resolution, high-resolution, and object-centric features. This approach enhances the performance of current Large Language Models (LLMs), as demonstrated in the experiments.
weaknesses: 1. The performance improvement on similarly sized LLMs in Table 2 and Table 3 appears modest.
2. The ablation study would benefit from visual comparisons to illustrate the impact of each component, such as case studies or visualizations of feature-level effects.
3. Some failure cases should be shown to provide insights into the method’s limitations.
4. It is unclear if the method can handle larger images, such as 1024p or 2k resolutions.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 4
questions: 1. The SEEDBench mentioned in the article uses SEEDBench-Image, but I checked the scores for leaderboard and the other methods mentioned in the paper, and they seem to correspond to SEEDBench-Avg (which contains both video and image), so it's not clear to me whether the comparison here includes scores from the video task.
2. If an open vocabulary detector is used, why is a tagger used to determine the bounding box instead of generating ROI directly based on text embedding?
3. The article suggests that this approach is intuitively better for small target comprehension or counting tasks, are there any datasets in this area that show that this approach has more significant performance gains on specific tasks?
4. I found that the Monkey model uses a similar idea to enhance the performance of the model and also proposes to augment the data with traditional CV methods for refinement, is there a comparison to this approach in the paper? For example, changing the base LLM to Qwen-7b to compare with Monkey (Li et al. 2023) and more models on this field.

[1] Li, Zhang, et al. "Monkey: Image resolution and text label are important things for large multi-modal models." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.
soundness: 3
strengths: The goal of this paper is to release the power of MLMs on fine-grained tasks. A high resolution visual encoder is introduced to make up for the complement of previous work. And some fusion and compression strategies are introduced to ease the computational pressure. In addition to this, the article demonstrates that this new framework achieves significantly higher scores on MLMs at several scales, which fully demonstrates the effectiveness of the method. Moreover, this is the first approach to introduce object-level features in the field of MLMs, and experimentally, the article demonstrates the ability of their method to achieve higher scores than private models under MMBench and SEEDBench.
summary: This paper aims to improve the visual capabilities of MLMs (multimodal large models) by proposing a new model MG-LLaVA. limited to resources, most of MLMs nowadays just have low resolution inputs, which are challenging on fine-grained tasks. Therefore, this papet proposes a novel framework that introduces object-level feature in addition to high resolution visual encoder. Based these, the article also uses a gating-based fusion strategy as well as explicit integration on object-level feature. These approaches reduce the computational pressure introduced by high resolution images and simultaneously improve performance on fine-grained tasks. On MMBench and SEEDBench, the model outperforms even the private models GPT-4V and GeminiPro-V. The article also conducts extensive experiments to show that their framework achieves competitive scores on multiple datasets of images or videos.
weaknesses: 1. As mentioned in the article itself, the introduction of multi-granularity and multi-scale to enhance model performance is a common approach to convolutional networks, and merely migrating this approach to the field of MLMs is hardly an innovative contribution. Some of the algorithms used in the article from object detection only do some information enhancement on the input side, while many MLMs can already accomplish the object detection task by themselves nowadays.
2. The scores achieved on both the MMBench as well as SEEDBench datasets, while respectable, are not compared to some of the more competitive models. I identified MMB as version 1 and SEEDBench as Avg based on the scores of Qwen-VL and MiniCPM-V2, and there are a number of scores on both leaderboards that are higher than the scores of MG-LLaVA work, eg. Honeybee (Cha et al., 2024), AllSeeing-v2 (Wang et al. 2024) based on Vicuna-13b at MMB-test. and then you can also find a lot of similar models with higher scores on the same substrate.
3. In addition to Perception Benchmarks. this problem can also be found in Visual QA and Video QA. such as on the MSRVTT-QA dataset. there are already many models with very high scores in 2024. Some of them also use some methods to improve the model's ability on fine-grained tasks. eg. Flash-VStream (Zhang et al. 2024) Monkey (Li et al. 2023). The article does not seem to compare these new 2024 models.

To summarize, I think the approach proposed in the article is valid, but MG-LLaVA does not do the job of making a difference, either from an innovation perspective or from a performance perspective.

[1] Cha, Junbum, et al. "Honeybee: Locality-enhanced projector for multimodal llm." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.

[2] Wang, Weiyun, et al. "The all-seeing project v2: Towards general relation comprehension of the open world." *arXiv preprint arXiv:2402.19474* (2024).

[3] Zhang, Haoji, et al. "Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams." *arXiv preprint arXiv:2406.08085* (2024).

[4] Li, Zhang, et al. "Monkey: Image resolution and text label are important things for large multi-modal models." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Although the method performs well on the general VQA, it lacks a comprehensive assessment of fine-grained perception capabilities. It would be more fair and convincing to compare it with region-level methods like Next-Chat and Osprey on the RefCOCO dataset. This could be accomplished by using the bounding box of the corresponding target as input.

2. It is evident that using object-level features can enhance the perception ability of MLLMs. However, incorporating additional detectors introduces extra computational costs and biases. An equitable efficiency comparison is necessary. if these added costs surpass the benefits from data expansion, parameter extension, or data filtering, it results in negative optimization, as I believe is the case with MG-LLaVA. From the performance comparison, when using Vicuna 7B, MMStar exhibits lower performance than other models, indicating data leakage risk and validating the risk of bias introduced by reliance on detectors.

3. Although MG-LLaVA shows improvements in general capabilities, these enhancements are marginal. The added expense of using additional detection models and object-level features should yield a greater performance boost. Moreover, during inference, reliance on detection results from other models is cumbersome. Transforming external dependencies into self-mining processes could significantly enhance the practical utility of the model.
soundness: 2
strengths: 1. The structure of paper is simple and easy to read, and the model implementation is very easy to follow.
2. The idea is very straightforward, and the experiments are solid. It is reasonable to introduce multi-granularity object-level features to enhance the perceptual capabilities of Multimodal Large Language Models (MLLMs).
summary: Summary:

MG-LLaVA is a multi-modal large language model (MLLM) designed to improve visual processing capabilities by using a multi-granularity vision flow. This includes low-resolution, high-resolution, and object-centric features to enhance perception tasks requiring detailed visual information. Extensive experiments have validated its effectiveness. 

Contributions:

1. Leveraging an additional open vocabulary detection model introduces multi-granularity object-level features to enhance the perceptual capabilities of MLLMs.
2. Extensive experiments demonstrate the effectiveness of the method.
weaknesses: 1. The idea appears incremental, as it simply integrates high-resolution image interpretation with region-level image understanding, resembling a trick 
2. Experimental evaluations and fair comparisons are notably lacking. Given that multi-granularity features are utilized to augment the model's perceptual abilities, evaluations should be conducted on fine-grained perception datasets. General VQA is inadequate for assessing the fine-grained perceptual capabilities of MLLM. 
3. Excessive reliance on additional detector inputs may result in suboptimal, non-end-to-end outcomes.