Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models

## Abstract

Abstract In this work, we introduce Mini-Gemini, a simple and effective framework enhancing multi-modality Vision Language Models (VLMs).
Despite the advancements in VLMs facilitating basic visual dialog and reasoning, a performance gap persists compared to advanced models like GPT-4 and Gemini.
We try to narrow the gap by mining the potential of VLMs for better performance and any-to-any workflow from three aspects, i.e. , high-resolution visual tokens, high-quality data, and VLM-guided generation.
To enhance visual tokens, we propose to utilize an additional visual encoder for high-resolution refinement without increasing the visual token count.
We further construct a high-quality dataset that promotes precise image comprehension and reasoning-based generation, expanding the operational scope of current VLMs.
In general, Mini-Gemini further mines the potential of VLMs and empowers current frameworks with image understanding, reasoning, and generation simultaneously.
Mini-Gemini supports a series of dense and MoE Large Language Models (LLMs) from 2B to 34B.
It is demonstrated to achieve leading performance in several zero-shot benchmarks and even surpasses the developed private models.
Code and models are available at https://github.com/dvlab-research/MiniGemini .

## 1 Introduction

With the rapid evolution in Large Language Models (LLMs) [, , ], empowering the impressive capabilities for multi-modality inputs is becoming an essential part of current Vision Language Models (VLMs) [, ].
To bridge the modality gap, several studies are conducted to marry vision with LLMs from images [, , ] to videos [, ].
Despite these advancements, a significant gap remains between academic initiatives and the prowess of well-established models like GPT-4 [] and Gemini [], which are trained with huge amounts of data and resources.

For vision itself, image resolution is a core part of explicitly despite the surrounding environment with minimal visual hallucination.
To this end, more attempts are performed to further improve the visual understanding in current VLMs.
For instance, LLaVA-Next [] and Otter-HD [] are proposed to enhance the ability based on previous work [, ] by improving the image resolution.
Increasing the number of visual tokens with higher resolution images undeniably enriches visual embeddings in LLMs.
However, this improvement comes with escalated computational demands and associated costs, particularly when processing multiple images.
Moreover, the existing data quality, model capabilities, and application scopes remain inadequate for accelerated training and development processes.
This scenario prompts a critical inquiry: how to push forward the VLMs approaching well-developed models with acceptable cost in an academic setting?

To answer this question, we explore the potential of VLMs from three strategic aspects, i.e., efficient high-resolution solution, high-quality data, and expanded applications.
Firstly, we utilize ConvNet to efficiently generate higher-resolution candidates, thus enhancing visual detail while maintaining the visual token count for LLMs.
To bolster data quality, we amalgamate high-quality datasets from diverse public sources, ensuring a rich and varied data foundation.
Furthermore, our approach integrates these enhancements with cutting-edge LLMs and generative models, aiming to elevate VLM performance and user experience.
This multifaceted strategy enables us to delve deeper into the capabilities of VLMs, achieving significant advancements within manageable resource constraints.

In general, our method employs an any-to-any paradigm, which is adept at handling both image and text as input and output.
In particular, we introduce an efficient visual token enhancement pipeline for input images, featuring a dual-encoder system.
It comprises twin encoders, one for high-resolution images and the other for low-resolution visual embedding, mirroring the cooperative functionality of the Gemini constellation.
During inference, they work in an attention mechanism, where the low-resolution one generates visual queries, and the high-resolution counterpart provides candidate keys and values for reference.
To augment the data quality, we collect and produce more data based on public resources, including high-quality responses [, ], task-oriented instructions [, , , ], and generation-related data [, ].
The increased amount and quality improve the overall performance and extend the capability of model.
Additionally, our model supports concurrent image and text generation, facilitated by the seamless integration of our VLM with advanced generative models [].
It leverages VLM guidance for image generation by providing the generated text from LLMs.

The Mini-Gemini framework, can be easily instantiated with a range of LLMs from 2B to 34B parameter scales, as detailed elaborated in Section 3.
Extensive empirical studies are conducted in Section 4 to reveal the effectiveness of the proposed method.
Remarkably, our approach attains leading performance in various settings and even surpasses the well-developed Gemini Pro [], Qwen-VL-Plus [], and GPT 4V [] in the complex MMB [] and MMU [] dataset, respectively.
These results underscore Mini-Gemini’s potential to set new benchmarks in the realm of VLMs, highlighting its advanced capabilities in handling complex multi-modal tasks.

## 2 Related Work

#### Large Language Models.

Recent progress in Natural Language Processing (NLP) has been dramatically accelerated by advancements in large language models (LLMs).
The seminal introduction of the Transformer framework [] served as a cornerstone, enabling a new wave of language models such as BERT [] and OPT [] that exhibit profound linguistic understanding.
The inception of the Generative Pre-trained Transformer (GPT) [] introduced a novel paradigm through auto-regressive language modeling, establishing a robust method for language prediction and generation.
The emergence of models like ChatGPT [], GPT-4 [], LLaMA [], and Mixtral [] further exemplified the field’s rapid evolution, each demonstrating enhanced performance on complex language processing tasks, attributable to their training on extensive textual datasets.
Instruction tuning [, ] has emerged as a key technique for refining the output of pre-trained LLMs, as evidenced by its application in the development of open-source models such as Alpaca [] and Vicuna [].
They iterate on the LLaMA [] with custom instruction sets.
Additionally, the integration of LLMs with specific tools for visual tasks [, ] highlights their adaptability and potential for broad application, underscoring the utility of LLMs in extending beyond traditional text-based processing to include multimodal interactions.
In this work, we take several pre-trained LLMs [, , ] as benchmarks and build multi-modality frameworks upon them to further extend the impressive reasoning ability.

#### Vision Language Models.

The convergence of computer vision and natural language processing has given rise to VLMs, which marry visual and linguistic models to achieve cross-modal comprehension and reasoning capabilities.
This integration has been pivotal in advancing tasks that require both visual understanding and language processing, as evidenced by models trained on diverse datasets for understanding [] and reasoning [, , ].
Groundbreaking models such as CLIP [] have further bridged the gap between language models and vision tasks, showcasing the feasibility of cross-modal applications.
Recent developments underscore a growing trend toward leveraging the robust capabilities of LLMs within the realm of VLMs.
Innovations like Flamingo [] and BLIP-2 [] have capitalized on massive collections of image-text pairs to fine-tune cross-modal alignment, significantly boosting learning efficiency.
Building upon these advancements, several models [, ] have focused on generating high-quality instructional data based on BLIP-2, leading to marked improvements in performance.
Furthermore, LLaVA [, ] adopts a simple linear projector to facilitate image-text space alignment with minimal learnable parameters.
It leverages tailored instruction data and exemplifies an efficient strategy that demonstrates the model’s potent capabilities.
Different from them, we aim to explore the potential for both comprehension and generation.

#### LLM as Generation Assistant.

Combining LLMs with image outputs has emerged as a pivotal area in recent multimodal research. Methods like InternLM-XComposer [, ] utilize image retrieval to produce interleaved text and image outputs, bypassing direct generation.
Conversely, auto-regressive token prediction approaches, exemplified by EMU [, ] and SEED [, ], enable LLMs to decode images through massive image-text data directly.
These methods require enormous training resources, and their auto-regressive nature leads to undesirable latency.
Recent studies [, , ] strive to align with latent diffusion models [] to streamline image generation.
They typically require designing text embeddings and additional optimization to achieve the desired generation effect.
This joint training can compromise the performance of VLMs in text generation.
Mini-Gemini distinguishes itself by adopting a text-data-driven approach to enable the model to generate high-quality images.
We leverage a mere 13K pure text data to activate the LLM’s ability as a high-quality re-captioner [] without undermining the fundamental performance of VLMs.

## 3 Mini-Gemini

Figure: Figure 2: The framework of Mini-Gemini with any-to-any workflow.

The framework of Mini-Gemini is conceptually simple: dual vision encoders are utilized to provide low-resolution visual embedding and high-resolution candidates;
patch info mining is proposed to conduct patch-level mining between high-resolution regions and low-resolution visual queries;
LLM is utilized to marry text with images for both comprehension and generation at the same time.

### 3.1 Dual Vision Encoders

In the Mini-Gemini framework, both text and image inputs can be processed, with the option to handle them individually or in combination.
For illustrative clarity, we consider the concurrent processing of both modalities.
As depicted in Figure 2, the processing begins with a high-resolution image $X_{H}\in\mathbb{R}^{H\times W\times 3}$, from which a corresponding low-resolution image $X_{L}\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times 3}$ is generated via bilinear interpolation, ensuring $H^{\prime}\leq H$.
Then, we process them and encode into multi-grid visual embeddings in two parallel image flows.
In particular, for the low-resolution (LR) flow, we maintain the traditional pipeline [, ] and employ a CLIP-pretrained ViT [] to encode the visual embedding $X^{\prime}_{L}\in\mathbb{R}^{N\times C}$, where $N$ denotes the number of visual patches.
In this way, the long-range relation among $N$ visual patches can be well preserved for subsequent interaction in LLMs.
As for the high-resolution (HR) flow, we adopt the CNN-based encoder for adaptive and efficient HR image processing.
For instance, to align with the LR visual embedding, the LAION-pretrained [] ConvNeXt [] is used to serve as an HR vision encoder.
Therefore, we can obtain the HR feature map $X^{\prime}_{H}\in\mathbb{R}^{N^{\prime}\times N^{\prime}\times C}$ by upsampling and concatenating the features from different convolutional stages to 1/4 input scale.
Here, $N^{\prime}=H/4\times W/4=N\times M^{2}$ denotes the number of HR features, where $M$ reflects the pixel-wise feature count within each HR segment, as illustrated in Figure 2.

Figure: (a) Details in patch info mining.

### 3.2 Patch Info Mining

With the above generated LR embedding $X^{\prime}_{L}$ and HR feature $X^{\prime}_{H}$, we propose patch info mining to extend the potential of VLMs with enhanced visual tokens.
In particular, to maintain the number of final visual tokens for efficiency in LLMs, we take the low-resolution visual embedding $X^{\prime}_{L}$ as query $Q\in\mathbb{R}^{N\times C}$, aiming to retrieve relevant visual cues from HR candidate.
Meanwhile, the HR feature map $X^{\prime}_{H}$ is taken as key $K\in\mathbb{R}^{N\times M^{2}\times C}$ and value $V\in\mathbb{R}^{N\times M^{2}\times C}$, as depicted in Figure 2.
Here, the low-resolution patch in $Q$ correlates with a corresponding high-resolution sub-region in $K$ and $V$, encompassing $M^{2}$ pixel-wise features.
Therefore, the patch info mining process can be formulated as

$$ $T_{V}={\mathrm{MLP}}(Q+{\mathrm{Softmax}}(\phi(Q)\times\phi(K)^{T})\times\phi( V)),$ (1) $$

where $\phi$ and ${\mathrm{MLP}}$ indicate a projection layer and a multi-layer perceptron, respectively.
As presented in Figure 3(a), this formula encapsulates the process of synthesizing and refining the visual cues, leading to generation of enhanced visual tokens $T_{V}$ for subsequent LLM processing.
It ensures that the mining for each query is confined to its corresponding sub-region in $X^{\prime}_{H}$ with $M^{2}$ features, thus preserving efficiency.
This design allows for the extraction of HR details without expanding the visual token count of $T_{V}$, maintaining a balance between richness of detail and computational feasibility.

Furthermore, visual token extension is also supported in the designed patch info mining.
As depicted in Figure 3(b), we can extend the visual token to $5N$ to capture more details.
This is achieved by incorporating the original image along with its $2\times$ upscaled counterpart, resulting in a batched input $X_{L}\in\mathbb{R}^{5\times H^{\prime}\times W^{\prime}\times 3}$.
And we can get the encoded visual embedding $X^{\prime}_{L}\in\mathbb{R}^{5\times N\times C}$ with the LR vision encoder, as detailed in Section 3.1.
Thanks to the flexible design of CNN-based HR vision encoder, it can adeptly handle the augmented visual token count during the patch info mining.
The only difference in the aforementioned procedure is the sub-region in $X^{\prime}_{H}$ should be changed according to the expanded visual embedding $X^{\prime}_{L}$.
We can also upsample the HR input to better support the higher resolution if needed, as experimentally analyzed in Table 2.

### 3.3 Text and Image Generation

With the mined visual tokens $T_{V}$ and input text tokens $T_{T}$, we concatenate them as the input to LLMs for auto-regressive generation, as presented in Figure 2.
Distinguished from traditional VLMs [, , ], the proposed Mini-Gemini supports both text-only and text-image generation as input and output, i.e., any-to-any inference.
Despite the image comprehension, we anchor Mini-Gemini’s ability to generate images on its outstanding image-text understanding and reasoning capabilities.
Unlike recent works [, , , ], which address the domain gap between text embeddings of LLMs and generation models, we choose to optimize the gap in the domain of language prompts.
Precisely, Mini-Gemini translates user instructions into high-quality prompts that produce context-relevant images in latent diffusion models [, ].
This approach is reflected in subsequent high-quality image generation frameworks, such as DALLE 3 [] and SORA [], which leverage the generation and understanding capabilities of VLMs to obtain higher-quality text conditions for generation tasks.

#### Text-image Instructions.

For better cross-modality alignment and instruction finetuning, we collect high-quality datasets from publicly available sources.
In particular, for cross-modality alignment, we utilize 558K image-caption pairs from the LLaVA-filtered CC3M dataset [] and 695K sampled GPT-4V-responded captions from the ALLaVA dataset [].
It brings about 1.2M image captions in total for projector pretraining.
As for instruction finetuning, we sample 643K single- and multi-turn conversations (excluding 21K TextCaps [] data) from the LLaVA [] dataset, 100K QA pairs from ShareGPT4V [], 10K LAION-GPT-4V [] captions, 700K GPT-4V-responded instruction pairs from ALLaVA dataset [], and 6K text-only multi-turn conversations from LIMA [] and OpenAssistant2 [].
To bolster the OCR-related abilities, we further collect 28K QA pairs that comprise 10K DocVQA [], 4K ChartQA [], 10K DVQA [], and 4K AI2D [] data.
In general, there are about 1.5M instruction-related conversations for image comprehension.
Moreover, we also collect 13K pairs for image-related generation that will be elaborated on subsequently.

Figure: Figure 4: Two types of our pure-text data are used for image generation. Left: Simple instruction re-caption and Right: In-context prompt generation. SDXL generates images with the output prompt.

#### Generation-related Instructions.

To support image generation, we further construct a 13K instruction-following dataset using GPT-4 Turbo.
As depicted in Figure 4, the training data encompasses two tasks:
(a) Simple instruction re-caption: we adopt 8K descriptive image captions from LAION-GPT-4V [] and let GPT-4 inversely infer the corresponding user’s short input and the target caption in the Stable Diffusion (SD) domain.
(b) In-context prompt generation: based on a few high-quality real-world conversation contexts in LIMA [] and OpenAssistant2 [], we generate prompts that produce images suitable for the conversation context, bringing 5K instructions in total.
For both kinds of data, in each query to GPT-4, we randomly sample 5 high-quality SD text-to-image prompts from GigaSheet [] as in-context examples to obtain target prompts for generation.
We format our data to use <GEN> as a trigger to initiate the generation process and wrap the target caption within <h>...</h>.
Following text generation, Mini-Gemini extracts target captions and utilizes SDXL [] to generate the corresponding image.
More details are discussed in Appendix A.

## 4 Experiments

**Table 1: Comparison with leading methods on zero-shot benchmarks. ^∗ and ^† denote images in train subset are included and the data is not publicly available, respectively. Our results are marked with ◼.**
| Method | LLM | Res. | VQA${}^{\text{T}}$ | MMB | MME | MM-Vet | MMMUv | MMMUt | MathVista |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Normal resolution setting |  |  |  |  |  |  |  |  |  |
| MobileVLM[] | MLLaMA 2.7B | 336 | 47.5 | 59.6 | 1289 | – | – | – | – |
| InstructBLIP [] | Vicuna-7B | 224 | 50.1 | 36.0 | – | 26.2 | – | – | 25.3 |
| InstructBLIP [] | Vicuna-13B | 224 | 50.7 | – | 1213 | 25.6 | – | – | – |
| Qwen-VL^† [] | Qwen-7B | 448 | 63.8^∗ | 38.2 | – | – | – | – | – |
| Qwen-VL-Chat^† [] | Qwen-7B | 448 | 61.5^∗ | 60.6 | 1488 | – | 35.9 | 32.9 | – |
| Shikra [] | Vicuna-13B | 224 | – | 58.8 | – | – | – | – | – |
| IDEFICS-80B [] | LLaMA-65B | 224 | 30.9 | 54.5 | – | – | – | – | – |
| LLaMA-VID [] | Vicuna-7B | 336 | – | 65.1 | 1521 | – | – | – | – |
| LLaMA-VID [] | Vicuna-13B | 336 | – | 66.6 | 1542 | – | – | – | – |
| LLaVA-1.5 [] | Vicuna-7B | 336 | 58.2 | 65.2 | 1511 | 31.1 | – | – | – |
| LLaVA-1.5 [] | Vicuna-13B | 336 | 61.3 | 69.2 | 1531/295 | 36.1 | 36.4 | 33.6 | 27.6 |
| Mini-Gemini | Gemma-2B | 336 | 56.2 | 59.8 | 1341/312 | 31.1 | 31.7 | 29.1 | 29.4 |
| Mini-Gemini | Vicuna-7B | 336 | 65.2 | 69.3 | 1523/316 | 40.8 | 36.1 | 32.8 | 31.4 |
| Mini-Gemini | Vicuna-13B | 336 | 65.9 | 68.5 | 1565/322 | 46.0 | 38.1 | 33.5 | 37.0 |
| Mini-Gemini | Mixtral-8x7B | 336 | 69.2 | 75.6 | 1639/379 | 45.8 | 41.8 | 37.1 | 41.8 |
| Mini-Gemini | Hermes-2-Yi-34B | 336 | 70.1 | 79.6 | 1666/439 | 53.0 | 48.7 | 43.6 | 38.9 |
| High resolution setting |  |  |  |  |  |  |  |  |  |
| OtterHD [] | Fuyu-8B | 1024 | – | 53.6 | 1314 | – | – | – | – |
| CogVLM-Chat [] | Vicuna-7B | 490 | 70.4^∗ | 63.7 | – | 51.1 | 41.1 | – | 34.5 |
| LLaVA-NeXT [] | Vicuna-7B | 672 | 64.9 | 68.1 | 1519/332 | 43.9 | 35.8 | – | 34.6 |
| LLaVA-NeXT [] | Vicuna-13B | 672 | 67.1 | 70.7 | 1575/326 | 48.4 | 36.2 | – | 35.3 |
| LLaVA-NeXT [] | Hermes-2-Yi-34B | 672 | 69.5 | 79.6 | 1631/397 | 57.4 | 51.1 | 44.7 | 46.5 |
| Mini-Gemini-HD | Vicuna-7B | 672 | 68.4 | 65.8 | 1546/319 | 41.3 | 36.8 | 32.9 | 32.2 |
| Mini-Gemini-HD | Vicuna-13B | 672 | 70.2 | 68.6 | 1597/320 | 50.5 | 37.3 | 35.1 | 37.0 |
| Mini-Gemini-HD | Mixtral-8x7B | 672 | 71.9 | 74.7 | 1633/356 | 53.5 | 40.0 | 37.0 | 43.1 |
| Mini-Gemini-HD | Hermes-2-Yi-34B | 672 | 74.1 | 80.6 | 1659/482 | 59.3 | 48.0 | 44.9 | 43.3 |
| Private models |  |  |  |  |  |  |  |  |  |
| Gemini Pro [] | Private | – | 74.6 | 75.2 | – | 64.3 | 47.9 | – | 45.2 |
| Qwen-VL-Plus [] | Private | – | 78.9 | 66.2 | – | – | 45.2 | 40.8 | 43.3 |
| GPT-4V [] | Private | – | 78.0 | 75.1 | – | 67.6 | 56.8 | 55.7 | 49.9 |

In this section, we first outline our experimental framework, commencing with the experimental setup.
Subsequently, we compare Mini-Gemini with leading methods on various benchmarks.
Component-wise analysis and qualitative results are given at the end of this section.

### 4.1 Experimental Setup

#### Implementation Details.

In this study, we instantiate Mini-Gemini with the CLIP-pretrained ViT-L [] for LR vision encoder and the LAION-pretrained ConvNext-L [] for HR vision encoder.
For efficient training, we keep two vision encoders fixed and optimize the projectors of patch info mining in all stages.
Meanwhile, we optimize the LLM during the instruction tuning stage only.
Regarding the training scheme, we optimize all the models for 1 epoch with the AdamW optimizer and a Cosine learning schedule.
In most cases, the initial learning rates for modality alignment and instruction tuning are respectively set at $1e^{-3}$ and $2e^{-5}$, with an adjusted rate of $1e^{-5}$ for the Mixtral-8$\times$7B and Hermes-2-Yi-34B to ensure stable instruction tuning.
The framework involves training on 8$\times$A800 GPUs for standard machine configurations.
For the largest model with Hermes-2-Yi-34B, we leverage 4 machines and complete the optimization within 2 days with DeepSpeed Zero3 strategy.
For the HD version, the total cost is enlarged to about 4 days because of the extended visual tokens in LLMs.

#### Datasets.

For model optimization, we construct high-quality data for cross-modality understanding and generation.
It mainly includes 1.2M caption pairs for modality alignment and 1.5M single- or multi-round conversations for instruction tuning, as elaborated in Section 3.3.
Moreover, we report results on widely-adopted zero-shot image-based benchmarks, including VQA${}^{\text{T}}$ (TextVQA) [], MMB (MMBench) [], MME [], MM-Vet [], MMMU [], and MathVista [] datasets.

### 4.2 Main Results

#### Normal Resolution.

In Table 1, we compare with previous leading approaches across several settings, including normal and high resolution, and also consider private models.
At normal resolution, Mini-Gemini consistently outperforms existing models across a wide range of LLMs.
In the efficient model category, Mini-Gemini, when configured with Gemma-2B [], demonstrates superior performance compared to the efficient MobileVLM [] and even surpasses InstructBLIP [] equipped with Vicuna-7B and even 13B.
The scalability of Mini-Gemini is evident when larger LLMs are employed.
Given the same LLM, the proposed Mini-Gemini is validated to surpass LLaVA-1.5 [] with a large margin across all benchmarks.
Notably, with the Hermes-2-Yi-34B LLM, Mini-Gemini achieves exceptional results, outpacing high-resource private models like Qwen-VL-Plus [] and Gemini Pro [] in some challenging benchmarks like MMMU [] and MMB [].

#### High Resolution.

To validate the framework for extended visual tokens, we perform experiments with an input size of 672 for LR visual encoder and 1536 for HR visual encoder in Table 1.
As discussed above, the HR visual encoder primarily serves to offer high-resolution candidate information.
Importantly, despite the increased resolution, the effective number of visual tokens processed by the LLM remains consistent with the LR input size of 672, ensuring computational efficiency.
The benefits of this approach are particularly evident in detail-oriented tasks.
For example, in the TextVQA [] benchmark, our method achieved a performance rate of 74.1% with the Hermes-2-Yi-34B configuration, closely matching the performance of the well-established Gemini Pro [].
Detailed results in Table 1 show that Mini-Gemini excels in more challenging benchmarks as well.
For instance, the proposed method is on par with Qwen-VL-Plus [] on the MathVista  [] and MMMU [] benchmark and even surpasses Gemini Pro and GPT-4V on the widely-adopted MMB [] benchmark.

**Table 2: Comparison with different info mining settings. The baseline is LLaVA-1.5 [] with Vicuna-7B using the same training data and strategy. Token Num indicates the number of visual tokens $T_{V}$ in Equation (1). ^∗ denotes that images in the train subset are included. Results with patch info mining are marked in ◼. We respectively set ConvNeXt-L, 336, and 768 for HR Vision Encoder (VE-HR), LR image resolution (LR), and HR image resolution (HR) by default.**
| Method | VE-HR | LR | HR | Token Num. | VQA${}^{\text{T}}$ | MME | MM-Vet |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline | – | 224 | – | 256 | 54.1^∗ |  |  | 1467.1 |  |  | 30.7 |  |  |
| + Info mining | ConvX-L | 224 | 512 | 256 | 58.1^∗ | +4.0 |  | 1485.2 | +18.1 |  | 31.3 | +0.6 |  |
| + Higher res. | ConvX-L | 224 | 768 | 256 | 59.8^∗ | +1.7 |  | 1478.3 | -6.9 |  | 31.9 | +0.6 |  |
| Baseline | – | 336 | – | 576 | 58.2^∗ |  |  | 1510.7 |  |  | 31.1 |  |  |
| + Info mining | ConvX-B | 336 | 768 | 576 | 58.4^∗ | +0.2 |  | 1451.7 | -59.0 |  | 33.8 | +2.7 |  |
| + Larger VE-HR | ConvX-L | 336 | 768 | 576 | 61.5^∗ | +3.1 |  | 1517.0 | +65.3 |  | 34.6 | +0.8 |  |
| + Larger VE-HR | ConvX-XXL | 336 | 768 | 576 | 62.0^∗ | +0.5 |  | 1505.7 | -11.3 |  | 33.8 | -0.8 |  |

**Table 3: Comparison with different models and data settings. We take LLaVA-1.5 [] with Vicuna-7B as our baseline. Token Num indicates the number of visual tokens $T_{V}$ in Equation (1). ^∗ denotes images in train subset are included. Ablation studies on model and data are marked with ◼ and ◼.**
| Method | LR | HR | Token Num. | VQA${}^{\text{T}}$ | MME | MM-Vet |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline | 336 | – | 576 | 58.2^∗ |  |  | 1510.7 |  |  | 31.1 |  |  |
| + Info mining | 336 | 768 | 576 | 61.5^∗ | +3.3 |  | 1517.0 | +6.3 |  | 34.6 | +3.5 |  |
| + ShareGPT4V | 336 | 768 | 576 | 63.2^∗ | +1.7 |  | 1527.6 | +10.6 |  | 34.2 | -0.4 |  |
| – TextCaps | 336 | 768 | 576 | 59.0 | -4.2 |  | 1465.2 | -62.4 |  | 35.0 | +0.8 |  |
| + LAION-GPT-4V | 336 | 768 | 576 | 58.7 | -0.3 |  | 1521.8 | +56.6 |  | 33.4 | -1.6 |  |
| + OCR-related | 336 | 768 | 576 | 61.6 | +2.9 |  | 1523.5 | +1.7 |  | 33.7 | +0.3 |  |
| + Gen-related | 336 | 768 | 576 | 62.2 | +0.6 |  | 1521.2 | -2.3 |  | 37.0 | +3.3 |  |
| + ALLaVA | 336 | 768 | 576 | 65.2 | +3.0 |  | 1523.3 | +2.1 |  | 40.8 | +3.8 |  |
| + Token extension | 672 | 1536 | 2880 | 68.4 | +3.2 |  | 1546.2 | +22.9 |  | 41.3 | +0.5 |  |

### 4.3 Component-wise Analysis

#### Patch Info Mining.

We first delve into the proposed patch info mining and report results in Table 2.
It is clear that the model achieves significant gains with the ConvNeXt-L integrated as the vision encoder for HR images.
For example, when the LR and HR are respectively set to 224 and 512, the model increases 4.0% and 18.1 in TextVQA and MME datasets.
Elevating the HR resolution to 768 further widens the performance margin, achieving a 5.7% uplift in TextVQA compared to the baseline.
These results underscore the substantial impact of patch info mining in harnessing more detailed visual cues.
When we further extend the LR resolution to 336, patch info mining still contributes consistent gains.
For instance, with the default ConvNeXt-L as vision encoder, it surpasses the baseline with 3.3%, 6.3, and 3.5% in TextVQA [], MME [], and MM-Vet [] dataset, respectively.
This proves the capability of designed modules with input resolution scaled up.

#### Vision Encoder.

To investigate the effect brought by mining candidates, we conduct experiments with various HR vision encoders in Table 2.
Compared with the default ConvNeXt-L, we add two encoders for contrast trials, i.e., ConvNeXt-B, and ConvNeXt-XXL.
With the basic ConvNeXt-B, the model performs better in TextVQA [] and MM-Vet [].
However, the ConvNeXt-L encoder consistently delivers peak results, especially in the MME and MM-Vet datasets, indicating a superior balance in handling detailed visual information.
We can conclude from the table that a larger vision encoder for HR images contributes more to the candidate quality, but the model converges with a too large encoder like ConvNeXt-XXL.
Hence, considering the balance between effectiveness and computational efficiency, ConvNeXt-L is chosen as the default HR vision encoder.
This decision is based on its ability to provide high-quality visual information mining while maintaining reasonable computational demands, as evidenced by the comparative performance across the benchmarks.

#### High-quality Data.

In this era, the significance of high-quality data for enhancing the capabilities of LLMs and VLMs cannot be overstated.
In our comprehensive analysis of data combination effects, presented in Table 3, we begin with a baseline model incorporating patch info mining.
The integration of high-quality captions from ShareGPT4V [] yields improved visual alignment and performance gains.
We validate the zero-shot performance on the TextVQA [] benchmark, notably removing TextCaps [] data from the training set in line with previous studies [].
This modification led to a notable performance decrease, underscoring the value of specific data types in training.
To counteract this decline, we incorporate additional high-quality captions from LAION-GPT-4V [] and OCR-specific data, thus enhancing the model’s OCR reasoning capabilities.
More details are provided in the appendix.
As elaborated in Section 3.3, we utilize generation-related instructions to expand the application.
It is interesting to find that such data also benefits the image understanding ability and brings 3.3% gains in MM-Vet dataset.
Moreover, with the high-quality GPT4V responses from ALLaVA [] dataset, the framework respectively pushes the baseline over 7% and 9% in TextVQA and MM-Vet datasets.
This comprehensive evaluation underscores the pivotal role of strategic high-quality data integration in amplifying the potential of the Mini-Gemini framework.

#### Visual Token Extension.

As depicted in Figure 3(b), the proposed patch info mining is adeptly designed to accommodate extended visual tokens, thereby generalizing its utility across different input resolutions.
We validate the effectiveness of the token extension in Table 3.
When increasing LR and HR input resolution, the model achieves significant gain in all benchmarks.
Notably, in detail-oriented tasks such as TextVQA, we observe a performance uplift of over 3%, indicating a significant enhancement in the model’s ability to handle complex visual data.
Our empirical observations suggest that the increase in resolution significantly diminishes visual hallucinations, leading to more accurate and reliable image comprehension.
Generally, with the increased visual token number, Mini-Gemini can be scaled up towards better capability.
We can also draw the same conclusion from high-resolution results in Table 1.

### 4.4 Qualitative Results

#### Visual Understanding.

To ascertain the visual comprehension prowess of Mini-Gemini in real-world settings, we apply it to a variety of understanding and reasoning tasks in Figure 5.
Thanks to the patch info mining and high-quality data, Mini-Gemini can well solve several complex cases.
For example, it is capable of recognizing plotted curves in graphical data and directly translating them into Python code for immediate application.
Beyond mere recognition, it exhibits a keen attention to detail, accurately describing intricate elements within complex indoor scenes, and demonstrating a nuanced understanding of character associations in memes.
Moreover, Mini-Gemini’s analytical capabilities extend to chart analysis and practical problem-solving, such as intelligence tests.

#### Image Generation.

In Figure 6, we provide a comprehensive evaluation of Mini-Gemini’s generation capabilities.
Compared with recent studies such as AnyGPT [] and ChatIllusion [], our stronger multi-modal understanding ability allows us to generate text-to-image captions that better align with the given instructions, resulting in more contextually appropriate image-text answers.
A noteworthy point, as shown in Figures 1 and 6, is its proficiency in generating high-quality content based on multi-modal human instructions, with text-only training data.
This capability underscores Mini-Gemini’s robust image-text alignment and semantic interpretation skills, which come into play effectively in the inference stage.
By leveraging the powerful reasoning ability of the LLM, it can produce reasonable image-text outputs in single or multi-round conversations.

Figure: Figure 5: Qualitative results in visual understanding with Mini-Gemini.

Figure: Figure 6: Qualitative results in image generation with Mini-Gemini. In the first two rows, we compare Mini-Gemini with ChatIllusion [] and AnyGPT [] with their official cases. In the following rows, we illustrate more cases to show the reasoning generation ability while preserving high-quality text generation. Image inputs (if have) are provided at the left-top corner. In some cases, we overlay the generated prompts on corresponding images.

## 5 Conclusion and Discussion

We presented Mini-Gemini, a streamlined and potent framework for multi-modality VLMs.
The essence of Mini-Gemini is to harness the latent capabilities of VLMs through strategic framework design, enriched data quality, and expanded functional scope.
At its core, patch info mining enables efficient extraction of detailed visual cues by engaging with high-resolution candidates.
From the data perspective, our meticulously compiled high-quality dataset ensures accurate vision-language alignment and bolsters strong instruction-following ability.
Furthermore, we support reasoning-based generation in Mini-Gemini and empower current VLMs with any-to-any workflow.
Extensive experiments on several zero-shot benchmarks prove the superiority of the proposed method, which surpasses previous leading approaches and even private models.
We hope the Mini-Gemini can serve as a strong benchmark for image understanding and VLM-guided generation.

Although Mini-Gemini achieves good results, it still has great potential to be further explored.
For visual comprehension, the counting ability and complex visual reasoning ability are still far from satisfactory.
This could be attributed to the lack of corresponding training data especially in the pretraining stage.
Meanwhile, for reasoning-based generation, we use text to bridge the VLM and diffusion model in this work because we do not find apparent gain with embedding-based approaches.
We will try to find a more advanced manner for visual understanding, reasoning, and generation.

## Appendix

## Appendix A Data Collection Details

#### Image-text Data Collection.

In this section, we delve into the specifics of OCR-related data collection.
Natural images can be easily annotated with detailed captions, but text-rich figures or diagrams, such as documents [], charts [], and scientific diagrams [], present a more intricate challenge for models in complex questions and answers.
Therefore, to facilitate the optimization process, we follow the strategy in TextVQA [] and incorporate OCR tokens for model reference in the training phase.
In particular, we utilize the PaddleOCR [] to initially identify and extract textual elements within each image.
Then, we append the text characters to the original conversations in a format of Reference OCR token:Text_1,...,Text_n, where Text_1 to Text_n indicates $n$ detected text strings.
This approach ensures that the model has access to textual representations from the images, enhancing its ability to understand the image content.
It is important to note that the OCR detector is utilized solely for generating enriched data and is not employed during testing.
This distinction underscores our objective to train Mini-Gemini with a comprehensive understanding of both textual and visual elements, thereby improving capability in complex image-text scenarios.

#### Generation Data Collection.

For the data generation collection described in Section 3.3, we provide specific examples of query prompts and their corresponding reference data sources for two generation tasks in Figure 7.
We commence with a corpus comprising 10K GPT4V caption data and 6K English-only LLM SFT data.
After filtering out results that did not meet the format requirements, we ultimately obtained 13K data points.
To enhance the contextuality and quality of the queries, we incorporate two distinct types of in-context examples: get_example_captions() and get_example_queries().
The former function randomly selects 5 high-quality Stable Diffusion (SD) Text-to-Image (T2I) prompts, while the latter extracts 3 instances from a repository of simple instructional templates.
These in-context examples serve as a foundational guide, providing diverse and representative prompts that significantly enrich the generation process.
This strategic approach ensures the production of high-quality, relevant data, effectively supporting the generative capabilities.

## Appendix B Extended Showcases

In this section, we further provide more cases to validate the generality and capability of Mini-Gemini in various environments.
As presented in Figures 8 and 9, Mini-Gemini can well answer detail-oriented questions and solve OCR-related and scientific problems.
For image generation, we present more examples in Figure 10 that include direct T2I generation, multi-round conversation, reasoning-based generation, storytelling, and in-context generation.
They further prove the superiority of Mini-Gemini in both visual comprehension and generation.

Figure: Figure 7: A detailed prompt design and data source illustration used for allocating image generation data. The total cost for the GPT-4 API to get all 13K data is around 80$.

Figure: Figure 8: Qualitative results of Mini-Gemini’s high-resolution understanding. We highlight a failed case in image understanding in orange.

Figure: Figure 9: Qualitative results of Mini-Gemini’s high-resolution understanding.

Figure: Figure 10: Rows 1-2: Comparison of Mini-Gemini with ChatIllusion [] and LLMGA [] on interactive image generation tasks. Mini-Gemini demonstrates superior adaptability and performance, capturing intricate details and maintaining coherence without further tuning on the text-image output alignment. Rows 3-6 showcase Mini-Gemini’s unique capabilities in generating images with its plug-and-play capability, reasoning generation, and multi-round storytelling. All Mini-Gemini-generated images adopt SDXL unless otherwise specified.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. I can see many experimental results in Table 1, but why not put more results in the radar chart (bottom left corner) in Figure 1?
2. In Table 4, when you utilize a stronger visual encoder ConvX-XXL, why is the performance of MME and MM-Vet in this setting lower than ConvX-L? Do you have more analysis about it?
soundness: 2
strengths: 1. The authors propose a new approach to use high-resolution images to elevate the model performance. Although this approach uses one more visual encoder, the cost is still relatively acceptable as shown in Table 2.
2. The authors use a series of experiments to showcase the performant results on different zero-shot benchmarks. Although there is still a gap between the proposed method and current SoTA private models, the Mini-Gemini has already outperformed many open-sourced models.
3. The authors employ a series of ablation studies to confirm the effectiveness of high-quality datasets and high-resolution images.
4. The paper is well-written and easy to follow. Not only the quantitative results, this paper also provides ample qualitative results.
summary: This paper introduces the Mini-Gemini, which is a simple and effective approach to enhance the VLMs. More specifically, the improvement comes from three aspects, which are higher resolution images, higher quality datasets, and VLM-guided generation. The Mini-Gemini can achieve performant results. In addition, this framework can support a series of dense or MoE LLMs from 2B to 34B.
weaknesses: ### 1. Efficiency and Novelty
From the experimental approach perspective, increasing the image resolution and dataset quality will elevate the performance of VLM[1]. 

In order to enhance the visual representation, the paper introduces one more visual encoder. 

Also, the authors collect a higher quality dataset to improve the performance. 

From an overall construction perspective, the authors leveraged additional resources to enhance the performance. However, from a research novelty standpoint, I feel this paper does not provide substantial new insights. Although the proposed approach is somewhat instructive, I believe this work primarily focuses on stacking resources to achieve better results rather than offering sufficient genuine innovation.

### 2. Error Analysis
All qualitative results showcase the capability of Mini-Gemini, but I think the error analysis also plays an important role in seeing why the current model fails and which direction we can improve in the future.

### 3. Some More Experiments
As far as I see, this work focuses on the zero-shot settings. However, nowadays, with the increase in training data, achieving a purely or strictly zero-shot scenario is quite difficult, as good zero-shot performance may often result from the downstream tasks containing elements similar to the training set. 

Therefore, rather than focusing on zero-shot scenarios, I am more interested in exploring few-shot capabilities. Few-shot not only allows us to observe the final performance but also reveals the model’s instruction-following ability. If the authors could conduct more few-shot experiments, I think it would make the study even more interesting.

[1] MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. I can understand the importance of increasing the image resolution for VLM, but a lot of work just crudely resizes the image without supporting arbitrary resolutions. Since the ConvNeXt is used as Visual Encoder, I am wondering why experiments are not conducted on the images with a larger aspect ratio. Because there is a greater loss of information when this type of image is fed into the VLM.
2. For patch info mining in Figure 3, I have some doubts about this module, such as: Is the channel dimension compressed? Why are the shapes K and V in the calculation NxM2X1? Besides, the entire cross-attention is an attention with window size M, will pixels located at the edges of the window be affected in the calculation? 
3. Please add some comparisons such as VILA, LLaVA-Next, Qwen-VL, and MiniCPM-V. 
4. In Table 2, is Mini-Gemini the final version? Has the token extension overhead been calculated? What is its latency？
5. In ablation study, is there an experimental comparison between Patch Info mining and Visual token extension? How effective is token extension alone? It seems like it improves the model very much.
6. Table 1 would be clearer if it were compared by model size or added line graphs.
7. Very happy to see a combination of VLMs and generative models. However, can Mini-Gemini enhance the ability to generate quality? With training on the dataset with Gen token, Mini-Gemini can generate images via SDXL, it's more of an agent to indicate when to use the SDXL model as well as adjust the prompt to increase the ability to Instruction-following for generating images. How does image understanding enhance image generation in multi-round? Another question is, is it possible to realize multiple rounds of modifications to the same image?
soundness: 2
strengths: 1. Empowering models for image understanding at higher resolution.
2. A new high-quality dataset for Large Vision-Language Models.
summary: This paper introduces a simple and effective VLMs Mini-Gemini. To enhance visual tokens in high-resolution data, the model utilizes a patch info mining to model the relationship between low- and high-resolution inputs as a means of achieving global and local modeling. Besides, this module can continue to be expanded for even higher performance. The authors construct a high-quality dataset to achieve reasoning-based generation. In conclusion, Mini-Gemini achieves leading performance in image understanding, reasoning, and generation.
weaknesses: 1. The model lacks comparisons with existing SOTA LVLM models (e.g., VILA, LLaVA-Next, Qwen-VL, MiniCPM-V)
2. Its support for higher resolutions comes at the cost of more computation.
3. There are no more technical contributions except for a simple high-resolution enhancement model and dataset.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: What's the performance on those in-domain tasks, like ChartQA and AI2D, which are used as training data?
soundness: 3
strengths: The experiments are comprehensive, which demonstrates the effectiveness of Mini-Gemini on benchmarks.
The image generation capability is an advantage of Mini-Gemini that makes it stand out in open-source MLLMs.
The paper is well-written and easy to follow.
summary: This paper proposes Mini-Gemini, a multi-modality Vision Language Model enhanced by high-resolution visual tokens, high-quality data, and VLM-guided generation. Mini-Gemini performs well in several vision-language understanding benchmarks and demonstrates good image generation ability.
weaknesses: 1. **Distilled from private models.** The main contribution of this work is mainly about data. However, most of the data is distilled from closed-sourced models like GPT-4V. Therefore, the model's capability is limited by the teacher model and also inherits the biases from closed models. This type of data collection may help open-sourced models to catch up but may never produce a better model, thus having limited insights for future research.

2. **Human evaluation of the usability of models.** Although the benchmark numbers look good, they may not correctly reflect models' usability, as many benchmarks have biases, unuseful output formats (e.g., multiple choice), disconnected from real-world use cases, etc. When applied in the wild, the proposed model may still have a big gap with GPT-4V. To quantify the usability, it is worth conducting human evaluations to compute the Elo rating.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. I believe that using multiple visual encoders and high-resolution self-mining can not only enhance the model's perception ability but also improve the model's visual robustness. Why not test Mini-Gemini on benchmarks like MMVP, RealWorldQA, and CV-Bench?

2. Is the Mini-Gemini architecture compatible with the approach of visual token drop at each layer in LLMs, such as in FastV?

3. Although  Mini-Gemini performs better than LLaVA-NEXT at the same input resolution, Mini-Gemini has used more training data. Is this comparison fair?
soundness: 3
strengths: 1.The idea of Mini-Gemini is very straightforward, and the experiments are solid. Mini-Gemini is success to narrow the gap by mining the potential of VLMs for better performance across various cross-modal tasks in an elegant way.

2. The structure of paper is simple and easy to read, and the model implementation is very easy to follow and expand.
summary: This study introduces Mini-Gemini, a simple yet effective framework designed to enhance the performance of multimodal visual language models (VLMs) by self-mining. It is success to narrow the gap by mining the potential of VLMs for better performance across various cross-modal tasks. Extensive experiments demonstrate the effectiveness of the method.
weaknesses: In addition to spatial compression of high-resolution images, I believe that temporal compression for videos and multi-image sequences can also be achieved. Mini-Gemini should be designed as a compression paradigm with spatiotemporal extensibility and undergo extensive testing in video understanding, allowing the approach to become more generalizable and impactful.