MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct

## Abstract

Abstract The development of Multimodal Large Language Models (MLLMs) has seen significant advancements with increasing demands in various fields (e.g., multimodal agents, embodied intelligence). While model-driven approaches attempt to enhance MLLMs capabilities through diverse architectures, the gains have become increasingly marginal. Conversely, data-driven methods, which scale up image-text instruction data, are more effective but face limited data diversity and complexity challenges. The absence of high-quality data constitutes a significant development barrier for MLLMs. To address the data quality bottleneck, we propose MMEvol , a novel multimodal instruction data evolution framework. This framework iteratively improve data quality through a refined combination of fine-grained perception, cognitive reasoning, and interaction evolution, generating a more complex and diverse image-text instruction dataset that empowers MLLMs with enhanced capabilities. Beginning with an initial set of instructions, SEED-163K, we utilize MMEvol to systematically broaden the diversity of instruction types, extend visual reasoning steps to improve cognitive reasoning abilities, and thoroughly explore fine-grained information within images to enhance visual understanding and robustness. To comprehensively evaluate the effectiveness of our approach, we conduct extensive qualitative analysis and quantitative experiments across 13 vision-language tasks. Compared to baseline models trained with the initial seed data, the results demonstrate that our method achieves an average accuracy improvement of 3.1 percentage points. Furthermore, our approach reaches state-of-the-art (SOTA) performance in nine tasks using significantly less data compared to state-of-the-art models.
The project page is available at https://mmevol.github.io/ .

## 1 Introduction

> “The True Acquisition of Knowledge Lies in Grasping the Most Subtle Details.” Aristotle, circa 4th century BCE

Multimodal Large Language Models (MLLMs) (Liu et al., ; ; Li et al., ; Dong et al., ; Sun et al., ; Dai et al., ; Luo et al., ; Qi et al., ) have seen rapid development over the past two years and have become the preferred approach for various vision-language tasks (Kembhavi et al., ; Fu et al., ; Zhang et al., ; Qian et al., ). By aligning visual encoders (Radford et al., ; Zhai et al., ; Sun et al., ) with LLMs (Touvron et al., ; Bai et al., ; Lu et al., ; Young et al., ; Tao et al., ), and employing large-scale coarse-grained image-text pre-training (Zhu et al., ; Schuhmann et al., ; ) followed by small-scale instruction-tuning (Chen et al., ; Liu et al., ), MLLMs have demonstrated impressive capabilities across numerous vision-language tasks and are widely applied in many domains (e.g., multimodal agents, embodied intelligence). Model-driven approaches (Luo et al., ; Liu et al., ; Tong et al., ; Zhang et al., ) aim to integrate knowledge from images and text more efficiently by designing different network architectures to improve MLLMs performance. However, their effectiveness is diminishing due to the lack of high-quality data, and redundant model designs struggle to realize their potential and push the boundaries of model intelligence. In contrast, data-driven methods (Liu et al., ; Chen et al., ; Yu et al., ; Liu et al., ; Fang et al., ; Chen et al., ), while more effective, often produce data with limited diversity and complexity. The lack of high-quality data remains a significant obstacle in the development of MLLMs. Therefore, the need of developing automated methods that can generate more challenging and diverse instructional data at a relatively low cost is urgent for empowering MLLMs with enhanced capabilities.

Figure: Figure 1: Overview of MMEvol. Instruction evolution and instruction elimination synergistically collaborate through multiple rounds to enhance the diversity and complexity of instruction data.
Refer to caption: x1.png

Analysis of existing data-driven methods for generating image-text instruction data reveals three common limitations: 1) Limited instruction diversity. Manually annotated instructions are constrained by the cognitive limitations of annotators, while model-generated instructions are limited by template presets, making it difficult to meet the diverse task requirements of the real world. This restricts the instruction-following ability of MLLMs. 2) Limited instruction complexity. Manual annotations often result in instructions of simple or moderate complexity, and automatically generated instructions tend to be brief and lacking in visual reasoning steps, which limits the model’s ability to handle complex tasks. 3) Insufficient alignment granularity. Both manually and model-generated instructions primarily focus on common objects, neglecting rare or small objects, resulting in limited granularity in image-text alignment. This affects the model’s visual perception robustness and resistance to hallucinations.

To address these limitations, we propose MMEvol, a novel method that utilizes advanced MLLMs for iterative evolution. This method automatically generates various types of open-domain instructions on a large scale, covering different difficulty levels to enhance the performance of MLLMs. Given that visual-language instruction data are constrained by visual content, the data generated through multiple iterations with Evol-Instruct (Xu et al., ; Luo et al., ; ) tend to include simple restatements and data unrelated to visual content, making deep and broad evolution challenging. Therefore, we have made several adjustments to the evolution prompting process, ultimately developing an image-text instruction evolution paradigm. These adjustments include a more refined image-text instruction data paradigm and the definition of three evolution directions: fine-grained perception evolution, cognitive reasoning evolution, and interaction evolution. The MMEvol mechanism is summarized in Fig. 1, with each evolution cycle comprising two main steps: instruction evolution and instruction elimination. Instruction evolution randomly selects one of fine-grained perception evolution, cognitive reasoning evolution, or interaction evolution, upgrading simple instructions to more complex or diverse ones. Specifically, fine-grained perception evolution aims to leverage visual information in images to generate data with more detailed information; cognitive reasoning evolution prolongs the visual operation reasoning steps of instructions to increase their complexity; and interaction evolution aims to enhance instruction diversity by providing a wider variety of instruction forms. To account for occasional failures in evolved instructions, we use instruction elimination to filter out failed evolution. MMEvol repeats the instruction evolution and elimination processes multiple times to obtain a complex instruction dataset containing various instruction forms.

Figure: Figure 2: SEED-163K: 163K Curated Seed Instruction Tuning Dataset for Evol-Instruct. Left: The inner circle shows the original distribution of SEED-163K. The outer circle shows the curated SEED-163K. Right: All the data sources in the SEED-163K dataset, as well as the ones filtered in data curation.
Refer to caption: x2.png

To validate the effectiveness of MMEvol, we perform three rounds of evolutionary iterations on 163K seed data, leading to 447K evolved samples. We fine-tuned the open-source LLaVA-NeXT (Liu et al., ) model with these evolved data and compared it with other advanced methods across 13 vision-language benchmarks. Our method achieves state-of-the-art (SOTA) performance, demonstrating the effectiveness and efficiency of MMEvol. Additionally, we conduct detailed qualitative analysis and ablation experiments to showcase the contribution of each component of our method. We hope that the released evolutionary data and code will assist the community in understanding that using a small amount of high-quality image-text instruction data is far more critical than training MLLMs with large-scale low-quality image-text instruction data.

Our main contributions can be summarized as follows:

- •
A image-text instruction evolution framework, MMEvol, is designed to leverage advanced MLLMs, automating the generation of open-domain image-text instruction data across varying difficulty levels to enhance the diversity and complexity of existing datasets.
- •
By utilizing instruction evolution data, a high-quality data recipe is composed, and the evolved data will be released to advance the capabilities of other open-source MLLMs further.
- •
We train an MLLM using this high-quality data recipe, achieving superior performance in various downstream visual-language tasks compared to other fully open-source methods.
- •
The effectiveness and efficiency of the proposed approach are validated through extensive qualitative and quantitative analyses.

Figure: Figure 3: Prefix-Prompt of MMEvol. The top block showcases the contexts such as caption and visual object locations, and the middle block demonstrates vision/la nguage-centered atomic propositions and evolution objective (described later). Additionally, we endow vision capabilities with pseudo-function calls to enhance visual reasoning during evolutionary processes. Finally, the bottom block further elucidates the organized seed sample, which is subsequently sent to the MLLM for rewriting.
Refer to caption: x3.png

## 2 Method

In this section, we first introduce the curation of seed instruction data and then elaborate on the methodological details of MMEvol. Due to the space limitation, we simplify the seed data curation process and prompt templates. More details can be found in the Appendix D.

### 2.1 Seed Data Curation

The seed instruction data are curated from LLaVA-Instruct (Liu et al., ) and ShareGPT4V  (Chen et al., ) datasets, supplemented with additional scientific and chart data sampled from Cambrain-1  (Tong et al., ). This process involved careful selection and refinement to ensure the quality and diversity of the instructions. For instructions with only captions, we use the OpenAI GPT-4o mini API to generate seed instruction data. Ultimately, after merging and filtering, we obtained a comprehensive dataset consisting of 163K instruction samples with unique images, which serve as the foundation for our subsequent Evol-Instruct. The seed data mixture is shown in Fig. 2.
Please refer to Appendix A for more details.

### 2.2 Methodological Details

Figure: Figure 4: Fine-grained perceptual evolution prompt and data example. Fine-grained perceptual evolution can generate samples with more detailed visual information, enhancing data diversity, which are marked with different colors for better visualization.
Refer to caption: x4.png

Figure: Figure 5: Cognitive reasoning evolution prompt template and example. Cognitive reasoning evolution can endow instruction data with a longer visual reasoning chain, increasing the complexity of the data. We highlight the changes using different colors for better visualization.
Refer to caption: x5.png

The evolution of image-text instruction data is constrained by visual information, requiring evolved instruction data to be relevant to image content to avoid hallucinations. This makes the diversity evolution of image-text instructions particularly challenging. Additionally, the complexity evolution process of image-text instruction data often results in shallow reasoning phenomena, with MLLMs struggling to provide complex answers. As shown in Fig. 1, to address these issues and improve the success rate of evolution, we include carefully designed domains such as visual objects, atomic capabilities, visual manipulations, and instruction formats to standardize each instruction data format. The visual object domain includes visual objects in the images involved in the instruction data, implicitly constraining the evolution data and reducing visual hallucinations. We also summarize nine types of atomic capabilities involved in image-text instruction data to populate the atomic capability domain, aiming to enhance data diversity. Specifically, this includes five vision-centric capabilities: localization, reference, computation, optical character recognition (OCR), and existence judgment, and four language-centric capabilities: relation description, scene understanding, behavior prediction, and world knowledge association. The visual manipulation domain includes visual manipulation chains for problem-solving, where each step of the visual manipulation is based on vision-centric atomic capabilities, explicitly defining the visual reasoning process to mitigate shallow reasoning. The instruction format domain specifies the interaction types of the instruction data. These adaptations enhance the diversity and complexity of image-text instruction data and improve the success rate of evolution.

Fine-grained Perceptual Evolution. The goal of fine-grained perceptual evolution is to maximize the extraction of available visual information from images, especially overlooked non-primary visual objects. We observe that most instruction data tend to construct questions involving primary objects in images while neglecting less frequent non-primary objects. This results in a lack of instructions related to long-tail distribution objects. Training with such data can lead to visual hallucinations and poor generalization and robustness. Fine-grained perceptual evolution generates questions involving new visual objects, uncovering usable and often overlooked visual information. The evolutionary prompt template and process are shown in Fig. 4.

Figure: Figure 6: Interactive evolution prompt template and example. Interactive evolution can automatically generate various types of non-predefined instruction formats, significantly enhancing the diversity of the data. The differences are highlighted using distinct colors for better visualization.
Refer to caption: x6.png

Figure: Figure 7: Instruction elimination prompt template. Instruction elimination is used to calculate the evolutionary gain and complexity level of the instruction data. We filter out harmful data that failed to evolve based on the evolutionary gain.
Refer to caption: x7.png

Cognitive Reasoning Evolution. Reasoning ability is one of the key capabilities of multi-modal large language models. However, most existing instruction data, such as LLaVA-Instruct (Liu et al., ), consists of simple question-and-answer pairs that lack detailed reasoning processes, making it difficult for trained models to accomplish complex tasks requiring reasoning capabilities, such as multi-modal agents and visual reasoning. We introduce the concept of a visual manipulation chain, abstracting four vision-centric reasoning capabilities into four visual operation functions described in text. By generating the necessary visual reasoning steps to solve problems, we define the complexity of the instruction data. During the cognitive reasoning evolution process, we evolve new instruction data by increasing the visual reasoning steps in the data to obtain more complex data. The evolutionary prompt template and process are shown in Fig. 5.

Interactive Evolution. Existing models generate instruction data in very few forms. For example, LLaVA-Instruct provides only dialogue-based question-answering, complex reasoning, and global description tasks. Handcrafted instruction data, such as ALLaVA (Chen et al., ), are limited by annotators’ experience, making it challenging to design various task forms. Models trained with such data often struggle to follow complex and diverse user-specified instructions or goals, limiting their practicality and applicability in real-world scenarios. To evolve instruction data with rich task forms and provide a good interaction experience, we design interactive evolution to generate instruction data with diverse task forms automatically. The evolutionary prompt template and process are demonstrated in Fig. 6.

Instruction Elimination. After each round of evolution, we score the evolved instruction data on multiple dimensions to assess the success of the evolution. We retain instruction data with evolutionary gains and discard those with failed evolution. The evolutionary elimination prompt template and process are shown in Fig. 7.

## 3 Experiments

### 3.1 Benchmarks

To comprehensively evaluate the effectiveness of our evolutionary method, we select 13 benchmarks, with their sources and tested skills illustrated in Table 1. MIA (Qian et al., ) is an open-domain instruction-following benchmark that thoroughly tests the model’s instruction-following abilities using extensive instruction data. MM-Self-Instruct (Zhang et al., ) is a novel visual reasoning benchmark that focuses on the model’s visual perception capabilities and performs common visual reasoning tasks encountered in daily life.

**Table 1: Benchmarks for evaluation with their sources and tested skills. The names are abbreviated due to space limitations. VQA^V2; GQA; VQA^T: TextVQA; MME^C: MME-Cognition; MathVista^M: MathVista-MINI; MMMU; AI2D; POPE; HallusionBench: HallBench; MIA; BLINK; RWQA: RealWorldQA; MMSInst: MM-Self-Instruct.**
| Skills | Sources | Skills | Sources |
| --- | --- | --- | --- |
| VQA | VQA^v2 (Goyal et al., ) | General Knowledge | MME^C (Fu et al., ) |
| Knowledge Leakage | MMStar (Chen et al., ) | MMMU (Yue et al., ) |  |
| Math Reasoning | MathVista^M (Lu et al., ) | Hallucination | POPE (Li et al., ) |
| OCR Related | AI2D (Kembhavi et al., ) | HallBench  (Guan et al., ) |  |
| Instruction Following | MIA (Qian et al., ) | Visual Reasoning | GQA (Hudson & Manning, ) |
| Visual Perception | BLINK (Fu et al., ), RWQA | MMSInst  (Zhang et al., ) |  |

### 3.2 Implementation Details

Data. During the pre-training phase, we use LLaVA-Pretrain-595K (Liu et al., ) for image-text alignment training. In ablation experiment settings, we fine-tune using both seed data and evolved data separately to ensure a fair comparison and validate the benefits of MMEvol. In SOTA setting experiments, we fine-tune using evolved instruction data combined with other publicly available datasets sampled from Cambrain-1 (Tong et al., ) and compare it with other methods. Additional details on training data recipes can be found in the Appendix B.

Model. We follow the architecture from LLaVA-NeXT, where a multimodal large model consists of three key components: an LLM for next token prediction, a visual encoder for extracting visual features, and an image-text projector to align the visual and text modalities. We use Llama3-8B-Instruct (Touvron et al., ) for ablation experiments. For comparisons with other methods, we switch to our previous SOTA settings with Llama3-8B-Instruct and Qwen2-7B-Instruct (Bai et al., ). We adapt CLIP-ViT-L (Radford et al., ) for the visual encoder and use simple linear layers to bridge the image and text modalities.

Training Strategies. We conduct MMEvol training following widely used two-stage settings. Vision-Language Pre-training and Visual Instruction-tuning. The language models and ViT are separately pre-trained, while the projector is randomly initialized. To initially align the feature space
between the visual and text modalities, we utilize the aligned dataset. Finally, we perform instruction tuning of the pre-trained model on visual language instruction datasets. Our experiments are conducted with 8$\times$A100 GPUs and a global batch
size of 128. We employ AdamW optimizer (Loshchilov, ) with learning rates $5\times 10^{-5}$ and $2\times 10^{-5}$ for aforementioned two stages respectively. Each stage is trained with one epoch with a 3% warmup
strategy. Please refer to the Appendix B for more details.

Figure: Figure 8: MMEvol continuously enhances instruction data complexity and diversity over evol-instruct. The sample is from SEED-163K. We mark fine-grained visual information in red, new instructions form in green, and longer reasoning steps in blue.
Refer to caption: x8.png

### 3.3 Qualitative Analysis

We sample a specimen from SEED-163K and display its evolution process in Fig. 8. In round 1, we perform fine-grained perceptual evolution, leading to instruction data with more precise details, including actions and attributes. In round 2, interaction evolution shifts instruction forms from general question answering to creative poetry generation, increasing the diversity of instruction formats. In round 3, cognitive reasoning evolution adds reasoning steps to the answers in the instruction data, enhancing its complexity. Through multiple rounds of instruction evolution, we improve the diversity and complexity of the seed data.

Figure: (a) Seed data
Refer to caption: x9.png

Additionally, we randomly sample 30K data points from the seed data and conduct qualitative analysis on the instruction data before and after evolution. As shown in Fig. 10, the evolved data is notably more complex. Specifically, each evolved instruction involves 0.68 more atomic abilities in Fig. 10(a) and has an average visual operation chain reasoning length of 0.86 longer compared with pre-evolution in Fig. 10(b). As we can see from Fig. 10(c), the average difficulty score of each evolution round increases progressively, demonstrating the effectiveness of cognitive reasoning evolution in increasing instruction data complexity.

We identify the verb-noun structures in the generated instructions to study the types of instructions generated and the diversity of evolved data. We use the Berkeley Neural Parser (Kitaev & Klein, ; Kitaev et al., ) to parse the instructions, extracting the verb closest to the root and its first direct noun object. Fig. 9 plots the root verbs and their direct noun objects with quantities exceeding 2K. We observe that the evolved data significantly enhances instruction diversity compared to pre-evolution, with diverse intents and textual formats in the evolved instructions. Furthermore, we conduct a long-tail distribution visualization analysis of the visual object domain in the instruction data before and after evolution to verify the effectiveness of fine-grained perceptual evolution. Fig. 11 shows that fine-grained perceptual evolution greatly improves the distribution of visual objects in the long tail, maximizing the extraction of usable visual information from images, refining the image-text alignment granularity in the instruction data, enhancing data diversity, which improves model generalization and reduces visual hallucinations.

Figure: (a)
Refer to caption: x11.png

Figure: Figure 11: The long-tail distribution of 200 visual objects between seed and evolved data. MMEvol significantly improves the long-tail distribution of visual objects in the seed data, providing more fine-grained visual information, thereby boosting the model’s generalization ability and robustness against hallucinations.
Refer to caption: x14.png

### 3.4 Ablation Study

We conduct ablation studies on seven vision-language benchmarks to explore the effects of instruction evolution and elimination. As shown in Table 2,
different evolution process can be orthogonally superimposed on each other to continuously enhances data diversity and complexity. leading to an average performance gain of 3.8 points across multiple vision-language benchmarks. However, the absence of instruction elimination introduces harmful data from failed evolutions, which inevitably reduces the model’s resistance to hallucinations by 1.2 points on POPE  (Li et al., ). When both instruction evolution and instruction elimination are employed, instruction elimination filters out harmful data from failed evolutions, further improving the quality and density of evolved data and enhancing the model’s performance by 0.9 points on average, particularly improving resistance to hallucinations by 1.7 points, which aligns with our qualitative analysis results in Section 3.3.

**Table 2: Ablation study on instruction evolution and instruction elimination. The application of instruction evolution alone enhances the complexity and diversity of the data, whereas the integration of instruction elimination further refines data quality, markedly reducing the occurrence of visual hallucinations.**
| FP-Evol | I-Evol | CR-Evol | I-Elim | MMStar | MathVista^M | POPE | AI2D | MME^C | MMMU^V | RWQA | AVG. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ✘ | ✘ | ✘ | ✘ | 36.5 | 25.3 | 84.8 | 53.9 | 31.5 | 32.3 | 43.5 | 44.0 |
| ✔ | ✘ | ✘ | ✘ | 37.3  (+0.8) | 25.6  (+0.3) | 85.0  (+0.2) | 54.2<br>(+0.3) | 33.0  (+1.5) | 32.5<br>(+0.2) | 46.7<br>(+3.2) | 44.9  (+0.9) |
| ✔ | ✔ | ✘ | ✘ | 38.2  (+1.7) | 26.2  (+0.9) | 83.8  (-1.0) | 54.5  (+0.6) | 35.6<br>(+4.1) | 32.9<br>(+0.6) | 48.9<br>(+5.4) | 45.8  (+1.8) |
| ✔ | ✔ | ✔ | ✘ | 38.9  (+3.4) | 27.3  (+3.0) | 83.6  (-1.2) | 54.7  (+0.8) | 40.1<br>(+8.6) | 34.4<br>(+0.9) | 54.4<br>(+10.9) | 47.6  (+3.8) |
| ✔ | ✔ | ✔ | ✔ | 40.3  (+3.8) | 28.6  (+3.6) | 86.5  (+1.7) | 55.2  (+1.3) | 39.9  (+8.4) | 35.3  (+3.0) | 55.3  (+11.8) | 48.7  (+4.7) |

### 3.5 Benchmark Comparison

After comprehensively validating our approach’s ability to enhance the complexity and diversity of instruction data, we perform a thorough comparison with previous SOTA methods across 13 vision-language benchmarks, summarizing the results in the Table 3. Notably, we observe that supported by enhanced and refined instruction data, our MLLM significantly advances performance boundaries in almost all benchmarks, consistent with the performance improvements observed in our ablation experiments in Section 3.4. Remarkably, compared to the fully open-source SOTA model Cambrain-1 (Tong et al., ), our method, although using seed data sampled from training data of Cambrain-1, achieves superior results with a substantial performance increase ($\uparrow$ 2.9 average points). This indicates that the quality of instruction data is more crucial than quantity.

In comparison to the open-source SOTA model MiniCPM-v2.5 (Yao et al., ), despite a considerable difference in training data volume, our model still delivers better results, particularly showing improvements in instruction following, visual hallucinations, and visual reasoning with gains of $\uparrow$3.1 points on HallBench, $\uparrow$2.5 points on MIA, and $\uparrow$13.6 points on MMSInst respectively. This demonstrates that our method enhances the model’s visual reasoning and instruction following, reduces visual hallucinations, and improves other general capabilities, consistent with our findings from ablation studies and qualitative analyses. By using our data and the leading large language model Qwen2, we can train a superior MLLM from scratch in only one day using 4×8 A100 GPUs, further validating that high-quality instruction data is more important than large-scale low-quality data.

**Table 3: Comparison with state-of-the-art methods on 13 visual-language benchmarks. Our models consistently improve LLaVA-NeXT under a head-to-head comparison, using the same prompts and the same base LLM, showing the effectiveness of enhanced pretraining data quality. We mark the best performance bold and the second-best underlined. “PT” denotes pre-training data scale, and “IT” denotes instruction tuning data scale.**
| Model | PT | IT | VQA^v2 | GQA | MME^C | MMStar | HallBench | MathVista^M | MMMU^V | AI2D | POPE | MIA | BLINK | RWQA | MMSInst | AVG. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Weight Open-Source |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Yi-VL-6B | 125M | 1M | – | – | 46.2 | 37.7 | 55.7 | 28.8 | 40.3 | 59.8 | 82.5 | 26.1 | 38.7 | 53.5 | – | 46.9 |
| DeepSeek-VL-7B | 275M | 50M | – | – | 37.1 | 40.5 | 53.9 | 36.8 | 38.3 | 65.3 | 85.6 | 61.0 | 40.9 | 49.7 | 26.7 | 48.7 |
| Qwen-VL-Chat-7B | 1.4B | 50M | 78.2 | 57.5 | 49.0 | 34.5 | 56.4 | 34.9 | 37.0 | 63.0 | 74.9 | 63.1 | 28.2 | 49.3 | – | 52.2 |
| CogVLM-Chat-17B | 1.5B | 5.1M | – | 65.2 | 37.4 | 39.9 | 55.1 | 34.7 | 37.3 | 63.3 | 88.0 | 60.0 | 41.5 | 60.3 | – | 53.0 |
| MiniCPM-V2.5-8B | 570M | 9.1M | 81.9 | 64.7 | 50.3 | 51.3 | 59.2 | 54.3 | 43.0 | 78.3 | 86.7 | 76.3 | 36.7 | 63.5 | 28.2 | 59.6 |
| Fully Open-Source |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| InstructBLIP-7B | 0.6M | 0.8M | – | 49.2 | 31.8 | 32.7 | 53.6 | 24.4 | 30.6 | 40.6 | 86.1 | 38.2 | 39.7 | 36.9 | – | 42.2 |
| LLaVA-1.5-7B | 0.6M | 0.8M | 78.5 | 62.0 | 37.8 | 33.1 | 48.8 | 25.6 | 35.7 | 55.5 | 86.1 | 62.2 | 38.0 | 54.8 | 15.4 | 48.7 |
| LLaVA-1.5-13B | 0.6M | 0.8M | 80.0 | 63.3 | 34.8 | 34.3 | 45.3 | 27.7 | 37.0 | 61.1 | 88.4 | 63.6 | 40.9 | 55.3 | – | 52.6 |
| LLaVA-NeXT-8B | 0.6M | 0.8M | 81.8 | 65.2 | 44.6 | 43.9 | 52.3 | 31.5 | 41.7 | 69.9 | 87.3 | 65.1 | 43.5 | 60.1 | 25.6 | 54.8 |
| LLaVA-NeXT-13B | 0.6M | 0.8M | 82.8 | 65.4 | 37.1 | 40.4 | 51.5 | 35.1 | 35.9 | 72.2 | 87.8 | 69.2 | 41.2 | 59.1 | 30.2 | 54.5 |
| VILA-1.5-8B | 50.5M | 6.0M | 80.9 | 61.9 | 39.0 | 39.7 | 55.8 | 37.3 | 36.9 | 58.8 | 85.5 | 66.1 | 37.0 | 43.3 | 21.6 | 51.1 |
| VILA-1.5-13B | 50.5M | 6.0M | 82.8 | 64.3 | 38.5 | 44.2 | 59.2 | 42.5 | 37.9 | 69.9 | 84.2 | 61.2 | 41.5 | 53.3 | 30.6 | 54.6 |
| Cambrian-1-8B | 2.5M | 7.0M | 81.2 | 64.6 | 41.1 | 50.7 | 47.8 | 47.0 | 41.8 | 74.6 | 86.4 | 68.7 | 44.9 | 64.2 | 28.3 | 57.1 |
| Cambrian-1-13B | 2.5M | 7.0M | 82.6 | 64.3 | 44.5 | 47.1 | 58.9 | 47.4 | 40.0 | 73.6 | 86.8 | 69.8 | 43.1 | 63.0 | 25.8 | 57.5 |
| Seed-8B | 0.6M | 1.1M | 82.5 | 64.8 | 41.3 | 47.4 | 60.8 | 47.7 | 38.0 | 72.1 | 85.3 | 69.4 | 44.2 | 59.9 | 26.2 | 56.9 |
| Evol-8B | 0.6M | 1.6M | 83.4 | 65.0 | 47.8 | 50.1 | 62.3 | 50.0 | 40.8 | 73.9 | 86.8 | 78.8 | 46.4 | 62.6 | 32.3 | 60.0 |
| Evol-Qwen2-7B | 0.6M | 1.6M | 83.1 | 65.5 | 55.8 | 51.6 | 64.1 | 52.4 | 45.1 | 74.7 | 87.8 | 77.6 | 47.7 | 63.9 | 41.8 | 62.4 |

## 4 Related Work

Multimodal Large Language Models (MLLMs). MLLMs have rapidly advanced in recent years due to the success of Large Language Models (LLMs) and the availability of diverse image-text instruction data from the internet. LLaVA (Liu et al., ) and MiniGPT-4 (Zhu et al., ) have demonstrated strong cross-task generalization by integrating visual encoders with large language models through simple connectors and training on instruction data. LLaVA-NeXT (Liu et al., ) has significantly enhanced visual perception by employing dynamic resolution techniques. Cambrain-1 (Tong et al., ) has improved model robustness through visual encoder routing, though it incurs higher training costs. DEEM (Luo et al., ) simplifies model architecture and enhances robustness by using diffusion models to extract visual features instead of traditional visual encoders. Subsequent work (Wang et al., ; Zhou et al., ; Xie et al., ) following DEEM combine diffusion models with LLMs to further enhance generative and understanding capabilities of MLLMs. However, these models still face challenges related to the quantity and quality of data, which limit performance improvements further.

Image-text Instruction Data Construction. LLaVA (Liu et al., ) has improved model capabilities by utilizing LLaVA-Instruct (Liu et al., ), a dataset labeled by advanced LLMs. However, this approach does not fully exploit visual information and have limited instruction types. ALLaVA (Chen et al., ), by manually crafting and rewriting instruction data, offers greater variety but suffers from high manual labeling costs, inefficiency, and overly simplistic problems. MMInstruct (Liu et al., ) generates instruction data automatically with advanced MLLMs, but the instruction complexity and diversity are constrained by predefined formats, failing to fully exploit effective visual information. VILA^2 (Fang et al., ) has generated extensive data through instruction evolution but lacks complexity and variety, limiting its utility for other models. In contrast, we address this challenge and propose MMEvol, which iteratively enhances instruction diversity and complexity through instruction evolution on limited data, aiming to extract more usable visual information and endow MLLMs with more powerful capabilities.

## 5 Conclusions

In this work, we propose an image-text instruction evolution framework and explore the techniques, insights, and benefits of Evol-Instruct for enhancing the quality and quantity of image-text instruction data. We employ three distinct evolution methods to increase the complexity and diversity of instruction data based on a limited seed dataset while utilizing instruction elimination to filter out harmful data. The data evolved through three rounds of evolution is used to train a new model, demonstrating state-of-the-art (SOTA) performance across a comprehensive set of benchmarks. Future directions include exploring integrating image generation models to synthesize new images and perform dual evolution of images and texts, aiming to train even more robust foundational models.

Limitations.
Due to resource constraints, we only performed evolution on 163K samples and conducted experiments with an 8B scale model. Expanding the dataset and using larger-scale models could yield even better results. We plan to explore these avenues in future work.

## Appendix

**Table 4: The mixture of training recipe datasets with corresponding categories and sources.**
| Category | Sources | Size | Ratio |
| --- | --- | --- | --- |
| VQA | VQAV2 (Goyal et al., ) | 83K | 5.1% |
| General Knowledge | OKVQA (Marino et al., ), A-OKVQA (Schwenk et al., ) | 243K | 14.9% |
| VG (Krishna et al., ), GeoQA (Gao et al., ) |  |  |  |
| Visual Reasoning | GQA (Hudson & Manning, ) | 72K | 4.4% |
| Visual Grounding | RefCOCO (Kazemzadeh et al., ) | 48K | 2.9% |
| OCR Related | OCR-VQA (Mishra et al., ), TextVQA (Singh et al., ) | 270K | 16.5% |
| AI2D (Kembhavi et al., ), ChartQA (Masry et al., ) |  |  |  |
| DocVQA (Mathew et al., ), DVQA (Kafle et al., ) |  |  |  |
| Synthdog-EN (Kim et al., ), Datikz (Belouadi et al., ) |  |  |  |
| TabMWP (Lu et al., ), ArxivQA (Li et al., ) |  |  |  |
| Instruct | MMEvol, ALLaVA (Chen et al., ) | 650K | 39.8% |
| Language | ShareGPT, WizardLM (Xu et al., ) | 183K | 11.2% |
| General Knowledge | Design2Code (Si et al., ), MathVision (Wang et al., ) | 85K | 5.2% |
| Geo170k (Gao et al., ), ScienceQA (Lu et al., ) |  |  |  |
| Websight (Li et al., ), Cambrain-Data-Engine (Tong et al., ) |  |  |  |

## Appendix A Curation Details of Seed Data

LLaVA-Instruct (Liu et al., ) is a dataset of image-text instructions based on the COCO (Chen et al., ) data source and generated using the OpenAI ChatGPT API. The image-text instruction format in this dataset primarily includes three types: dialogue-based question-answering, global descriptions, and complex reasoning. ShareGPT4V (Chen et al., ), on the other hand, is a dataset constructed or rewritten using the OpenAI GPT-4V API, based on image-text pairs from SAM (Kirillov et al., ), COCO, and other sources to introduce richer details into captions. Both LLaVA-Instruct and ShareGPT4V significantly advance the development of MLLMs (Hu et al., ; Li et al., ; Si et al., ) and are widely used. We integrate samples from these two datasets containing the same image by concatenating the corresponding instruction data lists. For samples with global descriptions but no instruction data, we use the GPT-4o-mini API to supplement the missing instruction data, similar to LLaVA-Instruct, resulting in a combined dataset of 133K samples. To ensure the diversity of the seed data, we also include additional scientific chart data. Specifically, we sample 30K entries from Cambrain-1 (Tong et al., ), covering various types of image-text instructions such as code generation, chart interpretation, scientific question-answering, document understanding, and mathematical reasoning, ultimately forming a seed dataset of 163K image-text instructions.

**Table 5: The detailed training setup for MMEvol and the hyper-parameters across the training stages.**
| Hyperparameter | Ablation Stage 1 | Ablation Stage 2 | SOTA Stage 1 | SOTA Stage 2 |
| --- | --- | --- | --- | --- |
| language model | LLaMA 3 8b | LLaMA 3 8b | LLaMA 3 8b | LLaMA 3 8b |
|  |  | Qwen 2 7b | Qwen 2 7b |  |
| global batch size | 128 | 128 | 128 | 128 |
| batch size | 4 | 4 | 4 | 4 |
| learning rate | 1e-3 | 5e-5 | 1e-3 | 5e-5 |
| lr schedule | cosine | cosine | cosine | cosine |
| lr warmup ratio | 0.03 | 0.03 | 0.03 | 0.03 |
| weight decay | 0 | 0 | 0 | 0 |
| epoch | 1 | 1 | 1 | 1 |
| optimizer | AdamW | AdamW | AdamW | AdamW |
| cost | 4h | 0.1h | 4h | 20h |
| dataset | LLaVA Pretrain | Seed-30K/Evol-30k | LLaVA Pretrain | Dataset Mixture |

## Appendix B Implementation Details

After three rounds of evolution and filtering, we obtain 447K high-quality image-text instruction data with diversity and complexity. This data, combined with the ALLaVA instruction dataset, forms the 600K instruction data segment of the training recipe. To ensure a fair comparison with other methods, we combine the instruction data with other commonly used image-text data into the final training recipe, as shown in the Table 4. Notably, we find that the DataEngine (Tong et al., ) data contains many harmful mismatched image-text pairs. We use OpenAI GPT-4o API to filter out harmful data and obtain 20K effective image-text instruction data. More details about training settings can be found in Table 5

## Appendix C Additional Visualization Results

We also present additional visualization results to demonstrate the capabilities of our model. As shown in Fig. 12, our model trained on this data exhibits strong visual reasoning, instruction following, and fine-grained perception capabilities. Additionally, it identifies nuances in meme content, validating the effectiveness and efficiency of MMEvol.

## Appendix D Complete Evolution Prompt Template

Due to the space limitations in the main text, we simplify the instruction evolution prompt template. We provide the complete detailed evolution templates as follows: the complete prefix-prompt template is shown in Fig. 13, the fine-grained perception evolution prompt template is in Fig. 14, the cognitive reasoning evolution prompt template is in Fig. 15, the interaction evolution prompt template is in Fig. 16, and the instruction elimination prompt template is in Fig. 17.

Figure: Figure 12: Examples of image-text dialogue with our Evol-8B Model.
Refer to caption: x15.png

Figure: Figure 13: Complete prefix-prompt template of MMEvol.
Refer to caption: x16.png

Figure: Figure 14: Complete fine-grained perceptual evolution prompt template.
Refer to caption: x17.png

Figure: Figure 15: Complete cognitive reasoning evolution prompt template.
Refer to caption: x18.png

Figure: Figure 16: Complete interactive evolution prompt template.
Refer to caption: x19.png

Figure: Figure 17: Complete instruction elimination prompt template.
Refer to caption: x20.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Addressing questions w.r.t. Technical clarify and evaluation in weakness session would impact my final judgement of the paper. The following questions are nice-to-have discussion which might not impact final score. 

Enhancing vision centric capabilities in fine-grained object, CoT, interaction seems to be effective in LLAVA style VLM, do you think it is because the pretrained ViT / LLM lacks these capabilities in the first place, or pretrained models already have learned enough knowledge but somehow forget it with poor instruction data during fine-tuning? Do you think this data will help other pretrained VLMs trained with tens of billions of image/text tokens?
soundness: 3
strengths: The problems this paper tries to address are spot on, current VLMs is limited in terms of complex visual reasoning, natural interaction with human, etc. The method to evolve current instruction fine-tune data to enhance these capabilities is one of interesting direction to improve the model’s capabilities, as shown in the benchmark results. 

The authors mention data and code will be released, and I think it’ll be a good contribution to the community to set a higher baseline for LLAVA style model and maybe even beyond.
summary: This paper introduce a method to systematically evolve seed instruction fine-tuning data for VLM to larger scale and enhance vision centric capabilities for fine-tuned visual understanding, visual reasoning and human interactive capabilities. The enhanced evolution dataset show signiificant improvement on a wide range of perception and reasoning dataset.
weaknesses: My major concern is lack of technical clarity and minor concern is insufficient evaluation. 

First, the prompt is clear in Fig 4 - Fig 7, can you explictly describe what model, commercial API or other method is used to generate the evolutoin instruction data from the prompt templates?

Second, it is mentioned pseudo function calling is used for visual reasoning evolution, can you describe the setup of function call, the model or template used to generate the function call, and how it is integrataed to the visual reason evolution process?

Third, it is mentioned MLLM is used for rewriting, can you describe details on what MLLM is used, how it is used (prompt used), evalution or abalation of the significance of this rewriting step?

On the evaluation side, I think a more comprehensive evaluation on different capabilities of the model and some more ablation would provide more insights to the method and data. 

Firrst, the seed dataset and one of the vision-centric capabilities is OCR, while there is few OCR related benchmark results, more results on OCRBench, ChratQA, DocVQA, TextVQA would be very insightful. 

Second, more ablation on ratio of the three evolution methods in each round, how to decide and eliminate failed evolution, what’s the success/fail ratio for each round, what is the model quality gain for reach round, etc. would be informative. 

Third, in comparison with other methods, InternVL2-8b (releaesd 2024/07) and Qwen2-VL-7b (released 2024/08) should be included in Table2 under weight open source section.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - How reliable is GPT-4 as a data rewriter? How can the quality of the rewritten data be evaluated?
- Is the rewritten output sensitive to changes in the prompt?
- What is the rationale behind the current prompt design? Have other prompt variations been compared?
- Why does **MMEval-8B** perform poorly on **MMMU**?
soundness: 2
strengths: - The paper addresses the issue of data scarcity in instruction datasets for training multimodal large language models, presenting a well-motivated problem.
- The authors designed a sophisticated pipeline to enhance data quality, featuring three evolution stages and an instruction elimination stage.
- The authors evaluate the effectiveness of the three proposed evolution stages, as shown in Table 1.
- The performance of **MMEval** MLLMs in Table 2 appears promising.
summary: This paper addresses the lack of high-quality data in building stronger multimodal large language models and proposes a solution through the development of a multimodal instruction data evolution framework, MMEval. The framework iteratively improves data quality across three evolution stages: fine-grained perception, cognitive reasoning, and interactive evolution. The authors demonstrate the effectiveness of the framework by refining a new dataset called SEED-163K and training a large multimodal model on this refined dataset. Evaluation results across 13 benchmarks further validate the effectiveness of the proposed framework.
weaknesses: - The entire data improvement framework relies on closed-source frontier models like GPT-4, which suggests a form of knowledge distillation from these models but may limit the ability to scale to larger datasets. Additionally, the strong dependence on models like GPT-4 reduces the framework's interpretability.
- The authors do not provide an analysis of the reliability of using GPT-4 as a data rewriter.
- The paper lacks a comparison of different prompts used in each evolution stage, leaving the impact of prompt templates on data refinement unclear.
- When comparing the results of **MMEval-8B** with **Cambrian-1 8B** in Table 2, although **MMEval-8B** shows overall improvements, it exhibits significant performance declines on key benchmarks like **MMMU**, **AI2D**, and **MMStar**.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: ### Strength
1. The paper is well-written and easy to follow
2. The proposed idea is novel and seems to work well

### Weakness
1. My main concern is the comparison fairness in Table 2. The LLaVA-Next baseline uses the seed-set (163K data),  while the MMEvol uses an augmented set with more data(447K additional data). This leads to an unfair comparison, as in most cases, more data leads to better performance. This problem also exists in Table 1. The unfair comparison hinders the understanding of the actual effectiveness of the proposed method and I think it is easy to become fairer as the seed-set is sampled from existing open-source datasets and can easily be scaled up to a similar size with the augmented one.

2. Both the evolution and elimination are realized by the same model (GPT4o-mini). Is the model capable of finding out bad cases generated by itself? Further, I'm wondering about the fail rate and elimination rate of the proposed method.

3. Following 2, the proposed method evolute the instruction multiple times, will this lead to the error accumulation problem?

4. The paper focuses on improving the training data quality, while the provided example is quite limited. More data samples will help better evaluate the data quality.

I like the proposed idea and give it a 6: marginally above the acceptance threshold. But there are still some unclear problems I mentioned above. I will adjust the final score based on the rebuttal.
soundness: 3
strengths: Please refer to Questions
summary: In this paper, the authors propose MMEvol, a novel multimodal instruction data evolution framework that augments existing multi-modal training data with better diversity and complexity. The experiment results show that the proposed method works well and helps the MLLMs get better performance
weaknesses: Please refer to Questions

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: Refer to Weaknesses.
soundness: 3
strengths: This work proposes an image-text instruction evolution framework, Evol-Instruct, to enhance the quality and quantity of instruction data. Using three distinct evolution methods and instruction elimination to remove harmful data, the approach increases the complexity and diversity of a limited seed dataset. After three rounds of evolution, the resulting data trains a new model that achieves SOTA performance across various benchmarks.
summary: MMEvol addresses data quality and diversity challenges by proposing an iterative evolution of image-text instruction data. Starting with SEED-163K, it expands instruction types, enhances visual reasoning, and strengthens fine-grained perception and cognitive abilities.
weaknesses: - Evolving multimodal dataset makes sense and is so interesting but actual performnace improvements are too marginal in my perspective: 2~3% because evaluating language ability can improve large margin if it is really contrbutional approach. 

- Experiments should be compared for fair comparison where same architecture, same dataset to provide the effectiveness of MMEvol.

- What kind of dataset samples are more effective to be applited by MMEvol like Math, code, or anyhting?

---

I will keep my score becuase it seems improvments are marginal than what I've expected (avg. 5%p~10%p).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Fig 12: why does negative scaling happen at step 6k-7k? Is it related to certain instruction augmentation techniques?
- Table 1: What are the number of instructions for each row?
- Table 2: What's Qwen2-7B baseline performance?
- The quality, and maybe quantity, of the seed instruction set may affect the quality of the generated instructions. Please provide a study on this to evaluate the robustness of the proposed method.
soundness: 3
strengths: Data Evolution Framework: The MMEvol framework effectively enhances the diversity and complexity of image-text instruction data through iterative evolution methods, such as fine-grained perceptual, cognitive reasoning, and interactive evolution. This approach significantly improves the quality of data used for training MLLMs, addressing a critical bottleneck in existing data-driven methods.

Empirical Improvement: The proposed method achieves an average accuracy improvement of 3.1 percentage points over baseline models and reaches state-of-the-art performance in nine vision-language tasks. This demonstrates the efficacy of MMEvol in enhancing model capabilities with less training data compared to other approaches.

Data Balancing and Quality Analysis: The generated instructions are more compositional, longer in reasoning chains, and more balanced between objects. The statistics of the instructions suggest improved quality of instructions.
summary: The paper addresses the challenge of enhancing the quality and diversity of training data for Multimodal Large Language Models (MLLMs). Traditional model-driven approaches face diminishing returns, while existing data-driven methods are limited by the complexity and variety of available data. To overcome this, the authors propose MMEvol, a framework that iteratively refines image-text instruction data through fine-grained perceptual evolution, cognitive reasoning evolution, and interactive evolution. This process generates a richer and more diverse dataset, improving the models' visual understanding and reasoning capabilities. The approach leads to a significant performance boost across multiple vision-language benchmarks, achieving state-of-the-art results in several tasks with less data compared to existing methods. This paper contributes by advancing the capability of MLLMs through an innovative data evolution method that emphasizes the quality of instructions over sheer data volume.
weaknesses: Evaluation Limitation: While the paper claims that the instruction generation method is a key contribution, it lacks a direct comparison with other multimodal instruction generation techniques, such as MIMIC-IT or LLaVA-NEXT. To strengthen this evaluation, I suggest that the authors generate instructions using MIMIC-IT and LLaVA-NEXT methods on the same seed data and compare the quality, diversity, and complexity of the resulting instructions with those generated by MMEvol. This would help demonstrate how MMEvol performs relative to similar methods in the field, addressing the largest weakness of the paper.

Absence of Failure Case Study: The paper does not sufficiently explore the limitations or failure scenarios of MMEvol. I recommend that the authors provide concrete examples of cases where MMEvol might fail or produce suboptimal results. Additionally, conducting an analysis of how performance changes with increasing rounds of evolution could help identify potential saturation points, offering insights into when the method might negatively impact model performance.

Technical Contribution: The paper's technical contribution feels incremental compared to previous works like MIMIC-IT[1] or MM-Instruct[2], which also iteratively refine instructions from image metadata. To clarify the novelty of MMEvol, it would be helpful for the authors to explicitly outline which aspects of their method are similar to these prior works and what specific contributions MMEvol makes beyond them. This will help position the paper relative to existing methods and highlight any unique advancements.

[1] Li et al., MIMIC-IT: Multi-Modal In-Context Instruction Tuning, 2023.

[2] Liu et al., MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment, 2024.

Meta-review: This paper introduces a framework to improve multimodal models by evolving image-text instruction data. It shows some performance improvements with less data. The approach is interesting and shows promising results. However, concerns were raised about the comparison with other methods, and there’s not enough exploration of failure cases or scalability. At this stage, the paper lacks strong support for acceptance, and the authors are encouraged to revise their work, providing more clarity in their experiments.

Decision: Reject