Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data (2024)

Abstract: The remarkable multimodal capabilities demonstrated by OpenAI’s GPT-4 have sparked significant interest in the development of multimodal Large Language Models (LLMs). A primary research objective of such models is to align visual and textual modalities effectively while comprehending human instructions. Current methodologies often rely on annotations derived from benchmark datasets to construct image-dialogue datasets for training purposes, akin to instruction tuning in LLMs. However, these datasets often exhibit domain bias, potentially constraining the generative capabilities of the models. In an effort to mitigate these limitations, we propose a novel methodology for data collection, which synchronously synthesizes images and dialogues for visual instruction tuning. This approach leverages the combined capabilities of generative text-to-image models and large language models, facilitating the *Equal contributions Project Leader Corresponding Author Project page: https://github.com/icoz69/StableLLAVA creation of a dataset that is both diverse and scalable, and more importantly, customized to enhance the models’ performance across a broad spectrum of tasks. Our research includes comprehensive experiments conducted on various datasets. The results emphasize substantial enhancements in more than ten commonly assessed capabilities. Additionally, our model achieves state-of-the-art results across multiple widely recognized multimodal benchmarks.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 14512–14531 August 11-16, 2024 ©2024 Association for Computational Linguistics
*Equal contributions †Project Leader ‡Corresponding Author
Project page: https://github.com/icoz69/StableLLAVA
creation of a dataset that is both diverse and scalable, and more importantly, customized to enhance the models’ performance across a broad spectrum of tasks. Our research includes comprehensive experiments conducted on various datasets. The results emphasize substantial enhancements in more than ten commonly assessed capabilities. Additionally, our model achieves state-of-the-art results across multiple widely recognized multimodal benchmarks.

heading: 1 Introduction
text: The launch of OpenAI’s ChatGPT(OpenAI, 2023) has marked a significant milestone in artificial intelligence (AI), showcasing the advanced capabilities of Large Language Models (LLMs). These models, exemplified by GPT-4(OpenAI, 2023), demonstrate exceptional versatility by handling not just images but also excelling in tasks once difficult to accomplish. This includes understanding humor within images and acting as intelligent agents (Zhang et al., 2023a), aspects that highlight its revolutionary potential.
14512
However, despite these notable achievements, a crucial aspect remains undisclosed: the specific mechanics underlying GPT-4, particularly concerning the seamless integration of multimodal information into LLMs. This knowledge gap has prompted a concerted research effort to address this puzzle. Among the promising approaches, an emerging method receiving considerable attention involves the utilization of adapter-based techniques (Zhang et al., 2023b; Gao et al., 2023; Luo et al., 2023), which allow the training of a visual-to-text adapter that converts features from pre-trained visual models into LLM tokens, showing promise in achieving results comparable to GPT-4. The effectiveness of adapter-based methods stems from their ability to leverage the extensive pre-existing knowledge in large visual models and LLMs. By focusing on training a lightweight adapter, these methods avoid the computational expense of training comprehensive models from scratch, thereby offering a more efficient pathway to enhancing LLMs’ multimodal integration capabilities.
A prerequisite for implementing these frameworks is the availability of paired vision-text image data. Such datasets are essential for aligning visual and textual information, facilitating the LLMs’ understanding of complex human instructions. Analogous to instruction tuning in LLMs (Wei et al., 2021), this process is commonly referred to as visual instruction tuning. Existing methods (Dai et al., 2023; Liu et al., 2023c; Peng et al., 2023; Liu et al., 2023a; Chung et al., 2022) typically construct visual instruction tuning datasets by leveraging existing datasets, extracting information such as image captions, spatial locations, and categories to form dialogues. This approach maximizes resource utilization, creating a comprehensive and efficient training dataset for multimodal LLMs.
Despite the efficiency and simplicity of this approach to dataset construction, certain limitations still persist. Existing large-scale vision-text datasets, such as LAION (Schuhmann et al., 2022) and CC (Changpinyo et al., 2021), often contain noise. Consequently, training only a subset may inadequately align visual-text features for immediate user requirements. Moreover, benchmark datasets (Changpinyo et al., 2021; Schuhmann et al., 2022; Lin et al., 2014) often exhibit a domain bias, primarily in terms of image styles. For instance, prevalent datasets such as COCO (Lin et al., 2014) predominantly feature images from everyday life, while stylized images like cartoons
are rarely represented. Additional, these vision annotations may also constrain the types of dialogues generated from them. For example, almost none of the current datasets contain data to directly enhance the model’s ability to comprehend jokes presented in the images, an impressive feature of GPT-4. Moreover, as multi-image dialogues become increasingly integral to the practical application of multimodal LLMs, the current lack of comprehensive multi-image datasets further underscores the need for data enrichment in this domain.
In addressing these limitations, we propose a novel data collection approach to enhance visual instruction tuning. Building upon recent successes in generative AI, we leverage generative models to produce image-dialogue pair data for visual instruction tuning. More concretely, we employ ChatGPT to create data that includes image-generating prompts and content-based dialogues. We then utilize the text-to-image diffusion model, StableDiffusion (Rombach et al., 2022), to generate images based on these prompts. Finally, the synthesized images and generated dialogues are employed to train multimodal LLMs. Simultaneously generating both image content and dialogues enables the production of diverse training data, affording greater control over its nature and quality. This flexibility allows us to construct multi-turn dialogues and datasets involving multi-image reasoning, which are challenging to obtain from other benchmarks.
Additionally, our methodology can potentially integrate more advanced image generative models, such as Dall-E 3 (Betker et al., 2023), to provide higher-level control over image contents like specifying complex spacial relations. This advanced control could generate more complex instructions to enhance image understanding capabilities. Examples from our synthesized visual instruction tuning datasets are shown in Figure 1. Building upon the flexible pipeline outlined above, users can tailor the generation of data to enhance specific capabilities based on their task requirements. Furthermore, our method of generating both images and dialogues eliminates constraints on data volume, thereby facilitating the production for limitless scaling of the datasets.
To demonstrate the effectiveness of our proposed pipeline, we conducted extensive experiments. Our main contributions are as threefold:
• We develop a novel pipeline for generating vi-
sual instruction tuning datasets by leveraging text-to-image diffusion models.
• To showcase its flexibility, we have built a dataset with various form of capabilities including multi-image data, and our results have shown improvements across all abilities.
• Extensive experimental analysis on multiple benchmarks shows the effectiveness of the proposed method, outperforming baseline and existing SOTA approaches.

heading: 2 Related Work
text: Recent research (Zhu et al., 2023; Liu et al., 2023c) efforts in multimodal Large Language Models (LLMs) have yielded promising strategies to efficiently align the embeddings of other modalities with language tokens. This has made it possible to effectively utilize pre-trained encoders from other modalities and LLMs, which effectively reduces the computational burden and training time. While there are alternative research approaches that include training-free methods leverage expert models (Wu et al., 2023; Shen et al., 2023; Yang et al., 2023a), these are not the focus of our work here.
Adapter-based LLMs represent a significant research direction, introducing methods to connect modalities through learnable interfaces with minimal training efforts. These approaches (Zhang et al., 2023b; Gao et al., 2023; Luo et al., 2023; Liu et al., 2023c; Zhu et al., 2023; Yang et al., 2023b) allow for the use of pre-trained modal encoders, reducing the need for training from scratch. Variations include direct training of projection layers for embedding alignment and the use of learnable queries for extracting modality-specific information, as seen in models like LLaVA (Liu et al., 2023c) and Flamingo (Alayrac et al., 2022). Innovations such as the LLaMA-Adapter (Zhang et al.,
2023b) and LaVIN (Luo et al., 2023) have introduced lightweight and mixed-modality adapters, respectively, enhancing the field’s diversity.
Visual instruction tuning datasets are crucial for training multimodal LLMs, focusing on aligning modalities and enabling instruction following. Most existing methods (Liu et al., 2023c; Rotstein et al., 2023; Li et al., 2023a; Yin et al., 2023; Li et al., 2023c; Liu et al., 2023a; Zhang et al., 2023c) rely on benchmark datasets for constructing visual instruction tuning datasets, which may be limited by the fixed categories in annotations. Our method leverages well-trained image generation models to produce controllable image data, enhancing multimodal LLM capabilities and allowing for the integration of advanced generative models for specific guidance forms, offering a more flexible and diverse approach to dataset construction. Recently, the concept of employing synthesized multimodal datasets has been extended to the training of models for chart and video understanding (Han et al., 2023; Song et al., 2024), showcasing its broad applicability and potential in advancing model training across various domains.

heading: 3 Preliminary
text: To assess the effectiveness of our data generation strategy, we chose the open-sourced LLaVA (Liu et al., 2023c,b) as our multimodal LLM model. It should be noted that our pipeline is model-agnostic, making it applicable for various models. This section serves as a foundation, briefly summarizing the LLaVA model’s design and training methods to prepare for a thorough exploration of our pipeline. The reader may refer to the original publication (Liu et al., 2023c) for detail.
Architecture. The LLaVA model integrates Vicuna-13B (Chiang et al., 2023) as the language model with a pre-trained CLIP visual encoder ViTL/14 (Radford et al., 2021) for extracting visual
features, transforming these features into language embedding tokens through a linear layer. This linear layer was updated in LLaVA-1.5 (Liu et al., 2023b) with a two-layer MLP, replacing Vicuna13B with Vicuna-13B-v1.5 and increasing input image size to 336x336. A detailed illustration of this model structure can be found in Figure 2.
Training and datasets. LLaVA’s training focuses on visual instruction tuning with data triplets: images, questions, and answers, aiming for predictive accuracy in an autoregressive manner. The training comprises two stages: the first emphasizes modality alignment using 595K image-text pairs, optimizing the linear layer with static visual encoder and LLM weights. The second stage, using 158K multimodal dialogue data from COCO, extends optimization to the LLM’s weights for comprehensive modality integration. LLaVA-1.5 further enriches the dataset by incorporating additional data like Region-level VQA (Kazemzadeh et al., 2014; Krishna et al., 2017; Mao et al., 2016) and GQA (Hudson and Manning, 2019), expanding the second-stage dataset to 665K examples.

heading: 4 Methods
text: This section outlines our dual-generation approach for creating visual instruction tuning datasets, which synthesizes images and their corresponding dialogues, as illustrated in Figure 3. We detail each component below.

heading: 4.1 Image Generation
text: We employ StableDiffusion (Rombach et al., 2022) to generate images based on prompts that include weighted keywords reflecting aspects like subject, scene, style, and visual elements such as image quality and lighting. Keywords at the prompt’s
start are prioritized, with the possibility of adding emphasis using brackets. To encourage diversity and stability during image generation, we add capability-specific instructions and cautions during prompting ChatGPT. For instance, in the task of generating images for joke understanding, we direct ChatGPT to create prompts that would result in the generation of abnormal images, like a “giraffe walking through a narrow corridor”, which are unlikely to be found in reality. When generating multi-image data, pairs of prompts can be generated concurrently based on predefined specific criteria. For maximum effect, we ensure that the most crucial keywords are placed at the beginning of the generated prompts, which are double-bracketed for additional emphasis. Furthermore, we instruct ChatGPT to avoid generating prompts that are nonvisual, such as the act of growing. The instruction template for prompt generation is provided in the left part of Figure 4. The generated prompts are then used with StableDiffusion to produce visually realistic images, which are subsequently encoded by LLaVA’s vision encoder into visual tokens for LLMs.

heading: 4.2 Dialogue Generation
text: Following the generation of images, we utilize ChatGPT to generate dialogues based on the same prompts used for image synthesis, aligning with LLaVA’s training objectives: the first stage focuses on aligning visual-text data, and the second on processing diverse instructions. Dialogues for the initial stage describe the images, where ChatGPT generates answers to predefined questions about the images’ content. Taking the example of the “giraffe walking through a narrow corridor”, a representative dialogue might be: “Question: What
is unusual in the image? Response: In reality, a giraffe is too tall and big to walk through a narrow home corridor.” The detailed instruction template for dialogue generation is shown in the right part of Figure 4.
For the second stage, dialogues aim to enhance reasoning across multiple images, addressing similarities, differences, and logical connections, and include multi-turn dialogues that blend image and text. We guide ChatGPT to produce a range of question types, steering clear of inherently ambiguous questions to ensure clarity and accuracy, detailed template can be found in the supplementary materials.

heading: 4.3 In-Context Examples
text: ChatGPT’s in-context learning capability, which allows it to grasp the essence of tasks from a few examples, is leveraged in our methodology. We incorporate in-context examples in the generation of StableDiffusion prompts and dialogues to enhance this learning process. During the data generation process, we observed that ChatGPT sometimes produced a lack of diversity. For example, when generating colors, the outputs frequently revolved around common color categories. To overcome this, we in-
dependently generate ability-related keywords such as color categories with ChatGPT, and utilize them as a reference during the prompting process. This additional step promotes a more diverse range of prompts, thereby enriching our visual instruction tuning dataset.
We further adopt a dynamic strategy to maintain and increase diversity: periodically substituting a portion of the original in-context examples with newly generated data. This continuous update prevents over-repetition and ensures the dataset’s comprehensiveness and representativeness, maintaining a balance that contributes to a richer and more diverse visual instruction tuning dataset.

heading: 4.4 Data Filtering Mechanism
text: To augment the quality and diversity of generated dataset, we introduced a data filtering mechanism. This mechanism encompasses several key aspects:
• Repetition Rate Filtering Initially, we filter the generated dataset based on the repetition rate of the extracted prompts. The selected prompts are subsequently utilized to generate corresponding images and dialogues, ensuring novelty and reducing redundancy. • Length-based Filtering We keep prompts
short, with no more than ten keywords, to make sure they’re clear and focused. Long prompts can make it hard for model to create accurate images. We also limit dialogues to 500 characters to keep responses concise. • Task-Specific Restrictions For certain categories, we implemented restrictions based on specific attributes of the capabilities. For example, when generating content related to construction workers, the model tended to focus on buildings. To address this, additional human attributes were incorporated into the prompts to ensure a more accurate prompt. • Alignment Check To ensure a high degree of alignment between generated images and dialogues, we employ the CLIP (Radford et al., 2021) model to compute matching scores for both texts and images. Data entries with scores exceeding a predefined threshold, set at γ = 0.25, are retained, thereby filtering out less relevant matches and elevating the overall data quality.

heading: 5 Experiments
text: In this section, we detail the experiments conducted to validate the effectiveness of our novel data collection approach for visual instruction tuning. We describe the training datasets, evaluation strategy, and both quantitative and qualitative outcomes.

heading: 5.1 Training Datasets
text: We generate a diverse and expansive dataset to show its versatility, covering single-image capabilities from basic recognition to complex visual reasoning. This includes understanding physical attributes, life features, and man-made items, among
others, amounting to 38K image-dialogue pairs for initial training. Each ability’s dataset was formulated following a standard template, illustrated in supplementary material. In addition, we also generated a dataset of 3K multi-image instances, encompassing descriptions of image similarity, difference, logical relations, and multi-turn dialogue data for the second stage. These datasets, in combination with the raw LLaVA dataset, provides a comprehensive training set in our experiments.

heading: 5.2 Evaluation Metrics
text: Evaluation datasets. To demonstrate our performance more clearly, we tested on a series of public multi-modal datasets, including VisWiz (Gurari et al., 2018), MM-Vet (Yu et al., 2023), MME (Fu et al., 2023), and MMBench (Liu et al., 2023d).
Subsequently, we established a real-image benchmark to evaluate training effectiveness across a wide range of single-image abilities, including 330 test samples of real images with associated question-answer dialogues, carefully selected and annotated from public repositories. This comprehensive benchmark aims to rigorously test the models’ single-image capabilities.
Simultaneously, we constructed a multi-image test set consisting of 30 dialogues to assess the models’ performance on this specific data type. This dataset evaluates the models across differences, similarities, and reasoning relationships among the images. The test data was sourced from publicly available datasets and manually annotated.
Evaluation strategy In terms of the evaluation process, we employ different testing strategies depending on the benchmarks used.
Our evaluation approach varies with the bench-
mark. For multimodal datasets like VizWiz (Gurari et al., 2018) and MMBench (Liu et al., 2023d), we follow official guidelines, converting test data to a compatible format for our model, and using official scripts or submission portals for assessment, primarily focusing on accuracy.
In evaluating the diverse capabilities we’ve generated, We adopted two evaluation methods, namely manual evaluation and evaluation based on GPT-4 score.
Initially, the participants were instructed to assess the answers produced by our model and those of the baseline for all abilities according to the label (1 for correct, 0 for incorrect), averaging these as the final metric.
Subsequently, inspired by (Chiang et al., 2023; Liu et al., 2023c), we leverage GPT-4 (OpenAI, 2023) to assist in scoring model outputs. We have established six scoring levels, ranging from 0 to 5. Each score level is accompanied by detailed descriptions of the evaluation criteria, and we assist GPT-4 in better assessment by providing a series of scoring examples. In particular, a score of 0 indicates that the predicted answer has no relevance to the reference answer, while a score of 5 signifies that the predicted answer aligns seamlessly with the annotated reference answer without any deviation. Drawing on our manual annotations, we conduct evaluations on the results produced by each model. The average GPT-4 score within each test set serves as the ultimate metric for our benchmark evaluations. The specific instruction template used for result evaluation is illustrated in supplementary material.

heading: 5.3 Quantitative Comparison with State-of-the-arts
text: Public multimodal benchmarks We perform quantitative performance comparisons against various state-of-the-art methods on different benchmarks, as illustrated in Table 2. Utilizing LLaVA1.5-13B as the baseline, we integrate our synthesized data with its original dataset for training. Training is carried out with identical parameter configurations as LLaVA-1.5. The outcomes demonstrate substantial improvements on many benchmarks, emphasizing the enhanced performance
achieved by our approach. Comparison of various abilities. To validate the effectiveness of our generated data, we conducted comprehensive tests on distinct capabilities, employing both manual evaluation and GPT4 score assessments. Employing LLaVA-13B as our baseline, the quantitative comparison of the baseline results and ours for manual evaluation are shown in Table 1, while the GPT-4 score assessment results are shown in the left part of Figure 5. Notably, our trained model consistently outperforms the LLaVA-13B baseline across all various capabilities on two metrics, which suggests the synthesized datasets’ generalizability and our pipeline’s robustness.
Besides, we conduct a comparison of subcategory performance on MMBench to better validate our superiority, using the LLaVA-1.5-13B as the baseline. The tested subcategories in MMBench encompass six aspects: attribute reasoning (AR), coarse perception (CP), fine-grained perception (cross-instance) (FP-C), fine-grained perception (instance-level) (FP-S), logic reasoning (LR), and relation reasoning (RR). The final results are shown in the right part of Figure 5, indicating better performance of subcategory on MMBench, which also attests to the high quality of our generated data.
Comparison on multi-image benchmark In order to validate the effectiveness of multi-image
capabilities, we manually curated a benchmark of real images. The evaluation metric used was the GPT-4 score mentioned. We used LLaVA-13B as the baseline and incorporated multi-image data in the second training phase. Since LLaVA itself lacks the capability for multi-image input, we modified the testing code for LLaVA to enable it to accept multiple sets of images simultaneously. The comparison with LLaVA results is shown in the Table 3, indicating a notable improvement across various multi-image capabilities despite adding less multiimage data in the process.

heading: 5.4 Qualitative Results
text: Supplementing the quantitative analysis, we provide a qualitative comparison between our model’s results and LLaVA-13B in Figure. 6 on multi-image data. Our model exhibits a heightened ability to adhere to question instructions, rendering more precise answers.
We compare our approach with the LLaVA-13B baseline, revealing its limitations: it struggles to differentiate between multi-image contents and
provides incomplete answers to questions. Our method, incorporating multi-image data, enhances the model’s understanding of multiple images, demonstrating its effectiveness. Additional qualitative results will be included in the supplementary materials.

heading: 6 Conclusion and Future Work
text: In the rapidly evolving realm of Large Language Models, efficiently integrating multimodal information is a key research focus. In this study, we introduced an innovative data collection method to enhance visual instruction tuning for multimodal models. Compared to existing strategies, our approach uniquely combines image and dialogue generation, effectively addressing limitations found in benchmark datasets. By carefully crafting instruction templates, our method ensures high-quality training data covering a broad range of crucial capabilities for multimodal models and users can generate customized data based on their specific requirements.
Our research opens avenues for exploration.
Moving forward, we aim to leverage advanced generative models to enhance model abilities, including spatial comprehension and fine-grained recognition. With promising results from our dualgeneration method, forward-thinking data collection techniques are poised to play a significant role in the future of LLM research.

heading: 7 Limitations
text: Due to constraints in text-to-image models like stable diffusion, generating certain data types, such as text-rich images and tables, is not effective in the current pipeline. We anticipate these constraints will be addressed with ongoing advancements in text-to-image generation techniques.

heading: 8 Ethics Statement
text: Our method leverages generative models to create synthetic images and dialogues. It is imperative to ensure that the generated content does not perpetuate or amplify biases present in existing datasets or societal prejudices. We have implemented data filtering mechanism to minimize the generation of potentially harmful or biased content. However, continuous vigilance and improvement of these filters are necessary as generative models evolve.
The enhanced capabilities of multimodal LLMs, facilitated by our data generation approach, could potentially be misused for creating deceptive or manipulative content. It is crucial to develop and adhere to guidelines that prevent the misuse of such technology, including transparent disclosure of synthetic content’s nature and purpose.

heading: 9 Acknowledgements
text: This project is particularly supported by ARC DP210101347 and DP240101349.

heading: Appendix for “Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue
text: Data”

heading: A Training Details
text: During the model training phase, we employed the original LLaVA configuration as the foundation for our training process. In both stages, we utilized 8 NVIDIA V100 GPUs. To conserve GPU memory, we employed deepspeed with zero3 during model training, disabling tf32 and opting for fp16. The remaining parameters, including epochs and learning rates, were set according to the original LLaVA configuration. For specific parameter details, please refer to the the original publication.

heading: B Multi-image Generation Template
text: We have illustrated the instruction template for generating multi-image data in Figure 7. To enhance the diversity of the data, we first use ChatGPT to generate a series of paired prompts. Subsequently, based on these paired prompts, we generate corresponding captions. As shown in the Figure 7, we can generate different types of data by configuring different examples and replacing similarities with difference or logical relations.
C Interleaved multi-turn dialogues Generation Template
We present a template for generating interleaved multi-turn dialogues as Figure 8. Similar to multiimage generation, we randomly generate a series of non-repeating phases to enhance data diversity. Based on these phases, we generate interleaved multi-turn dialogues. The red sections are replaceable. In our generated dialogues, the focus is primarily on instructional content related to the use of everyday objects and recipes. Users can make specific adjustments according to their needs. Finally, the brackets part represents the prompt used for stable diffusion to generate images. During the model training process, the content within the brackets will be replaced with an image_placeholder.

heading: D GPT-4 Scoring Criteria
text: As shown in the Figure 9, we present our detailed GPT-4 scoring system. We have established a scale of 0-5 with six levels of scores, and for each score, we provide detailed evaluation criteria along with specific examples for assessment. Utilizing the
template in the Figure 9, evaluations are conducted for each model, and the average of the results is taken as the final score.

heading: E More qualitative Results.
text: We showcase more generated data as shown in the Figure 10 11, which includes various capabilities, as well as multi-image dialogues and interleaved multi-turn dialogues, to validate the effectiveness and diversity of our pipeline.
We present additional outcomes of our methodology on more real images in Figure 12. Spanning diverse capabilities like profession recognition, color categorization, or anomaly detection, our approach consistently exhibits efficacy. In particular, our method demonstrates a robust capability in discerning abnormal images.
Simultaneously, we validated our model on multi-turn interactive dialogues, and the results are shown in the Figure 13 14. It can be observed that our model can provide detailed descriptions of each step, which is attributed to the high-quality data we generated.

heading: Future Work
text: With promising results from our dualgeneration method, forward-thinking data collection techniques are poised to play a significant role in the future of LLM research