Generative Visual Instruction Tuning

## Abstract

Abstract We propose to use automatically generated instruction-following data to improve the zero-shot capabilities of a large multimodal model with additional support for generative and image editing tasks.
We achieve this by curating a new multimodal instruction-following set using GPT-4V and existing datasets for image generation and editing. Using this instruction set and the existing LLaVA-Finetune instruction set for visual understanding tasks, we produce GenLLaVA, a Generative Large Language and Visual Assistant.
GenLLaVA is built through a strategy that combines three types of large pretrained models through instruction finetuning: Mistral for language modeling, SigLIP for image-text matching, and StableDiffusion for text-to-image generation.
Our model demonstrates visual understanding capabilities superior to LLaVA and additionally demonstrates competitive results with native multimodal models such as Unified-IO 2, paving the way for building advanced general-purpose visual assistants by effectively re-using existing multimodal models.
We open-source our dataset, codebase, and model checkpoints to foster further research and application in this domain 1 1 1 https://github.com/jeffhernandez1995/GenLlaVA.git .

## 1 Introduction

The field of multimodal models has become increasingly popular in the research community as they are one of the key building blocks for general-purpose assistants (Achiam et al., ; Gemini Team et al., ; Bai et al., ). One of the main directions researchers have pursued is to combine Large Language Models (LLMs) with Vision Models for multimodal tasks *i.e*. creating LVLMs. The recently proposed LLaVA model (Liu et al., ) is among the latest wave of works that have demonstrated the effectiveness of instruction tuning for multimodal models (Liu et al., ; Zhao et al., ; Wang et al., ; Karamcheti et al., ). In these works, a two-stage pipeline is followed: (1) multimodal pre-training where the unimodal models are combined and trained on a large corpus of captioning data Schuhmann et al. (); Ordonez et al. (); and (2) a supervised fine-tuning (SFT) Liu et al. (, ) stage where the model is trained on domain-specific data and enables it to better perform various downstream tasks of interest.

Visual generation is another research direction that combines visual and language modalities. There are two common approaches for text-to-image generation. One approach employs diffusion models (Rombach et al., ), which have shown unprecedented performance in image synthesis, becoming the de facto method for visual generation. The other line of work converts visual content into discrete tokens using vector quantization (VQ) and then leverages an autoregressive transformer for high-quality image synthesis (Chang et al., ; Lee et al., ).

As visual understanding and generation capabilities advance rapidly and independently, a growing trend is to combine these into a unified Large Multimodal Model (LMM). There are two main approaches to achieving such unification. Many LMMs (Koh et al., ; Sun et al., ; Fu et al., ) produce conditional embeddings to be used by a pretrained diffusion model for image generation. On the other hand, there are LMMs (Lu et al., ; Chameleon Team, ; Yu et al., ) that adopt VQ encoders to project visual inputs into discrete tokens and use the same next-token prediction paradigm as Language Models.

There has been a considerable amount of work building on top of the LLaVA model
ranging from image generation (Koh et al., ; Sun et al., , ), grounding (You et al., ), image editing (Fu et al., ) to video understanding (Lin et al., ). These works share the same principles; they extend the ideas of visual instruction tuning to one or more capabilities.
However, after adding a new capability (i.e., image generation), the resulting models often lose some, if not most, of their visual and language understanding capabilities.
We propose a strategy that leads to a model that can perform generative tasks while retaining multimodal understanding capabilities.

Figure: Figure 1: Comparison of GenLLaVA against recent architectures. Unlike BLIP-2 (Li et al., ), we use a Linear projector similar to the LlaVA architecture (Liu et al., ). Generation capabilities are added using a diffusion model, but unlike GILL (Koh et al., ), we use a Q-former as the generation head. Finally, our model benefits from using a stronger visual encoder, namely SigLIP(Zhai et al., ); a stronger LLM, namely Mistral-7b (Jiang et al., ); and a stronger diffuser, namely SDv1.4 (Rombach et al., ). ^∗ L stands for Linear projection, and Q stands for Q-former resampler.
Refer to caption: x1.png

In this paper, we present generative visual instruction tuning, an approach in which we teach a Large Multimodal Model (LMM) image understanding, image generation, and image editing tasks without diminishing the performance of each individual capability. (See Fig. 1 for an overview of our method.)
To our knowledge, this is the first time such capability has been achieved, and our findings pave the way for building a general-purpose visual assistant. Our contributions are the following:

- •
Generative multimodal instruction-following data. Inspired by Liu et al. (), which curated an instruction set for image understanding tasks, we curate a multimodal instruction tuning set that combines image understanding, image generation, and image editing data.
- •
A single composite model, *i.e*. GenLLaVA, which unifies visual understanding and generation using open-source models. GenLLaVA is trained using a single-stage training recipe, unlike its predecessor LLaVA Liu et al. ().
- •
Open source. We will publicly release our generated multimodal instruction data, code to replicate our results, model checkpoints, and a visual chat demo.

## 2 Related Work

#### Large Multimodal Models (LMMs).

Large Multimodal Models (LMMs) refer to large language models that can understand various modalities beyond human language. Some research efforts are focused on combining image, audio, video, and other modalities with language  (Zhan et al., ; Lu et al., ), while others aim to enhance the fusion of vision knowledge and language. For example, BLIP-2 (Li et al., ) created a large-scale image captioning dataset and paired a language model with a vision encoder to produce a robust multimodal model. Following this, LLaVA (Liu et al., ) developed a cost-effective approach to train an advanced LLM through visual instruction tuning. Although LLaVA-NeXT (Liu et al., ) improved performance for single-image tasks, it required over 2,000 tokens per image, which is about four times more than the original LLaVA. More recent models such as QwenVL (Bai et al., ), CogVLM (Wang et al., ), and Yi-VL (Young et al., ) follow architectures similar to those of LLaVA. Our proposed method not only focuses on models for multimodal understanding but also on adding generative capabilities to such models.

#### Diffusion-based LMMs for visual generation

We review works that combine diffusion with autoregressive prediction to create LMMs for generative tasks.
For instance, GILL (Koh et al., ) translates the hidden representations of an LLM into embeddings that correspond to a text-to-image model by learning a neural network to perform efficient mapping using the text encoder of the diffusion model.
MGIE (Fu et al., ) adapts the text embedder, image input adapter, and LM head output parameters of an LMM jointly with a diffusion model for image editing from instructions.
DreamLLM (Dong et al., ) uses the same paradigm as GILL and MGIE but instead trains on interleaves documents for visual generation and understanding synergy.
Transfusion (Zhou et al., ) combines the next-token production objective with the using a bidirectional casual mask on the visual tokens in a single unified model to understand and generate both discrete
and continuous modalities. Show-O (Xie et al., ) is concurrent to Transfusion and similarly uses a bidirectional casual mask on the visual tokens but uses an extra masking loss similar to MaskGIT (Chang et al., ).

#### Token-based LMMs for visual generation

We review works that project visual features into discrete tokens and use next-token prediction for generative tasks.
AnyGPT (Zhan et al., ) discretizes data from multiple modalities, extends the existing LLM vocabulary to add the extra modalities, and incorporates new randomly initialized parameters that enable additional input embeddings and prediction outputs.
Codi-2 (Tang et al., ) follows a similar line and adds multiple decoders for image, audio, and video generation to condition an LLM to generate interleaved outputs from different modalities.
CM3leon (Yu et al., ) proposes an early-fusion token-based decoder-only mixed modal model based on the CM3 architecture that is capable of both text and image generation and editing.
Unified-IO 2 (Lu et al., ), Chameleon (Chameleon Team, ) and GPT-4o OpenAI () take the early-fusion fully multimodal approach training the model from scratch to be able to expand the number of supported tasks and modalities.

## 3 Method

Figure: Figure 2: Editing capabilities of our model. GPT4 currently uses a version of the DALLE text-to-image model as a tool and, hence, is not directly able to edit images. GPT4o instead uses tools through Python-generated code to accomplish the requested action. Our model, GenLLaVA, connects input features obtained from CLIP to a language model that also produces output embeddings for a text-to-image StableDiffusion model, achieving an end-to-end editing task with a multimodal model.
Refer to caption: x2.png

### 3.1 Background: Large Multimodal Models

Large language models (LLMs) excel in natural language generation, while Large Multimodal Models enhance LLMs with the ability to interpret images and respond accordingly. Built upon a pre-trained LLM, the LMM incorporates a visual encoder (e.g., CLIP (Radford et al., )) to derive visual features $f$, along with an adapter $\mathcal{W}$ that maps $f$ into the language domain. Following the training methodology of LLaVA (Liu et al., ), this process is encapsulated in the equation:

$$ $\displaystyle\mathcal{C}$ $\displaystyle=\{x_{1},x_{2},\ldots,x_{l}\},$ (1) $\displaystyle f$ $\displaystyle=\text{Enc}_{\text{vis}}(\mathcal{V}),$ $\displaystyle x_{t}$ $\displaystyle=\text{LMM}(\{x_{1},\ldots,x_{t-1}\}~{}|~{}\mathcal{W}(f)),$ $$

where $l$ represents the number of tokens within $\mathcal{C}$. The set $\mathcal{C}$ can represent an image caption (Features Alignment) or multimodal instruction-following data (Instruction Tuning). The LMM employs the standard autoregressive method for next-token prediction, allowing it to function as a visual assistant across diverse tasks such as visual question answering and complex reasoning. Despite gaining visual perceptive abilities through this training, its responses are currently constrained to text.

### 3.2 Visual Generation in Large Multimodal Models

We append $N$ visual tokens [IMG] after the instruction $\mathcal{E}$, with their word embeddings being trainable. The LMM learns to generate these tokens through its language modeling (LM) head. These visual tokens represent visual-related instruction comprehension within $\mathcal{E}$ and form a bridge between the language and vision modalities. We follow the same visual generation framework of GILL (Koh et al., ) and MGIE (Fu et al., ) in extracting visual features, which we summarize here for succinctness.

We employ a generation head $\mathcal{T}$ to convert [IMG] into concrete visual guidance. The model $\mathcal{T}$ is a sequence-to-sequence model that translates the sequential visual tokens from the LMM into the semantically meaningful latent set $\mathcal{U}=\{u_{1},u_{2},\ldots,u_{L}\}$ for visual guidance:

$$ $u_{t}=\mathcal{T}(\{u_{1},\ldots,u_{t-1}\}~{}|~{}\{e_{\texttt{[IMG]}}+h_{ \texttt{[IMG]}}\}),$ (2) $$

where $e$ denotes the word embedding and $h$ is the hidden state (from the final layer of the LMM before the LM head) of [IMG]. Specifically, the transformation applied to $e$ serves as a broad visual representation, while $h$ provides an instance-specific visual latent that reflects both the original image and the text conditioning the generation.

To guide image generation with the visual latent information $\mathcal{U}$, we employ a latent diffusion model (Rombach et al., ), incorporating a variational autoencoder (VAE) for handling denoising diffusion in the latent space. First, we encode the desired visual output via the diffusion model encoder $o=\text{Enc}_{\text{VAE}}(\mathcal{O})$; this output may be intended for image generation or editing tasks. The diffusion process progressively introduces noise into $o$ as $z_{t}$, increasing the noise level over timesteps $t$. We then train the UNet $\epsilon_{\theta}$ to predict the added noise (Ho et al., ). The diffusion process is conditioned on the visual latent information $\mathcal{U}$ through a cross-attention layer, defined as $\text{Attention}(Q,K,V)=\text{softmax}(\frac{QK^{T}}{\sqrt{\text{dim}}})\cdot V$, where:

$$ $Q=W^{(i)}_{Q}\cdot\varphi_{i}(z_{t}),K=W^{(i)}_{K}\cdot\mathcal{U},V=W^{(i)}_{ V}\cdot\mathcal{U},$ (3) $$

with $\varphi$ representing the flattening operation, and $W^{(i)}_{Q}$, $W^{(i)}_{K}$, and $W^{(i)}_{V}$ being learnable attention matrices. We apply classifier-free guidance (Ho & Salimans, ), where the score estimation $s_{\theta}$ is extrapolated to deviate from the unconditional $\varnothing$, following standard practices in diffusion models.

## 4 Experiment Settings

Figure: Figure 3: Qualitative conversational example of our model. The dashed line indicates that the conversation has to be restarted from the beginning due to the model losing track of it.
Refer to caption: x3.png

### 4.1 Generative Visual Instruction Data

Multimodal instruction tuning is a crucial process that equips the model with a wide range of skills and capabilities across different modalities while also enabling it to adapt to novel and unique instructions. We build the multimodal instruction tuning dataset by aggregating a diverse set of supervised datasets and tasks. Each task is provided with a clear prompt, either by using existing prompts or crafting new ones using GPT4-V.

#### Natural Language. [1.93%]

We use the publicly available ShareGPT (ShareGPT, ) dataset, which was used to train the Vicuna LLM (Chiang et al., ). This dataset contains mostly English natural language conversations but also contains code and markdown. We filter inappropriate or low-quality entries following the same methodology of Chiang et al. (). As a final preprocessing step, entries that surpass 2048 tokens are truncated rather than split into multiple conversations. This results in $\sim$40K conversations.

#### Image Editing. [9.63%]

We create a subset of the Instruction Prompt-to-Prompt dataset (IPr2Pr) (Brooks et al., ) for editing our editing data. We use $\sim$200K from the CLIP-filtered data version of IPr2Pr, where editing instructions are generated by GPT-3, and images are synthesized by the Prompt-to-Prompt model (Hertz et al., ).

#### Image Generation. [26.88%]

For text-to-image generation, we use the same image & text pairs that were used to pre-train the LLaVA model. This dataset, named LLaVA-Pretrain, is inverted and presented to our model in the format (caption, image) with a dynamically pre-generated *e.g*. “Please generate an image of caption”. These prefixes are created using the GPT4 language model. This dataset contains $\sim$558K data points originally sourced from the LAION (Schuhmann et al., ), SBU (Ordonez et al., ), and CC3M (Changpinyo et al., ) datasets and captioned by the BLIP-2 model (Li et al., ).

#### Image Understanding.[61.56%]

For image understanding, we combine the dataset used to fine-tune the LLaVA model. This dataset, named LLaVA-Finetune, contains $\sim$665K samples. We also add the LVIS-INSTRUCT4V (Wang et al., ) dataset—a new visual instruction tuning dataset constructed in the same way as the original LlaVA dataset but using GPT4-V (Achiam et al., ) as the captioner instead of BLIP-2 (Li et al., ). We remove duplicates from the resulting dataset; this results in $\sim$880K samples. We additionally add the following instruction datasets:

- •
LRV-Instruction (Liu et al., ) ($\sim$80K) a diagram undestanding and hallucination reduction dataset.
- •
laion-gpt4v-dataset ($\sim$15K) a subset of the LAION (Schuhmann et al., ) dataset with high-quality captions created using GPT-4V (Achiam et al., ).
- •
ShareGPT4V (Chen et al., ) ($\sim$100K) a conversational dataset created using publicly available conversations that users had with the GPT-4V model.
- •
Datasets for documents, chart and OCR understanding such as DocVQA (Mathew et al., )($\sim$50K), SynDog-EN (Kim et al., )($\sim$65K), ChartQA (Masry et al., )($\sim$23K), DVQA (Kafle et al., ) ($\sim$50K) and AI2D (Kembhavi et al., ) ($\sim$15K).

### 4.2 Training details.

In this section, we evaluate our model on a broad range of tasks that require visual understanding and generation. We do not perform task-specific finetuning in any experiments. The Supplementary section details additional results on GenLLaVA ’s instruction capabilities.

We adopt LLaVA-v1.5-7B (Liu et al., ) architecture, then tune it on the constructed GVIT-mix-2076K. We named this model GenLLaVA and it is made of the following components:

- •
Image Processing & Visual Representations. We implement all image processing logic using the default image transforms provided by torchvision and the TIMM library (Wightman, ). We normalize pixel values using the default ImageNet values. The default backbone employed by all visual representations $\text{Enc}_{\text{vis}}$ that we evaluate in this work is a Vision Transformer (Dosovitskiy et al., ); we extract patch features from the penultimate layer, following LLaVA (Liu et al., ).
- •
Vision-Language Projector. We use a simple 2-layer GELU MLP as the projector $\mathcal{W}$, which projects each patch independently into the embedding space of the language model.
- •
Language Model. We choose the Mistral-7B LLM (Jiang et al., ). In order to combine the projected visual patch embeddings, we perform simple sequence-wise concatenation, placing the patch embeddings before the text embeddings.
- •
Visual Generation Head. The generation head $\mathcal{T}$ is a lightweight 4-layer encoder-decoder Transformer, which takes word embeddings $e$ and hidden states $h$ from the [IMG] tokens, as well as $L$ learnable query tokens as the input and generates the visual latent $\mathcal{U}$, we use the $L=77$, and the dimension of each $u_{t}\in\mathcal{U}$ is $768$.
- •
Diffusion Image Decoder. We adopt Stable Diffusion v1.4 (SDv1.4) (Rombach et al., ) trained on 512×512 resolution. Similar to the visual encoder, the SD model is frozen without any modifications or training throughout the whole process.

**Table 1: Main result. Comparison of various models across advanced knowledge and general understanding. ^⋆ MGIE was not originally designed for these tasks, as it is purely an editing model. For VQA, we take the generated caption as the answer, and when asking it to generate entirely new images, we provide a blank image as the prompt. We intend to show that models lose previous capabilities when we add a new one.**
| Model name | Advanced Knowledge | General Understanding | Editing | Generation |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MathVista | MMMU | MMVet | SEED-B | MMB | EVR | CC3M | COCO |  |
| GILL (Koh et al., ) | 18.6 | 26.8 | 13.0 | 29.4 | 38.2 | 30.4 | 15.3 | 0.67 |
| AnyGPT (Zhan et al., ) | 24.4 | 24.0 | 14.8 | 28.0 | 36.0 | 40.3 | 14.3 | 0.65 |
| MGIE^⋆ (Fu et al., ) | 15.5 | 25.6 | 13.0 | 28.8 | 6.6 | 71.5 | 13.6 | 0.66 |
| Chameleon (Chameleon Team, ) | 22.3 | 22.4 | 10.3 | 30.5 | 15.4 | - | 10.2 | 0.78 |
| Unified-IO 2 (Lu et al., ) | 28.3 | 35.5 | 36.6 | 61.6 | 57.9 | 50.2 | 13.4 | 0.72 |
| GenLLaVA (Ours) | 30.5 | 37.1 | 35.8 | 64.5 | 66.8 | 66.9 | 12.5 | 0.73 |

We implement our training codebase in PyTorch. We train all models in BF16 mixed precision. For a fair comparison, the rest of the model training protocol is kept unchanged from the original LLaVA. Generative Visual Instruction tuning takes about 48 hours for both full-parameter tuning and LoRA tuning on 8 NVIDIA Tesla A100 GPUs, each with 48GB memory, with DeepSpeed ZeRO Stage 3 (Rajbhandari et al., ) to distribute training across GPUs.

#### Single-stage training.

Unlike its predecessor LLaVA (Liu et al., ), our model does not use a two-stage training pipeline and instead directly finetunes the Vision-Language projector, the Language model, and the Visual Generation Head. We found that the logical extension of the original pipeline—a three-stage training pipeline—consisting of (1) multimodal alignment, (2) instruction tuning, and (3) image generation tuning to teach the models progressively simply does not work, and the model performance on visual understanding tasks decreases significantly. We instead choose a single-stage pipeline as it has been shown to work previously in the work of Karamcheti et al. (). This comes with other unintentional advantages: training cost is reduced by 20%, and we can use the LLaVA-Pretrain data as image-to-text data instead of having to collect more.

#### Task Tokens.

To perform unified learning on multimodal understanding and generation, we
introduce special tokens, which we name task tokens, to format the data. Specifically, we create the tokens [T2I] and [I2T] to indicate the learning (generation or understanding) task for the input sequence. We keep the original special token from the language model that indicates the start and end of the text. Similarly, [SOI] and [EOI] are pre-defined special tokens marking the start and end of visual tokens for generation. Without these task tokens, the model has trouble inferring the user intention and would generate an image when it is not necessary. We remark that this methodology is not new and has been used before by others (Lu et al., ; Xie et al., ).

### 4.3 Evaluation Details.

#### Visual Understanding

We evaluate vision-language performance and compare it against other generalist models, i.e., models that can do visual generation and understating. Results on a collection of 5 vision/language benchmarks are shown in Table 1. These benchmarks are designed to probe advanced knowledge and general understanding. MMBench (Liu et al., ) evaluates answer robustness through comprehensive shuffling of multiple-choice options. SEED-Bench (Li et al., ) tests model performance on images and videos with multiple-choice questions. MM-Vet (Yu et al., ) evaluates the ability to engage in visual conversations and assess the accuracy and helpfulness of responses. Mathvista (Lu et al., ) consolidates mathematical reasoning benchmarks, focusing on logical and algebraic reasoning with puzzle tests. MMMU (Yue et al., ) covers 57 subjects across STEM, humanities, and social sciences, ranging from elementary to advanced professional levels, and tests both world knowledge and problem-solving skills.

Figure: Figure 4: (Left) Results on selected Visual Question answering datasets. (Right) A qualitative example of our model.
Refer to caption: x4.png

#### Visual generation.

For visual generation evaluation, we measured the Fréchet Inception Distance (FID) (Heusel et al., ) on the CC3M validation set (Changpinyo et al., ) (which is a measure of image realism) and CLIP Similarity on the MS-COCO dataset (Lin et al., ) (which is a measure of the alignment between the text prompt and the generated image). For image editing, we measured the DINOScore on the 5.7K validation examples of the EVR dataset (Tan et al., ) following the same protocol as Fu et al. ().

### 4.4 Main Result

We evaluate GenLLaVA against similar models across various tasks, covering advanced knowledge, general understanding, and visual generation tasks. As shown in Table 1, GenLLaVA demonstrates consistent improvements, surpassing models like GILL, AnyGPT, and MGIE in most cases.

For advanced knowledge tasks such as MathVista and MMMU, GenLLaVA performs exceptionally well, achieving the top scores compared to other models. In general understanding, GenLLaVA excels, particularly on SEED-B and MMB, significantly outperforming its competitors. Although MGIE specializes in image editing and maintains a slight edge in this area, GenLLaVA holds its own and outperforms other general models on the EVR dataset.

In visual generation tasks, GenLLaVA shows competitive performance, with results that closely match those of Unified-IO 2 on CC3M and CLIP Similarity while consistently maintaining an edge over other models. These results demonstrate GenLLaVA ’s versatility and ability to achieve superior outcomes across multiple modalities.

GenLLaVA surpasses Unified-IO 2 in visual understanding performance. It shows strong and balanced performance across visual generation tasks, highlighting its versatility and robustness as a generalist vision-language model with significantly less training time.

Figure: Figure 5: Comparisons of VQA capabilities among GenLLaVA, Unified-IO 2, MGIE, and GILL. One can observe that GenLLaVA is able to describe the image in detail and respond to commonly asked questions, even addressing the unusual aspects within an image. Hallucinations made by the models are highlighted in red.
Refer to caption: x5.png

### 4.5 Results on selected Visual Question Answering datasets.

We evaluate various models’ performance across diverse visual question-answering datasets, including VQAv2, GQA, VizWiz, TextVQA, and ScienceQA. Our results show that Unified-IO 2 and GenLLaVA consistently perform well across most datasets. Specifically, Unified-IO 2 achieves the highest scores on the ScienceQA (86.2%) and TextVQA (67%), while GenLLaVA demonstrates strong performance on VQAv2 (79.3%) and a competitive score on GQA (62.9%). In contrast, GILL and MGIE exhibit generally lower performance across all datasets, with MGIE notably struggling on VizWiz (20.1%) and TextVQA (26.3%). AnyGPT shows moderate effectiveness, with its best performance on ScienceQA (61%). We used VLMEvalKit from Duan et al. () to get the results for these datasets, which perform a generation-based evaluation using the LLM-as-a-judge protocol.(^2^22We used GPT-4 (0409) as the judge.) The results can be seen in Fig. 4 and Fig. 5.

### 4.6 Ablations

We investigate the effect of scaling the data used to train GenLlaVA, the effect of using different image backbones, and the number of visual tokens used for image generation.

#### Instruction data.

We start with the original instruction tuning dataset from LlaVA-1.5, basically reproducing the original results using a one-stage training recipe. We add the LLaVA-Pretrain and generation head to our model, and we notice that adding generation capabilities significantly affects the visual understanding capabilities, with all metrics degrading between 2.3% (MM-Vet) and 7.9% (MathVista). To compensate for this loss in performance, we modify the ratio of image generation to image understanding data in our dataset from $\sim$50%-50% to $\sim$70%-30%, by adding more image instruction data from LVIS-INSTRUCT4V, LRV-Instruction and other chart understanding datasets. This results in a model with significant generation capabilities that maintain its image-understanding capabilities. We finally add image-generation capabilities using our selected subset of the IPr2Pr dataset. This reduces the image understanding capabilities, but we consider this a small enough change that balances the three tasks while maintaining commendable performance. Finally, task tokens are added to the resulting instruction set to condition the model on the desired task.
The results can be seen in Table 2(a).

**Table 2: Ablation experiments on different datasets. We present evaluation results across various ablation types.**
| Model Variation | Advanced Knowledge | General Understanding | Editing | Generation |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MathVista | MMMU | MM-Vet | SEED-B | MMB | EVR | CC3M | COCO |  |
| LLaVA-Finetune | 24.7 | 28.7 | 30.1 | 54.7 | 65.6 | - | - | - |
| Generation | 16.8 | 27.5 | 27.8 | 52.1 | 59.8 | 30.2 | 13.9 | 0.73 |
| Extra Knowledge | 28.2 | 31.8 | 32.4 | 59.7 | 64.1 | 28.5 | 14.0 | 0.72 |
| IPr2Pr-200K | 24.9 | 29.7 | 33.1 | 63.5 | 65.0 | 64.7 | 14.3 | 0.71 |
| Task Tokens | 30.5 | 37.1 | 35.8 | 64.5 | 66.8 | 66.9 | 12.5 | 0.73 |

#### Choice of Visual encoder.

The quality of the vision encoder can also have big effects on the final LMM performance. We start by using a CLIP/B, which has the lowest performance, then we compare against a stronger visual encoder and create versions of GenLLaVA, which is trained with CLIP (Radford et al., ) and SigLIP (Zhai et al., ), respectively. We can see in Table 2(b) that the SigLIP encoder generally achieves better performance than the CLIP encoder. This shows that SigLIP is a better vision encoder for LMM development.

#### Number of visual generation tokens.

We experiment with varying the number of visual generation tokens, $N$, to determine the optimal number required for balancing image generation and understanding tasks. We noticed that we need more visual generation tokens than GILL ($N=4$) and MGIE ($N=8$) to achieve the best performance. We find that $N=16$ is the best choice for our model. We hypothesize that this is because our model has to balance image generation and editing in the same head and thus needs more visual tokens to capture the complexity of the tasks. The results can be seen in Table 2(c).

### 4.7 Comparison with SOTA.

When compared with state-of-the-art models, GenLLaVA maintains similar performance to models of the LLaVA family when evaluated using the average of the scores on the MathVista, MMMU, MMVet, SEED-B, and MMB datasets. However, it lags behind larger and more specialized models. However, some of these models lack the generative capabilities present in GenLLaVA.

When compared with models of similar size and setup ($\sim 7$b parameters), our model surpasses the original LlaVAv1(Liu et al., ) model by  9% points (37.5% vs. 46.9%), and LlaVA-1.5 (Liu et al., ) by $\sim$1% points (45.3% vs. 46.9%). It is surpassed by the LlaVA-Next (Liu et al., ) family of models by $\sim$4% points; by Idefics2 (Laurençon et al., ) by $\sim$8% points (55.7% vs. 46.9%) and by the newly released MiniCPM-Llama3 (Xu et al., ) model by $\sim$13% points (60.6% vs. 46.9%). Compared with the absolute state-of-the-art open-source models, GenLLaVA lags behind Yi-VL (Young et al., ) by $\sim$2% points; and surpasses Emu2 (Sun et al., ) by $\sim$1% points. It lags behind LlaVA-Next (Liu et al., ) (34b) by $\sim$12% points, and InternVL 1.5 (Chen et al., ) by $\sim$15% points. Compared with the absolute state-of-the-art closed models, GenLLaVA lags behind GPT-4o by $\sim$26% points, GPT-4V (Achiam et al., ) by $\sim$20% points; and the Gemini family (Gemini Team et al., ) of models by  13% points.

## 5 Conclusion

In this paper, we have introduced the Generative Large Language and Visual Assistant (GenLLaVA), a comprehensive framework for enabling Large Multimodal Models (LMMs) to excel simultaneously in image understanding, generation, and editing, while maintaining competitive performance. By balancing multimodal capabilities within a single model through the curation of a diverse multimodal instruction dataset and the development of an innovative single-phase training methodology, GenLLaVA sets a new benchmark in the development of multimodal systems. Our results show that unifying generation and understanding under a single framework is possible without compromising their strengths. Our work sets a new standard for building visual assistants with extra capabilities, and we hope our open-source contributions, including datasets, codebase, and model checkpoints, will serve as valuable resources for the research community, driving further advancements in the field of multimodal AI. GenLLaVA’s capabilities can be extended to video understanding, audio-visual tasks, and more advanced real-time multimodal interactions.

## Acknowledgements

The authors would like to thank Google Cloud and the TPU Research Cloud program from Google for providing funding for this research effort. We are also thankful for support from the Department of Computer Science at Rice University.

## Appendix A Appendix

### A.1 Implementation details

We adopt the hyperparameters in Table A1 for all our experiments.

**Table A1: Training Hyperparameters**
| Hyperparameter | Value |
| --- | --- |
| Batch Size | 128 |
| Max Gradient Norm | 1.0 |
| Weight Decay | 0.1 |
| Learning Rate | 2e-5 |
| Optimizer | AdamW |
| Scheduler | Warmup & Cosine Decay |
| Warmup Ratio | 0.03 |

### A.2 Evaluation Details

During the evaluation with VLMEvalKit (Duan et al., ), we used the prompt templates listed in Fig A1

Figure: Figure A1: Prompt templates. (a) Short VQA includes VQAv2, VizWiz, GQA, and TextVQA and ScienceQA. (b) Long VQA includes MMB, SEED-B, and MM-Vet. (c) Advanced knowledge includes MathVista and MMMU. (d) Generation includes CC3M and MS-COCO. (e) Editing includes EVR. <IMAGE> is the image representation, <QUESTION> denotes each specific question, <ANSWER> is the generated answer, <DESCRIPTION> is an image description for generation or editing, and <GENERATED IMAGE> is the output of the generation head..
Refer to caption: x6.png

### A.3 Additional Qualitative Examples

In Tables A2 and A3, we present a qualitative comparative analysis of VQA results between our model, GenLLaVA, and other state-of-the-art models: GPT-4 (Achiam et al., ), LLaVA (Liu et al., ), GPT-4o (OpenAI, ), and InternVL-1.5 (Chen et al., ). Our model, while smaller than state-of-the-art models, is still able to give detailed and precise responses to given questions, avoiding the introduction of hallucinations, unlike LLaVA-1.5.

**Table A2: Visual question answering example comparison of GenLLaVA to LLaVA-1.5-7B(Liu et al., ), GPT-4V (Achiam et al., ), GPT-4o (OpenAI, ), and InternVL-1.5 (Chen et al., ). This table follows the format from Bubeck et al. ().**
| Visual input example, Extreme Ironing: |  |
| --- | --- |
|  |  |
|  | Source: [https://www.barnorama.com/wp-content/uploads/2016/12/03-Confusing-Pictures.jpg](https://www.barnorama.com/wp-content/uploads/2016/12/03-Confusing-Pictures.jpg) |
| User | What is unusual about this image? |
| GenLLaVA (Ours) | The unusual aspect of this image is that a man is ironing clothes on the back of a yellow SUV in the middle of a city street. This is not a common sight, as ironing is typically done indoors, in a private setting. The presence of a taxi cab in the background further emphasizes the unexpected nature of the scene, as it juxtaposes the mundane task of ironing with the hustle and bustle of urban life. |
| User | What is unusual about this image? |
| LLaVA (Liu et al., ) | The unusual aspect of this image is that a man is ironing clothes while standing on the back of a moving car. This is not a typical scene, as ironing clothes is usually done indoors, in a stationary position, and not while the vehicle is in motion. The man’s actions are both unconventional and potentially dangerous, as ironing clothes on a moving car can be hazardous and may cause accidents or injuries. |
| User | What is unusual about this image? |
| GPT-4V (Achiam et al., ) | The unusual aspect of this image is the man ironing on a portable ironing board mounted to the back of a yellow taxi in the middle of a bustling city street. This activity is not typically associated with such a setting, making it an extraordinary and humorous sight. This represents an extreme form of “extreme ironing,” a performance sport where people iron clothes in unexpected and often difficult locations. |
| User | What is unusual about this image? |
| GPT-4o (OpenAI, ) | The image shows a man ironing clothes on an ironing board that is attached to the back of a moving vehicle, which appears to be a yellow taxi. This is highly unusual because ironing is typically done indoors, on stable surfaces, and definitely not while attached to a moving car. This extreme scenario might be an example of “extreme ironing,” a humorous and unconventional sport where people iron clothes in extreme or unusual situations. The image itself blends an ordinary chore with a risky and absurd setting, making it quite striking and humorous. |
| User | What is unusual about this image? |
| InternVL-1.5 (Chen et al., ) | This image depicts an unusual scene in which a person is ironing clothes on an ironing board attached to the back of a taxi while it is parked on a city street. Several elements make this situation peculiar: (1) Location: Ironing is typically done indoors at home or in a laundry room where there are proper facilities for such tasks. (2) Practicality: It’s impractical to set up an ironing board and use an iron outdoors, especially in a busy urban environment where distractions and interruptions could occur frequently. |
|  | The juxtaposition of a mundane domestic activity with the bustling cityscape creates a humorous and unexpected scenario. |

**Table A3: Visual question answering example comparison of GenLLaVA to LLaVA-1.5-7B(Liu et al., ), GPT-4V (Achiam et al., ), GPT-4o (OpenAI, ), and InternVL-1.5 (Chen et al., ). This table follows the format from Bubeck et al. ().**
| Visual input example, Chicken Nugget Map: |  |
| --- | --- |
|  |  |
| User | Can you explain this meme in detail? |
| GenLLaVA (Ours) | The meme is a play on the phrase "beauty is in the eye of the beholder," which means that what one person finds beautiful may not be the same for another. In this case, the person creating the meme finds beauty in the shape of the Earth formed by pieces of fried chicken arranged on a baking tray. It’s a humorous take on appreciating simple things and finding artistic or aesthetic value in everyday objects or food items. |
| User | Can you explain this meme in detail? |
| LLaVA (Liu et al., ) | The meme in the image is a creative and humorous take on food, with a focus on chicken nuggets, as the center of the universe. The meme begins with the phrase “Sometimes I just look at pictures of the Earth from space and I marvel at how beautiful it all is…” The main image features a tray of cooked chicken nuggets, with the chicken pieces positioned to resemble the continents and islands on a world map.<br>The meme playfully suggests that the chicken nuggets represent the Earth, and the various locations depicted in the photo are actually chicken nugget versions of different places. The punchline of the meme appears in smaller text at the bottom, reading “I mean, it’s not the real Earth, but how beautiful it is all is.” This lighthearted and entertaining meme combines food and humor, using the image of chicken nuggets to create a funny and imaginative representation of the world. |
| User | Can you explain this meme? |
| GPT-4V (Achiam et al., ) | This meme cleverly uses chicken nuggets shaped like continents to make a humorous comparison. The caption suggests that the person admires pictures of Earth from space for their beauty, but the image shows chicken nuggets on a baking tray arranged to look like a world map. This juxtaposition creates a playful contrast between the expected awe-inspiring images of our planet and the mundane, yet amusing arrangement of food. The humor lies in the unexpected substitution and the notion that even chicken nuggets can evoke a sense of wonder if seen from the right perspective. |
| User | Can you explain this meme in detail? |
| GPT-4o (OpenAI, ) | This meme humorously compares the shapes of fried chicken nuggets to a map of the world, as though the person is marveling at the beauty of the Earth from space. The chicken nuggets are arranged in a way that resembles the continents, and the caption plays off the idea of someone gazing at these shapes with awe, as if they’re looking at Earth, rather than just pieces of food. |
|  | The humor comes from the exaggerated comparison of something mundane (fried chicken nuggets) to something grand and majestic (the Earth viewed from space). It’s a playful and lighthearted take on how our brains often find patterns in random objects that resemble familiar things. |
| User | Can you explain this meme in detail? |
| InternVL-1.5 (Chen et al., ) | This meme humorously contrasts the profound and contemplative act of admiring the beauty of Earth from space with a more mundane, everyday activity. The top part of the image includes text that says: "Sometimes I just look at pictures of the earth from space and I marvel at how beautiful it all is." Following this statement, instead of showing an image of Earth from space, there is a picture of several pieces of fried chicken arranged on a baking sheet. |
|  | The joke lies in the unexpected and humorous twist, where the "beautiful sight" referenced in the text turns out to be something as ordinary and relatable as fried chicken. This creates a light-hearted contrast between grandeur (Earth viewed from space) and simplicity (fried chicken), playing on the idea that both can evoke similar feelings of wonder or satisfaction. |

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Could you provide a detailed explanation of how the model integrates outputs from large language models with Diffuser models specifically for image generation and editing? 
2. The paper references similarities with existing models like GILL and Unified-IO 2, particularly in Figure 1, where the only discernible differences appear to be the use of certain image encoders, language models, and Diffuser models. Could you elaborate on the original aspects of GLLaVA, if any, beyond component substitution? 
3. While the paper claims to achieve balanced performance across image understanding, generation, and editing tasks, Table 1 suggests that the model falls short of peers in image generation, and no comparisons are made against other multimodal understanding models. Could you address this by clarifying if factors other than inter-task conflicts influence this performance gap?
4. Your decision to adopt a single-stage pipeline, inspired by Prismatic VLMS, is an essential component of the methodology, yet no ablation studies are provided to justify this choice over a multi-stage approach. Could you explain if any experiments were conducted to compare the performance of the single-stage pipeline with a more conventional multi-stage process?
5. Table 2(a) indicates a notable performance improvement with Task Tokens, yet no explanation is provided for this boost. Could you delve into the role of Task Tokens in explicitly defining the task format during inference?
6. The paper proposes an improvement in understanding capabilities by adding more data, yet no specific comparison is provided for image understanding performance alone. Would you be able to offer more insights into the impact of this additional data specifically for image understanding?
soundness: 1
strengths: Originality:
- New Dataset Curation: The creation of a new multimodal instruction-following dataset that amalgamates image understanding, generation, and editing is innovative. It addresses the need for diverse training data to support complex multimodal tasks. 
- Single-Stage Training Strategy: Moving away from the traditional multi-stage training pipeline to a single-stage training recipe is a significant departure that simplifies the training process while maintaining performance.

Quality:
- State-of-the-Art Comparisons: With new dataset, GenLLaVA establishes its competitive standing within the field comparing to LLaVA in multi-modal understanding, GILL, AnyGPT, Unified-IO 

Significance:
- The work moves towards the development of a unified MLLM, which is able to simultaneously perform image generation and image understanding. This is a rather important direction for the development of MLLM.
summary: GenLLaVA, a novel approach aimed at enhancing the zero-shot capabilities of a single unfied large multimodal model to perform image understanding, image generation, and image editing tasks is proposed with the following contributions:
1. Construction of a new multimodal instruction-following dataset that combines image understanding, image generation, and image editing data.
2. A unified model, GenLLaVA, is trained in a single-stage training process, which is a departure from its predecessor, LLaVA.
3. GenLLaVA outperforms LLaVA and is competitive with native multimodal models like Unified-IO 2, setting a new benchmark for building advanced general-purpose visual assistants by effectively reusing existing multimodal models.
weaknesses: 1. Conciseness of the Methodology Section: The methodology section lacks sufficient depth, particularly in explaining how the model integrates outputs from large language models with Diffuser models for tasks in image generation and editing. Specific details, such as the configuration of attention masks, inputs and target outputs during image generation, and the loss functions employed, are absent. Including these would enhance clarity.

2. Limited Originality in Methodology: Due to its brevity, the methodology section does not adequately establish the originality of the proposed approach. A more detailed explanation of the unique methods used is essential to distinguish this work from similar studies, such as GILL[1] and Unified-IO 2[2] As depicted in Figure 1, the only observable difference between GLLaVA and GILL appears to be the choice of image encoders, language models, and Diffuser models—none of which are novel. If these component substitutions introduce significant innovation, or if additional design elements are integral, these should be clearly outlined to affirm the model’s originality in achieving image understanding, generation, and editing capabilities.

3. Experimental Support for Claims: The experiments presented do not convincingly substantiate the paper's claims of achieving image understanding, generation, and editing without compromising individual performance. As Table 1 indicates, the model underperforms in image generation tasks compared to its peers, with no comparison provided against other multimodal understanding models. Additionally, while the model uses data from ShareGPT[3], the performance is lower, scoring 66.8 on MMB[4] for a 7B LLM compared to ShareGPT’s 68.8. If this discrepancy is due to factors beyond inter-task conflicts, ablation studies could help clarify. Further, the single-stage pipeline is chosen and no ablation is performed, the author follows Prismatic vlms[5]. It's a main contribution claimed in paper which should contains some experiments to support it.

4. Inadequacy of Ablation Studies: The ablation studies presented are insufficient in addressing inter-task conflicts. Table 2(a) reveals a significant drop in performance when handling all three tasks concurrently. Although the authors propose enhancing understanding performance by adding more understanding data, no specific performance comparison is provided for image understanding alone. Moreover, the use of Task Tokens boosts performance in Table 2(a), but the underlying reasons for this improvement remain unexplored. This suggests that explicit task format definition is essential during inference, warranting further analysis in the discussion. 


[1] Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. In NeurIPS, 2024.

[2] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. In CVPR, 2024.

[3] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In ECCV, 2024.

[4] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, 2024.

[5] Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. In ICML, 2024.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: 1.	What makes single-stage training preferable over multi-stage training in GenLLaVA.
2.	The performance of GenLLaVA significantly trails behind recent state-of-the-art models. For instance, the new version of LLaVA, LLaVA-OV [A], achieves an 80.8 on MMB, outperforming GenLLaVA by 14 points. I'm curious how the proposed dataset would perform if a stronger model and large-scale understanding data were used. Would a new mix of data be necessary, or would the current amount of generative data suffice?

[A] Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., ... & Li, C. (2024). Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326.
soundness: 3
strengths: 1.	Developing generative multimodal instruction-following data could be highly valuable for future research and applications.
2.	The model shows strong performance across image understanding, image editing, and image generation tasks.
3.	The paper is well-presented, making it easy to follow and understand.
summary: This paper proposes a composite model, GenLLaVA, for image understanding, generation, and editing. The authors create a generative multimodal instruction-following dataset using GPT-4V and existing datasets for image generation and editing. They also explore various training strategies and model designs, including single-stage training, the selection of a vision encoder, and the use of task tokens. Quantitative results across multiple benchmarks demonstrate GenLLaVA's effectiveness across all three tasks.
weaknesses: 1.	Although the authors promise to open-source all materials, including data, code, and pre-trained weights, none of these resources are provided for review. Since the dataset is a key contribution of this work, it would strengthen the paper to include these materials in the revised manuscript (with anonymity). I would consider recommending acceptance only if these resources are included in the final version.
2.	The novelty of this paper appears limited, as the model seems to be a straightforward combination of existing techniques, such as the framework, task tokens, vision encoder, and single-stage training.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Similar as putting MGIE as a baseline model, which is a domain expert on image editing, it would be great if other domain expert models can be put as references as well. The proposed model doesn't have to beat these models, but it will give readers a more comprehensive understanding of how good a combined model can be compared with domain experts.

The paper acknowledges that task specific tokens such as [T2I] and [I2T] are needed to achieve this balanced performance. In real user queries these tokens are not likely to be provided, I wonder if there are other more natural approaches to improve user intent understanding.
soundness: 3
strengths: While the task and the approach are not completely new, the paper proposed several improvements on current approach to build MLLMs, and demonstrated their effectiveness using relatively comprehensive evaluations, resulting in a new model that is strong on both visual understanding and visual generation.

The paper is largely clear about the goal, approach and results, though it will be better if more details on the reason of several design choices can be made clear.
summary: This paper presents a new MLLM that has multi modal capabilities on both input and output. It also presents a new instruction following dataset, which is a non-trival combination and selection from a large amount of existing datasets for various specific tasks.
The paper adopts a less commonly used single-stage instruction tuning pipeline, which according to the text is more effective than extending previous multi-stage pipeline. 
Result evaluation shows that the model surpasses many major MLLMs on visual understanding tasks, and is roughly on par with previous models emphasizing visual generation tasks. 

The author claims that model checkpoint and the new dataset will be released.
weaknesses: The biggest weakness is the incremental nature of the work.  For example, 
the paper claims it unifies visual understanding and generation, but there are similar models such as SEED-X, which is also capable of both, as well as the compared work such as Unified-IO 2.

the paper claims they contribute a dataset for instruction tuning, but the dataset is composed of multiple existing datasets. The novelty to me would be insights on why these sets are selected, and why for some datasets such as IPr2Pr only a subset is selected. The paper can be much more impactful if there are more discussions on these decisions.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Please see weaknesses.
soundness: 2
strengths: 1. The authors build a multimodal instruction tuning dataset by aggregating a diverse set of supervised datasets and tasks including text understanding, image comprehension, generation and editing.
2. The paper is generally easy to follow.
summary: This paper introduces a multimodal instruction tuning set that combines image understanding, image generation, and image editing data. It further introduces a model GenLLaVA, which unifies visual understanding and generation using open-source models.
weaknesses: 1. Unified multimodal understanding and generation has become a highly popular direction recently, with many significant models introduced this year that can achieve image understanding, generation, and editing. However, this method simply adopts the same framework as GILL, which translates the hidden representations of an LLM into text embeddings of a pre-existing stable diffusion model. The mention of "using a Q-former as the generation head" in the paper cannot be considered a distinguishing feature from the GILL approach. From a methodological standpoint, this work appears to lack any innovation.
2.  This paper only compares a few outdated and underperforming models as baselines. The recent multimodal LLMs with unified comprehension and generation should be compared including Emu [1], Emu-2.0 [2], SEED-LLaMA [3], SEED-X [4], Mini-Gemini [5], etc.
3. The claim that "To our knowledge, this is the first time such capability has been achieved" is unfounded. As I mentioned earlier, several works have already accomplished unified image understanding, generation, and editing.

[1] Generative pretraining in multimodality
[2] Generative multimodal models are in-context learners
[3] Making llama see and draw with seed tokenizer
[4] SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation
[5] Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models

Meta-review: This paper introduces GenLLaVA for unified image understanding and generation. After rebuttal, it received scores of 3356. Several major concerns still remain: (1) very limited novelty of the model, and (2) weak experimental results given nowadays a large volume of papers on this topic. The rebuttal is unfortunately not convincing enough. Therefore, the AC would like to recommend rejection of the paper.

Decision: Reject