Title: InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Vision-Language Models

Instruction Tuning

Zero-shot
Abstract: Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence. However, building general-purpose vision-language models is challenging due to the rich input distributions and task diversity resulting from the additional visual input. Although vision-language pretraining has been widely studied, vision-language instruction tuning remains under-explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pretrained BLIP-2 models. We gather 26 publicly available datasets, covering a wide variety of tasks and capabilities, and transform them into instruction tuning format. Additionally, we introduce an instruction-aware Query Transformer, which extracts informative features tailored to the given instruction. Trained on 13 held-in datasets, InstructBLIP attains state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and larger Flamingo models. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA questions with image contexts). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models. All InstructBLIP models are open-source.

InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

Abstract

Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence. However, building general-purpose vision-language models is challenging due to the rich input distributions and task diversity resulting from the additional visual input. Although vision-language pretraining has been widely studied, vision-language instruction tuning remains under-explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pretrained BLIP-2 models. We gather 26 publicly available datasets, covering a wide variety of tasks and capabilities, and transform them into instruction tuning format. Additionally, we introduce an instruction-aware Query Transformer, which extracts informative features tailored to the given instruction. Trained on 13 held-in datasets, InstructBLIP attains state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and larger Flamingo models. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA questions with image contexts). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models. All InstructBLIP models are open-source.

1 Introduction

A longstanding aspiration of Artificial Intelligence (AI) research is to build a single model that can solve arbitrary tasks specified by the user. In natural language processing (NLP), instruction tuning [1, 2] proves to be a promising approach toward that goal. By finetuning a large language model (LLM) on a wide range of tasks described by natural language instructions, instruction tuning enables the model to follow arbitrary instructions. Recently, instruction-tuned LLMs have also been leveraged for vision-language tasks. For example, BLIP-2 [3] effectively adapts frozen instructiontuned LLMs to understand visual inputs and exhibits preliminary capabilities to follow instructions in image-to-text generation.

Compared to NLP tasks, vision-language tasks are more diverse in nature due to the additional visual inputs from various domains. This poses a greater challenge to a unified model that is supposed to generalize to diverse vision-language tasks, many unseen during training. Most previous work can be grouped into two approaches. The first approach, multitask learning [4, 5], formulates various vision-language tasks into the same input-output format. However, we empirically find multitask learning without instructions (Table 4) does not generalize well to unseen datasets and tasks. The

What could have happened based on the

current scene?

Based on the current scene in the image, it is

possible that a hurricane or severe weather event

caused significant damage to the buildings and

infrastructure in the area. The person walking

through the debris-covered area suggests that they

are inspecting the damage caused by the disaster.

The presence of palm trees indicates that the

location may be in a tropical or sub-tropical region,

which can be more susceptible to hurricanes and

other natural disasters.

Introduce me this painting

in detail.

The painting depicts a young girl with a pearl earring,

also known as "The Girl with the Pearl Earring" or

"Girl with a Pearl Earring". She is wearing a blue

dress and has a pearl earring on her left ear. Her hair

is pulled back into a bun, and she is looking directly

at the viewer. This painting is a famous work of art

It is considered one of his masterpieces and is widely

recognized for its intricate details and realistic

portrayal of the young girl's facial expression.

Can you tell me about this image in detail?

The image depicts a collection of various

vegetables including carrots, cucumbers,

tomatoes, and nuts arranged on a stone surface.

There are several jars filled with different types of

ingredients, such as peanuts, cashews, sunflower

seeds, and pumpkin seeds. These ingredients are

likely to be part of a healthy meal or snack.

Can you give me an example of such meal?

An example of a healthy meal or snack that

incorporates some of the ingredients in the image

could be a salad made with mixed greens, carrots,

cucumbers, tomatoes, and nuts. The salad can be

dressed with a homemade vinaigrette made with

olive oil, lemon juice, salt, and pepper.

Additionally, the salad can be topped with

pumpkin seeds or sunflower seeds for added

crunch and nutritional value.

Sounds good! Show me steps of making this salad.

Sure! Here are the steps to make this healthy salad:

1. Wash and chop the mixed greens, carrots,

cucumbers, and tomatoes into bite-sized pieces.

2. In a small bowl, whisk together olive oil, lemon

juice, salt, and pepper to create a homemade

vinaigrette dressing.

3. Place the chopped vegetables in a large bowl and

toss them with the vinaigrette dressing.

4. Sprinkle pumpkin seeds or sunflower seeds on top

of the salad for added crunch and nutritional value.

5. Serve the salad immediately or store it in the

refrigerator until ready to enjoy.

Can you describe this image

in detail?

The image depicts a person standing in front of an

open doorway, looking out into space. They are

wearing a jacket and appears to be contemplating the

vastness of the universe beyond the doorway. In the

background, there are multiple stars visible, creating

a stunning celestial scene. The person's gaze is

directed towards the center of the image, where a

bright star can be seen shining brightly. There are

also smaller stars scattered throughout the scene,

adding to the overall cosmic atmosphere.

What if they jump over the doorway?

If the person were to jump over the doorway, they

would be leaving the confines of their current location

and venturing into the vastness of space. This could be

a metaphorical representation of stepping out of one's

comfort zone or embarking on a new adventure.

However, it is important to note that such an action

would be extremely dangerous and potentially life-

threatening due to the harsh conditions of space.

What makes this dog special?

The dog in the image is wearing futuristic armor and

has a glowing green eye, giving it a unique and

futuristic appearance. This combination of elements

makes the dog stand out from other dogs and adds an

element of fantasy or sci-fi to the image.

Additionally, the dog's posture and facial expression

convey a sense of confidence and determination,

further emphasizing its unique features.

second approach [3, 6] extends a pre-trained LLM with additional visual components, and trains the visual components with image caption data. Nevertheless, such data are too limited to allow broad generalization to vision-language tasks that require more than visual descriptions.

To address the aforementioned challenges, this paper presents InstructBLIP, a vision-language instruction tuning framework that enables general-purpose models to solve a wide range of visuallanguage tasks through a unified natural language interface. InstructBLIP uses a diverse set of instruction data to train a multimodal LLM. Specifically, we initialize training with a pre-trained BLIP-2 model consisting of an image encoder, an LLM, and a Query Transformer (Q-Former) to bridge the two. During instruction tuning, we finetune the Q-Former while keeping the image encoder and LLM frozen. Our paper makes the following key contributions:

We perform a comprehensive and systematic study on vision-language instruction tuning. We transform 26 datasets into the instruction tuning format and group them into 11 task categories. Weuse 13 held-in datasets for instruction tuning and 13 held-out datasets for zero-shot evaluation. Moreover, we withhold four entire task categories for zero-shot evaluation at the task level. Exhaustive quantitative and qualitative results demonstrate the effectiveness of InstructBLIP on vision-language zero-shot generalization.

We propose instruction-aware visual feature extraction, a novel mechanism that enables flexible and informative feature extraction according to the given instructions. Specifically, the textual instruction is given not only to the frozen LLM, but also to the Q-Former, so that it can extract instruction-aware visual features from the frozen image encoder. Also, we propose a balanced sampling strategy to synchronize learning progress across datasets.

We evaluate and open-source a suite of InstructBLIP models using two families of LLMs: 1) FlanT5 [2], an encoder-decoder LLM finetuned from T5 [7]; 2) Vicuna [8], a decoder-only LLM finetuned from LLaMA [9]. The InstructBLIP models achieve state-of-the-art zero-shot performance on a wide range of vision-language tasks. Furthermore, InstructBLIP models lead to state-of-the-art finetuning performance when used as the model initialization on individual downstream tasks.

2 Vision-Language Instruction Tuning

InstructBLIP aims to address the unique challenges in vision-language instruction tuning and provide a systematic study on the models' improved generalization ability to unseen data and tasks. In this section, we first introduce the construction of instruction-tuning data, followed by the training and evaluation protocols. Next, we delineate two techniques to improve instruction-tuning performance from the model and data perspectives, respectively. Lastly, we present the implementation details.

2.1 Tasks and Datasets

To ensure the diversity of instruction tuning data while considering their accessibility, we gather comprehensive set of publicly available vision-language datasets, and transform them into the instruction tuning format. As shown in Figure 2, the final collection covers 11 task categories and 26 datasets, including image captioning [10, 11, 12], image captioning with reading comprehension [13], visual reasoning [14, 15, 16], image question answering [17, 18], knowledge-grounded image question answering [19, 20, 21], image question answering with reading comprehension [22, 23], image question generation (adapted from the QA datasets), video question answering [24, 25], visual conversational question answering [26], image classification [27], and LLaVA-Instruct-150K [28]. We include detailed descriptions and statistics of each dataset in Appendix C.

For every task, we meticulously craft 10 to 15 distinct instruction templates in natural language. These templates serve as the foundation for constructing instruction tuning data, which articulates the task and the objective. For public datasets inherently favoring short responses, we use terms such as short and briefly into some of their corresponding instruction templates to reduce the risk of the model overfitting to always generating short outputs. For the LLaV A-Instruct-150K dataset, we do not incorporate additional instruction templates since it is naturally structured in the instruction format. The full list of instruction templates can be found in Appendix D.

Image Captioning

COCO Caption

Web CapFilt

NoCaps

Flickr30K

Image Question Answering

VQAv2

Visual Reasoning

GQA

Visual Spatial

Reasoning

IconQA

Image Question Generation

VQAv2

OKVQA

A-OKVQA

Visual Conversational QA

Visual Dialog

Video Question Answering

MSVD QA

MSRVTT QA

iVQA

LLaVA-Instruct-150K

Visual Conversation

Image Classification

VizWiz

Description

HatefulMemes

Image Captioning

Reading Comprehension

TextCaps

Image Question Answering

Reading Comprehension

OCR-VQA

TextVQA

Knowledge Grounded

Image Question Answering

OKVQA

A-OKVQA

ScienceQA

2.2 Training and Evaluation Protocols

To ensure sufficient data and tasks for training and zero-shot evaluation, we divide the 26 datasets into 13 held-in datasets and 13 held-out datasets, indicated by yellow and white respectively in Figure 2. We employ the training sets of the held-in datasets for instruction tuning and their validation or test sets for held-in evaluation.

For held-out evaluation, our aim is to understand how instruction tuning improves the model's zero-shot performance on unseen data. We define two types of held-out data: 1) datasets not exposed to the model during training, but whose tasks are present in the held-in cluster; 2) datasets and their associated tasks that remain entirely unseen during training. Addressing the first type of held-out evaluation is nontrivial due to the data distribution shift between held-in and held-out datasets. For the second type, we hold out several tasks completely, including visual reasoning, video question answering, visual conversational QA, and image classification.

To avoid data contamination, datasets are selected carefully so that no evaluation data appear in the held-in training cluster across different datasets. One exception is the Visual Dialog [26] dataset, which has overlap of images with some held-in data. The Visual Dialog [26] dataset is an exception, as it has an overlap of images with some training data. Nevertheless, since it's virtually the only high-quality visual dialog dataset available for quantitative evaluation, we still incorporated it in our evaluation as a reference. During instruction tuning, we mix all the held-in training sets and sample instruction templates uniformly for each dataset. The models are trained with the standard language modeling loss to directly generate the response given the instruction. Furthermore, for datasets that involve scene texts, we add OCR tokens in the instruction as supplementary information.

2.3 Instruction-aware Visual Feature Extraction

Existing zero-shot image-to-text generation methods, including BLIP-2, take an instruction-agnostic approach when extracting visual features. That results in a set of static visual representations being fed into the LLM, regardless of the task. In contrast, an instruction-aware vision model can adapt to the task instruction and produce visual representations most conducive to the task at hand. This is clearly advantageous if we expect the task instructions to vary considerably for the same input image.

We show the architecture of InstructBLIP in Figure 3. Similarly to BLIP-2 [3], InstructBLIP utilizes a Query Transformer, or Q-Former, to extract visual features from a frozen image encoder. The input to the Q-Former contains a set of K learnable query embeddings, which interact with the

Queries

Q-Former

…

Instruction

Image

Encoder

Instruction

Queries

Feed Forward

Self Attention

Feed Forward

Cross Attention

Q-Former

Image

Embeddings

…

…

…

…

Instruction

Image Embeddings

…

Fully

Connected

…

Instruction

…

LLM

Response

left one

Choose the correct option to

the following question:

which picture shows the

pizza inside the oven?

Options

: (a) left one (b)

right one.

Answer

:

image encoder's output through cross attention. The output of the Q-Former consists of K encoded visual vectors, one per query embedding, which then go through a linear projection and are fed to the frozen LLM. As in BLIP-2, the Q-Former is pretrained in two stages using image-caption data before instruction tuning. The first stage pretrains the Q-Former with the frozen image encoder for vision-language representation learning. The second stage adapts the output of Q-Former as soft visual prompts for text generation with a frozen LLM . After pretraining, we finetune the Q-Former with instruction tuning, where the LLM receives as input the visual encodings from the Q-Former and the task instruction.

Extending BLIP-2, InstructBLIP proposes an instruction-aware Q-former module, which takes in the instruction text tokens as additional input. The instruction interacts with the query embeddings through self-attention layers of the Q-Former, and encourages the extraction of task-relevant image features. As a result, the LLM receives visual information conducive to instruction following. We demonstrate empirically (Table 2) that instruction-aware visual feature extraction provides substantial performance improvements for both held-in and held-out evaluations.

2.4 Balancing Training Datasets

Due to the large number of training datasets and the significant differences in the size of each dataset, mixing them uniformly could cause the model to overfit smaller datasets and underfit larger datasets. To mitigate the problem, we propose to sample datasets with probabilities proportional to the square root of their sizes, or the numbers of training samples. Generally, given D datasets with sizes { S 1 , S 2 , . . . , S D } , the probability of a data sample being selected from a dataset d during training is p d = √ S d ∑ D i =1 √ S i . On top of this formula, we make manual adjustments to the weights of certain datasets based on their individual convergence time to improve optimization. This is warranted by inherent differences in the datasets and tasks that require varying levels of training intensity despite similar sizes. Specifically, we lower the weight of A-OKVQA, which features multiple-choice questions, and increase the weight of OKVQA, which requires open-ended text generation. In Table 2, we show that the balanced dataset sampling strategy improves overall performance for both held-in evaluation and held-out generalization.

2.5 Inference Methods

During inference time, we adopt two slightly different generation approaches for evaluation on different datasets. For the majority of datasets, such as image captioning and open-ended VQA, the instruction-tuned model is directly prompted to generate responses, which are subsequently compared to the ground truth to calculate metrics. On the other hand, for classification and multi-choice VQA

tasks, we employ a vocabulary ranking method following previous works [1, 29, 30]. Specifically, we still prompt the model to generate answers, but restrict its vocabulary to a list of candidates. Then, we calculate log-likelihood for each candidate and select the one with the highest value as the final prediction. This ranking method is applied to ScienceQA, IconQA, A-OKVQA (multiple-choice), HatefulMemes, Visual Dialog, MSVD, and MSRVTT datasets. Furthermore, for binary classification, we expand the positive and negative labels into a slightly broader set of verbalizers to exploit word frequencies in natural text (e.g., yes and true for the positive class; no and false for the negative class).

For the video question-answering task, we utilize four uniformly-sampled frames per video. Each frame is processed by the image encoder and Q-Former individually, and the extracted visual features are concatenated before being fed into the LLM.

2.6 Implementation Details

Architecture. Thanks to the flexibility enabled by the modular architectural design of BLIP-2, we can quickly adapt the model to a wide range of LLMs. In our experiments, we adopt four variations of BLIP-2 with the same image encoder (ViT-g/14 [31]) but different frozen LLMs, including FlanT5XL (3B), FlanT5-XXL (11B), Vicuna-7B and Vicuna-13B. FlanT5 [2] is an instruction-tuned model based on the encoder-decoder Transformer T5 [7]. Vicuna [8], on the other hand, is a recently released decoder-only Transformer instruction-tuned from LLaMA [9]. During vision-language instruction tuning, we initialize the model from pre-trained BLIP-2 checkpoints, and only finetune the parameters of Q-Former while keeping both the image encoder and the LLM frozen. Since the original BLIP-2 models do not include checkpoints for Vicuna, we perform pre-training with Vicuna using the same procedure as BLIP-2. We set the number of query embeddings to 32 and do not observe any improvement when increasing it in our settings. However, it could potentially be beneficial for future tasks with more complex visual inputs.

Training and Hyper-parameters. We use the LAVIS library [32] for implementation, training, and evaluation. All models are instruction-tuned with a maximum of 60K steps and we validate model's performance every 3K steps. For each model, a single optimal checkpoint is selected and used for evaluations on all datasets. We employ a batch size of 192, 128, and 64 for the 3B, 7B, and 11/13B models, respectively. The AdamW [33] optimizer is used, with β 1 = 0 . 9 , β 2 = 0 . 999 , and a weight decay of 0.05. Additionally, we apply a linear warmup of the learning rate during the initial 1,000 steps, increasing from 10 -8 to 10 -5 , followed by a cosine decay with a minimum learning rate of 0. For decoding, we adopt beam search with a beam size of 1 for HatefulMemes, VSR, and OCR-VQA, 3 for NoCaps, and 5 for the other tasks. All models are trained utilizing 16 Nvidia A100 (40G) GPUs and are completed within 1.5 days.

3 Experimental Results and Analysis

3.1 Zero-shot Evaluation

We first evaluate InstructBLIP models on the set of 13 held-out datasets with instructions provided in Appendix E. We compare InstructBLIP with the previous SOTA models BLIP-2 and Flamingo. As demonstrated in Table 1, we achieve new zero-shot SOTA results on all datasets. InstructBLIP consistently surpasses its original backbone, BLIP-2, by a significant margin across all LLMs, demonstrating the effectiveness of vision-language instruction tuning. For instance, InstructBLIP FlanT5XL yields an average relative improvement of 15.0% when compared to BLIP-2 FlanT5XL. Furthermore, instruction tuning boosts zero-shot generalization on unseen task categories such as video QA. InstructBLIP achieves up to 47.1% relative improvement on MSRVTT-QA over the previous SOTA despite having never been trained with temporal video data. Finally, our smallest InstructBLIP FlanT5XL with 4B parameters outperforms Flamingo-80B on all six shared evaluation datasets with an average relative improvement of 24.8%.

For the Visual Dialog dataset, we choose to report the Mean Reciprocal Rank (MRR) over the Normalized Discounted Cumulative Gain (NDCG) metric. This is because NDCG favors generic and uncertain answers while MRR prefers certain responses [34], making MRR better aligned with the zero-shot evaluation scenario.

3.2 Ablation Study on Instruction Tuning Techniques

To investigate the impact of the instruction-aware visual feature extraction (Section 2.3) and the balanced dataset sampling strategy (Section 2.4), we conduct ablation studies during the instruction tuning process. As illustrated in Table 2, the removal of instruction awareness in visual features downgrades performance significantly across all datasets. The performance drop is more severe in datasets that involve spatial visual reasoning (e.g., ScienceQA) or temporal visual reasoning (e.g., iVQA), where the instruction input to the Q-Former can guide visual features to attend to informative image regions. The removal of the data balancing strategy causes unstable and uneven training, as different datasets achieve peak performance at drastically different training steps. The lack of synchronized progress over multiple datasets harms the overall performance.

3.3 Qualitative Evaluation

Besides the systematic evaluation on public benchmarks, we further qualitatively examine InstructBLIP with more diverse images and instructions. As illustrated in Figure 1, InstructBLIP demonstrates its capacity for complex visual reasoning. For example, it can reasonably infer from the visual scene what could have happened and deduce the type of disaster from the location of the scene, which it extrapolates based on visual evidence like the palm trees. Moreover, InstructBLIP is capable of connecting visual input with embedded textual knowledge and generate informative responses, such as intruducing a famous painting. Furthermore, in descriptions of the overall atmosphere, InstructBLIP exhibits the ability to comprehend metaphorical implications of the visual imagery.

40

45

50

55

46.1

46.3

45.5

46.8

52.9

60

75

90

105

67.8

92.5

89.0

93.7

93.8

BLIP-2 Zero-shot

Train w/ Plain Input

Eval w/ Instruction

Train w/ Dataset Name

Eval w/ Instruction

Train w/ Dataset Name

Eval w/ Dataset Name

InstructBLIP

Multi-task

Held-out Avg.

Held-in Avg.

Finally, we show that InstructBLIP can engage in multi-turn conversations, effectively considering the dialog history when making new responses.

In Appendix B, we qualitatively compare InstructBLIP with concurrent multimodal models (GPT4 [36], LLaVA [28], MiniGPT-4 [37]). Although all models are capable of generating long-form responses, InstructBLIP's outputs generally contains more proper visual details and exhibits logically coherent reasoning steps. Importantly, we argue that long-form responses are not always preferable. For example, in Figure 2 of the Appendix, InstructBLIP directly addresses the user's intent by adaptively adjusting the response length, while LLaVA and MiniGPT-4 generate long and less relevant sentences. These advantages of InstructBLIP are a result of the diverse instruction tuning data and an effective architectural design.

3.4 Instruction Tuning vs. Multitask Learning

A direct analogue to instruction tuning is multitask learning, a widely used method that involves the simultaneous training of multiple datasets with the goal of improving the performance of each individual dataset. To investigate whether the improvement in zero-shot generalization mainly comes from the instruction format or merely from multi-task learning, we conduct a comparative analysis.

Following [1], we consider two multitask training approaches. In the first approach, the model is trained using the vanilla input-output format of the training datasets without instructions. During evaluation, instructions are still provided to the model to specify which task to perform. However, an exception is made for image captioning, as the model performs better when receiving the image as the only input. For the second approach, we take a step toward instruction tuning by prepending a [Task:Dataset] identifier to the text input during training. For example, we prepend [Visual question answering:VQAv2] for the VQAv2 dataset. During evaluation, we explore both instructions and the identifier. Particularly, for held-out datasets, we only use the task name as the identifier since the model never sees the dataset name.

In Figure 4, we show the results for zero-shot, multitask training, and instruction tuning. All models are based on the BLIP-2 FlanT5XL backbone and adhere to the identical training configurations in Section 2. Overall, we make two observations. First, instruction tuning and multitask learning exhibit similar performance on the held-in datasets. This suggests that the model can fit these two different input patterns comparably well, as long as it has been trained with such data. Second, instruction tuning yields a significant improvement over multitask learning on unseen held-out datasets, whereas multitask learning performs on par with the original BLIP-2. This indicates that instruction tuning is the key to zero-shot generalization.

Table 3: Results of finetuning BLIP-2 and InstructBLIP on downstream datasets. Compared to BLIP-2, InstructBLIP provides a better weight initialization model and achieves SOTA performance on three out of four datasets.

3.5 Finetuning InstructBLIP on Downstream Tasks

We further finetune the InstructBLIP models to investigate its performance on learning a specific dataset. Compared to most previous methods (e.g., Flamingo, BLIP-2) which increase the input image resolution and finetune the visual encoder on downstream tasks, InstructBLIP maintains the same image resolution (224 × 224) as used in instruction tuning and keeps the visual encoder frozen during finetuning. This significantly reduces the number of trainable parameters from 1.2B to 188M, thus greatly improves finetuning efficiency.

The results are in Table 3. Compared to BLIP-2, InstructBLIP leads to better finetuning performance on all datasets, validating InstructBLIP as the better initialization for task-specific finetuning. InstructBLIP sets new state-of-the-art finetuning performance on ScienceQA (image-context), OCR-VQA, A-OKVQA, but is outperformed on OKVQA by PaLM-E [38] with 562B parameters.

Additionally, we observe that the FlanT5-based InstructBLIP is superior at multi-choice tasks, whereas Vicuna-based InstructBLIP is generally better at open-ended generation tasks. This disparity can be primarily attributed to the capabilities of their frozen LLMs, as they both employ the same image encoder. Although FlanT5 and Vicuna are both instruction-tuned LLMs, their instruction data significantly differ. FlanT5 is mainly finetuned on NLP benchmarks containing many multi-choice QA and classification datasets, while Vicuna is finetuned on open-ended instruction-following data.

4 Related Work

Instruction tuning aims to teach language models to follow natural language instructions, which has been shown to improve their generalization performance to unseen tasks. Some methods collect instruction tuning data by converting existing NLP datasets into instruction format using templates [1, 2, 42, 43]. Others use LLMs (e.g., GPT-3 [44]) to generate instruction data [8, 45, 46, 47] with improved diversity.

Instruction-tuned LLMs have been adapted for vision-to-language generation tasks by injecting visual information to the LLMs. BLIP-2 [3] uses frozen FlanT5 models, and trains a Q-Former to extract visual features as input to the LLMs. MiniGPT-4 [37] uses the same pretrained visual encoder and Q-Former from BLIP-2, but uses Vicuna [8] as the LLM and performs training using ChatGPT [48]-generated image captions longer than the BLIP-2 training data. LLaVA [28] directly projects the output of a visual encoder as input to a LLaMA/Vinuca LLM, and finetunes the LLM on vision-language conversational data generated by GPT-4 [36]. mPLUG-owl [49] performs low-rank adaption [50] to a LLaMA [9] model using both text instruction data and vision-language instruction data from LLaVA. A separate work is MultiInstruct [51], which performs vision-language instruction tuning without a pretrained LLM, leading to less competitive performance.

Compared to existing methods, InstructBLIP uses a much wider range of vision-language instruction data, covering both template-based converted data and LLM-generated data. Architecture wise, InstructBLIP proposes an instruction-aware visual feature extraction mechanism. Furthermore, our paper provides a comprehensive analysis on various aspects of vision-language instruction tuning, validating its advantages on generalizing to unseen tasks.

5 Conclusion

In this paper, we present InstructBLIP, a simple yet novel instruction tuning framework towards generalized vision-language models. We perform a comprehensive study on vision-language instruction tuning and demonstrate the capability of InstructBLIP models to generalize to a wide range of unseen tasks with state-of-the-art performance. Qualitative examples also exhibit InstructBLIP's various capabilities on instruction following, such as complex visual reasoning, knowledge-grounded image description, and multi-turn conversations. Furthermore, we show that InstructBLIP can serve as an enhanced model initialization for downstream task finetuning, achieving state-of-the-art results. We hope that InstructBLIP can spur new research in general-purpose multimodal AI and its applications.

6 Acknowledgments

Anthony Meng Huat Tiong is supported by Salesforce and Singapore Economic Development Board under the Industrial Postgraduate Programme. Junqi Zhao and Boyang Li are supported by the Nanyang Associate Professorship and the National Research Foundation Fellowship (NRF-NRFF132021-0006), Singapore. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not reflect the views of the funding agencies.

References

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In ICLR , 2022.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 , 2022.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML , 2023.

Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text generation. arXiv preprint arXiv:2102.02779 , 2021.

Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In CVPR , 2020.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikoł aj Bi´ nkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, NeurIPS , 2022.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 2020.

Vicuna. https://github.com/lm-sys/FastChat , 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014.

Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In ICCV , pages 8948-8957, 2019.

Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics , 2, 2014.

Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioningwith reading comprehension. 2020.

Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR , 2019.

Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics , 2023.

Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In NeurIPS Track on Datasets and Benchmarks , 2021.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR , July 2017.

Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR , 2018.

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR , 2019.

Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. Aokvqa: A benchmark for visual question answering using world knowledge. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, ECCV , 2022.

Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS , 2022.

Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR , 2019.

Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR , pages 8317-8326, 2019.

Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM International Conference on Multimedia , page 1645-1653, 2017.

Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In ICCV , pages 1686-1697, 2021.

Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jose M. F. Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In CVPR , 2017.

Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. In NeurIPS , 2020.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023.

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS , 2021.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML , 2022.

Yuxin Fang, Wen Wang, Binhui Xie, Quan-Sen Sun, Ledell Yu Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. ArXiv , abs/2211.07636, 2022.

Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven C. H. Hoi. Lavis: A library for language-vision intelligence, 2022.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR , 2019.

Vishvak Murahari, Dhruv Batra, Devi Parikh, and Abhishek Das. Large-scale pretraining for visual dialog: A simple state-of-the-art baseline. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV , 2020.

Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4566-4575, 2015.

OpenAI. Gpt-4 technical report. ArXiv , abs/2303.08774, 2023.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023.

Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model, 2023.

Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language, 2022.

Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A. Smith, and Jiebo Luo. Promptcap: Promptguided task-aware image captioning, 2023.

Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. Prompting large language models with answer heuristics for knowledge-based visual question answering. Computer Vision and Pattern Recognition (CVPR) , 2023.

Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. In ICLR , 2022.

Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. SuperNaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In EMNLP , 2022.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020.

Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. ArXiv , abs/2212.09689, 2022.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. ArXiv , abs/2212.10560, 2022.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github. com/tatsu-lab/stanford_alpaca , 2023.

Chatgpt. https://openai.com/blog/chatgpt , 2023.

Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yi Zhou, Junyan Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qiang Qi, Ji Chao Zhang, and Feiyan Huang. mplug-owl: Modularization empowers large language models with multimodality. 2023.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR , 2022.

Zhiyang Xu, Ying Shen, and Lifu Huang. Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning. ArXiv , abs/2212.10773, 2022.

Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2015.

A Broader Impact

InstructBLIP utilizes off-the-shelf frozen LLMs. Therefore it inherits some of the shortcomings of the original LLMs, such as hallucinating ungrounded text or generating outputs with biases. We mitigate such shortcomings by improving the model's grounding on the vision and instruction input, and performing vision-language instruction tuning on a diverse set of high-quality datasets. Nevertheless, we do not recommend applying InstructBLIP models to any downstream applications without a prior assessment on safety and fairness specific to that application. Additionally, since we did not finetune the parameters of the LLM, InstructBLIP may not always precisely follow given instructions. We believe that employing a more advanced LLM and meticulously finetuning its parameters during instruction tuning could address this concern. We reserve this for future work.

B More Case Studies

[Input Image]

What is unusual about this image?

[Model Response]

InstructBLIP

The unusual aspect of this image is that a man is ironing clothes on the back of a yellow taxi cab while it is driving down a bus city street. He is standing on top of the car's trunk, holding an iron and using it to press his clothing items. This situation is not only unconventional but also potentially hazardous, as the man could be at risk of falling off the vehicle or causing an accident due to his actions.

LLaVA

The unusual aspect of this image is a man ironing clothes on the back of a minivan or van. This is not a typical place to perform this activity, as one would usually iron clothes in a more stationary and safe location, such as a home, using a regular ironing board. The scene depicted in the image is peculiar as it involves a makeshift ironing setup on a vehicle, which can be both unsafe and unconventional. Additionally, it is not clear how the man is able to maintain balance and stability while ironing clothes in such an unstable environment.

MiniGPT-4

The image shows a man standing on the back of a yellow taxi, with a yellow shirt and black pants, and a blue backpack on his back. The taxi is driving on a city street with othr cars and taxis in the background.

The unusual thing about this image is that a man is ironing clothes on an ironing board attached to the roof of a moving taxi.

[Input Instruction]

[Input Image]

Do you know who drew this painting?

[Model Response]

InstructBLIP

LLaVA

MiniGPT-4

The painting depicts a woman, commonly believed to be Mona Lisa, the famous artwork by Leonardo da Vinci. It is a portrait painting that showcases the woman's enigmatic smile and has become one of the most famous and iconic art pieces in the world. The original work is displayed in the Louvre Museum in Paris, and it is known for its intricate details, use of oil paint, and the artist's innovative techniques that contributed to its enduring appeal and mystery.

The painting is the Mona Lisa, also known as the Mona Lisa, by Leonardo da Vinci. It is one of the most famous paintings in the world and is known for its enigmatic smile and the mysterious identity of the sitter. The painting is housed in the Louvre Museum in Paris, France.

[Input Instruction]

[Input Image]

[Input Instruction]

Introduce me this painting in detail.

[Model Response]

InstructBLIP

LLaVA

MiniGPT-4

The painting features a woman with a headscarf, possibly of Dutch origin,

dressed in blue and white clothing. She is wearing a large blue hat and has

a red mouth, which adds a vibrant touch to the portrait. The woman is

looking directly at the viewer, creating a sense of connection. The

background of the painting is predominantly blue, further emphasizing the

color theme of the piece. The overall composition and attire of the woman

suggest that the painting might be inspired by the style of Rembrandt.

The painting depicts a young woman with a pearl earring, wearing a blue

dress and a white apron. She is shown with a serious expression, looking

directly at the viewer. The background is a dark, neutral tone, with no

visible objects or elements. The overall mood of the painting is somber

and contemplative.

The painting is well-executed, with a high level of detail and realism in

the depiction of the woman's features and clothing. The use of shading

and lighting creates a sense of depth and dimension, making the woman

appear three-dimensional. The artist has used a limited color palette, with

the woman's dress and apron providing the only sources of color in the

painting.

The painting is well-composed, with the woman's head and shoulders

positioned in the center of the canvas, creating a sense of balance and

symmetry. The use of negative space around the woman's figure adds to

the overall somber mood of the painting.

Overall, this is a well-executed and thought-provoking painting that

effectively conveys a sense of contemplation and introspection.

The painting depicts a young girl with a pearl earring, also known as "The

Girl with the Pearl Earring" or "Girl with a Pearl Earring". She is wearing

a blue dress and has a pearl earring on her left ear. Her hair is pulled back

into a bun, and she is looking directly at the viewer. This painting is a

famous work of art by Dutch painter Johannes Vermeer, created in 1665.

It is considered one of his masterpieces and is widely recognized for its

intricate details and realistic portrayal of the young girl's facial

expression.

C Instruction Tuning Datasets

D Instruction Templates

E Instructions for Zero-shot Inference

We provide instructions used for zero-shot inference. Note that for instructions with options, we separate options with the alphabetical order, e.g. (a) blue (b) yellow (c) pink (d) black.

GQA, VizWiz, iVQA, MSVD, MSRVTT <Image> Question: {} Short answer: NoCaps, Flickr30k <Image> A short image description: TextVQA <Image> OCR tokens: {}. Question: {} Short answer: IconQA <Image> Question: {} Options: {}. Short answer: ScienceQA <Image> Context: {} Question: {} Options: {}. Answer: HatefulMemes <Image> This is an image with: "{}" written on it. Is it hateful? Answer: VSR <Image> Based on the image, is this statement true or false? "{}" Answer: Visual Dialog <Image> Dialog history: {}\n Question: {} Short answer:

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a significant step towards a general-purpose vision-language model with instruction-following abilities. It is built upon the existing BLIP-2 model and instruction-following dataset LLaVA-Instruct-150K [1]. It further extends the scale of instruction-tuning by automatically converting public vision-language datasets such as VQAv2 into instruction-following formats. The authors compare the instruction-tuning paradigm with classic multi-task learning and show that language instruction is the key to generalizing to unseen task instructions.\n\nSTRENGTHS:\nThis work extends multi-modal instruction-tuning to a wide range of datasets and tasks spanning 11 diverse categories. Robust evaluation is performed by instructing-tuning on 13 held-in datasets and zero-shot evaluating on another 13 held-out datasets. The authors demonstrate the generalization capabilities of their proposed approach by comparing with classic multi-task learning on both held-in and held-out datasets. The paper is well-written and easy to follow. All assets used in this work including the instruction-tuned models and datasets are released to the public. Implementation details such as instruction-aware Q-Former, balanced dataset sampling, and inference strategies like vocabulary reranking are empirically effective and technically sound.\n\nWEAKNESSES:\nInstructBLIP uses LLaVA-Instruct-150K [1] as one of the instruction-tuning datasets. However, there is no direct comparison to the open-sourced LLaVA model which is trained solely on this dataset.\n\nApart from this, I do not find any major weaknesses from this work. However, I would be curious to know if InstructBLIP has stronger vision-language reasoning capabilities than widely-adopted CLIP-like models, which are known to exhibit bag-of-word behaviors [2,3,4,5,6].\n\n[1] Visual Instruction Tuning. Liu et al. 2023.\n\n[2] When and why vision-language models behave like bags-of-words, and what to do about it? Yuksekgonul et al. 2022.\n\n[3] Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality. Thrush et al. 2022.\n\n[4] CREPE: Can Vision-Language Foundation Models Reason Compositionally? Ma et al. 2022.\n\n[5] Equivariant Similarity for Vision-Language Foundation Models. Wang et al. 2023.\n\n[6] Visio-Linguistic Reasoning with Multimodal Generative Pre-Training Scores. Lin et al. 2023.\n\nQUESTIONS:\nThere are some minor suggestions to improve the clarity of the paper:\n- L98: What is the \"standard\" language modeling loss given multimodal instruction-following samples? Do you adopt the same LM loss as LLaVA or did you use the captioning loss as BLIP-2? In other words, is the LM loss enforced on all tokens including both instruction and response tokens?\n- L132-134: How do you decide the dataset balancing ratio of each task? Do you use held-in or held-out scores?\n- L140: What are the evaluation metrics for open-ended generation task? What is the sampling procedure? i.e., nucleus/top-k with temperature? \n\nI would be happy to revise my rating if the authors can address my above-mentioned weaknesses and questions.\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents InstructBLIP, a vision-language instruction tuning framework to solve a wide range of visual-language tasks through a unified multimodal interface. The authors conduct a comprehensive study on vision-language instruction tuning, transforming 26 datasets into the instruction tuning format and grouping them into 11 task categories. They propose instruction-aware Qformer, to enable BLIP models with instruction following capabilities. InstructBLIP models achieve state-of-the-art zero-shot performance on a wide range of held-out vision-language tasks.\n\nSTRENGTHS:\n- The paper provides a comprehensive study of vision-language instruction tuning, covering a wide variety of tasks.\n- The proposed InstructBLIP extends BLIP with an instruction-aware Q-Former and finetunes the model on both instruction-following datasets and VQA datasets, showing better performance than previous models.\n- The paper is generally well-written and ablates the model's design of the instruction-aware Q-Former.\n\nWEAKNESSES:\n1. The paper does not provide enough analysis and the comparison on the open-ended out-of-domain multimodal question answering capability. This is one of the most impressive results that GPT-4, MiniGPT-4, and LLaVA shows. The paper claims in L208 \"Although all models are capable of generating long-form responses, InstructBLIP’s outputs generally contains more proper visual details and exhibits logically coherent reasoning steps. Importantly, we argue that long-form responses are not always preferable.\" Without enough evidence, it is hard to justify this point.\n\nQUESTIONS:\n1. How does the instruction-aware Q-former work on multi-turn conversations? For example, `<image><Q1><A1><Q2>`, what corresponds to the `<instruction>` part in the Fig. 3? Is it `<Q1><A1><Q2>` or `<Q2>` only? In both cases, user can ask complex questions that may be hard for a lightweight LLM like Qformer to interpret. For example, (1) \"can you explain your answer with more detail?\" (2) \"That is not correct, that is not what I am asking. Please try again.\" In these cases, it seems to be tricky for Qformer to extract the corresponding visual features, either because the region to extract visual features is unclear from `<Q2>` only (1), or it seems hard to correctly extract the information without sophisticated reasoning capabilities (2). What is the benefit of Qformer in these cases?\n\n2. For the Q-former architecture in Fig. 3, to my understanding, there will be several consecutive yellow blocks and instruction tokens will be updated via self-attention and feed-forward (green) throughout different blocks, while the updated tokens will not be fed into the LLM. Is this understanding correct? If so, the figure does not reflect (1) multiple consecutive blocks; (2) instruction tokens being updated between blocks.\n\nLIMITATIONS:\nThe author does not discuss limitations in the paper or appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper presents a study of instruction finetuning for vision-language tasks. The paper follows the design of FLAN for instruction tuning and borrows ideas from Flamingo for image/text model freezing and query network. Experimental results suggest FLAN-style instruction tuning also works for vision-language tasks, and this paper provides the first comprehensive study.

STRENGTHS:
1. To the best of my knowledge, this is the first work of FLAN-style instruction tuning in the VL domain. It provides a comprehensive analysis for future development in this field.
2. The experiment shows the effectiveness of the proposed instruction-aware query network, which is novel and insteresting.

WEAKNESSES:
1. Overall, the experimental results are expected, as demonstrated by existing LLM papers. To that end, novelty is somewhat limited, as similar designs have been explored in LLM and Flamingo.
2. The model is called InstructBLIP yet they are based on T5 and Vicuna (LLaMA). This puzzled me for a second. I feel the key component (frozen LLM + query network) of the model is first shown work by Flamingo and thus find the naming to be a bit misleading. Perhaps a completely new name is more appropriate.

QUESTIONS:
None

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes InstructBLIP, a vision and language instruction tuning framework that enables general-purpose models to solve a wide range of visual language tasks through a unified natural language. It uses a diverse set of instruction data to train a multimodal LLM.\nThe model is initialized with a pre-trained BLIP-2 model consisting of an image encoder, an LLM, and a Query Transformer to bridge the two. The LLM and Q-Former is kept frozen while fine-tuning the image encoder. The paper makes the following contributions –\n\n1. A comprehensive and systematic study on vision-language instruction tuning. \n\n2. It proposes instruction aware visual feature extraction, a novel mechanism that enables flexible and informative feature extraction according to the given instructions. \n\n3. The InstructBLIP models are evaluated and open-sourced using two families of LLM’s: \n- FlanT5, and encode-decoder LLM finetuned from T5.\n- Vicunna, a decoder-only LLM finetuned from LLaMA.\n\nSTRENGTHS:\nFollowing are the strengths of the paper --\n\n1. The paper is well written and easy to follow.\n\n2. The proposed framework is evaluated on a large variety of tasks and datasets and beats the SOTA.\n\n3. The paper provides both the qualitative and quantitative evaluation for the models.\n\nWEAKNESSES:\nThe paper has couple of weaknesses -\n\n1. The human evaluation for the model is missing.\n\n2. The model is evaluated on the static datasets which poses a question on its generalizability to the real world scenarios.\n\nQUESTIONS:\nQ1. A human evaluation for the model should be provided.\n\nQ2. A evaluation on the commercial datasets should also be provided to check it's generalizability in real world settings.\n\nLIMITATIONS:\nThe paper has couple of weaknesses -\n\n1. The human evaluation for the model is missing.\n\n2. The model is evaluated on the static datasets which poses a question on its generalizability to the real world scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes InstructBLIP, which is built on BLIP-2 and further perform instruction tuning to enable the instruction following ability of BLIP-2 models. The InstructBLIP model is trained on a set of template-based converted instruction-following data for different tasks (e.g. image captioning and VQA) and LLM-generated instruction-following data (e.g. LLaVA-Instruct-150K). The model delivers better zero-shot performance on selected datasets compared with BLIP-2 and Flamingo.\n\nSTRENGTHS:\n1. The paper is clear and well-organized. The motivations, techinical settings and details are clearly illustrated.  \n2. The proposed InstructBLIP model serves as a sound contribution to the research community of general-purpose large multi-modal models.  \n3. This paper provides insightful analysis on the efficacy and generalization ability of instruction tuning in the ablation studies.\n\nWEAKNESSES:\n1. Most of the instruction-following data are converted from existing image-text datasets in a template-based fasion, except for the LLaVA-Instruct-150K dataset which was composed of GPT generated contents. Simply converting existing image-text datasets with some handcrafted templates may result in lack of diversity of instructions.  \n2. The selected tasks and datasets do not include some mainstream image and image-text datasets such as ImageNet and CIFAR for image classification. \n3. The proposed framework is only applicable to image-level tasks and still cannot handle object-level tasks such as referring object detection / region description. Meanwhile, the proposed framework cannot handle image generation tasks.\n\nQUESTIONS:\n1. The authors provide per-task detailed zero-shot comparisons on held-out datasets. How about the performance on each held-in task? Meanwhile, for the held-in tasks, they can be compared fairly with previous jointly trained multitask generalist methods such as [1-3]. How is the performance compared with these methods?\n\n[1] Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks. In ICLR 2023.  \n[2] Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Tasks. In CVPR 2023.  \n[3] A Unified Sequence Interface for Vision Tasks. In NeurIPS 2022.\n\nLIMITATIONS:\nAs discussed in the weakness section, the proposed framework is only applicable to image-level tasks and still cannot handle object-level tasks such as referring object detection / region description. Meanwhile, the proposed framework cannot handle image generation tasks.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-13"}