InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
Authors: Wenliang Dai
Junnan Li
Dongxu Li
Anthony Tiong
Junqi Zhao
Weisheng Wang
Boyang Li
Pascale Fung
Steven Hoi
Venue: NeurIPS 2023 poster (2023)
Keywords: Vision-Language Models
Instruction Tuning
Zero-shot
TL;DR: We present InstructBLIP, an instruction tuning framework towards general-purpose vision-language models. InstructBLIP achieves state-of-the-art results in zero-shot and finetuning with various instruction following abilities. All models are released.

Abstract: Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence. However, building general-purpose vision-language models is challenging due to the rich input distributions and task diversity resulting from the additional visual input. Although vision-language pretraining has been widely studied, vision-language instruction tuning remains under-explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pretrained BLIP-2 models. We gather 26 publicly available datasets, covering a wide variety of tasks and capabilities, and transform them into instruction tuning format. Additionally, we introduce an instruction-aware Query Transformer, which extracts informative features tailored to the given instruction. Trained on 13 held-in datasets, InstructBLIP attains state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and larger Flamingo models. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA questions with image contexts). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models. All InstructBLIP models are open-source.

InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
Wenliang Dait
Junnan Lifes,1
Dongxu Li
Anthony Meng Huat Tiong 1.3 Pascale Fung" Steven Hoi
Junqi Zhao
Weisheng Wang Boyang Li
Salesforce Research
HHong Kong University of Science and Technology
"Nanyang Technological University, Singapore
Abstract
Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence. However, building general-purpose purpose vision-language models is challenging due to the rich input dis- tributions and task diversity resulting from the additional visual input. Although vision-languae pretraining has been widely studied, vision-language instruction tuning remains under-explored. In this paper, we conduct a systematic and compre- hensive study vision-languse instruction tuning based on the pretrained BLIP-2 models. We gather publicly available datasets, covering a wide variety of tasks and capabilities, and transform them into instruction tuning format. Additionally, we introduce an instruction aaaare Query Transformer, which extracts informative features tailored to the given instruction. Trained on 13 held-in datasets, Instruct- BLIP attains state-of-the-arr zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and larger Flamingo models. Our models also lead to statc-of-the-as performance when finetuned on individual downstream tasks (c.g., 90.7% accuracy on ScienceQA questions with image contexts). Further- more, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models. All InstructBLIP models are open-source.
Introduction
A longstanding aspiration of Artificial Intelligence (Al) research is to build single model that can solve arbitrary tasks specified by the user. In natural language processing (NLP), instruction proves to be promising approach toward that goal. By finetuning large language model (LLM) on wide range of tasks described by natural language instructions, instruction tuning enables the model to follow arbitrary instructions. Recently, instruction-tunel LLMs have also been leveraged for vision-language tasks. For example, BLIP-2 3 effectively adapts frozen instruction- tuned LLMs to understand visual inputs and exhibits preliminary capabilities to follow instructions in image-to text generation.
tuning [1 2
Compared to NLP tasks, vision- languag tasks are more diverse in nature due to the additional visual inputs from various domains. This poses a greater challenge to a unified model that is supposed to generalize to diverse vision-languas be grouped into two approaches. The first approach, multitask learning 44 5. formulates various vision- languae tasks into the same input-output format. However, we empirically find multitask learning without instructions (Table 4 does not generalize well to unseen datasets and tasks. The
tasks, many unseen during training. Most previous work can
Work done during internship at Salesforce. Equal contribution. Corresponding authors: (junnan.li,shol @salesforce.com
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduce this puinting
detail.
What could have huppened bused on the
The painting depicts young girl with pearl eurring. also known as "The Girl with the Pearl Earring" "Girl with a Pearl Farring", Shee is wearing blue dress and has a pearl carring on her left car. Her hair is pulled back into a bun, and she is looking directly at the viewer. This painting is famous work of art by Dutch painter Johannes Vermeer, created in 1665. It is considered one of his musterpieces and widely recognized for its intricate details and realistic portrayal of the young girl's 's facial expression.
current scene?
Based on the current seene in the image, it is possible hurricane or severe weather event caused significant dumage to the buildings and infrastructure in the area. The person walking through the debris-covered urea suggests that they are inspecting the damage caused by the disaster. The presence of palm trees indicates that the location may be in tropical or sub-tropical region. which can be more susceptible to hurricanes and other natural disasters.
Can you deseribe this image
detail?
The image depicts a person standing in front of an open doorway, looking out into space. They are wearing a jacket and appears to be contemplating the vustness of the universe beyond the doorway. In the background, there multiple stars visible, creating a stunning celestial scene. The person's guze is directed towards the center of the image, where hright star can he seen shining brightly, There are also smaller stars scattered throughout the scene, adding to the overall cosmic atmosphere.
Can you tell about this imuge in detail?
The image depicts collection of various vegetables including carrots, cucumbers, tomatocs, and nuts arranged on stone surface.
There are several jars filled with different types of ingredients, such as peanuts, cushews, sunflower seeds, and pumpkin seeds. These ingredients are likely to be part of'u healthy meal or snack.
Can you give me an example of such meal?
What if they jump over the doorway?
An example of healthy meal snack that incorporates some of the ingredients in the image could be salad made with mixed greens, carrols, cucumbers, tomatocs, and nuts. The salad can be dressed with homemade vinaigrette made with olive oil, lemon juice, salt, and pepper. Additionally, the salad can be topped with pumpkin seeds sunflower seeds for added crunch and nutritional value.
If the person were to jump over the doorway, they would be leaving the confines of their current location und venturing into the vastness of spuce. This could be a metaphorical representation stepping out of one's comfort zone embarking However, it important to note that such an action would be extremely dangerous and potentially life- threatening due to the harsh conditions of space.
new adventure.
Sounds good! Show me steps of making this salad.
Sure! Here are the steps to make this healthy salad: Wash and chop the mixed greens, carrots. cucumbers, and tomatoes into bite-sized pieces.
ab
What makes this dog special?
2. In small bowl, whisk together olive oil, lemon juice, salt, and pepper to create a homemade vinaigrette dressing.
The dog in the image is wearing futuristic armor and has a glowing green eye, giving it unique and futuristic appearance. This combination of elements makes the dog stand out from other dogs and adds element of fantasy or sci-fi to the image.
Place the chopped vegetables in large bowl and them with the vinaigrette dressing.
Sprinkle pumpkin seeds or sunflower seeds on top of the salad for added crunch and nutritional value.
Additionally, the dog's posture and facial expression convey sense of confidence and determination, further emphasizing its unique feutures.
Serve the salad immeedutely or store it in the refrigerator until ready to enjoy.
Figure 1: A few qualitative examples generated by our InstructBLIP- Vicuna model. Here, we demonstrate diverse range of capabilities, including complex visual scene understanding and reasoning, knowledge-groo -gruundde image description, multi-turn visual conversation, etc.

---

second approach B 6 extends pre-trained LLM with additional visual components, and trains the visual components with image caption data. Nevertheless, such data are too limited to allow broad generalization to vision-language tasks that require more than visual descriptions.
To address the aforementioned challenges, this paper presents InstructBLIIP, vision-language instruction tuning framework that enables general-purpose models to solve a wide range of visual- language tasks through unified natural language interface. InstructBLIP uses diverse set of instruction data to train a multimodal LLM. Specifically, we initialize training with a pre-trained BLIP-2 model consisting of an image encoder, an LLM, and a Query Transformer (Q-Former) to bridge the two. During instruction tuning. we finetune the Q-Former while keeping the image encoder and LLM frozen. Our paper makes the following key contributions:
We perform a comprehensive and systematic study on vision- -angugg instruction tuning. We transform 26 datasets into the instruction tuning format and group them into 11 task categories. We use 13 held-in datasets for instruction tuning and held-out datasets for zero-shot evaluation. Moreover, we withhold four entire task categories for zero-shot evaluation at the task level. Exhaustive quantitative and qualitative results demonstrate the effectiveness of InstructBLIIP on vision-language zero-shot generalization.
We propose instruction-aware visual feature extraction, novel mechanism that enables flexible and informative feature extraction according to the given instructions. Specifically, the textual instruction given not only to the frozen LLM, but also the Q-Former, so that can extract instruction-wwan aware visual features from the frozen image encoder. Also, we propose balanced sampling strategy synchronize learning progress across datasets.
We evaluate and open-source a suite of Instruct models using two families of LLMs: 1) Flan'T5 (2) encoder-decoder LLM finetuned from T5 77 2) Vicuna (8) decoder-only L.M finetuned from LLaMA [9]. The InstructBLIIP models achieve state-of-the-ard zero-shot performance wide range of vision-language tasks. Furthermore, InstructBLIP models lead to state-of-the-ar finetuning performance when used as the model initialization on individual downstream tasks.
2 Vision-Language Instruction Tuning
InstructBLIP aims to address the unique challenges in vision-language instruction tuning and provide systematic study the models' improved generalization ability to unseen data and tasks. In this section, we first introduce the construction of instruction-tuning data, followed by the training and evaluation protocols. Next, we delineate two techniques improve instruction-tuii tuniing performance from the model and data perspectives, respectively. Lastly, we present the implementation details.
2.1 Tasks and Datasets
To ensure the diversity of instruction tuning data while considering their accessibility, we gather comprehensive set of publicly available vision-language datasets, and transform them into the instruction tuning format. As shown Figure the final collection covers 11 task categories and 26
datasets, including image captioning [10 [1] image captioning with reading comprehension [13 visual reasoning [4
15 16), image question answering [17 18), knowledge-groundeed image
question answering 19 20 21 image question answering with reading comprehension 22 23
image question generation (adapted from the QA datasets), video question answering [24] 25 visual
conversational question answering (26), image classification [27], and LLa -A-ntttttttt (28
We include detailed descriptions and statisties of each dataset in Appendix C.
For every task, we meticulously craft 10 to 15 distinct instruction templates in natural language. These templates serve as the foundation for constructing instruction tuning data, which articulates the task and the objective. For public datasets inherently favoring short responses, we use terms such as short and briefly into some of their corresponding instruction templates to reduce the risk of 150K dataset, we
the model overfitting to always generating short outputs. For the do not incorporate additional instruction templates since is naturally structured the instruction format. The full list of instruction templates can be found in Appendix D.
3

---

Image Captioning
Visual Reasoning
Visual Conversational QA
Video Question Answering
COCO Caption
GQA
Visual Dialog
MSVD QA
Web Caphilt
Visual Spatial
MSRVII
Reasoning
NoCaps
IVQA
IconQA
Knowledge Grounded Image Question Answering
Flickr30K
OKVQA
Image Captioning
Reading Comprehension
A-OKVQA
Image Question Answering Reading Comprehension
SciencoQA
TextCaps
OCR-VQA
Image Question Generation
LLa LLnV
Image Question Answering
VQAv2
Visual Conversation
TexVVQ
VQA+2
OKVQA
Complex Reasoning
Image Classification
VizWiz
A-OKVQA
Detailed Image Description
HatefulMemes
Figure 2: Tasks and their corresponding datasets used for vision- larionl instruction tuning. The held-in datasets are indicated by yellow and the held-out datasets by white.
2.2 Training and Evaluation Protocols
To ensure sufficient data and tasks for training and zero-shot evaluation, we divide the 26 datasets into
13 held-in datasets and 13 held-out datasets, indicated by yellow and white respectively in Figure 2 We employ the training sets of the held-in datasets for instruction tuning and their validation or test sets for held-in evaluation.
For held-out evaluation, our aim to understand how instruction tuning improves the model's zero-shot performance on unseen data, We define two types of held-out data: 1) datasets not exposed to the model during training, but whose tasks are present the held-in cluster; 2) datasets and their associated tasks that remain entirely unseen during training. Addressing the first type of held-out evaluation is nontrivial due to the data distribution shift between held-in and held-out datasets. For the second type, we hold out several tasks completely, including visual reasoning, video question answering, visual conversational QA, and image classification.
To avoid data contamination, datasets are selected carefully so that no evaluation data appear in the held-in training cluster across different datasets. One exception the Visual Dialog (26] dataset, which has overlap of images with some held-in data. The Visual Dialog (26) dataset is an exception, as it has an overlap of images with some training data. Nevertheless, since it's virtually the only high-quality visual dialog dataset available for quantitative evaluation, we still incorporated it in our evaluation as a reference. During instruction tuning, we mix all the held-in training sets and sample instruction templates uniformly for each dataset. The models are trained with the standard language modeling loss to directly generate the response given the instruction. Furthermore, for datasets that involve scene texts, we add OCR tokens in the instruction as supplementary information.
2.3 Instruction-aware Visual Feature Extraction
Existing zero-shot image-to-text generation methods, including BLIP-2, take an instruction-agnostii approach when extracting visual features. That results in set of static visual representations being fed into the LLM, regardless of the task. contrast, an instruction- aware vision model can adapt to the task instruction and produce visual representations most conducive to the task at hand. This is clearly advantageous if we expect the task instructions to vary considerably for the same input image.
We show the architecture of InstructBLIP in Figure 3 Similarly to BLIP-2 3 InstructBLIP utilizes a Query Transformer, or Q-Former, to extract visual features from frozen image encoder. The input to the Q-Former contains set of K learnable query embeddings, which interact with the

---

Instruction
Response
left one
Choose the correct option to the following question:
LLM
which picture shows the
Q-Former
C
pirza inside the oven?
Options: (a) one (b)
right one Answer:
Instruction
Feed Forward
Forward
Fully Connected
Image Embeddings
Cross Attention
Imuge Encoder
Self Attention
Q-Former
Image Embeddings
Queries
Instruction
C--00
Input Image
Queries
Instruction
Figure 3: Model architecture of InstructBLIP. The Q-Former extracts instruction- visual features from the output embeddings of the frozen image encoder, and feeds the visual features soft prompt input to the frozen LLM. We instruction-tune the model with the language modeling loss to generate the response.
image encoder's output through cross attention. The output of the Q-Former consists of encoded visual vectors, one per query embedding, which then go through linear projection and are fed to the frozen LLM. As BLIP-2, the Q-Former is pretrained in two stages using image-caption data before instruction tuning. The first stage pretrains the Q Q-Former with the frozen image encoder for vision-language representation learning. The second stage adapts the output of Q-Former as soft visual prompts for text generation with a frozen LLM After pretraining, we finetune the Q-Forme with instruction tuning. where the LLM receives as input the visual encodings from the Q-Former and the task instruction.
Extending BLIP-2, InstructBLIP proposes an instruction-aware Q-former module, which takes in the instruction text tokens as additional input. The instruction interacts with the query embeddings through self-attention layers of the Q-Former, and encourages the extraction of task-relevar image features. As a result, the LLM receives visual information conducive to instruction following, We demonstrate empirically (Table that instruction aware visual feature extraction provides substantial performance improvements for both held-in and held-out evaluations.
2.4 Balancing Training Datasets
Due the large number of training datasets and the significant differences in the size of each dataset, mixing them uniformly could cause the model to overfit smaller datasets and underfit larger datasets. To mitigate the problem, we propose sample datasets with probabilities proportional the square root of their sizes, or the numbers of training samples. Generally, given D datasets with sizes Sp}, the probability of data sample being selected from a dataset d during training is Pa VSr
(S1. S2,
On top of this formula, we make manual adjustments to the weights of certain
VS."
datasets based on their individual convergence time to improve optimization. This is warranted by inherent differences in the datasets and tasks that require varying levels of training intensity despite similar sizes. Specifically, we lower the weight of A-OKVQA, which features multiple-choice questions, and increase the weight of OKVQA, which requires open-ended text generation. In Table 2 we show that the balanced dataset sampling strategy improves overall performance for both held-in evaluation and held-out generalization.
2.5 Inference Methods
During inference time, we adopt two slightly different generation approaches evaluation on different datasets. For the majority of datasets, such as image captioning and open-ended VQA, the instruction-tun model directly prompted to generate responses, which are subsequently compared to the ground truth calculate metrics. On the other hand, for classification and multi-choice VQA
5

---

tasks, we employ a vocabulary ranking method following previous works [1, 30 Specifically, we still prompt the model to generate answers. but restrict its vocabulary to a list of candidates. Then, we calculate log-likelihood for each candidate and select the one with the highest value as the final prediction. This ranking method is applied to ScienceQA, IconQA, A-OKVQA (multiple-choice), MatefulMemes, Visual Dialog, MSVD, and MSRVTT datasets. Furthermore, for binary classification, we expand the positive and negative labels into a slightly broader set of verbalizers to exploit word frequencies in natural text (e.g., yes and true for the positive class; no and false for the negative class).
For the video presston-ansseeniee task, we utilize four uniformly-sampled frames per video. Each frame is processed by the image encoder and Q-Former individually, and the extracted visual features are concatenated before being fed into the LLM.
2.6 Implementation Details
Architecture. Thanks to the flexibility enabled by the modular architectural design of BLIP-2, we can quickly adapt the model to wide range of LLMs. In our experiments, we adopt four variations of BLIP-2 with the same image encoder (ViT-g/14 (31) but different frozen LLMs, including F5-
XL. (3B), FlanT5-XXI, (11B), Vicuna-7B and Vicuna- 13B. FlanT5 [2] is instruction-tuned model based on the incoder-decode Transformer T5 77. Vicuna (8] on the other hand, is a recently released decoder-only Transformer instruction-ture instruction tuning, we initialize the model from pre-trained BLIP-2 checkpoints, and only finetune the parameters of Q-Former while keeping both the image encoder and the LLM frozen. Since the original BLIP-2 models do not include checkpoints for Vicuna, we perform pre-training with Vicuna using the same procedure as BLIP-2 We set the number of query embeddings 32 and do not observe any improvement when increasing it in our settings. However, it could potentially be beneficial for future tasks with more complex visual inputs.
from LLaMA (9] During vision-language
We use the LAVIS library 332 for implementation, training, and evaluation. All models are instruction-tuned with maximum of 60K steps and we validate model's ' performance every 3K steps. For each model, a single optimal checkpoint is selected and used for evaluations on all datasets. We employ a batch size of 192, 128, and 64 for the 3B, 7B, and 1 11/13B models, respectively. The Adam W
Training and Hyper-parameters
[3] optimizer is used, with 31 0.9, B22 0.999, and a weight decay of 0.05. Additionally, we apply linear warmup of the learning rate during the initial 1,000 steps, increasing from 10-8 to 10 5 followed by a cosine decay with a minimum learning rate of O. For decoding, we adopt beam search with a beam size of for HatefulMemes, VSR, and OCR-VQA 3 for NoCaps, and for the other tasks. All models are trained utilizing 16 Nvidia A100 (40G) GPUs and are completed within 1.5 days.
3 Experimental Results and Analysis
3.1 Zero-shot Evaluation
We first evaluate InstructBLIP models on the set of 13 held-out datasets with instructions provided in Appendix E
We compare InstructBLIP with the previous SOTA models BLIP-2 and Flamingo. As demonstrated in [ we achieve new zero-shot SOTA results all datasets. InstructBLIP consistently surpasses its original backbone, BLIP-2, by significant margin across all LLMs, demonstrating the effectiveness of vision-language instruction tuning. For instance, InstructBIIP FlanT5xx yields an average relative improvement of 15.0% when compared to BLIP-2 F5an Furthermore, instruction tuning boosts zero-shot generalization on unseen task categories such as video QA. InstructBLIP achieves up to 47.1% 1% relative improvement on MSRVTT-QA over the previous SOTA despite having never been trained with temporal video data. Finally, our smallest Instruct FlanT5xx. with 4B parameters outperforms Flamingo-80B on all six shared evaluation datasets with average relative improvement of 24.8%.
For the Visual Dialog dataset, we choose to report the Mean Reciprocal Rank (MRR) over the Normalized Discounted Cumulative Gain (NDCG) metric. This is because NDCG favors generic and uncertain answers while MRR prefers certain responses 34 making MRR better aligned with the zero-shot evaluation scenario.

---

NoCaps
Flickr 30K
GQA VSR IconQA
TexVVQA Visdial
HM VizWiz
SciQA image
MSVD
MSRVIT
IVQA
QA
QA
Flamingo 3B 6
60.6
30.1
53.7
28.9
27.5
11.0
32.7
Flamingo 9B
61.5
31.8
57.0
28.8
30.2
13.7
35.2
Flamingo-008
67.2
35.0
46.4
31.6
35.6
40.7
BLIP-2 (FlanTTx))))
104.5
76.1
44.0
60.5
45.5
43.1
45.7
53.0
29.8
54.9
33.7
16.2
40.4
BI.IP-2 2 (5xa
98.4
73.7
44.6
68.2
45.4
44.1
46.9
52.0
29.4
64,5
34.4
17.4
45.8
BLIP (Vicuma 7B)
107.5
74.9
38.6
50.0
39.7
40.1
44.9
50.6
25.3
53.8
18.3
9.2
27.5
BLIP-2 (Vicuna 13B)
103.9
40.6
42.5
45.1
20.3
23.5
InstructBLIP (Fl.)
119.9
84.5
64.8
50.0
46.6
46.6
56.6
32.7
70.4
43.4
53.1
InstructBLIP (5lan)5xxu.)
120.0
83,5
47.9
65.6
51.2
46.6
48.5
54,1
30,9
70.6
44.3
25.6
53.8
InstructBLIP IP (Vicuna-78)
123.1
82.4
49.2
54.3
43.1
50.1
45.2
59.6
34.5
60.5
41.8
22.1
52.2
InstructBLIP (Vicuna-13B)
121.9
44.8
50.7
45.4
57.5
33.4
41.2
51.0
Table 1:
Zero-shot results on the held-out datasets. Here, Visdial, HM and SciQA denote the Visual Dialog, HatefulMemes and ScienceQA datasets, respectively. For ScienceQA, only evaluate the set with image context. Following previous works [6] 25 34), we report the CIDEr score [35] for NoCaps and Flickr30K, iVQA accuracy for iVQA, AUC score for HatefulMemes,, and Mean Reciprocal Rank (MRR) for Visual Dialog. For all other datasets, we report the top-1 accuracy (%).
Science (image-context)
Model
Held-in Avg.
GQA
IconQA
VizWiz
iVQA
InstructBLIP (FlanT5xa.)
94.1
48.4
70.4
50.0
32.7
53.1
w/o Instruction-awee Visual Features
89.8
45.9 (12.5)
63.4 (17.0)
45.8 (14.2)
25.1 (U7.6)
47.5 (U5.6)
w/o Data Balancing
92.6
46.8 (11.6)
66.0 (14.4)
49.9 (10.1)
31.8 (J0.9)
51.1 (12.0)
InstructBLIP (Vicuna-7B)
100.8
49.2
60.5
43.1
34.5
52.2
w/o Instruction caruettoon-wwre Visual Features
98.9
48.2 (41.0)
55.2 (15.3)
41.2 (11.9)
32.4 (U2.1)
36.8 (115.4)
w/o Data Balancing
98.8
47.8 (11.4)
(L1.1)
43.5 (10.4)
32.3 (12.2)
50.3 (11.9)
Table Results of ablation studies that remove the instruction-awarc Visual Features (Section 2.3 and the balanced data sampling strategy (Section 2.4). For held-in evaluation, we compute the average score of four datasets, including COCO Caption, OKVQA, A-OKVQA, and TextCaps. For held-out evaluation, we show five datasets from different tasks.
3.2 Ablation Study on Instruction Tuning Techniques
To investigate the impact of the instruction-aware visual feature extraction (Section and the balanced dataset sampling strategy (Section 2.4). we conduct ablation studies during the instruction tuning process. As illustrated in Table 2 the removal of instruction awareness in visual features downgrades performance significantly across all datasets. The performance drop is more severe in datasets that involve spatial visual reasoning (e-g.. ScienceQA) or temporal visual reasoning (e.g., iVQA), where the instruction input to the Q-Former can guide visual features to attend to informative image regions. The removal of the data balancing strategy causes unstable and uneven training, as different datasets achieve peak performance at drastically different training steps. The lack of synchronized progress over multiple datasets harms the overall performance.
3.3 Qualitative Evaluation
Besides the systematic evaluation on public benchmarks, we further qualitatively examine Instruct- BLIP with more diverse images and instructions. As illustrated in Figure InstructBLIP demonstrates its capacity for complex visual reasoning. For example, it can reasonably infer from the visual scene what could have happened and deduce the type of disaster from the location of the scene, which extrapolates based on visual evidence like the palm trees. Moreover, InstructBIIIP is capable of connecting visual input with embedded textual knowledge and generate informative responses, such as intruducing famous painting. Furthermore, in descriptions of the overall atmosphere, InstructBLIP exhibits the ability to comprehend metaphorical implications of the visual imagery.
7

---

BLIP-2 Zero-shot
45.1
67.8
Train Plain Input Eval w/ Instruction
Multi-taok
Train w/ Dataset Name
Eval w/ Instruction
Train Dataset Name
46.8
93.
Eval w/ Dataset Name
InstruciBLIP
40
45
50
53
60
75
90
105
Held-out Avg.
Held-in Avg.
Figure 4: Comparison of instruction tuning and multitask training based on BLIP-2 FlanT5xx. backbone. For held-in evaluation, we compute the average score across all held-in datasets. For held-out evaluation, we compute the average score across GQA, TexIVQA, VSR, HatefulMemes, IconQA, ScienceQA, iVQA, VizWiz.
Finally, show that InstructBLIP can engage in multi-turn conversations, effectively considering the dialog history when making new responses.
In Appendix B, we qualitatively compare InstructBLIIP with concurrent multimodal models (GPT- 4 [36]. LLaVA (28), MiniGPT-4 (37). Although all models are capable of generating long-form responses, InstructBLIP's outputs generally contains more proper visual details and exhibits logically coherent reasoning steps. Importantly, we argue that long-form responses are not always preferable. For example, in Figure of the Appendix, InstructBLIP directly addresses the user's intent by adaptively adjusting the response length, while LLaVA and MiniGPT-4 generate long and less relevant sentences. These advantages of InstructBLIP result of the diverse instruction tuning data and an effective architectural design.
Instruction Tuning vs. Multitask Learning
A direct analogue to instruction tuning is multitask learning, widely used method that involves the simultaneous training of multiple datasets with the goal of improving the performance of each individual dataset. To investigate whether the improvement zero-shot generalization mainly comes from the instruction format or merely from multi-task learning, we conduct a comparative analysis.
Following M. we consider two multitask training approaches. In the first approach, the model is trained using the vanilla input-output format of training datasets without instructions. During evaluation, instructions are still provided to the model to specify which task to perform. However, an exception is made for image captioning. as the model performs better when receiving the image as the only input. For the second approach, we take step toward instruction tuning by prepending a [Task: Dataset] identifier to the text input during training. For example, we prepend [Visual question answering: VQAv2] for the VQAv2 dataset. During evaluation, we explore both instruc- tions and the identifier. Particularly, for held-out datasets, we only use the task name as the identifier since the model never sees the dataset name.
In Figure 4 we show the results for zero-shot, multitask training, and instruction tuning. All models are based on the BLIP-2 FlanT5xx. backbone and adhere to the identical training configurations in Section Overall, we make two observations. First, instruction tuming and multitask learning exhibit similar performance on the held-in datasets. This suggests that the model can fit these two different input patterns comparably well, as long as it has been trained with such data. Second, instruction tuning yields significant improvement over multitask learning on unseen held-out datasets, whereas multitask learning performs on par with the original BI.P-2. This indicates that instruction tuning the key to zero-shot generalization.
8

---

A-OKVQA
ScienceQA (image-context)
OCR-VQL VQA
OKVQA
Direct Answer
Multi-choice
Val
Test
Val
Test
LLaVA
28
GIT 33
PaLM-E (562B)
R8
40
A1
A40
A1
Previous SOTA
89.0
70.3
66.1
56.3
61.6
73.2
73.6
BLIP-2 (FlanT5xxx)
89.5
72.7
54.7
57.6
53.7
80.2
76.2
InstructBLIP (Flan
90.7
73.3
55.5
57.1
54.8
81.0
76.7
BLIP-2 (Vicuna-7B)
77.3
69.1
59.3
60.0
58.7
72.1
69.0
InstructBLIP (Vicuna-7R)
79.5
72.8
62.1
64.0
62.1
75.7
73.4
Table 3: Results of finetuning BLIIP- and InstructBLIIP downstream datasets. Compared to BLIP-2, InstructBLIP provides better weight initialization model and achieves SOTA performance on three out of four datasets.
3.5 Finetuning InstructBLIP on Downstream Tasks
We further finetune the InstructBLIP models to investigate its performance on learning a specific dataset. Compared to most previous methods (e.g.. Flamingo, BLIP-2) which increase the input image resolution and finetune the visual encoder on downstream tasks, InstructBLIP maintains the same image resolution (224x224) used in instruction tuning and keeps the visual encoder frozen during finetuning. This significantly reduces the number trainable parameters from 1.2B to 188M, thus greatly improves finetuning efficiency.
The results are in 3 Compared BLIP-2, InstructBI.IP leads to better finetuning performance on all datasets, validating InstructBLIIP as the better initialization for task-specific finetuning. Instruct- BLIP sets new state-of-the-an finetuning performance on ScienceQA (image-contex)) OCR-VQA A-OKVQA, but is outperformed on OK VQQ by PaLM-E 38 with 562B parameters.
Additionally, we observe that the Flan InstructBLIP superior at multi-choice tasks, whereas Vicuna-basse InstructBLIP is generally better at open-ended generation tasks. This disparity can be primarily attributed the capabilities of their frozen LLMs, as they both employ the same image encoder. Although FlanTS and Vicuna are both instruction settucttionttueedd LLMs, their instruction data significantly differ. FlanT5 is mainly finetuned on NLP benchmarks containing many multi-choice QA and classification datasets, while Vicuna is finetuned on open-ended eestructionfoollwwiin data.
4 Related Work
Instruction tuning aims to teach language models to follow natural language instructions, which has been shown to improve their generalization performance to unseen tasks. Some methods collect instruction tuning data by converting existing NLP datasets into instruction format using
templates 2 42 43), Others use LLMs (e.g.. GPT-3 44) generate instruction data 8 45 46 with improved diversity.
Instruction-tunae LLMS have been adapted for vision-to-langue generation tasks by injecting visual information to the LLMs. BLIP-2 3 uses frozen FlanT5 models, and trains Q-Former to extract visual features input to the LLMs. MiniGPT-4 37 uses the same pretrained visual encoder and Q-Former from BLIP-2, but uses Vicuna g as the LLM and performs training using ChatGPT [48] -enerated image captions longer than the BLIP-2 training data. LLaVA (28) directly projects the output of a visual encoder as input to LLaMA/Vinuca LLM, and finetunes the LLM on vision- languge conversational data generated by GPT-4 (36). mPLUG-owl [49 performs low-rank adaption [50] to a LLaMA 9 model using both text instruction data and vision-language instruction data from LLaVA. A separate work is MultiInstruct [5I] which performs vision-language instruction tuning without a pretrained LLM, leading to less competitive performance.
Compared to existing methods, InstructBLIP uses a much wider range of vision- langualla instruction data, covering both template-based converted data and LLM-ggnerated data. Architecture wise, InstructBI IIP proposes an instruction-aware visual feature extraction mechanism. Furthermore, our paper provides comprehensive analysis on various aspects of vision-language instruction tuning, validating its advantages generalizing to unseen tasks.

---

5 Conclusion
In this paper, we present InstructBLIP, a simple yet novel instruction tuning framework towards gen- eralized vision-language models. We perform comprehensive study on vision-language instruction tuning and demonstrate the capability of InstructBLIIP models to generalize a wide range of unseen tasks with state-of-the-ee performance. Qualitative examples also exhibit InstructBLIP's various capabilities on instruction following, such as complex visual reasoning, description, and multi-turn conversations. Furthermore, we show that InstructBLIP can serve as an enhanced model initialization for downstream task finetuning, achieving state-of-the-an results. We hope that InstructBLIP can spur research in general purpose multimodal Al and its applications.
newwledee-grouddee image
6 Acknowledgments
Anthony Meng Huat Tiong is supported by Salesforce and Singapore Economic Development Board under the Industrial Postgraduate Programme. Junqi Zhao and Boyang Li are supported by the Nanyang Associate Professorship and the National Research Foundation Fellowship (NRF-NRFFI3 2021-0006), Singapore. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not reflect the views of the funding agencies.
References
[1] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models zero-shot learners. In ICLR, 2022.
[2] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gauray Mishra, Adams Yu. Vincent Zhao, Yanping Huang. Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instuuetio-finetuueed language models. arXiv preprint 221: 11.0//14110 /1416, 2022.
[3] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2:
Bootstrapping language-inage pre-training with frozen image encoders and large language models. In ICML, 2023.
[4] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-langee -anguagg tasks via text generation. arXiv preprint arXiv:2102.0277 022.0779. 2021.
[5] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In CVPR, 2020.
[6] Jean- -aa-tttpistt Alayrac, Jeff Donahue, Pauline Luc, Antoine Micch, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong. Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sehastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikot aj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: visual language model for few-shot learning. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, NeurlPS, 2022.
[7] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Reseurch, 2020.
[8] Vicuna. https:
2023.
[9] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Erie Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[10] Tsung-Y Yi Lin, Michael Maire, Serge Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zimicck. Microsoft coco: Common objects in context. In ECCV, 2014.
[11] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In ICCV, pages 8948 8957, 2019.
10

---

[12] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference event descriptions. Transactions of the Association for Computational Linguistics, 2. 2014.
[13] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: dataset for image captioningwith reading comprehension. 2020.
[14] Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. CVPR, 2019.
[15] Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 2023.
[16] Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In NeurlPS Track on Datasets and Benchmarks, 2021.
[17] Yash Goyal, Tejas Khot, Douglas Summers-Stay. Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, July 2017.
[18] Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, 2018.
[19] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering henchmark requiring external knowledge. In CVPR, 2019.
[20] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- okvqa: A benchmark for visual question answering using world knowledge. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, ECCV, 2022.
[21] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NewrlPS, 2022.
[22] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019.
[23] Amanprect Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, pages 8317-8326, 2019.
[24] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM International Conference on Multimedia, page 1645-1653, 2017.
[25] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In ICCV, pages 1686-1697, 2021.
[26] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jose M. F. Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In CVPR, 2017.
[27] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech multimodal memes. In NewrlPS, 2020.
[28] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. 2023.
[29] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NewrlPS, 2021.
[30] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip:
Bootstrapping language-ing -mage pre- pe-training for unified uision-languag understanding and generation. In ICML, 2022.
[31] Yuxin Fang, Wen Wang, Binhui Xie, Quan-Sen Sun, Ledell Yu Wu, Xinggang Wang, Tiejun Huang. Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation leamning at scalc. ArXiv, abs/221 1.0636,, 2022.
[32] Dongxu Li, Junnan Li, Hung Le. Guangsen Wang, Silvio Savarese, and Steven C. H. Hoi. Lavis: A library for language- vision intelligence, 2022.
11

---

[33] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[34] Vishvak Murahari, Dhruy Batra, Devi Parikh, and Abhishek Das. Large-scale pretraining for visual dialog: A simple state-of-the-art baseline. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV, 2020.
[35] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566 4 2015.
[36] OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
[37] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023.
[38] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, Wenlong Huang. Yevgen Chehotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine. Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model, 2023.
[39] Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Cc Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language, 2022.
[40] Yushi Hu, Hang Hua, Zhengyuan Yang. Weijia Shi, Noah A. Smith, and Jiebo Luo. Promptcap: Prompt- guided task-aware image captioning, 2023.
[41] Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. Prompting large language models with answer heuristies for knowledge-basek visual question answering. Computer Vision and Pattern Recognition (CVPR), 2023.
[42] Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, An- toine Chaffin, Arnaud Stiegler. Arun Raja. Manan Dey, Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Dehajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. In ICLR. 2022.
[43] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolahashi, Yeganeh Kordi, Amirreza Mirzaci, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit. Neeraj Varshney. Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri. Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A. Sumanta Patro, Tanay Dixit, and Xudong Shen. Super- Naturallnstruccions:: Generalization via declarative instructions on 1600+ NLP tasks. In EMNLP, 2022.
[44] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse. Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever. and Dario Amodei. Language models are fow-shot leamners. arXiv preprint arXiv:2005. 14 165,, 2020.
[45] Or Honovich, Thomas Scialom, Omer Levy, and Timo Schiek. Unnatural instructions: Tuning language models with (almost) no human labor. ArXiv, abs/22 2022.
[46] Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language model with self generated instructions. ArXiv, 1is/2212.10500. 2022.
[47] Rohan Taori, Ishaan Guirajani. Tianyi Zhang, Yann Dubois. Xuechen Li, Carlos Guestrin, Percy Liang. and Tatsunori B. Hashimoto. Stanford alpaca: An instruction following llaaa model. https://ggithub com/ Com/tatsu- lab/stanford_alpsc.... 2023.
[48] Chatgpt. https::/opena:: wtt.tl/wccooootce.ee...to chatgpt 2023.
[49] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yi Zhou, Junyan Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qiang Qi, Ji Chao Zhang, and Feiyan Huang. mplug-owl: owl:: Modularization empowers large language models with multimodality. 2023.
12

---

[50] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
[51] Zhiyang Xu, Ying Shen, and Lifu Huang. Multinsttruc: Improving multi-modal zero-shot learning via instruction tuning. ArXiv, abs/2212.10773, 2022.
[52] Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2015.
13

---

A Broader Impact
InstructBLLPP utilizes off-the-shelf frozen L.Mss. Therefore inherits some of the shortcomings of the original LLMs, such as hallucinating ungrounded text or generating outputs with biases. We mitigate such shortcomings by improving the model's grounding performing vision-language instruction tuning on diverse set of high-quality datasets. Nevertheless, do not recommend applying InstructBLIP models to any downstream applications without prior assessment on safety and fairness specific to that application. Additionally, since we did not finetune the parameters of the LLM, InstructBLIP may not always precisely follow given instructions, We believe that employing more advanced LLM and meticulously finetuning its parameters during instruction tuning could address this concern. We reserve this for future work.
the vision and instruction input, and
B
More Case Studies
[Input Image]
[Input Instruction]
What unusual about this image?
[Model Response]
InstructRL.IP
The unusual aspect of this image is that a man is ironing clothes on the back of yellow taxi cab while it is driving down bus city street. He is using it to press his
standing on top of car's trunk, holding an clothing items. This situation is not only unconventional but also potentially hazardous, the could be risk of falling off the vehicle or causing an accident due to his actions.
GPT-4
The unusual thing about this image that man ironing clothes on an ironing board attached to the roof moving taxi.
LLaVA
The unusual aspect of this image minivan or van, This is not typical place to perform this activity, as one would usually iron clothes in more stationary and safe location, such as home, using a regular ironing board. The scene depicted the image is peculiar it involves makeshift ironing setup vehicle, which can be both unsafe and unconventional. Additionally, it not clear how the man is able to maintain balance and stability while ironing clothes in such unstable environment.
ironing clothes on the back of
MiniGPT-4
The image shows standing the back of yellow taxi, with yellow shirt and black pants, and blue backpack on his back. The taxi is driving city street with othr cars and taxis in the background.
Figure 5: The response from InstructBIIIP is more comprehensive than GPT-4, more visually- grounded than LLaVA, and more logical than MiniGPT-4, The responses of GPT4 and LLaVA are obtained from their respective papers, while the official demo is used for MiniGPT-4. Source: https: / //www barnorama. com
Pictures.
14

---

[Input Image]
[Input Instruction]
Do you know who drew this painting?
[Model Response]
InstructBLIP
Leonardo da Vinci.
LLaVA
The painting depicts a woman, commonly believed to be Mona Lisa, the famous artwork by Leonardo da Vinci. It is portrait painting that showcases the woman's enigmatic smile and has become one of the most famous and iconic art pieces in the world. The original work is displayed in the Louvre Museum in Paris, and it known for its intricate details, use of oil paint, and the artist's innovative techniques that contributed to enduring appeal and mystery.
MiniGPT-4
The painting the Mona Lisa, also known the Mona Lisa, by Leonardo da Vinci. It of the most famous paintings in the world and is known for its enigmatic smile and the mysterious identity of the sitter. The painting is housed in the Louvre Museum in Paris, France.
Figure 6: In this example, we illustrate that long-form responses are not always preferable. Instruct-
BLIP can directly address the user's intent by adaptively adjusting the response length, while other models tend to generate lengthy paragraphs with less-relevant sentences. The response from LLaVA taken from the paper, and for MiniGPT-4, we utilize its official demo.
15

---

[Input Image]
[Input Instruction]
Introduce me this painting detail.
[Model Response]
InstructBLIP
The painting depicts young girl with pearl earring, also known as "The Girl with the Pearl Earring" or "Girl with a Pearl Earring". She is wearing blue dress and a pearl earring on her ear. Her hair is pulled back into bun, and she is looking directly at the viewer. This painting is famous work of art by Dutch painter Johannes Vermeer, created in 1665. It considered one of his masterpieces and widely recognized for its intricate details and realistic portrayal of the young girl's facial expression.
LLaVA
The painting features woman with headscarf, possibly of Dutch origin, dressed in blue and white clothing. She is wearing a large blue hat and has red mouth, which adds vibrant touch the portrait. The woman is looking directly at the viewer, creating sense of connection. The background of painting is predominantly blue, further emphasizing color theme of the piece. The overall composition and attire of the woman suggest that the painting might be inspired by the style of Rembrandt.
MiniGPT-4
The painting depicts young woman with pearl earring, wearing blue dress and a white apron. She is shown with serious expression, looking directly at the viewer. The background is dark, neutral tone, with no visible objects elements. The overall mood of the painting somber and contemplative.
The painting well-executed, with a high level of detail and realism the depiction of the woman's 'ns features and clothing. The use of shading and lighting creates sense of depth and dimension, making the woman appear three-dimensional. The artist has used a limited color palette, with the woman's 's dress and apron providing the only sources of color in the painting.
The painting well-composed, with the woman's head and shoulders positioned in the center of the canvas, creating sense of balance and symmetry. The use of negative space around the woman's 's figure adds to the overall somber mood of the painting.
Overall, this is well-executed and tlought-provokee painting that effectively conveys sense of contemplation and introspection.
Figure 7: InstructBLIP shows the ability to connect textual knowledge with visual input, while the other models depict the picture plainly. MiniGPT-4 exhibits poorer results, which may be due to its training with only long captions. Responses of L.LaVA and MiniGPT-4 are generated by their official demos.
16

---

C
Instruction Tuning Datasets
Dataset Name
Held-out
Dataset Description
COCO Caption 10
x
We the large-scale COCO dataset for the image captioning task. Specifically. Karpathy split [52] is used, which divides the data into 82K/5K/SK images for the train/val/test scts,
Web CapFilt
x
14M imago-text pairs collected from the web with additional BLIP-geaerated syothetic captions, used in BLIP (80 and BLIP-2 [..
NoCaps [1]
/ (val)
NoCaps contains 15,100 images with 166,100 human written captions for novel object image captioning.
Flickr30K 12
/ (test)
The Flickr30k dataset consists of 31K images collected from Flickr, each imuge has five ground tuuth captions. use test split beld-out which contains IK images.
Tex:Caps
x
TextCaps an image captioning dataset that requires the model to compechend and reason the text in images. Its train/val/test sets contain 21K/3K/3K images. respectively.
VQAv2 17
VQAv2 is dataset open-ended image question answering. is split 82K440K/1K for train/val/test.
VizWiz [R]
(test-dev)
dataset contains visual questions asked by people who are blind. 8K images used for the beld-out ovaluation.
GQA 4
/ (test-dev)
GQA contains image questions for scene understanding and reasoning. We use the balanced test dev set as held out.
Visual Spatial Reasoning
v
(test)
VSR is collection of image-text pairs, in which the text describes the spotial relation of two objects the image. Models are required classily trueffalse for description. We use the zero zero-shoe data split given in its official githuh repository.
IconQA IB
(test)
IconQA measures the abstract diagram undorstanding and comprehensive cognitive rea- soning abilities of models. We use the test set of its multi-text-cheice task for held-out
evaluntion.
OKVQA 19
x
OKVQA contains visual questions that require outside knowledge to answer. It hus been split into 9K/SK for train and test.
A-OKYQA
x
A-OKVQA a successor of OKYQA with more challenging and diverso questions. It has 17K/1K/6K questions for train/vaiiet.
ScienceQA 21
(test)
SeicnseQA covers diverse science topics with corresponding loctures and explonations. out settings, we only use the part with image context (IMG).
Visual Dialog 26
/ (val)
Visual dialog conversational question answering dataset. We the val split as the held-out, which contains 2,064 images and each has 10 rounds.
OCR-VQA 2
x
OCR-VQA contains visual questions that require models to read text in the image. It has 800K/100K/100K for train/val/tst, respectively.
TeaVQA 28
V (val)
TexVQQA requires models to comprehend visual text to answer questions.
HatefulMemes +
/ (val)
binary classification dataset to justify whether mome contains bateful content.
LLaVA-
28
An instruction tuning dataset which has three parts: detailed caption (23K), reasoning
(77K), conversation (58K).
MSVD-QA 24
/ (lest)
We the set (13K video QA pairs) of MSVD-QA for held out lesting.
MSRVTT-QA
(test)
MSRVTT-QA QA has more complex scenes than MSVD, with 72K video QA pairs test set.
IVQA 25
(test)
IVQA video dataset with mitigated language biases. bas 6K/2K/2K samples for train/vailtst.
Table 4: Description of datasets in our held-in instruction tuning and held-out zero-shot evaluations.
17

---

D Instruction Templates
Task
Instruction Template
<Image>A short image caption:
<lmage>A short image description:
<Image>A photo of
<Image>An image that shows
<Image>Wriic a short description for the image.
<lmage>Write description
the photo.
Image
Provide description of what is presented in the photo.
Captioning
<Imuge>Brielly describe the content of the image.
<lmage>Can you briefly explain what you see in the image?
<lmage>Cuuld you use a few words to describe what you perceive photo?
<Imuge>>leasee provide short depiction of the picture.
Jiagg>>Using language, provide a short account of the image.
< <Immge>>U UseeessUUe see few words to illustrate what is happening in the picture.
<Image> Beange>/Queseee Questioon
<lmage>Question:: [Question]
<Imuge>> Question] short
the question
<Image>Q: (Question) A:
<
[Question] Short unswer:
VQA
<lmage>>Given image, answer the following question with no than three [Question]
Emages-Based the image, respond this question with short answer: [Question]. Answer: <Image>Use Use the provided image to answer the question: [Question] Provide your answer as short as possible:
<lmagesWWhi What is the answer to the following question? "[ "Quuetion]"
<Image> The question "[ "[Question]
be answered using image.
answer is
<lmagezSGiven image, generate question whose
[Answer). Question:
<lmaae>BBased on the image, provide question with the answer: [Anserr). Question:
<lmage>Giivn the visual representation, create question for which the answer is "[Answer]".
VQG
<lmages-From the image provided, craft question that leads to the reply: (Answer). Question:
<lnsiesCunsidering picture, come up question where answer (Answer). <lmage>Taking the image into account, generate an question that has the answer: [Answer). Question:
Table 5: Instruction templates used for transforming held-in datasets into instruction tuming data. For datasets with OCR tokens, we simply add "OCR tokens:" after the image query embeddings.
E
Instructions for Zero-shot Inference
We provide instructions used for zero-shot inference. Note that for instructions with options, separate options with the alphabetical order, e.g. (a) blue (b) yellow (c) pink (d) black.
GQA, VizWiz, IVQA, MSVD, MSRVTT <Image> Question: { Short answer:
NoCaps, Flickr30k
<Image> A short image description:
TextVQA
<Image> OCR tokens: (). Question: 1 Short answer:
IconQA <Image> Question: {} Options: (). Short answer:
ScienceQA
<Image> Context: () Question: () Options: (). Answer:
HatefulMemes
<Image> This is an image with: "(1" written on Is it hateful? Answer:
VSR <Image> Based on the image, this statement true or false? "()" Answer:
Visual Dialog <Image> Dialog history: () Question: () Short answer:
18


Reviews:
[{"review_id": "JH8ECZGTc2", "reviewer": "Reviewer_b9mY", "summary": "This paper proposes InstructBLIP, which is built on BLIP-2 and further perform instruction tuning to enable the instruction following ability of BLIP-2 models. The InstructBLIP model is trained on a set of template-based converted instruction-following data for different tasks (e.g. image captioning and VQA) and LLM-generated instruction-following data (e.g. LLaVA-Instruct-150K). The model delivers better zero-shot performance on selected datasets compared with BLIP-2 and Flamingo.", "questions": "1. The authors provide per-task detailed zero-shot comparisons on held-out datasets. How about the performance on each held-in task? Meanwhile, for the held-in tasks, they can be compared fairly with previous jointly trained multitask generalist methods such as [1-3]. How is the performance compared with these methods?\n\n[1] Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks. In ICLR 2023.  \n[2] Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Tasks. In CVPR 2023.  \n[3] A Unified Sequence Interface for Vision Tasks. In NeurIPS 2022.  ", "limitations": "As discussed in the weakness section, the proposed framework is only applicable to image-level tasks and still cannot handle object-level tasks such as referring object detection / region description. Meanwhile, the proposed framework cannot handle image generation tasks.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is clear and well-organized. The motivations, techinical settings and details are clearly illustrated.  \n2. The proposed InstructBLIP model serves as a sound contribution to the research community of general-purpose large multi-modal models.  \n3. This paper provides insightful analysis on the efficacy and generalization ability of instruction tuning in the ablation studies.", "weaknesses": "1. Most of the instruction-following data are converted from existing image-text datasets in a template-based fasion, except for the LLaVA-Instruct-150K dataset which was composed of GPT generated contents. Simply converting existing image-text datasets with some handcrafted templates may result in lack of diversity of instructions.  \n2. The selected tasks and datasets do not include some mainstream image and image-text datasets such as ImageNet and CIFAR for image classification. \n3. The proposed framework is only applicable to image-level tasks and still cannot handle object-level tasks such as referring object detection / region description. Meanwhile, the proposed framework cannot handle image generation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you a lot for your insightful review, we appreciate it a lot. Our response to your comments is as follows:\n\n----\n\n**Q1**: Diversity of instructions.\n\n**A1**: Converting from existing human-annotated datasets provides high-quality instruction-following data. The data has a reasonable level of diversity due to three reasons:\n\n1. The use of a wide range of datasets across 11 different tasks; \n2. The diversity within each dataset as a result of the human annotation procedure (e.g., in VQA, different annotators have different styles of asking questions); \n3. The use of 10 unique templates for each task. \n\nSince it is difficult to quantify the level of diversity, we directly verify the effectiveness of our instruction-following data by evaluating the InstructBLIP models’ zero-shot performance on unseen datasets.\n\n----\n\n**Q2**: Image classification datasets.\n\n**A2**: We do not include these image classification datasets because our primary focus is on vision-language tasks that involve both language reasoning and visual perception. However, follow-up work [1] has tested InstructBLIP on image classification (ImageNet-1K, CIFAR-10, Pets37, and Flowers102), object counting, and Multi-class Identification. InstructBLIP achieves the highest average score (0.928) among 8 large vision-language models, with a large gap ahead the second best (0.858).\n\n----\n\n**Q3**:  Object-level tasks and image generation tasks.\n\n**A3**: InstructBLIP focuses on image- and video-level vision-language tasks, showing state-of-the-art performance on a wide range of well-established benchmarks. It is a generic framework that could be extended to object-level vision-language tasks. For example, [2] evaluates InstructBLIP on a series of instance-level tasks and InstructBLIP achieves the best performance among 18 vision-language models. Image generation is not the main focus of our paper. However, our learned multimodal representation can potentially benefit image generation tasks, as shown by [3]. \n\n----\n\n**Q4**:  Performance on held-in tasks.\n\n**A4**:  The main goal of instruction tuning is to enhance the model's generalization ability to unseen tasks. As such, our primary focus is on held-out evaluations. For held-in evaluations, the average scores are provided in Section 3.4, and the finetuning results are shown in Section 3.5. When compared to previous multitask methods, InstructBLIP generally achieves better performance on held-in datasets. For instance, InstructBLIP attains 62.1% accuracy on OKVQA, while UNIFIED-IO XL reaches 54.0%. On COCO Caption, InstructBLIP achieves 142.6 CIDEr, whereas Uni-Perceiver v2 (large) reports 122.5. We will include more detailed held-in results in the Appendix of the next version of our paper.\n\n----\n\n**References**\n\n[1] LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models, Xu et al., 2023.\n\n[2] SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension, Li et al., 2023.\n\n[3] Planting a SEED of Vision in Large Language Model, Ge et al., 2023."}, {"review_id": "tDK18nk5C5", "reviewer": "Reviewer_prhX", "summary": "The paper proposes InstructBLIP, a vision and language instruction tuning framework that enables general-purpose models to solve a wide range of visual language tasks through a unified natural language. It uses a diverse set of instruction data to train a multimodal LLM.\nThe model is initialized with a pre-trained BLIP-2 model consisting of an image encoder, an LLM, and a Query Transformer to bridge the two. The LLM and Q-Former is kept frozen while fine-tuning the image encoder. The paper makes the following contributions –\n\n1. A comprehensive and systematic study on vision-language instruction tuning. \n\n2. It proposes instruction aware visual feature extraction, a novel mechanism that enables flexible and informative feature extraction according to the given instructions. \n\n3. The InstructBLIP models are evaluated and open-sourced using two families of LLM’s: \n- FlanT5, and encode-decoder LLM finetuned from T5.\n- Vicunna, a decoder-only LLM finetuned from LLaMA.\n", "questions": "Q1. A human evaluation for the model should be provided.\n\nQ2. A evaluation on the commercial datasets should also be provided to check it's generalizability in real world settings.", "limitations": "The paper has couple of weaknesses -\n\n1. The human evaluation for the model is missing.\n\n2. The model is evaluated on the static datasets which poses a question on its generalizability to the real world scenarios.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Following are the strengths of the paper --\n\n1. The paper is well written and easy to follow.\n\n2. The proposed framework is evaluated on a large variety of tasks and datasets and beats the SOTA.\n\n3. The paper provides both the qualitative and quantitative evaluation for the models.\n", "weaknesses": "The paper has couple of weaknesses -\n\n1. The human evaluation for the model is missing.\n\n2. The model is evaluated on the static datasets which poses a question on its generalizability to the real world scenarios.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and providing your insights. We value your feedback and have responded to your concerns as follows:\n\n----\n\n**Q1**: Human evaluation.\n\n**A1**: Our paper evaluates the InstructBLIP models on a wide range of well-established benchmarks, which sufficiently verifies the advantage of our vision-language instruction-tuning framework in generalizing to unseen tasks. Human evaluation, while potentially beneficial in certain applications, is not an essential consideration in this context.\n\n----\n\n**Q2**: Generalizability.\n\n**A2**: For a more comprehensive understanding of InstructBLIP's performance, we refer to three follow-up papers that provide further evaluations from different perspectives.\n\n[1] SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension, Li et al., 2023.\n\nSEED-Bench evaluates Multimodal LLMs across 12 evaluation dimensions: Scene Understanding, Instance Identity, Instance Attributes, Instance Location, Instance Counting, Spatial Relations, Instance Interaction, Visual Reasoning, Text Recognition, Action Recognition, Action Prediction, and Procedure Understanding, utilizing 19K human-annotated data. Among the 18 evaluated models, InstructBLIP achieves the best average performance, significantly outperforming the other multimodal LLMs. \n\n[2] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models, Fu et al., 2023.\n\nThis paper provides a comprehensive evaluation of 12 advanced Multimodal Large Language Models (MLLMs) across 14 perception and cognition tasks. InstructBLIP achieves top-2 performance in both the overall perception and the cognition test scores, and ranks among the top 3 in 10 out of 14 tasks. Specifically, InstructBLIP attains state-of-the-art performance on tasks including Existence, Count, Color, Scene, and Commonsense Reasoning.\n\n[3] Evaluating Object Hallucination in Large Vision-Language Models, Li et al., 2023.\n\nThis paper evaluates large VL models for the hallucination problem, a common issue observed in modern LLMs. Among the five tested large VL models, InstructBLIP performs significantly better than the others, whether on existing benchmarks or on their proposed POPE pipeline, which incorporates a more real-world environment. For example, InstructBLIP achieves a 77.32 F1 score on the most challenging adversarial examples, while the second-best model achieves 70.42.\n\nWe hope these supplementary evaluations provide a more well-rounded view of InstructBLIP's capabilities and can address the reviewer’s concern.\n"}, {"review_id": "jjiipV0swf", "reviewer": "Reviewer_whUx", "summary": "This paper presents a study of instruction finetuning for vision-language tasks. The paper follows the design of FLAN for instruction tuning and borrows ideas from Flamingo for image/text model freezing and query network. Experimental results suggest FLAN-style instruction tuning also works for vision-language tasks, and this paper provides the first comprehensive study.", "questions": "None", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. To the best of my knowledge, this is the first work of FLAN-style instruction tuning in the VL domain. It provides a comprehensive analysis for future development in this field.\n2. The experiment shows the effectiveness of the proposed instruction-aware query network, which is novel and insteresting.", "weaknesses": "1. Overall, the experimental results are expected, as demonstrated by existing LLM papers. To that end, novelty is somewhat limited, as similar designs have been explored in LLM and Flamingo.\n2. The model is called InstructBLIP yet they are based on T5 and Vicuna (LLaMA). This puzzled me for a second. I feel the key component (frozen LLM + query network) of the model is first shown work by Flamingo and thus find the naming to be a bit misleading. Perhaps a completely new name is more appropriate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing and providing valuable comments. We address your questions in the following.\n\n----\n\n**Q1**: Novelties of InstructBLIP.\n\n**A1**: Our paper proposes a novel vision-language instruction tuning framework that has not been previously explored. We delineate our novelties as follows.\n\n1. **Vision-Language Instruction Tuning vs. Text-only Instruction Tuning.** Although instruction tuning has been applied to text-only LLMs, it has not been systematically investigated in vision-language LLMs. Vision-language (VL) instruction tuning is challenging, as the additional visual modality introduces a high level of variety to the input, making the model harder to generalize. We propose novel approaches to tackle challenges unique in VL instruction tuning, such as the instruction-aware visual feature extraction, which leads to non-trivial improvements and state-of-the-art performance.\n\n2. **InstructBLIP vs. Flamingo.** InstructBLIP is drastically different from Flamingo. From an architectural perspective, InstructBLIP is based on the BLIP-2 backbone. The Q-former in InstructBLIP is different from the perceiver resampler in Flamingo. The Q-former in InstructBLIP has been pre-trained with vision-language representation learning, which enables our proposed instruction-aware visual feature extraction. We refer the reviewer to the BLIP-2 paper [1] for more details on the Q-former pre-training. \nFrom a model training perspective, InstructBLIP is also different from Flamingo. InstructBLIP is trained on a wide variety of vision-language instruction data, whereas Flamingo is pre-trained with image-caption pairs similar to BLIP-2. Our experiments validates the significant advantage of InstructBLIP over both Flamingo and BLIP-2.\n\n----\n\n**Q2**: Naming of InstructBLIP.\n\n**A2**: We name our model InstructBLIP because it is an instruction-tuned model based on the BLIP-2 [1] backbone. InstructBLIP is a generic vision-language instruction-tuning framework that is flexible to incorporate any LLMs. As discussed above, InstructBLIP is drastically different from Flamingo.\n\n----\n\n**References**\n\n[1] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models, Li et al., 2023."}, {"review_id": "GfhsDq0DIz", "reviewer": "Reviewer_r68F", "summary": "The paper presents InstructBLIP, a vision-language instruction tuning framework to solve a wide range of visual-language tasks through a unified multimodal interface. The authors conduct a comprehensive study on vision-language instruction tuning, transforming 26 datasets into the instruction tuning format and grouping them into 11 task categories. They propose instruction-aware Qformer, to enable BLIP models with instruction following capabilities. InstructBLIP models achieve state-of-the-art zero-shot performance on a wide range of held-out vision-language tasks.", "questions": "1. How does the instruction-aware Q-former work on multi-turn conversations? For example, `<image><Q1><A1><Q2>`, what corresponds to the `<instruction>` part in the Fig. 3? Is it `<Q1><A1><Q2>` or `<Q2>` only? In both cases, user can ask complex questions that may be hard for a lightweight LLM like Qformer to interpret. For example, (1) \"can you explain your answer with more detail?\" (2) \"That is not correct, that is not what I am asking. Please try again.\" In these cases, it seems to be tricky for Qformer to extract the corresponding visual features, either because the region to extract visual features is unclear from `<Q2>` only (1), or it seems hard to correctly extract the information without sophisticated reasoning capabilities (2). What is the benefit of Qformer in these cases?\n\n2. For the Q-former architecture in Fig. 3, to my understanding, there will be several consecutive yellow blocks and instruction tokens will be updated via self-attention and feed-forward (green) throughout different blocks, while the updated tokens will not be fed into the LLM. Is this understanding correct? If so, the figure does not reflect (1) multiple consecutive blocks; (2) instruction tokens being updated between blocks.", "limitations": "The author does not discuss limitations in the paper or appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a comprehensive study of vision-language instruction tuning, covering a wide variety of tasks.\n- The proposed InstructBLIP extends BLIP with an instruction-aware Q-Former and finetunes the model on both instruction-following datasets and VQA datasets, showing better performance than previous models.\n- The paper is generally well-written and ablates the model's design of the instruction-aware Q-Former.", "weaknesses": "1. The paper does not provide enough analysis and the comparison on the open-ended out-of-domain multimodal question answering capability. This is one of the most impressive results that GPT-4, MiniGPT-4, and LLaVA shows. The paper claims in L208 \"Although all models are capable of generating long-form responses, InstructBLIP’s outputs generally contains more proper visual details and exhibits logically coherent reasoning steps. Importantly, we argue that long-form responses are not always preferable.\" Without enough evidence, it is hard to justify this point.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable review and comments. We would like to address your questions as follows:\n\n----\n\n**Q1**: Analysis and comparison on the open-ended out-of-domain multimodal question answering.\n\n**A1**: We have provided qualitative comparisons with GPT-4, MiniGPT-4, and LLaVA on open-ended out-of-domain visual questions. The comparisons are included in the appendix that can be viewed in the supplementary material. Our claims are supported by such comparisons. Since qualitative comparisons can be subjective to interpret, our paper provides a systematic quantitative evaluation on a wide range of benchmarks, showing the state-of-the-art performance of InstructBLIP models on many out-of-domain vision-language tasks, including multimodal question answering.\n\nThese follow-up papers [1][2][3][4] also demonstrate the strong performance of InstructBLIP over existing models (MiniGPT-4, LLaVA) on a wide range of challenging vision-language tasks. For example, [4] tests object hallucination with adversarial objects, where InstructBLIP performs the best compared to other VL models.\n\n[1] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models, Fu et al., 2023.\n\n[2] SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension, Li et al., 2023.\n\n[3] LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models, Xu et al., 2023.\n\n[4] Evaluating Object Hallucination in Large Vision-Language Models, Li et al., 2023.\n\n----\n\n**Q2**: Instruction format.\n\n**A2**: In multi-turn conversations, we use all historic context as the instruction. Therefore, for a two-turn conversation, the instruction is formatted as <Q1><A1><Q2>.\n\n----\n\n**Q3**: Q-Former feature extraction.\n\n**A3**: Since all dialog context is present in the instruction, the Q-former can learn to extract visual features according to both the current question and the historic context. The instruction-aware Q-former mechanism is designed to benefit cases where the instruction provides useful guidance to extract specific visual features. In cases where the instruction does not contain useful information, the Q-former can simply ignore the instruction and extract general visual features.\n\n----\n\n**Q4**: About Fig. 3.\n\n**A4**: The reviewer’s understanding is correct. The Q-former contains multiple consecutive transformer blocks, where the output from the previous block will be given as input to the next block, and only the queries' features from the last layer will be fed to the LLM. We will revise the figure to make this clear.\n\n----\n\n**Q5**: About limitations.\n\n**A5**: We have included a section called \"Broader Impact\" in the appendix, which discusses limitations of our work. We will make it more complete and clear."}, {"review_id": "jDZd2cP0UC", "reviewer": "Reviewer_WE2s", "summary": "This paper presents a significant step towards a general-purpose vision-language model with instruction-following abilities. It is built upon the existing BLIP-2 model and instruction-following dataset LLaVA-Instruct-150K [1]. It further extends the scale of instruction-tuning by automatically converting public vision-language datasets such as VQAv2 into instruction-following formats. The authors compare the instruction-tuning paradigm with classic multi-task learning and show that language instruction is the key to generalizing to unseen task instructions.", "questions": "There are some minor suggestions to improve the clarity of the paper:\n- L98: What is the \"standard\" language modeling loss given multimodal instruction-following samples? Do you adopt the same LM loss as LLaVA or did you use the captioning loss as BLIP-2? In other words, is the LM loss enforced on all tokens including both instruction and response tokens?\n- L132-134: How do you decide the dataset balancing ratio of each task? Do you use held-in or held-out scores?\n- L140: What are the evaluation metrics for open-ended generation task? What is the sampling procedure? i.e., nucleus/top-k with temperature? \n\nI would be happy to revise my rating if the authors can address my above-mentioned weaknesses and questions.", "limitations": "Yes, the authors adequately addressed the limitations in the appendix.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This work extends multi-modal instruction-tuning to a wide range of datasets and tasks spanning 11 diverse categories. Robust evaluation is performed by instructing-tuning on 13 held-in datasets and zero-shot evaluating on another 13 held-out datasets. The authors demonstrate the generalization capabilities of their proposed approach by comparing with classic multi-task learning on both held-in and held-out datasets. The paper is well-written and easy to follow. All assets used in this work including the instruction-tuned models and datasets are released to the public. Implementation details such as instruction-aware Q-Former, balanced dataset sampling, and inference strategies like vocabulary reranking are empirically effective and technically sound.", "weaknesses": "InstructBLIP uses LLaVA-Instruct-150K [1] as one of the instruction-tuning datasets. However, there is no direct comparison to the open-sourced LLaVA model which is trained solely on this dataset.\n\nApart from this, I do not find any major weaknesses from this work. However, I would be curious to know if InstructBLIP has stronger vision-language reasoning capabilities than widely-adopted CLIP-like models, which are known to exhibit bag-of-word behaviors [2,3,4,5,6].\n\n[1] Visual Instruction Tuning. Liu et al. 2023.\n\n[2] When and why vision-language models behave like bags-of-words, and what to do about it? Yuksekgonul et al. 2022.\n\n[3] Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality. Thrush et al. 2022.\n\n[4] CREPE: Can Vision-Language Foundation Models Reason Compositionally? Ma et al. 2022.\n\n[5] Equivariant Similarity for Vision-Language Foundation Models. Wang et al. 2023.\n\n[6] Visio-Linguistic Reasoning with Multimodal Generative Pre-Training Scores. Lin et al. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you a lot for your review and insightful comments. We have addressed your questions in the following.\n\n----\n\n**Q1**: Comparison to LLaVA.\n\n**A1**: Our paper provides two comparisons between InstructBLIP and LLaVA: \n\n1. A qualitative comparison using the same images and prompts (Appendix B).\n2. Finetuning comparison on the ScienceQA benchmark (Table 3), where InstructBLIP achieves 90.7% accuracy over LLaVA’s 89.0%. We were unable to provide further quantitative comparisons, as the LLaVA paper did not evaluate their model on other public benchmarks.\n\nFor a more comprehensive understanding of InstructBLIP's performance, we refer to three follow-up papers [1,2,3] that provide systematic evaluations from different perspectives, which demonstrate the superior performance of InstructBLIP over LLaVA in most scenarios.\n\n----\n\n**Q2**: Bag-of-word behaviors.\n\n**A2**: We believe that InstructBLIP has stronger vision-language reasoning capabilities and less bag-of-word behaviors than CLIP-like models. Intuitively, LLMs (which CLIP does not have) may provide strong language representations, which are sensitive to word ordering, and our generative instruction tuning objective may encourage finer-grained representations than the contrastive learning objective of CLIP. \n\nIn the paper, we demonstrate SOTA zero-shot results on GQA, a QA dataset focused on compositionality. The dataset requires locating objects described by attributes (e.g., green door, large container) and spatial relations (e.g., Is there a bag to the right of the green door?). Bag-of-words representations that are insensitive to word ordering will likely fail on GQA questions. For example, “is there a green door to the right of the bag” is a completely different question from “is there a bag to the right of the green door”. \n\nFollow-up evaluation work lends further support to our claim. [1] shows that InstructBLIP achieves the best performance (among 18 models) on tasks such as scene understanding, instance identity, instance attributes, instance location, instance counting, and spatial relations. [2] finds that InstructBLIP has the least hallucination when compared with mPLUG-Owl, LLaVA, MiniGPT-4, and MultiModal-GPT. [3] conducts a comprehensive evaluation of 12 VL models on two critical abilities: perception and cognition, where InstructBLIP demonstrates very strong performance. These achievements require robust reasoning and precise recognition of attributes, objects, and relations. Overall, these results empirically demonstrate the strong vision-language reasoning capabilities of InstructBLIP.\n\n----\n\n**Q3**: Suggestions to improve the clarity of the paper.\n\n**A3**: Thank you for your valuable suggestions, we will revise and improve the clarity of these points accordingly in the next version of our paper.\n\n1. The LM loss is only enforced on the response tokens.\n2. We first use the equation on L130 $p_d = \\frac{\\sqrt{S_d}}{\\sum_{i=1}^D \\sqrt{S_i}}$ ($\\{S_1, S_2, \\dots, S_D\\}$ are dataset sizes and $p_d$ is the probability of a sample being selected from the dataset $d$) to compute the data sampling ratio, where datasets with more samples have higher chances to be sampled. Then, we make manual adjustments according to the difficulty of individual tasks. Specifically, we finetune the model on each individual dataset and check how many epochs it takes to converge using the validation sets. We increase the sampling ratio for datasets that take longer to converge.\n3. We adopt beam search with a beam size of 1 for HatefulMemes, VSR, and OCR-VQA, 3 for NoCaps, and 5 for the other tasks.\n\n----\n\n**References**\n\n[1] SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension, Li et al., 2023.\n\n[2] Evaluating Object Hallucination in Large Vision-Language Models, Li et al., 2023.\n\n[3] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models, Fu et al., 2023."}]

Author rebuttal:
Thank you to all the reviewers for your insightful and constructive feedback. We deeply appreciate the time and effort you have dedicated to reviewing our work. We have responded to your comments and questions inside each individual review. We hope these responses provide a more comprehensive view of our paper. Please kindly consider increasing your rating if your concerns have been addressed.

Meta-review: This paper proposes InstructBLIP, which is built on BLIP-2 and further performs instruction tuning on a combined dataset: a set of template-based converted instruction-following data for different academic tasks (e.g. image captioning and VQA) and LLM-generated instruction-following data (e.g. LLaVA-Instruct-150K). The model delivers good zero-shot performance on selected datasets, especially the academic task related benchmarks.

The paper is thoroughly discussed among reviewers and authors, the pros and cons are well received. While the performance of InstructBLIP is great on selected datasets, the paper is not quite significant in (1) novelty, as the results are expected due to a similar conclusion in NLP; (2) The performance on real world scenarios such as visual chat is becoming weak, as it ``over-fits'' to academic benchmarks due to combined datasets. Overall, InstructBLIP can serve a strong baseline to encourage more future research, though novelty and insights are limited.
Decision: Accept (poster)