SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Zero-shot results on the held-out datasets. Here, Visdial, HM and SciQA denote the Visual Dialog, Hateful Memes and ScienceQA datasets, respectively. For ScienceQA, we only evaluate on the set with image context. Following previous works [6, 25, 34], we report the CIDEr score [35] for NoCaps and Flickr30K, iVQA accuracy for iVQA, AUC score for HatefulMemes, and Mean Reciprocal Rank (MRR) for Visual Dialog. For all other datasets, we report the top-1 accuracy  $(\%)$ .
Model	Held-in Avg.	GQA	ScienceQA (image-context)	IconQA	VizWiz	iVQA
InstructBLIP (FlanT5XL)	94.1	48.4	70.4	50.0	32.7	53.1
w/o Instruction-aware Visual Features	89.8	45.9 (↓2.5)	63.4 (↓7.0)	45.8 (↓4.2)	25.1 (↓7.6)	47.5 (↓5.6)
w/o Data Balancing	92.6	46.8 (↓1.6)	66.0 (↓4.4)	49.9 (↓0.1)	31.8 (↓0.9)	51.1 (↓2.0)
InstructBLIP (Vicuna-7B)	100.8	49.2	60.5	43.1	34.5	52.2
w/o Instruction-aware Visual Features	98.9	48.2 (↓1.0)	55.2 (↓5.3)	41.2 (↓1.9)	32.4 (↓2.1)	36.8 (↓15.4)
w/o Data Balancing	98.8	47.8 (↓1.4)	59.4 (↓1.1)	43.5 (↑0.4)	32.3 (↓2.2)	50.3 (↓1.9)
Table 2: Results of ablation studies that remove the instruction-aware Visual Features (Section 2.3) and the balanced data sampling strategy (Section 2.4). For held-in evaluation, we compute the average score of four datasets, including COCO Caption, OKVQA, A-OKVQA, and TextCaps. For held-out evaluation, we show five datasets from different tasks.
3.2 Ablation Study on Instruction Tuning Techniques
To investigate the impact of the instruction-aware visual feature extraction (Section 2.3) and the balanced dataset sampling strategy (Section 2.4), we conduct ablation studies during the instruction tuning process. As illustrated in Table 2, the removal of instruction awareness in visual features downgrades performance significantly across all datasets. The performance drop is more severe in datasets that involve spatial visual reasoning (e.g., ScienceQA) or temporal visual reasoning (e.g., iVQA), where the instruction input to the Q-Former can guide visual features to attend to informative image regions. The removal of the data balancing strategy causes unstable and uneven training, as different datasets achieve peak performance at drastically different training steps. The lack of synchronized progress over multiple datasets harms the overall performance.
3.3 Qualitative Evaluation
Besides the systematic evaluation on public benchmarks, we further qualitatively examine InstructBLIP with more diverse images and instructions. As illustrated in Figure 1, InstructBLIP demonstrates its capacity for complex visual reasoning. For example, it can reasonably infer from the visual scene what could have happened and deduce the type of disaster from the location of the scene, which it extrapolates based on visual evidence like the palm trees. Moreover, InstructBLIP is capable of connecting visual input with embedded textual knowledge and generate informative responses, such as intruding a famous painting. Furthermore, in descriptions of the overall atmosphere, InstructBLIP exhibits the ability to comprehend metaphorical implications of the visual imagery.
Figure 4: Comparison of instruction tuning and multitask training based on BLIP-2 FlanT5XL backbone. For held-in evaluation, we compute the average score across all held-in datasets. For held-out evaluation, we compute the average score across GQA, TextVQA, VSR, Hateful Memes, IconQA, ScienceQA, iVQA, VizWiz.
Finally, we show that InstructBLIP can engage in multi-turn conversations, effectively considering the dialog history when making new responses.
In Appendix B, we qualitatively compare InstructBLIP with concurrent multimodal models (GPT-4 [36], LLaVA [28], MiniGPT-4 [37]). Although all models are capable of generating long-form responses, InstructBLIP's outputs generally contain more proper visual details and exhibits logically coherent reasoning steps. Importantly, we argue that long-form responses are not always preferable. For example, in Figure 2 of the Appendix, InstructBLIP directly addresses the user's intent by adaptively adjusting the response length, while LLaVA and MiniGPT-4 generate long and less relevant sentences. These advantages of InstructBLIP are a result of the diverse instruction tuning data and an effective architectural design.
3.4 Instruction Tuning vs. Multitask Learning
A direct analogue to instruction tuning is multitask learning, a widely used method that involves the simultaneous training of multiple datasets with the goal of improving the performance of each individual dataset. To investigate whether the improvement in zero-shot generalization mainly comes from the instruction format or merely from multi-task learning, we conduct a comparative analysis.
Following [1], we consider two multitask training approaches. In the first approach, the model is trained using the vanilla input-output format of the training datasets without instructions. During evaluation, instructions are still provided to the model to specify which task to perform. However, an exception is made for image captioning, as the model performs better when receiving the image as the only input. For the second approach, we take a step toward instruction tuning by prepending a [Task:Dataset] identifier to the text input during training. For example, we preprocess [Visual question answering:VQAv2] for the VQAv2 dataset. During evaluation, we explore both instructions and the identifier. Particularly, for held-out datasets, we only use the task name as the identifier since the model never sees the dataset name.
In Figure 4, we show the results for zero-shot, multitask training, and instruction tuning. All models are based on the BLIP-2  $\mathrm{FlanT5_{XL}}$  backbone and adhere to the identical training configurations in Section 2. Overall, we make two observations. First, instruction tuning and multitask learning exhibit similar performance on the held-in datasets. This suggests that the model can fit these two different input patterns comparably well, as long as it has been trained with such data. Second, instruction tuning yields a significant improvement over multitask learning on unseen held-out datasets, whereas multitask learning performs on par with the original BLIP-2. This indicates that instruction tuning is the key to zero-shot generalization.
ScienceQA (image-context)	OCR-VQA	OKVQA	A-OKVQA
Direct Val	Answer Test	Multi-choice Val	Test
Previous SOTA	LLaVA [28]	GIT [39]	PaLM-E (562B) [38]	[40]	[41]	[40]	[41]
89.0	70.3	66.1	56.3	61.6	73.2	73.6
BLIP-2 (FlanT5XXL)	89.5	72.7	54.7	57.6	53.7	80.2	76.2
InstructBLIP (FlanT5XXL)	90.7	73.3	55.5	57.1	54.8	81.0	76.7
BLIP-2 (Vicuna-7B)	77.3	69.1	59.3	60.0	58.7	72.1	69.0
InstructBLIP (Vicuna-7B)	79.5	72.8	62.1	64.0	62.1	75.7	73.4
Table 3: Results of finetuning BLIP-2 and InstructBLIP on downstream datasets. Compared to BLIP-2, InstructBLIP provides a better weight initialization model and achieves SOTA performance on three out of four datasets.
3.5 Finetuning InstructBLIP on Downstream Tasks
We further finetune the InstructBLIP models to investigate its performance on learning a specific dataset. Compared to most previous methods (e.g., Flamingo, BLIP-2) which increase the input image resolution and finetune the visual encoder on downstream tasks, InstructBLIP maintains the same image resolution  $(224\times 224)$  as used in instruction tuning and keeps the visual encoder frozen during finetuning. This significantly reduces the number of trainable parameters from 1.2B to 188M, thus greatly improves finetuning efficiency.
The results are in Table 3. Compared to BLIP-2, InstructBLIP leads to better finetuning performance on all datasets, validating InstructBLIP as the better initialization for task-specific finetuning. InstructBLIP sets new state-of-the-art finetuning performance on ScienceQA (image-context), OCR-VQA, A-OKVQA, but is outperformed on OKVQA by PaLM-E [38] with 562B parameters.
Additionally, we observe that the FlanT5-based InstructBLIP is superior at multi-choice tasks, whereas Vicuna-based InstructBLIP is generally better at open-ended generation tasks. This disparity can be primarily attributed to the capabilities of their frozen LLMs, as they both employ the same image encoder. Although FlanT5 and Vicuna are both instruction-tuned LLMs, their instruction data significantly differ. FlanT5 is mainly finetuned on NLP benchmarks containing many multi-choice QA and classification datasets, while Vicuna is finetuned on open-ended instruction-following data.
Instruction tuning aims to teach language models to follow natural language instructions, which has been shown to improve their generalization performance to unseen tasks. Some methods collect instruction tuning data by converting existing NLP datasets into instruction format using templates [1, 2, 42, 43]. Others use LLMs (e.g., GPT-3 [44]) to generate instruction data [8, 45, 46, 47] with improved diversity.
Instruction-tuned LLMs have been adapted for vision-to-language generation tasks by injecting visual information to the LLMs. BLIP-2 [3] uses frozen FlanT5 models, and trains a Q-Former to extract visual features as input to the LLMs. MiniGPT-4 [37] uses the same pretrained visual encoder and Q-Former from BLIP-2, but uses Vicuna [8] as the LLM and performs training using ChatGPT [48]-generated image captions longer than the BLIP-2 training data. LLaVA [28] directly projects the output of a visual encoder as input to a LLaMA/Vinuca LLM, and finetunes the LLM on vision-language conversational data generated by GPT-4 [36]. mPLUG-owl [49] performs low-rank adaption [50] to a LLaMA [9] model using both text instruction data and vision-language instruction data from LLaVA. A separate work is MultiInstruct [51], which performs vision-language instruction tuning without a pretrained LLM, leading to less competitive performance.
Compared to existing methods, InstructBLIP uses a much wider range of vision-language instruction data, covering both template-based converted data and LLM-generated data. Architecture wise, InstructBLIP proposes an instruction-aware visual feature extraction mechanism. Furthermore, our paper provides a comprehensive analysis on various aspects of vision-language instruction tuning, validating its advantages on generalizing to unseen tasks.
In this paper, we present InstructBLIP, a simple yet novel instruction tuning framework towards generalized vision-language models. We perform a comprehensive study on vision-language instruction tuning and demonstrate the capability of InstructBLIP models to generalize to a wide range of unseen tasks with state-of-the-art performance. Qualitative examples also exhibit InstructBLIP's various capabilities on instruction following, such as complex visual reasoning, knowledge-grounded image description, and multi-turn conversations. Furthermore, we show that InstructBLIP can serve as an enhanced model initialization for downstream task finetuning, achieving state-of-the-art results. We hope that InstructBLIP can spur new research in general-purpose multimodal AI and its applications.
Anthony Meng Huat Tiong is supported by Salesforce and Singapore Economic Development Board under the Industrial Postgraduate Programme. Junqi Zhao and Boyang Li are supported by the Nanyang Associate Professorship and the National Research Foundation Fellowship (NRF-NRFF13-2021-0006), Singapore. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not reflect the views of the funding agencies.
[1] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In ICLR, 2022.
[2] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022.
[3] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
[4] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text generation. arXiv preprint arXiv:2102.02779, 2021.
[5] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In CVPR, 2020.
[6] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangoooi, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikol aj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, NeurIPS, 2022.
[7] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020.
[8] Vicuna. https://github.com/lm-sys/FastChat, 2023.
[9] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[10] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dóllár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[11] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In ICCV, pages 8948-8957, 2019.
[50] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
[51] Zhiyang Xu, Ying Shen, and Lifu Huang. Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning. ArXiv, abs/2212.10773, 2022.
[52] Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
InstructBLIP utilizes off-the-shelf frozen LLMs. Therefore it inherits some of the shortcomings of the original LLMs, such as hallucinating ungrounded text or generating outputs with biases. We mitigate such shortcomings by improving the model's grounding on the vision and instruction input, and performing vision-language instruction tuning on a diverse set of high-quality datasets. Nevertheless, we do not recommend applying InstructBLIP models to any downstream applications without a prior assessment on safety and fairness specific to that application. Additionally, since we did not finetune the parameters of the LLM, InstructBLIP may not always precisely follow given instructions. We believe that employing a more advanced LLM and meticulously finetuning its parameters during instruction tuning could address this concern. We reserve this for future work.
B More Case Studies
[Input Image]
[Input Instruction]	What is unusual about this image?
[Model Response]
InstructBLIP	The unusual aspect of this image is that a man is ironing clothes on the back of a yellow taxi cab while it is driving down a bus city street. He is standing on top of the car's trunk, holding an iron and using it to press his clothing items. This situation is not only unconventional but also potentially hazardous, as the man could be at risk of falling off the vehicle or causing an accident due to his actions.
GPT-4	The unusual thing about this image is that a man is ironing clothes on an ironing board attached to the roof of a moving taxi.
LLaVA	The unusual aspect of this image is a man ironing clothes on the back of a minivan or van. This is not a typical place to perform this activity, as one would usually iron clothes in a more stationary and safe location, such as a home, using a regular ironing board. The scene depicted in the image is peculiar as it involves a makeshift ironing setup on a vehicle, which can be both unsafe and unconventional. Additionally, it is not clear how the man is able to maintain balance and stability while ironing clothes in such an unstable environment.
MiniGPT-4	The image shows a man standing on the back of a yellow taxi, with a yellow shirt and black pants, and a blue backpack on his back. The taxi is driving on a city street with other cars and taxis in the background.
Figure 5: The response from InstructBLIP is more comprehensive than GPT-4, more visually-grounded than LLaVA, and more logical than MiniGPT-4. The responses of GPT-4 and LLaVA are obtained from their respective papers, while the official demo is used for MiniGPT-4. Source: https://www.barnorama.com/wp-content/uploads/2016/12/03-Confusing-Pictures.jpg
[Input Image]
[Input Instruction]	Do you know who drew this painting?
[Model Response]
InstructBLIP	Leonardo da Vinci.
LLaVA	The painting depicts a woman, commonly believed to be Mona Lisa, the famous artwork by Leonardo da Vinci. It is a portrait painting that showcases the woman's enigmatic smile and has become one of the most famous and iconic art pieces in the world. The original work is displayed in the Louvre Museum in Paris, and it is known for its intricate details, use of oil paint, and the artist's innovative techniques that contributed to its enduring appeal and mystery.
MiniGPT-4	The painting is the Mona Lisa, also known as the Mona Lisa, by Leonardo da Vinci. It is one of the most famous paintings in the world and is known for its enigmatic smile and the mysterious identity of the sitter. The painting is housed in the Louvre Museum in Paris, France.
Figure 6: In this example, we illustrate that long-form responses are not always preferable. Instruct-BLIP can directly address the user's intent by adaptively adjusting the response length, while other models tend to generate lengthy paragraphs with less-relevant sentences. The response from LLaVA is taken from the paper, and for MiniGPT-4, we utilize its official demo.
[Input Image]
[Input Instruction]	Introduce me this painting in detail.
[Model Response]
InstructBLIP	The painting depicts a young girl with a pearl earring, also known as "The Girl with the Pearl Earring" or "Girl with a Pearl Earring". She is wearing a blue dress and has a pearl earring on her left ear. Her hair is pulled back into a bun, and she is looking directly at the viewer. This painting is a famous work of art by Dutch painter Johannes Vermeer, created in 1665. It is considered one of his masterpieces and is widely recognized for its intricate details and realistic portrayal of the young girl's facial expression.
LLaVA	The painting features a woman with a headscarf, possibly of Dutch origin, dressed in blue and white clothing. She is wearing a large blue hat and has a red mouth, which adds a vibrant touch to the portrait. The woman is looking directly at the viewer, creating a sense of connection. The background of the painting is predominantly blue, further emphasizing the color theme of the piece. The overall composition and attire of the woman suggest that the painting might be inspired by the style of Rembrandt.
MiniGPT-4	The painting depicts a young woman with a pearl earring, wearing a blue dress and a white apron. She is shown with a serious expression, looking directly at the viewer. The background is a dark, neutral tone, with no visible objects or elements. The overall mood of the painting is somber and contemplative.
The painting is well-executed, with a high level of detail and realism in the depiction of the woman's features and clothing. The use of shading and lighting creates a sense of depth and dimension, making the woman appear three-dimensional. The artist has used a limited color palette, with the woman's dress and apron providing the only sources of color in the painting.
The painting is well-composed, with the woman's head and shoulders positioned in the center of the canvas, creating a sense of balance and symmetry. The use of negative space around the woman's figure adds to the overall somber mood of the painting.
Overall, this is a well-executed and thought-provoking painting that effectively conveys a sense of contemplation and introspection.
Figure 7: InstructBLIP shows the ability to connect textual knowledge with visual input, while the other models depict the picture plainly. MiniGPT-4 exhibits poorer results, which may be due to its training with only long captions. Responses of LLaVA and MiniGPT-4 are generated by their official demos.
C Instruction Tuning Datasets
Table 4: Description of datasets in our held-in instruction tuning and held-out zero-shot evaluations.
D Instruction Templates
Task	Instruction Template
Image Captioning	<A image>A short image caption:<Image>A short image description:<Image>A photo of <Image>An image that shows <Image>Write a short description for the image.<Image>Write a description for the photo.<Image>Provide a description of what is presented in the photo.<Image>Briefly describe the content of the image.<Image>Can you briefly explain what you see in the image? <Image>Could you use a few words to describe what you perceive in the photo? <Image>Please provide a short depiction of the picture.<Image>Using language, provide a short account of the image.<Image>Use a few words to illustrate what is happening in the picture.
VQA	{Question}<Image>Question:{Question}<Image{Question} A short answer to the question is <Image>: {Question} A:<Image>Question:{Question} Short answer:<Image>Given the image, answer the following question with no more than three words. {Question}<Image>Based on the image, respond to this question with a short answer: {Question}. Answer:<Image>Use the provided image to answer the question: {Question} Provide your answer as short as possible:<Image What is the answer to the following question? {"Question"}<Image>The question {"Question"} can be answered using the image. A short answer is
VQG	Given the image, generate a question whose answer is: {Answer}. Question:<Image>Based on the image, provide a question with the answer: {Answer}. Question:<Image>Given the visual representation, create a question for which the answer is {"Answer)}. <Image>From the image provided, craft a question that leads to the reply: {Answer}. Question:<Image>Considering the picture, come up with a question where the answer is: {Answer}. Taking the image into account, generate an question that has the answer: {Answer}. Question:
Table 5: Instruction templates used for transforming held-in datasets into instruction tuning data. For datasets with OCR tokens, we simply add "OCR tokens:" after the image query embeddings.
E Instructions for Zero-shot Inference
We provide instructions used for zero-shot inference. Note that for instructions with options, we separate options with the alphabetical order, e.g. (a) blue (b) yellow (c) pink (d) black.
GQA, VizWiz, iVQA, MSVD, MSRVTT <Image> Question: {} Short answer:
NoCaps, Flickr30k <Image> A short image description:
TextVQA <Image> OCR tokens: {}. Question: {} Short answer:
IconQA <Image> Question: {} Options: {}. Short answer:
ScienceQA <Image> Context: {} Question: {} Options: {}. Answer:
HatefulMemes <Image> This is an image with: "\{\}" written on it. Is it hateful? Answer:
VSR <Image> Based on the image, is this statement true or false? {"}" Answer:
Visual Dialog <Image> Dialog history: {}\n Question: {} Short answer: