SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haotian Liu $^{1*}$ , Chunyuan Li $^{2*}$ , Qingyang Wu $^{3}$ , Yong Jae Lee $^{1}$
<sup>1</sup>University of Wisconsin-Madison <sup>2</sup>Microsoft Research <sup>3</sup>Columbia University https://llava-vl.github.io
Instruction tuning large language models (LLMs) using machine-generated instruction-following data has been shown to improve zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. We present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and an LLM for general-purpose visual and language understanding. To facilitate future research on visual instruction following, we construct two evaluation benchmarks with diverse and challenging application-oriented tasks. Our experiments show that LLaVA demonstrates impressive multimodal chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a  $85.1\%$  relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of  $92.53\%$ . We make GPT-4 generated visual instruction tuning data, our model, and code publicly available.
Humans interact with the world through many channels such as vision and language, as each individual channel has a unique advantage in representing and communicating certain concepts, and thus facilitates a better understanding of the world. One of the core aspirations in artificial intelligence is to develop a general-purpose assistant that can effectively follow multi-modal vision-and-language instructions, aligned with human intent to complete various real-world tasks in the wild [4, 26].
To this end, the community has witnessed an emergent interest in developing language-augmented foundation vision models [26, 16], with strong capabilities in open-world visual understanding such as classification [39, 21, 56, 53, 38], detection [28, 61, 32], segmentation [25, 62, 57] and captioning [49, 27], as well as visual generation and editing [41, 42, 55, 15, 43, 29]. We refer readers to the Computer Vision in the Wild reading list for a more up-to-date literature compilation [12]. In this line of work, each task is solved independently by one single large vision model, with the task instruction implicitly considered in the model design. Further, language is only utilized to describe the image content. While this allows language to play an important role in mapping visual signals to language semantics—a common channel for human communication, it leads to models that usually have a fixed interface with limited interactivity and adaptability to the user's instructions.
Large language models (LLM), on the other hand, have shown that language can play a wider role: a universal interface for a general-purpose assistant, where various task instructions can be explicitly represented in language and guide the end-to-end trained neural assistant to switch to the task of interest to solve it. For example, the recent success of ChatGPT [34] and GPT-4 [35] have demonstrated the power of aligned LLMs in following human instructions, and have stimulated tremendous interest in developing open-source LLMs. Among them, LLaMA [48] is an open-source LLM that matches the performance of GPT-3. Alpaca [47], Vicuna [9], GPT-4-LLM [37]
utilize various machine-generated high-quality instruction-following samples to improve the LLM's alignment ability, reporting impressive performance compared with proprietary LLMs. Importantly, this line of work is text-only.
In this paper, we present visual instruction-tuning, the first attempt to extend instruction-tuning to the language-image multimodal space, to pave the way towards building a general-purpose visual assistant. In particular, our paper makes the following contributions:
- Multimodal instruction-following data. One key challenge is the lack of vision-language instruction-following data. We present a data reformation perspective and pipeline to convert image-text pairs into an appropriate instruction-following format, using ChatGPT/GPT-4.
- Large multimodal models. We develop a large multimodal model (LMM), by connecting the open-set visual encoder of CLIP [39] with the language decoder Vicuna [9], and fine-tuning end-to-end on our generated instructional vision-language data. Our empirical study validates the effectiveness of using generated data for LMM instruction-tuning, and suggests practical tips for building a general-purpose instruction-following visual agent. When ensembled with GPT-4, our approach achieves SoTA on the Science OA [33] multimodal reasoning dataset.
- Multimodal instruction-following benchmark. We present LLaVA-Bench with two challenging benchmarks, with a diverse selection of paired images, instructions and detailed annotations.
- Open-source. We release the following assets to the public: the generated multimodal instruction data, the codebase, the model checkpoints, and a visual chat demo.
Multimodal Instruction-following Agents. In computer vision, existing works that build instruction-following agents can be broadly categorized into two classes: (i) End-to-end trained models, which are separately explored for each specific research topic. For example, the vision-language navigation task [3, 19] and Habitat [46] require the embodied AI agent to follow natural language instructions and take a sequence of actions to complete goals in visual environments. In the image editing domain, given an input image and a written instruction that tells the agent what to do, InstructPix2Pix [6] edits images by following the human instructions. (ii) A system that coordinates various models via LangChain [1] / LLMs [34], such as Visual ChatGPT [52], X-GPT [62], MM-REACT [54], VisProg [18], and ViperGPT [45]. While sharing the same goal in building instruction-following agents, we focus on developing an end-to-end trained language-vision multimodal model for multiple tasks.
Instruction Tuning. In the natural language processing (NLP) community, to enable LLMs such as GPT-3 [7], T5 [40], PaLM [10], and OPT [59] to follow natural language instructions and complete real-world tasks, researchers have explored methods for LLM instruction-tuning [36, 51, 50], leading to instruction-tuned counterparts such as InstructGPT [36]/ChatGPT [34], FLAN-T5 [11], FLAN-PaLM [11], and OPT-IML [22], respectively. It turns out that this simple approach can effectively improve the zero- and few-shot generalization abilities of LLMs. It is thus natural to borrow the idea from NLP to computer vision. More broadly, the teacher-student distillation ideas with foundation models have been studied in other topics such as image classification [14]. Flamingo [2] can be viewed as the GPT-3 moment in the multimodal domain, due to its strong performance on zero-shot task transfer and in-context-learning. Other LMMs trained on image-text pairs include BLIP-2 [27], FROMAGe [24], and KOSMOS-1 [20]. PaLM-E [13] is an LMM for embodied AI. Based on the recent "best" open-source LLM LLaMA, OpenFlamingo [5] and LLaMA-Adapter [58] are open-source efforts that enable LLaMA to use image inputs, paving the way to build open-source multimodal LLMs. While these models present promising task transfer generalization performance, they are not explicitly tuned with vision-language instruction data, and their performance in multimodal tasks usually falls short compared to language-only tasks. In this paper, we aim to fill this gap and study its effectiveness. Finally, note that visual instruction tuning is different from visual prompt tuning [23]: the former aims to improve the model's instruction-following abilities, while the latter aims to improve the parameter-efficiency in model adaptation.
Context type 1: Captions
A group of people standing outside of a black vehicle with various luggage. Luggage surrounds a vehicle in an underground parking area
Context type 2: Boxes
person: [0.681, 0.242, 0.774, 0.694], backpack: [0.384, 0.696, 0.485, 0.914], suitcase: ...<omitted>
Response type 1: conversation
Question: What type of vehicle is featured in the image?
Answer: The image features a black sport utility vehicle (SUV) ...<omitted>
Response type 2: detailed description
The image is an underground parking area with a black sport utility vehicle (SUV) parked. There are three people in the scene, with one person standing closer to the left side of the vehicle, another person in the middle, and the third person on the right side. They are all working together to pack their luggage into the SUV for a trip. ...<omitted>
Response type 3: complex reasoning
Question: What challenges do these people face?
Answer: In the image, a group of people is standing outside a black SUV in a parking area, surrounded by various pieces of luggage, including suitcases and backpacks. They are facing the challenge of fitting all their luggage into the black SUV. There are multiple suitcases and backpacks to be packed, which suggests that the group has a significant amount of belongings ...<omitted>
Table 1: One example to illustrate the instruction-following data. The top block shows the contexts such as captions and boxes used to prompt GPT, and the bottom block shows the three types of responses. Note that the visual image is not used to prompt GPT, we only show it here as a reference.
3 GPT-assisted Visual Instruction Data Generation
The community has witnessed a surge in the amount of public multimodal data such as image-text pairs, ranging from CC [8] to LAION [44]. However, when it comes to multimodal instruction-following data, the available amount is limited, partially because the process for creating such data is time-consuming and less well-defined when human crowd-scouring is considered. Inspired by the success of recent GPT models in text-annotation tasks [17], we propose to leverage ChatGPT/GPT-4 for multimodal instruction-following data collection, based on the widely existing image-pair data.
For an image  $\mathbf{X}_{\mathrm{v}}$  and its associated caption  $\mathbf{X}_{\mathrm{c}}$ , it is natural to create a set of questions  $\mathbf{X}_{\mathrm{q}}$  with the intent to instruct the assistant to describe the image content. We prompt GPT-4 to curate such a list of questions (see details in Appendix). Therefore, a simple way to expand an image-text pair to its instruction-following version is Human:  $\mathbf{X}_{\mathrm{q}}\mathbf{X}_{\mathrm{v}}<\text{STOP}>$  Assistant:  $\mathbf{X}_{\mathrm{c}}<\text{STOP}>$ . Though cheap to construct, this simple expanded version lacks diversity and in-depth reasoning in both the instructions and responses.
To mitigate this issue, we leverage language-only GPT-4 or ChatGPT as the strong teacher (both accept only text as input), to create instruction-following data involving visual content. Specifically, in order to encode an image into its visual features to prompt a text-only GPT, we use two types of symbolic representations: (i) Captions typically describe the visual scene from various perspectives; (ii) Bounding boxes usually localize the objects in the scene, and each box encodes the object concept and its spatial location. One example is shown in the top block of Table 14.
This symbolic representation allows us to encode the image as an LLM-recognizable sequence. We use COCO images [30] and generate three types of instruction-following data. One example per type is shown in the bottom block of Table 14. For each type, we first manually design a few examples. They are the only human annotations we have during data collection, and are used as seed examples in in-context-learning to query GPT-4.
- Conversation. We design a conversation between the assistant and a person asking questions about this photo. The answers are in a tone as if the assistant is seeing the image and answering the question. A diverse set of questions are asked about the visual content of the image, including the object types, counting the objects, object actions, object locations, relative positions between
objects. Only questions that have definite answers are considered. Please see Appendix for the detailed prompt.
- Detailed description. To include a rich and comprehensive description for an image, we create a list of questions with such an intent. We prompt GPT-4 then curate the list (see detailed prompts and curation process in Appendix). For each image, we randomly sample one question from the list to ask GPT-4 to generate the detailed description.
- Complex reasoning. The above two types focus on the visual content itself, based on which we further create in-depth reasoning questions. The answers typically require a step-by-step reasoning process by following rigorous logic.
We collect 158K unique language-image instruction-following samples in total, including 58K in conversations, 23K in detailed description, and 77k in complex reasoning, respectively. We ablated the use of ChatGPT and GPT-4 in our early experiments, and found that GPT-4 consistently provides higher quality instruction-following data, such as spatial reasoning.
4 Visual Instruction Tuning
The primary goal is to effectively leverage the capabilities of both the pre-trained LLM and visual model. The network architecture is illustrated in Figure 1. We choose Vicuna [9] as our LLM  $f_{\phi}(\cdot)$  parameterized by  $\phi$ , as it has the best instruction following capabilities in language tasks among publicly available checkpoints [47, 9, 37].
Figure 1: LLaVA network architecture.
For an input image  $\mathbf{X}_{\mathrm{v}}$ , we consider the pre-trained CLIP visual encoder ViT-L/14 [39], which provides the visual feature  $\mathbf{Z}_{\mathrm{v}} = g(\mathbf{X}_{\mathrm{v}})$ . The grid features before and after the last Transformer layer are considered in our experiments. We consider a simple linear layer to connect image features into the word embedding space. Specifically, we apply a trainable projection matrix  $\mathbf{W}$  to convert  $\mathbf{Z}_{\mathrm{v}}$  into language embedding tokens  $\mathbf{H}_{\mathrm{v}}$ , which have the same dimensionality as the word embedding space in the language model:
$$
\mathbf {H} _ {\mathrm {v}} = \mathbf {W} \cdot \mathbf {Z} _ {\mathrm {v}}, \text {w i t h} \mathbf {Z} _ {\mathrm {v}} = g (\mathbf {X} _ {\mathrm {v}}) \tag {1}
$$
Thus, we have a sequence of visual tokens  $\mathbf{H}_{\mathrm{v}}$ . Note that our simple projection scheme is lightweight, which allows us to iterate data centric experiments quickly. More sophisticated schemes to connect the image and language representations can also be considered, such as gated cross-attention in Flamingo [2] and Q-former in BLIP-2 [27]. We leave exploring possibly more effective and sophisticated architecture designs for LLaVA as future work.
For each image  $\mathbf{X}_{\mathrm{v}}$ , we generate multi-turn conversation data  $(\mathbf{X}_{\mathrm{q}}^{1}, \mathbf{X}_{\mathrm{a}}^{1}, \dots, \mathbf{X}_{\mathrm{q}}^{T}, \mathbf{X}_{\mathrm{a}}^{T})$ , where  $T$  is the total number of turns. We organize them as a sequence, by treating all answers as the assistant's response, and the instruction  $\mathbf{X}_{\mathrm{instruct}}^{t}$  at the  $t$ -th turn as:
$$
\mathbf {X} _ {\text {i n s t r u c t}} ^ {t} = \left\{ \begin{array}{l l} \text {R a n d o m l y c h o o s e} & \left[ \mathbf {X} _ {\mathrm {q}} ^ {1}, \mathbf {X} _ {\mathrm {v}} \right] \text {o r} \left[ \mathbf {X} _ {\mathrm {v}}, \mathbf {X} _ {\mathrm {q}} ^ {1} \right], \quad \text {t h e f i r s t t u r n} t = 1 \\ & \mathbf {X} _ {\mathrm {q}} ^ {t}, \quad \text {t h e r e m a i n i n g t u r n s} t > 1 \end{array} \right. \tag {2}
$$
This leads to the unified format for the multimodal instruction-following sequence illustrated in Table 2. We perform instruction-tuning of the LLM on the prediction tokens, using its original auto-regressive training objective.
Xsystem-message<STOP> Human:  $\mathbf{X}_{\mathrm{instruct}}^{1}$  <STOP> Assistant:  $\mathbf{X}_{\mathrm{a}}^{1}$  <STOP> Human:  $\mathbf{X}_{\mathrm{instruct}}^{2}$  <STOP> Assistant:  $\mathbf{X}_{\mathrm{a}}^{2}$  <STOP>
Table 2: The input sequence used to train the model. Only two conversation turns are illustrated here; in practice, the number of turns varies based on the instruction-following data. In our current implementation, we follow Vicuna-v0 [9] to set the system message  $\mathbf{X}_{\mathrm{system - message}}$  and we set  $<\text{STOP}>$  = _____. The model is trained to predict the assistant answers and where to stop, and thus only green sequence/tokens are used to compute the loss in the auto-regressive model.
Specifically, for a sequence of length  $L$ , we compute the probability of the target answers  $\mathbf{X}_{\mathrm{a}}$  by:
$$
p \left(\mathbf {X} _ {\mathrm {a}} \mid \mathbf {X} _ {\mathrm {v}}, \mathbf {X} _ {\text {i n s t r u c t}}\right) = \prod_ {i = 1} ^ {L} p _ {\boldsymbol {\theta}} \left(x _ {i} \mid \mathbf {X} _ {\mathrm {v}}, \mathbf {X} _ {\text {i n s t r u c t}, <   i}, \mathbf {X} _ {\mathrm {a}, <   i}\right), \tag {3}
$$
where  $\theta$  is the trainable parameters,  $\mathbf{X}_{\mathrm{instruct}, < i}$  and  $\mathbf{X}_{\mathrm{a}, < i}$  are the instruction and answer tokens in all turns before the current prediction token  $x_{i}$ , respectively. Please see Table 2 for an illustration of the prediction tokens. For the conditionals in (3), we explicitly add  $\mathbf{X}_{\mathrm{v}}$  to emphasize the fact that the image is grounded for all answers, and we omit  $\mathbf{X}_{\mathrm{system - message}}$  and all previous  $\langle \mathrm{STOP} \rangle$  for better readability. For LLaVA model training, we consider a two-stage instruction-tuning procedure.
Stage 1: Pre-training for Feature Alignment. To strike a balance between concept coverage and training efficiency, we filter CC3M to 595K image-text pairs. Please see Appendix for details of the filtering process. These pairs are converted to the instruction-following data using the naive expansion method describe in Section 3. Each sample can be treated as a single-turn conversation. To construct the input  $\mathbf{X}_{\mathrm{instruct}}$  in (2), for an image  $\mathbf{X}_{\mathrm{v}}$ , a question  $\mathbf{X}_{\mathrm{q}}$  is randomly sampled, which is a language instruction to request the assistant to describe the image briefly. The ground-truth prediction answer  $\mathbf{X}_{\mathrm{a}}$  is the original caption. In training, we keep both the visual encoder and LLM weights frozen, and maximize the likelihood of (3) with trainable parameters  $\theta = \mathbf{W}$  (the projection matrix) only. In this way, the image features  $\mathbf{H}_{\mathrm{v}}$  can be aligned with the pre-trained LLM word embedding. This stage can be understood as training a compatible visual tokenizer for the frozen LLM.
Stage 2: Fine-tuning End-to-End. We always keep the visual encoder weights frozen, and continue to update both the pre-trained weights of the projection layer and LLM in LLaVA; i.e., the trainable parameters are  $\theta = \{\mathbf{W},\phi \}$  in (3). We consider two specific use case scenarios:
- Multimodal Chatbot. We develop a Chatbot by fine-tuning on the 158K language-image instruction-following data in Section 3. Among the three types of responses, conversation is multi-turn while the other two are single-turn. They are uniformly sampled in training.
- Science QA. We study our method on the ScienceQA benchmark [33], the first large-scale multimodal science question dataset that annotates the answers with detailed lectures and explanations. Each question is provided a context in the form of natural language or an image. The assistant provides the reasoning process in natural language and selects the answer among multiple choices. For training in (2), we organize the data as a single turn conversation, the question & context as  $\mathbf{X}_{\text {instruct }}$ , and reasoning & answer as  $\mathbf{X}_{\mathrm{a}}$ .
We assess the performance of LLaVA in instruction-following and visual reasoning capabilities with two primary experimental settings: multimodal chatbot and the ScienceQA dataset, respectively. We train all models with  $8 \times \mathrm{A}100\mathrm{s}$ , following Vicuna's hyperparameters [9]. We pre-train our model on the filtered CC-595K subset for 1 epoch with a learning rate of 2e-3 and a batch size of 128, and fine-tune on the proposed LLaVA-Instruct-158K dataset for 3 epochs, with a learning rate of 2e-5 and a batch size of 32. See Appendix for more training details.
5.1 Multimodal Chatbot
We developed a chatbot demo to show the image understanding and conversation abilities of LLaVA, and to study how well LLaVA is able to digest visual inputs and exhibit instruction-following
Visual input example, Extreme Ironing:
Source: https://www.barnorama.com/up-content/uploads/2016/12/03-Confusing-Pictures.jpg
User	What is unusual about this image?
LLaVA	The unusual aspect of this image is a man ironing clothes on the back of a minivan or van. This is not a typical place to perform this activity, as one would usually iron clothes in a more stationary and safe location, such as a home, using a regular ironing board. The scene depicted in the image is peculiar as it involves a makeshift ironing setup on a vehicle, which can be both unsafe and unconventional. Additionally, it is not clear how the man is able to maintain balance and stability while ironing clothes in such an unstable environment.
[Start a new conversation, and clear the history]
User	What's happening in the scene?
LLaVA	The image depicts a man standing on top of a yellow SUV in a busy city street. He is holding a portable ladder, seemingly ironing clothes while standing on the vehicle. Around the scene, there are other cars, a traffic light, and a fire hydrant. Additionally, there are two other people in the vicinity, one close to the man on the car and another further away. The man on the ladder appears to be attracting attention due to his unconventional choice of ironing his clothes on top of a moving car. The city street around him is bustling with activity, adding to the unique nature of the scene.
User	What is unusual about this image?
GPT-4 [35]	The unusual thing about this image is that a man is ironing clothes on an ironing board attached to the roof of a moving taxi.
User	What is unusual about this image?
BLIP-2	a man is sitting on the back of a yellow cab
User	What is unusual about this image?
OpenFlamingo	The man is drying his clothes on the hood of his car.
Table 3: Example prompt from GPT-4 paper [35] to compare visual reasoning and chat capabilities. Compared to BLIP-2 [27] and OpenFlamingo [5], LLaVA accurately follows the user's instructions, instead of simply describing the scene. LLaVA offers a more comprehensive response than GPT-4. Even when merely asked to describe the image, LLaVA identifies atypical aspects of the image.
capabilities. We first use the examples in the original GPT-4 paper [35], shown in Table 3 (more examples in Appendix), that require in-depth image understanding. For comparisons, we quote the prompt and response of the multimodal GPT-4 from their paper, and query BLIP-2 and OpenFlamingo model checkpoints to get their response.
Surprisingly, although LLaVA is trained with a small multimodal instruction-following dataset ( $\sim 80\mathrm{K}$  unique images), it demonstrates quite similar reasoning results with multimodal GPT-4 on these examples. Note that while these images are out-of-domain for LLaVA, LLaVA is still able to understand the scenes and follow the question instruction to provide a reasonable response. In contrast, BLIP-2 and OpenFlamingo focus on describing the image, instead of following the user instruction to answer in an appropriate manner.