Visual Instruction Tuning
Authors: Haotian Liu
Chunyuan Li
Qingyang Wu
Yong Jae Lee
Venue: NeurIPS 2023 oral (2023)
Keywords: visual instruction tuning
instruction tuning
multimodal
LLM
GPT

Abstract: Instruction tuning large language models (LLMs) using machine-generated instruction-following data has been shown to improve zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. We present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and an LLM for general-purpose visual and language understanding. To facilitate future research on visual instruction following, we construct two evaluation benchmarks with diverse and challenging application-oriented tasks. Our experiments show that LLaVA demonstrates impressive multimodal chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%. We make GPT-4 generated visual instruction tuning data, our model, and code publicly available.

Visual Instruction Tuning
Haotian Liu'r, Chunyuan Li Qingyang Wu², Yong Jae Lee'
University of Wisconsin Madiison Microsoff Research $Columbia University https://laaa
Abstract
Instruction tuning large language models (LLMs) using machine- machine-generated instruction-following data has been shown improve zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. We present the first attempt use language-only GPT-4 generate multimodal language-image catructtionfollowiini data. By instruction tuning on such generated data, we in- troduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and an LLM for general- purpose visual and language understanding. To facilitate future research on visual instruction following, we construct two evaluation benchmarks with diverse and challenging application-oriented tasks. Our experiments show that LLaVe VA demon- strates impressive multimodal chat abilities, sometimes exhibiting the behaviors multimodal GPT-4 on unseen images/instruutions, and yields a 85.1% rela- tive score compared with GPT-4 synthetic multimodal instruction-followin dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves new state-of-the of-the-art accuracy of 92.53%. We make GPT-4 generated visual instruction tuning data, our model, and code publicly available.
Introduction
Humans interact with the world through many channels such vision and language, each individual channel has a unique advantage in representing and communicating certain concepts, and thus facilitates a better understanding of the world. One of the core aspirations artificial intelligence is to develop a general general-purpose purpose assistant that can effectively follow multi modall rision-and and-language instructions. aligned with human intent to complete various real-workh tasks in the wild [4, 26].
To this end, the community has witnessed an emergent interest in developing
foundation vision models [26, 16], with strong capabilities in world visual understanding
such as classification [39, 21, 56, 53, 38], detection [28, 61, 321, segmentation [25, 62, 57] and captioning [49, 27], well as visual generation and editing [41, 42, 55, 15, 43, 29]. We refer readers to the Computer Vision in the Wild reading list for more up-to-date literature compilation [12]. In this line of work, each task is solved independently by one single large vision model, with the task instruction implicitly considered in the model design. Further, language is only utilized to describe the image content. While this allows language play an important role in mapping visual signals to language semantics- common channel for human communication, it leads to models that usually have a fixed interface with limited interactivity and adaptability to the user's instructions.
Large language models (LLM),, on the other hand, have shown that language can play a wider role: universal interface for general-pun purposse assistant, where various task instructions can be explicitly represented in language and guide the end-to-end trained neural assistant to switch to the task of interest to solve it. For example, the recent success of ChatGPT [34] and GPT-4 [35] have demonstrated the power of aligned LLMs in following human instructions, and have stimulated tremendous interest in developing open-son open-source LLMs. Among them, LLaMA [48] is an open- source LLM that matches the performance of GPT-3. Alpaca [47], Vicuna [9], GPT-4-LLM [37]
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

utilize various machine machine-generated high-quality instruction foolloiing samples to improve the LLM's alignment ability, reporting impressive performance compared with proprietary LLMs. Importantly this line of work is text-only.
In this paper, we present visual instruction-tuning the first attempt to extend instruction-tuning to the language-image multimodal space, to pave the way towards building a general-purposs visual assistant. In particular, our paper makes the following contributions:
Multimodal cestructiio-ollooiinn data. One key challenge is the lack of vision-language pnstruction-folloviin data. We present a data reformation perspective and pipeline to convert image-text pairs into an appropriate instruction woolowwwg format, using ChatPPT/GPT-4
Large multimodal models. We develop large multimodal model (LMM), by connecting the open-set visual encoder of CLIP [39] with the language decoder Vieuna [9], and line-tuning end-to-end on our generated instructional vision-language data. Our empirical study validates the effectiveness of using generated data for LMM instruction-tuning and suggests practical tips for building a general general-purpose pprpose instruction
visual agent. When ensembled with GPT-4, our approach achieves SoTA on the Science QA [33] multimodal reasoning dataset.
Multimodal instruction followiig benchmark. We present LLa VA-Bench with two challenging benchmarks, with diverse selection of paired images, instructions and detailed annotations. Open-source. We release the following assets to the public: the generated multimodal instruction data, the codebase, the model checkpoints, and a visual chat demo
2 Related Work
Multimodal Instruction-folloving Agents. In computer vision, existing works that build cnstruction-followiie agents can be broadly categorized into two classes: (i) End-to-end trained models, which are separately explored for each specific research topic. For example, the vision- language navigation task [3, 19] and Habitat [46] require the embodicd Al agent to follow natural language instructions and take sequence of actions to complete goals in visual environments. In the image editing domain, given an input image and a written instruction that tells the agent what to do, InstructPix2PPix [6] edits images by following the human instructions. (ii) A system that coordinates various models via LangChain [1] / LLMs [34], such as Visual CharGPT [52], X-GPT [62], MM-REACT [54], VisProg [18), and ViperGPT [45]. While sharing the same goal in building corruaccioo-folloiini agents, we focus on developing an end-to-end trained language-vision multimodal model for multiple tasks.
Instruction Tuning. In the natural language processing (NLP) community, to enable LLMs such as GPT-3 [7]. T5 [40], PaLM [10], and OPT [59] to follow natural language instructions and complete real-world tasks, researchers have explored methods for LLM instruction-tuning [36, 50J, leading to instruction-tune tuned counterparts such as InstructGPT [36]/ChatGPT [34], FLAN-T5 [11], FLAN-PaLM [11], and OPT-IML [22], respectively. It turns out that this simple approach can effectively improve the zero- and few-shot generalization abilities of LLMs. It is thus natural to borrow the idea from NLP to computer vision. More broadly, the teacher-student distillation ideas with foundation models have been studied in other topics such image classification [14]. Flamingo [2] can be viewed as the GPT-3 moment in the multimodal domain, due to its strong performance on zero-shot task transfer and in-context-leammigg Other LMMs trained on image- text pairs include BLIP-2 [27], FROMAGe [24]. and KOSMOS-I [20]. PaLM-E [13] is an LMM for embodied AI. Based on the recent "best" open-source LLM LLaMA, OpenFlamingo [5] and LLaMA-Adapte [58] are open-source efforts that enable LLaMA to use image inputs, paving the way to build spunesource multimodal LLMs. While these models present promising task transfer generalization performance, they are not explicitly tuned with vision-language instruction data, and their performance in multimodal tasks usually falls short compared to language-only tasks. this paper, we aim to fill this gap and study its effectiveness. Finally, note that visual instruction tuning different from visual prompt tuning [23]: the former aims to improve the model's 's instruction- following abilitics, while the latter aims to improve the parameter-efficient in model adaptation.

---

Context type 1: Captions
group of people standing outside black vehicle with various luggage. Luggage surrounds a vehicle in an underground parking area
People try to fit all of their luggage in an SUV.
The sport utility vehicle is parked in the public garage, being packed for trip
Some people with luggage near van that transporting it.
Context type 2: Boxes
person: [0.681, 0.242, 0.774, 0.694]. backpack: [0.384, 0.696, 0,485, 0.914], suitcase: ...omitted>
Response type I: conversation
Question: What type vehicle featured the image?
Answer: The image features a black sport utility vehicle ...oiitddt Response type 2: detailed description
underground parking area with black sport utility vehicle (SUV) parked. There three people in the scene, with one person standing closer to the left side of the vehicle, another person in the middle, and the third person on right side. They are all working together to pack their luggage into the SUV for a trip. <..omitted>>
The image
Response type 3: complex reasoning
Question: What challenges do these people face?
Answer: In the image, group people standing outside a black SUV parking area, surrounded by various pieces of luggage, including suitcases and backpacks. They are facing the challenge of fitting all their luggage into the black SUV. There are multiple suitcases and backpacks be packed, which suggests that the group has significant amount of belongings.. <..coiited>>>
Table 1:
One example to illustrate the instructionnfollooingg data. The top block shows the contexts such as captions and boxes used to prompt GPT, and the bottom block shows the three types of responses. Note that the visual image is not used to prompt GPT, we only show it here reference.
3 GPT-assisted Visual Instruction Data Generation
The community has witnessed surge in the amount of public multimodal data such as image-text pairs, ranging from CC [8] to LAION [44]. However, when it comes to multimodal instruction- following data, the available amount limited, partially because the process for creating such data is time-consuming and less well-defined when human crowd-scouring is considered. Inspired by the success of recent GPT models in text-annotation tasks [17], we propose leverage ChatGPT/GPT-44 for multimodal instruction- custucttion-oolloiing following data collection, based on the widely existing image-pair data.
For an image X, and its associated caption X., it is natural create set of questions X, with the intent to instruct the assistant to describe the image content. We prompt GPT-4 to curate such a list of questions (see details in Appendix). Therefore, simple way expand an image-fext pair its instruction -owltiioonoooloiingg version is Human X. X, <STOP> Assistant X, <STOP>. Though cheap to construct, this simple expanded version lacks diversity and in-depth reasoning in both the instructions and responses.
To mitigate this issue, leverage language-only GPT-4 or ChatGPT the strong teacher (both accept only text as input), to create coltuuttiooooooowing data involving visual content. Specifically, two types of
in order to encode image into its visual features to prompt text-only GPT, symbolic representations: (i) Captions typically describe the visual scene from various perspectives; (ii) Bounding boxes usually localize the objects in the scene, and each box encodes the object concept and its spatial location. One example is shown in the top block of Table 14.
This symbolic representation allows to encode the image an LLM ALM-rroognizable sequence. We use COCO images [30] and generate three types of costruction-follwwwee is shown in the bottom block of Table 14. For each type, we first manually design few examples. They are the only human annotations we have during data collection, and are used as seed examples in context-learning
data. One example per type
query GPT-4.
Conversation. We design a conversation between the assistant and person asking questions about this photo. The answers are in tone as if the assistant is seeing the image and answering the question. diverse set of questions are asked about the visual content of the image, including the object types, counting the objects, object actions, object locations, relative positions between
3

---

objects. Only questions that have definite answers are considered. Please see Appendix for the detailed prompt.
Detailed description. To include a rich and comprehensive description for an image, we create a list of questions with such intent. We prompt GPT-4 then curate the list (see detailed prompts and curation process in Appendix). For each image, we randomly sample one question from the list ask GPT-4 to generate the detailed description.
Complex reasoning. The above two types focus on the visual content itself, based on which we further create in-depth reasoning questions. The answers typically require a step-by-step reasoning process by following rigorous logic.
We collect 158K unique language-imann iingee-imeeeeee enstructionfoolloing samples in total, including 58K in conversations, 23K in detailed description, and 77k in complex reasoning, respectively. We ablated the use of ChatGPT and GPT-4 in our early experiments, and found that GPT-4 consistently provides higher quality -oolooi data, such spatial reasoning.
4
Visual Instruction Tuning
4.1
Architecture
The primary goal is to effectively leverage the capabilities of both the pre-trained LLM and visual model. The network archtecture is illustrated in Figure 1. We choose Vicuna [9] as our LLM (-) parameterized by Φ. it has the best instruction following capabilities in language tasks among publicly available checkpoints 9,7,9,37 37].
Language Response
X
Language Model fo
Projection W
H
H
Z
Vision Encoder
X,
Image
X
Language Instruction
Figure 1: LLaVA network architecture.
For an input image X,, we consider the pre-trained CLIP visual encoder ViT-L/14 [39], which provides the visual feature Z, g(x)). The grid features before and after the last Transformer layer are considered in our experiments. We consider simple linear layer to connect image features into the word embedding space. Specifically, we apply trainable projection matrix W to convert Z, into language embedding tokens H,, which have the same dimensionality as the word embedding space in the language model:
H, = W Zu, with Z. = -X((.)
(1)
Thus, we have a sequence of visual tokens H,, Note that our simple projection scheme is lightweight, which allows us to iterate data centric experiments quickly. More sophisticated schemes to con- nect the image and language representations can also considered, such as gated cross-attention in Flamingo [2] and Q-former in BLIP-2 [27]. We leave exploring possibly more effective and sophisticated architecture designs for LLaVA VA as future work.
4.2 Training
For each image X,, we generate multi-turn conversation data (X) _.... ".. ( where T is the total number of turns. We organize them sequence, by treating all answers the assistant's response, and the instruction X i at the 1-th turn as:
Randomly choose x
[X] X2] or [x., X!
q): the first turn the remaining turns
X instruct =
(2)
This leads to the unified format for the multimodal instruction-following sequence illustrated in Table 2. We perform instruction-tuning of the L.MM on the prediction tokens, using original auto-regressive training objective.

---

X nossage <STOP>
Human X nnttruct <STOP> Assistant: X <STOP>
Human X Instruct <STOP> Assistant: X <STOP>...
Table 2: The input sequence used to train the model. Only two conversation turns are illustrated here; in practice, the number of turns varies based on the instruction-folloiing data. In our current implementation, we follow Vicuna-v0 [9] to set the system message Xsysten- and we set <STOP> ### The model is trained to predict the assistant answers and where stop, and thus only green sequence/tokens are used to compute the loss in the auto-regressive model.
Specifically, for sequence of length L, compute the probability of the target answers X, by:
T. pe (x) Xy; X Xinatruct, <o Xa,st),
p(Xa|X Xv: Xinarrct))
II
where 0 is the trainable parameters, inntrruct<<i and ai are the instruction and answer tokens in all turns before the current prediction token 22, respectively. Please see Table 2 for illustration of the prediction tokens. For the conditionals in (3), we explicitly add X., to emphasize the fact that the image is grounded for all answers, and we omit eysstnnnnesaage and all previous <STOP> for better readability. For LLaVA model training, we consider a two-stage instruction-tuning procedure.
Stage 1: Pre-training for Feature Alignment. To strike balance between concept coverage and training efficiency, we filter CC3M to 595K image-text pairs. Please see Appendix for details of the filtering process. These pairs expansion method describe in Section 3. Each sample can be treated as a single-turn conversation. To construct the input X instruct in (2), for image X,, question X. is randomly sampled, which is language instruction to request the assistant to describe the image briefly. The ground-truth prediction answer X, is the original caption. In training, frozen, and maximize the likelihood of (3) with trainable parameters 0 = W (the projection matrix) only. In this way, the image features H, can be aligned with the pre-trained LLM word embedding. This stage can be understood as training compatible visual tokenizer for the frozen LLM.
converted the instruction-follon
data using the naive
keep both the visual encoder and LLM weights
Stage 2: Fine-tuning End-to-End. We always keep the visual encoder weights frozen, and con- tinue to update both the pre-tedined weights of the projection layer and LLM in LLaVA; i.e., the trainable parameters are 0 (W. Φ)} in (3). We consider two specific use scenarios:
Multimodal Chatbot. We develop Chatbot by fine-tuning on the 158K language-image instruction-following data in Section 3. Among the three types of responses, conversation is multi-turn while the other two are single-turn. They are uniformly sampled in training.
Science QA. We study our method on the ScienceQA benchmark [33], the first large-scale multimodal science question dataset that annotates the answers with detailed lectures and explanations. Each question is provided a context in the form of natural language or an image. The assistant provides the reasoning process in natural language and selects the answer among multiple choices. For training in (2), we organize the data as a single turn conversation, the question & context as Xinstructs and reasoning & answer X.-
5 Experiments
We assess the performance of LLaVA in instruction- for.uuttioooooooiinngoooo and visual reasoning capabilities with two primary experimental settings: multimodal chathot and the ScienceQA dataset, respectively. We train all models with 8x A A100s, following Vicuna's hyperparameters [9]. We pre-train our model on the filtered CC-595K subset for epoch with a learning rate of 2e-3 and batch size of 128, and fine-tune on the proposed -LaVA-Instruct-1558 dataset for 3 epochs, with a learning rate of 2e-5 and batch size of 32. See Appendix for more training details.
5.1 Multimodal Chatbot
We developed a chatbot demo to show the image understanding and conversation abilities LLaVV, and to study how well LLaVA is able to digest visual inputs and exhibit cilructtoo-following
5

---

Visual input example, Extreme Ironing:
Source: betper//wvs, barnorons. com//p- oosteessiiiooooaa000/////// Corfusing Ficoures.
User LLaVA
What unusual about this image?
ironing clothes the back of minivan van. This is not typical place to perform this activity, as one would usually iron clothes more stationary and safe location, such as a home, using regular ironing hoard. The scene depicted in the image is peculiar as it involves a makeshift ironing setup on vehicle, which can be both unsafe and unconventional. Additionally, is not clear how the man is able to maintain balance and stability while ironing clothes in such an unstable environment.
The unusual aspect of this image is
[Start new conversation, and clear the history] What's 's happening in the scene? The image depicts holding portable ladder, scemingly ironing clothes while standing on the vehicle, Around scene, there are other cars, traffic light, and fire hydrant. Additionally, there are two other people the vicinity, one close to the man on car and another further away. The man on the ladder appears to be attracting attention due to his unconventional choice of ironing his clothes on top of a moving car. The city street around him is bustling with activity, adding to the unique nature of the scene.
User LLaVA
standing on top of yellow SUV in busy city street. He
User GPT-4 [35]
What unusual about this image?
The unusual thing about this image is that man ironing clothes on an ironing board attached to the roof a moving taxi.
User
What is unusual about this image?
BIII-2
man is sitting on the back of a yellow cab
User
What is unusual about this image?
OpenFlamingo
The man is drying his clothes on the hood of his car.
Table 3: Example prompt from GPT-4 paper [35] compare visual reasoning and chat capabilities. Compared to BLIP-2 [27] and OpenFlamingo [5], LLaVA accurately follows the user's instructions, instead of simply describing the scene. LLaVA offers a more comprehensive response than GPT-4. Even when merely asked to describe the image, LLaVA identifies atypical aspects of the image.
capabilities. We first use the examples in the original GPT-4 paper [35], shown in Table (more examples in Appendix), that require in-depth image understanding, For comparisons, quote the prompt and response the multimodal GPT-4 from their paper, and query BLIP-2 and OpenFlamingo model checkpoints to get their response.
Surprisingly, although LLaVA is trained with small multimodal Instruction-followinng dataset (~88K unique images). it demonstrates quite similar reasoning results with multimodal GPT-4 on these examples. Note that while these images are oft-of-domain for LLaVA, LLaVA is still able to understand the scenes and follow the question instruction to provide reasonable response. In contrast, BLIP-2 and OpenFlamingo focus on describing the image, instead of following the user instruction to answer in an appropriate manner.
Quantitative Evaluation. To gain a systematic understanding of the performance of LLaVA, we propose a quantitative metric measure the model's enstruction-folloiin capability on multimodal data. Inspired by [9], we leverage GPT-4 to measure the quality of generated responses. Specifically, we create triplets consisting of image, ground-trutt textual descriptions, and question. The candidate models (e.g., LLaVA) predict the answers based on the question and the image. To provide an approximate theoretical upper bound, we create reference prediction based on the question and the ground-true truth textual descriptions, using the text-only GPT-4. After obtaining the responses from

---

Conversation
Detail description
Complex reasoning
All
Full data
83.1
75.3
96.5
85.1
Detail Complex
81.5 (-tm))
73.3 (ax:
90.8 cs.b
81.9 (-1:)
Conv + 5% Detail + 10% Complex
81.0 (an
68.4 67.8
91.5 (550))
80.5 (44)
Conversation No Instruction Tuming
76.5 (6.0)
59.8 (1122)
84.9 (:144)
73.8 (11.)
22.0 (SL.))
24.0 (51.3)
18.5 (:78.)
21.5 643.0)
Table 4: Ablation on LLaVA-Bench (COCO) with different training data. We report relative scores
w.r.f. text-only GPT-4 model that uses ground truth image captions and bounding boxes visual input. We prompt GPT-4 with the answers from our model outputs and the answers by GPT-4 (text-only), and let it compare between both responses and give rating with an explanation.
Conversation
Detail description
Complex reasoning
All
OpenFlamingo [5]
19.3 + 0,5
19.0 + 0,5
19.1 + 0.7
19.1 + 0.4
BLIP-2 [27]
54.6 + 1.4
29.1 + 1.2
32.9 + 0.7
38.1 + 1.0
I.aaV
57.3 + 1.9
52.5 + 6.3
81.7 + 1.8
67.3 + 2.0
LLaVA
58.8 0.6
49.2 0.8
81.4 0.3
66.7 = 0.3
Table 5: Instruction-followin capability comparison using relative scores on LLaVA-Benc' (In-the- Wild). The results are reported in the format of mean + std. For the first three rows, we report three inference runs. LLaVA performs significantly better than others. For a given set of LLaVA decoding sequences, we evaluate by querying GPT-4 three times; GPT-4 gives a consistent evaluation.
both models, we feed the question, visual information (in the format of textual descriptions), and the generated responses from both assistants, to the judge (i.e., text-only GPT-4). It evaluates the helpfulness, relevance, accuracy, and level of detail of the responses from the assistants, and gives an overall score on a scale of I to 10, where a higher score indicates better overall performance. It is also asked to provide a comprehensive explanation for the evaluation, for us to better understand the models. We report relative scores w.E.t the text-only GPT-4 model that uses the textural ground truth description as visual input. We create two benchmarks to evaluate the model's performance.
LLaVA Ve-Bench (COCO). We randomly select 30 images from COCO-Val-2014, and for each image, we generate three types of questions (conversation, detailed description, complex reasoning) using the proposed data generation pipeline in Sec. 3, totaling 90 questions. This benchmark studies the model's 's aligmment behavior and capabilities with consistent visual inputs. We vary the training datasets study the effectiveness of different types of iostruction-followinng data, and show the results in Table 4. First, with instruction tuning, the model's ability of following user instructions improves significantly by over 50 points. Second, adding small amount of detailed description and complex reasoning questions contributes to a considerable improvement of the model's overall capability by points. Furthermore, it also improves the model's performance on conversational questions, suggesting that improvements in reasoning capabilities complement conversational abilities. Finally, we show that having all three types of data yields the best performance 85.1% 15.
LLaVA- (n-the-Wild), To evaluate the model's capability in more challenging tasks and generalizability to novel domains, we collect a diverse set of 24 images with 60 questions in total, including indoor and outdoor scenes, memes, paintings, sketches, etc., and associate each image with a highly-detailed and manually-curated description and a proper selection of questions. We compare LLa BLIP, and OpenFlamingo in Table 5. Thanks to visual instruction tuning, LLaVA achieves significantly better performance compared with BLIP-2 (+29%) and OpenFlamingo (+48%). Compared to the text-only GPT-4 that has access to ground truund-truth labels, LLaVA achieves an impressive 81.7% performance on complex reasoning questions, with an overall score of 67.3%.
Limitations. This ALLVA--Bence (In-the-Wild is designed to be challenging and reveal a model's weaknesses. We provide two examples with associated captions and questions in Table 6. For the ramen example (left), to correctly answer the name of the restaurant, it requires the model to have a large knowledge coverage and multilingual understanding capability; to correctly describe the side dishes, the model may need to retrieve relevant multimodal information from Internet. For the fridge example (right), perceiving the correct brand of the yogurt requires the model to process high resolution images and possess extensive knowledge coverage. We also observed an interesting failure of VA. it responds with yes when asked if strawberry- flavored yogurt is present, even though
7

---

Challenging examples from LLaVh-Bench (n-the-Wild):
ICHIRAN Ramen [source]
Filled fridge [source]
Annotation
A close-up photo of a meal at ICHI- RAN. The chashu ramen bowl with a spoon is placed in the center. The ramen is seasoned with chili sauce. chopped scallions, and served with pieces of chashu. Chopsticks are placed to the right of the bowl, still in their paper wrap, not yet opened. The ramen also served with nori on the left. On top, from left to right, the fol- lowing sides are served: a bowl of or- ange spice (possibly garlic sauce), a smoke-flavored flavored stewed pork
An open refrigerator filled with a variety of food items. In the left part of the compartment, towards the front, there is a plastic box of strawberries with a small bag of baby carrots top. Towards the back. there is a stack of sauce containers. In the middle part of the compartment, lowards the front, there is green plastic box, and there plastic bag placed on it. Towards the back, there is a carton of milk. In the right part of the compartment, towards the front, there is box of blucherries with three yogurts stacked on top. The large bottle of yogurt is Fage non non-fat yogurt, and one of the smaller cups is Fage blueberry yogurt. The brand and flavor of the other smaller cup unknown. Towards the back, there is container with unknown content.
unidentified
plate of smoke- with chopped scallions, and cup of matcha green tea.
Question 1
What's the name of the restaurant?
What is the brand of the blueberry- flavored yogurt? Is there strawberry-favord yogurt the fridge?
Question 2
Describe this photo in detail.
Table 6: Challenging examples from LLLaVA-Benci (In-theWWild), provide extremely-detailes annotation for each image for an accurate evaluation. Some questions require the model to extract details from high resolution image and to have broad knowledge coverage.
the fridge contains only yogurt and strawberries. This indicates that, at times, LLaVA perceives the image as a "bag of patches", failing to grasp the complex semantics within the image. We hope LLaVA serves as a solid baseline on the benchmarks, on which our findings can inspire future work in developing more capable LMMs.
5.2 ScienceQA
ScienceQA [33] contains 21k multimodal multiple choice questions with rich domain diversity across 3 subjects, 26 topics, 127 categories. and 379 skills. The benchmark dataset is split into training, validation, and test splits with 12726, 4241, and 4241 examples, respectively. We consider two representative methods, including GPT-3.5 model (text-davinci-onl -002) with and without chain- of-thought (CoT), LLaMA-Adapter [58], as well as multimodal chain-of-thouuhtt (MM-CoT) [60], which is the current SoTA method on this dataset. For more baseline numbers, please see [33].
The results are reported in Table 7. For LLaVA, we use the visual features before the last layer, ask the model to first predict reasons and then the answer, and train it for 12 epochs. It yields 90.92% accuracy, which is quite close to the SoTA 91. 11.68% To explore the limit of L.MMs, we also prompt GPT-4 using 2-shot in-context-ccaminng and achieve 82.69% accuracy, which is 7.52% absolute gain compared with 75.17% from GPT-3.5. For a substantial number of questions, we note that GPT-4 fails simply because it reports that there insufficient context such as images or plots. We consider two schemes to combine the outcomes from our model and GPT-4. (i) A GPT-4 complement. Whenever GPT-4 fails to provide answers, we use the prediction from our method. This schemes yields 90.97% accuracy, which is almost the same as applying our method alone. (tt) GPT-4 as the judge. Whenever GPT-4 and LLaVA produce different answers, we prompt GPT-4 again, asking to provide its own final answer based on the question and two outcomes. The spirit is similar with CoT, but with the external knowledge from the other model. Surprisingly, this scheme able to provide consistent improvement over all question classes, and achieves new SoTA accuracy of 92.33. Interestingly.
8

---

Subject
Context Modality
Grade
Method
Average
NAT
SOC
LAN
TXT
IMG
NO
G1-6
G7-12
Representative & SoTA methods with numbers reported in the literature
Human [33]
90.23
84.97
87.48
89.60
87.50
88. 18
91.59
82.42
88.40
GPT-3.5 [35]
74.64
69.74
76.00
74.44
67.28
77.42
76.80
68.89
73.97
GPT-3.5 w/ CoT [33]
75.44
70.87
78.09
74.68
67.43
79.93
78.23
69.68
75.17
ALaMA-Adapter [58]
84.37
88,30
84,36
83.72
80.32
86.90
85.83
84.05
85.19
MM MM-CoT [60]
87.52
77.17
85,82
87.88
82.90
86,83
84.65
85.37
84.91
MM-CoTLarey Large [60]
95,91
82.00
90.82
95.26
88,80
92.89
92.44
90.31
91.68
Results with our own experiment runs
GPT-4
84.06
73.45
87.36
81.87
70.75
90.73
84.69
79.10
82.69
LLaVA
90.36
95.95
88.00
89.49
88.00
90.66
90.93
90.90
90.92
LLaVA+GPT-4F (complement)
90.36
95.50
88.55
89.05
87.80
91.08
92.22
88.73
90.97
LLaVA+GPT-44 (judge)
91.56
96.74
91.09
90.62
88.99
93.52
92.73
92.16
92.53
Accuracy (%) on Science QA dataset. Question categories: NAT = natural science, SOC= social science, LAN language science, TXT text context, IMG image context, NO context, G1-6 = grades 1-6, G7-12 = grades 7-12. TText-only GPT-4, our eval. Our novel model ensembling with the text-only GPT-4 consistently improves the model's performance under all categories, setting the new SoTA performance.
Table 7:
the text-only GPT-4, which cannot process images, improves the overall performance of the model on questions that have image context. This is because some of these questions do not actually require the image context for a correct answer. The GPT-4 judge can identify such cases and correct some of the errors that LLa VA makes. See the example in Appendix. To the best of our knowledge, this is the first time that GPT-4 used for model ensembling. We hope this finding can encourage future research to explore more effective methods leverage LLMs for model ensembling.
Ablations. We ablate several design choices on ScienceQA in Table 8. (i) Visual features. We tried using the last layer feature from CLIP vision encoder, which yields 89.96% and is 0.96% lower than the feature before the last layer. We hypothesize that this is because CLIP's last layer features may focus more global and abstract image properties compared to the layer before it, which focus more localized properties that are useful for under-
Visual features
Before
Last
Best variant
90.92
89.96 89.77 (4:15)
Predict answer first
Training from scratch
85.81 (6:.1)
7B model size
89.84 (11.8)
Table 8: Design choice ablations (%). The differ- ence with the best variant is reported in red text.
standing specific image details. (ii) Chain-of-thought. To decide the order between the answer and reasoning process in the model prediction, we run both variants and observe that answer-first reports the best number 89.77% accuracy 12 epochs, while reasoning-firs can quickly reach 89.77% accuracy in 6 epochs, but no further improvement with more training. Training the model for 24 epochs does not improve the performance. We conclude that CoT-like reasoning first strategy can largely improve convergence, but contributes relatively little to the final performance. (iii Pre-iraining. We skip pre-training and directly train on Science QA from scratch - performaarorcee drops to 85.81% % accuracy. The 5.1 1 % absolute degradation indicates the importance of our pre-training stage, in aligning multimodal features while preserving the vast pre-trained knowledge, (iv) Model size. We keep all configurations the same as our best 13B model, and train a 7B model. This yields 89.84% accuracy, which is 1.08% lower than 90.92% demonstrating the importance of model scale.
6 Conclusion
This paper demonstrated the effectiveness of visual instruction tuning. We presented an automatic pipeline to create language-image instruction-folloviin data, based on which we train LLaVA, a multimodal model to follow human intent to complete visual tasks. It achieves the new SoTA accuracy when fine-tuned on ScienceQA, and excellent visual chat capabilities when fine-tuned on multimodal chat data. Besides, we present the first benchmark to study multimodal instruction- following capability. This paper an initial step in visual instruction tuning, and mainly focuses on real-life tasks. For more quantitative results of LLaVA on academic benchmarks, please refer to the

---

improved baselines with visual instruction tuning [31]. We hope our work can inspire future research on building more capable multimodal models.
Acknowledgements. We thank Baolin Peng and Pan Lu for valuable discussions on instruction- tuning language models and Science QA, respectively. We thank the ILaMMA team for giving us access to their models, and open-source projects, including Alpaca and Vicuna. This work was supported part by NSF CAREER IIS2150012 and Institute of Information communications Technology Planning & Evaluation(IITP) grants funded by the Korea government (No. 2022- 0-00871, Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration) and (No. RS RS-2022-001882338, Development of Large Korean Language Model Technology for Efficient Pre-training).
References
[1] Langchain. https://githin httptt/bb.ccc//ccccaaa///aaaaaaa 2022. 2
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson. Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022. 2,4
[3] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and- Lasion-andlaanguage navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE
conference on computer vision and pattern recognition, 2018. 2
[4] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. arXiv preprint arXiv:2112.0080. 2021.
[5] Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo, March 2023. 2, 6.
[6] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instruct pix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:221 1,09800, 2022. 2
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 2
[8] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail tail visual concepts. In CVPR, 2021.
[9] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. 1, 2, 5,
[10] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint (tW.ii::2244023111 2022.
[11] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling cetrucctionfineturee language models. arXiv preprint arXiv:2210.11416. 2022.
[12] CVinW.
Computer vision in the wild. Computer- tean-ii-the--iiddd WWWiiWWWeedins...... 2022. 1
https://github. com/
[13] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM-F; An embodied multimodal language model. arXiv preprint arXiv:2303.033: 2023, 2
10

---

[14] Fartash Faghri, Hadi Pouransari. Sachin Mehta, Mehrdad Farajtabar, Ali Farhadi, Mohammad Rastegari, and Oncel Tuzel. Reinforee data, multiply impact: Improved model accuracy and robustness with dataset reinforcement. arXiv preprint arXiv:2303.089%. 2023. 2
[15] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. ArXiv, abs/2203.131311 2022.
[16] Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao, et al. Vision- language pre-training: Basics, recent advances, and future trends. Foundations and Trends@ in Computer Graphics and Vision, 2022.
[17] Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303. 1506, 2023.
[18] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. arXiv preprint arXiv:2211.11559, 2022.
[19] Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning a generic agent for vision-and anden--nd--aaae -andlagg navigation via pre-training. In CVPR, 2020. 2
[20] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao
Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302. 14045, 2023. 2
[21] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. If you this software, please cite it as below.
[22] Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang. Qing Liu, Punit Singh Koura, et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint 22:::2222.11111 2022.
[23] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022. 2
[24] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal generation. arXiv preprint arXiv:2301. 13823, 2023.
[25] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language- driven semantic segmentation. ICLR, 2022.
[26] Chunyuan Li, Haotian Liu, Liunian Harold Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, and Jianfeng Gao. ELEVATER: A bench- mark and toolkit for evaluating language-augmente visual models. In NeurIPS Track on Datasets and Benchmarks, 2022.
[27] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023. 2, 4, 6,7
[28] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, al. Grounded language-image pre-training, In CVPR, 2022. 1
[29] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. arXiv preprint arXiv:2301.0700 2023.
[30] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 3
11

---

[31] Haotian Liu, Chumyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. 2023. 10, 14
[32] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre- vee-training for open-set object detection. arXiv preprint arXiv:2303. 05499, 2023.
[33] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Kei Chang, Long-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information P Processing Systems,
2022. 2, 5, 9
[34] OpenAl. ChatGPT. attpp://openne http:/aaii.commbooo/cgtgggg/ 2023,
2023. 1,2
[35] OpenAl. Gpt-4 technical report, 2023. 1, 6, 15
[36] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022. 2
[37] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. arXiv preprint arXiv:2304.032777 032277, 2023. 1,
[38] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, al. Combined scaling for open-vocabulary image classification. arXiv preprint arXiv: 2/ 2111. 10050, 2021.
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 2021. 1,
[40] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020. 2
[41] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. ArXiv, abs/2204.06125, 2022.
[42] Robin Rombach, A. Blattmann, Dominík Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. CVPR, pages 10674-10685, 2022.
[43] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes. Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. ArXiv, 11882 2022.
[44] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:22 10. 08402, 2022.
[45] Dídac Surís. Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint 203.022333..8818, 2023. 2
[46] Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 2
12

---

[47] Rohan Taori, Ishaan Gulrajani. Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Perey Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An insllwctionfoollowingg Ilama model. https:// http//ggtttubbb.. thub. com
ord alpaca, 2023. 1,
[48] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 13977, 2023. 1
[49] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv: 14100, 2022.
[50] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions. arXiv preprint arXiv:2212.10560, 2022. 2
[51] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashii Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07700, 2022. 2
[52] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.0466) 2023. 2
[53] Jianwei Yang. Chunyuan Li, Pengchuan Zhang, Bin Xiao, Lu Yuan, Ce Liu, and Jianfeng Gao. Unified contrastive learning in image-text-label space. CVPR, 2022. 1
[54] Zhengyuan Yang. Linjie Li, Jianfeng Wang. Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint 2rXiv:2303.13300 2023. 2
[55] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfci Yang, Burcu Karagol Ayan, Benton C. Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. ArXiv, Abs/2206.10789 2022.
[56] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2 I. 1432, 2021.
[57] Hao Zhang. Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianfeng Gao, Jianwei Yang. and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. arXiv preprint ar arXiv:2303.00 08131, 2023. 1
(58) Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199 2023. 8, 9
[59] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. arXiv preprint wrXiv:2205001066 2022. 2
[60] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multi- modal chain-of-thoon -ffthoughtt reasoning in language models. arXiv preprint arX1v:2302.00922, 2023.
8, 9
[61] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 16793-16803, 2022. 1
[62] Xueyan Zou, Zi-Yi Dou, Jianwei Yang. Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. arXiv preprint arXiv:2212.112200 2022. 1,2
13

---

A Broader Impact
The broader impact of LLaVA, a general- visual assistant, has potential benefits and risks associated with its deployment and release. Some considerations unique to LLaVA due to its visual nature, while others share similarities with existing eesructiion-ollooiig LLMs (e.g. Alpaca, Vicuna, efc.). As LLaVA is built upon LLaMA, Vicuna, and CLIP, it inherits some of the issues associated with LLMs and vision encoders. In the following, we outline both the risks and mitigation strategies place for the release of this model.
Malicious input. To minimize potential misuse and harmful consequences, employ two pre- cautionary measures for LLaVA: (1) OpenAl Filter API for user input text to prevent harmful or inappropriate text instructions from being processed by the model, and (2) NSFW Filter for uploaded user images to detect and block Not Safe For Work (NSFW) content or any other potentially harmful image inputs.
Hallucination. Similar to LLMs, LLaVA might generate outputs that aren't grounded in facts or input data, This raises concerns about inferences made, especially in critical applications (e.g., medical).
Biases. Bias can be transferred from the base models to LLaVA, both from the vision encoder (CLIP) and the language decoder (LLaMA/Vicuna) This may lead to biased outcomes or unfair representations of diverse content.
Energy consumption. Though energy consumption is not primary concern for LLaVA due to smaller pretraining dataset (see details in Sec. C), it may become concern when scaling up the pretraining dataset or increasing the model size, e.g., to larger LLaMA version like the 65B model.
Evaluation complexities. Assessing the performance of LLaVA challenging as involves both language and visual tasks. Our evaluation benchmark covers several aspects, including accuracy, concept coverage, reasoning ability, and creativity. However, additional aspects need consideration, such the degree of visual content hallucination and fine-grained understanding of visual content. While text-only GPT-4 based multimodal evaluation is consistent and accurate our study, its robustness in different situations and capability to evaluate other unexplored aspects are subjects for future work.
Despite these risks, we believe that the benefits of releasing LLaVA to the research community outweigh the potential harm. allows for ongoing investigation and improvement of the model and engages the community in developing better mitigation strategies to address these concerns. Moreover, the release of LA can spur the development of new applications and research directions, ultimately contributing to the progress and responsible deployment of foundation models in vision-language tasks.
B More Results
We present more qualitative results of LLaVA to analyze its emergent behaviors and observed weaknesses. For more quantitative results LLaVA on academic benchmarks, please refer to the improved baselines with visual instruction tuning [31]. In Table 9, ILLVVA demonstrates a similar behavior as GPT-4 in another cxample from its paper. Similar to the GPT-4 live demo by OpenAL, LLaVA is capable of generating the HTML/JS/CSS code for interactive joke website based on a simplified user input sketch in Fig. 2, despite a minor error. As shown in Fig, 3, LLaVA can follow user's s instructions in a conversational style and provide detailed responses or creative writings. Furthermore, LLaVA is able to relate the visual content to the textual knowledge from the pretrained LLM, as demonstrated in Fig. and Fig. 5.
One interesting emergent behavior of LLaVA is that it is able to understand visual contents that are not covered in the training. For example, in Fig. 6, it is able to recognize Elon Musk both in headshot and in a humorous meme where he dressed as a doge, even though Elon Musk never appears in the training data for either the visual feature alignment visual instruction tuning stages of
14

---

VA. LLaVA also demonstrates impressive OCR (optical character recognition) ability in Table 9 and Fig. 2, which is rarely covered in our training data.
We hope these additional results and observations showcase the potential of LLaVA in various application areas. In future work, it is important to investigate these emergent behaviors more thoroughly and to understand the underlying mechanisms that enable LLaVA to demonstrate such generalization abilities. This will pave the way towards building better LMMs, including enhancing robustness, reducing biases, and improving the alignment and the scope of the learned vision-language representations.
Visual input example, Chicken Nugget Map:
Sometimes just look pictures of earth space manvel how beoutiful alla.
User ILaVA
Can you explain this meme in detail?
The meme the image is creative and humorous take on food, with focus chicken nuggets, the center the universe. The meme begins with the phrase "Sometimes I just look pictures of the Earth from space and marvel how heautiful it all is..." 'The main image features a tray of cooked chicken nuggets, with the chicken pieces positioned to resemble the continents chicken nuggets represent the Earth, and the various locations depicted in the photo are actually chicken nugget versions of different places. The punchline of the meme appears in smaller text at the hottom, reading "I mean, it's not the real Earth, but how beautiful it is all is." " This lighthearted and entertaining meme combines food and humor, using the image of chicken nuggets to create funny and imaginative representation of the world.
islands on a world map. The meme playfully suggests that the
User GPT [35]
Can you explain this meme?
This meme is a joke that combines two unrelated things: pictures of the earth from space and chicken nuggets. The text of the meme suggests that the image helow is a beautiful picture of the earth from space. However, the image is actually of chicken nuggets arranged to vaguely resemble a map of the world. The humor in this meme comes from the unexpected juxtaposition of the text and the image. The text sets up expectation of majestic image of the earth, but the image is actually something mundane and silly.
User
Can you explain this meme in detail? sometimes i just look at pictures of the earth from space and marvel how beautiful it is
BLIP-2
User
Can you explain this meme detail?
OpenFlamingo
It's picture of a chicken nugget on the International Space Station.
Table 9: Example prompt comparing LLaVA, GPT-4, BLIP-2, and OpenFlamingo's visual reasoning capabilities in understanding the humor. BLIP-2 and OpenFlamingo fail to follow the user's s instructions. LLaVA and GPT-4 both explain the meme and its humor, while GPT-4 produces a more concise answer.
15

---

My Joke Webbite
User
[ Funn y Jokel [push to revol purbllnne
Turn this mock-up into interactive webs using heml/js, replace joke.
<IDOCTYPE hembs
chead>
LLavA
<till>My Joke Website-/ttte>> <style>
body background color: font-familyy:.o Artal, sans-sertt:)
container [ display: flos; flex-directin: column; align-items: center; justify-content conter; height: 100wh;;
h1 font-size: 36or mergin-bottom: 20px;
p font-siie 20px;]
button & background color: blue; color: white: border: nome, padding: 10px 20px; font slee: 18px, cursor: pointer:)
</styles
</head>
<hady>
<div class "container">
<h1bMy loke Webuite</ht>>
<p "punchiine"
my joke
punchlinets/p>
<button onclike" "revealuunhhlinef" SPush mel</button>
</divs
escript>
document getf smentByldd" punchiine"].iwwrHHMMM - "The punchline 'Wity was the math book sad? Because it had too many
problems."
s/script>
</body>
</htmb>
Place the generated HTML code into on HTML renderer render the web
My Joke Website
My Joke Website
Welcome to my joke website. Push the button to reveal the punchline!
The punchline is: 'Why was the math book sad? Because had too many problems.
Push me!
Push me!
Rendered website
Rendered website (dlccked the button)
Figure 2: LLaVA generates HTML/IS code for an interactive website based on user sketch inputs. The interactive interface works after fixing minor error (in red) in the generated output. There is room for improvement LLaVA's output, such as splitting the joke and punchline into two rows, and only revealing the punchline upon button click, to better reflect the user's intent.
16

---

User
What meals that cook these?
prepare stmaberries, blueberries, and carrons, Additionall, you can make a delicious smoothie using the fruits vegetables, which would healthy and nutritious snack or breadfast option. The milk and eggs can be used to create various diches, such as milk based smoothie, refreshing homemade popsicle. The
stared refrigerator,
LLaVA
baked casserole, scrambled The juice also be make fresh fruit and depend your preferences and dietary needs.
possibilities vast, meals you can
would to do the fruit salad. Show me the recipe.
Certainlyl simple recipe using the Items your refrigeratio
User
Ingredients:
LLaVA
cups strawberris, sliced
-1 cup fresh blueberries
1 large carrot, grated
half leman
handful thopped fresh parsley or mint
A fow ice cubes
Optional: honey maple syrup jadjust to taste)
Instructions:
1. a large bowl, combine the sliced strawberries, blueberries, and carrot. Gendly toss them together mix. the gently If desired, add drovle honey or maple syrup for touch of sweetness, Adjust the amount according your taste preferences.
Drizzle
half
fruit vegetables parsley
add refreshing
Sprinkll chopped
more.
Cover the bowl and refrigerate the fruit salad for lesst minutes to few hours. This allows the flavors meld and creates
refreshing
Serve the fruit salad in cups or bowis, garnished with sprinkee fresh parsley or mint, destred.
fruit saladi perfect healthy easy put together quickly.
with ingredients available
PG
User
What are the things should be cautious about when visit here?
this
extending
things
about. First pay attention to the weather conditons, as sudden changes weather, especially strong winds or storms, can cause the water become rough and pose risks to the stabilit of the pier. Second, be minaful the structural integrity of the pier, avoid overloading it with too collaase. Additionally, them way. Finally, you
LLaVA
people excessive weight, this could potentially damage the pier
of the surrounding fishing, ensure that you adhere to local
respect their space; fishing regulations and use appropriate fishing techniques to keep the aake ecosystem healthy and maintain the balance of the food chaln.
Campase an engaging travel blog post about trip to this place, highlighting the beautiful sceneries, experiences,
Traveling to picturesque location pier extending over the calm water provided escellent vontage point from which to appreciate the breathtaking sceneny, As stood on the couldn't
mountains background was refreshing and rejusennatnn experience.
User
connection
peaceful atmosphere.
LLaVA
One of the highlights of my trip was undoubsedly the fishing experience. The area teeming with life,, and was thrille to land sizeable catch, despite the modest size of the lake. The joy fishing in beautiful setting was truly rewarding, and ailowee me to take break immerse
my life
tranquiiity
While exploring the surrounding area, came across family of birds that had built nest close to the ples adding the charm of the enriching experience that cherish
location. Interacting with the wildiie and observing their behavior
conclusion, this ityylcc destination provided me with an opportunity to unwind and recharge, surrounded by mesmerizing ing natural beauty and the gontle sounds of water and wielife. you're looking for serene escape from the hustle and bustle of everyday life, this pier overiooking place find
lake
Figure 3: LLaVA is capable of recognizing the visual content following the user's intent, without directly prompting for visual recognition. It also provides detailed response when prompted with follow-up request, and the generated response closely related to the provided visual content.
17

---

User
What's the onding this movie?
the image, young woman leaning on man in front of boautiful sunset, with both of them standing on boat. They appear be enjoying romantic moment togecher. The scene suggests that they could be depicted the characters lack and Rose from the popular movie Titanic.
passionate kiss ship siniss into oceen. The ending the movie known its emotional and heartfelt spenes, where characters are forced to say goodbye to each other they are separated by the sinking ship. The image evokes serse of romance and nostalgis, paying tribute to the iconic love portrayed the Titani.
fictional
ultimately
Figure LLaVA relates the movie scenes the textual knowledge from the pretrained LLM.
User
Do you know who drew this painting?
artwork by Leonardo showcases the woman's enigmatic omile and has become one of the most famous and iconic art pieces the world. The original work is displayed the Louvre Muscum Paris, and known for its intricate detalls, use paint, and the artis's innovative techniques that contributed
painting depicts woman, commonly believed
portratt pa inting
LLavA
cleored.
User
painting?
The painting depicts a dog humorous situation, where It appears to wearing woman's dress and standing front of landscape. traditional portralt style, replacing a human figure dog
painting explicitly identified including Mona painting is Iklly a creativv dressed in woman's dathing.making it an interesting and amusing visuel piece.
Although
LLaVA
Figure 5: LLaVA recognizes the famous art work, Mona Lisa, by Leonardo da Vinci. When start a new conversation, it also explains the humourous artwork created on the web, mimicking the Mona Lisa.
18

---

User
What the name of the man in the photo?
LLavA
man photo
Start new conversation, history
User
What
photo?
LLaVA
Figure 6: An interesting emergent behavior of LLaVA is its ability to recognize Elon Musk both in headshot and in humorous meme where he is dressed as doge. This implies that the pre-trained CLIP vision encoder may have seen images of Elon Musk. However, is still surprising because Elon Musk never appears in the training data for either the visual feature alignment visual instruction tuning stages of LLaVA, which indicates that the base language model generalizes to unseen visual concepts.
Question:
Which material is this rocking chair made of?
Options: (A) wood (B) silk
LLaVA answer:
LECTURE: A material is type of matter.
Wood, glass, metal, and plastic are common materials.
Some objects made of just material.
Most nails are made metal.
Other objects are made of more than one material.
This hammer is made of metal and wood.
SOLUTION: Look the picture of the rocking chair. The rocking chair is made of two different materials. The legs are made af wood, and the back and seat made of silk.
answer GPT-4 (text-only) answer:
The answer is A. BECAUSE: Rocking chairs are typically made of sturdy materials that can support the weight of person and withstand the rocking motion. Wood is making rocking chairs due to strength, durability, natural appearance.
material used for
GPT-4 (text-only) judge:
Since there is context image refer to, cannot independently analyze the rocking chair in question. However, can try to analyze the reasoning given by the assistants. Assistant who claims to have seen image, mentions that the legs made of wood, and the back and seat are made of silk, implying that it's a combination of two materials. However, this response seems less plausible, silk is not common material used for the construction of rocking chairs due issues with stability and durability. Typically, would expect frame, legs, and seat back to made wood, fabrics, such as cotton or polyester.
rocking chair are
cushions and padding made
Assistant argues that racking chairs are typically made of sturdy materials, and wood is common material used for making rocking chairs due to its strength, durability, and natural appearance. This reasoning stands and based on common knowledge. Given the information available, most plausible answer is that rocking chair made wood, as suggested by Assistant 1. The answer is A.
Table 10:
One example on how the text-only GPT-4 acts as a judge to ensemble the predictions from LLaVA and text-only GPT-4, and gives correct final answer.
19

---

C
Training Details
We pre-train our model on the filtered CC-595K subset for epoch with learning rate of 2e-3 and batch size of 128, and fine-tune on the proposed LLaVA-Instruct- 158K dataset for 3 epochs, with learning rate of 2e-5 and batch size of 32. Following Vicuna, we use the Adam optimizer with no weight decay and cosine learning rate with warmup ratio of 3%. During finetuning, FSDP (Full Shard Data Parallel) and gradient checkpointing is used to save GPU memory, and offloading not used. BF16 and TF32 are enabled to achieve balance between speed and precision.
We train all models with x A100s. Pretraining on CC-595K completes within 4 hours. Finetuning on Instruct- 158K completes within 10 hours. Finetuning on ScienceQA completes within 4 hours.
D Assets
Our source code, generated instruction-tuning data, proposed benchmark are uploaded to the anonymized GitHub repository: LLa LLaVA-Annonyymus/LLaa
1. Source Code: link
2. README: link
3. Instructions to launch the demo: link
4. All prompts and few shot examples for querying GPT-4; link
5. LLaAA-Instruct-158K link
6. LLaVA- VLaBe-Bench: COCO, In-The-Will
7. Model checkpoints. The size of the model checkpoints after compression is 25GB, which exceeds the 5GB limit of GitHub LFS (Large File Storage). We'll release the checkpoint to the public, or upon request with reviewers for this submission.
E Data
Instructions for brief image description. The list of instructions used to briefly describe the image content are shown in Table 11. They present the same meaning with natural language variance.
"Descrihe the image concisely."
Proovdde brief description of the given image."
"Offer a succinct explanation of the picture presented."
"Summarize the visual content the image."
"Give a short and clear explanation
subsequent image."
"Share a concise interpretation the image provided."
"Present compact description of the photo's key features."
"Relay brief, clear account of the picture shown."
"Render a clear and concise summary of the photo."
"Write a terse but informative summary of the picture."
"Create a compact narrative representing the image presented."
Table 11: The list of instructions for brief image description.
Instructions for detailed image description. The list of instructions used describe the image content in detail are shown in Table 12. They present the same meaning with natural language variance.
We extract noun- phrases using Spacy for each caption over the whole CC3M dataset, and count the frequency of each unique noun-phrase. skip noun-phrases whose frequency is smaller than 3, they usually rare combinations concept and attributes that has already been covered
CC3M.
20

---

Deeeriibe the following image in detail"
Provide detailed description of the given image"
"Give an elaborate explanation of the image you see"
Share a comprehensive rundown of the presented image"
"Offer a thorough analysis the image"
image before you" "Explain various aspects
"Clarify the contents of the displayed image with great detail"
"Characterize the image using well-detailed description"
"Break down the elements of the image in detailed manner"
"Walk through the important details of the image"
PPortryy the image with a rich, descriptive narrative"
"Narrtte the contents the image with precision"
Analyze image a comprehensive and detailed manner"
"Illustrate the image through descriptive explanation"
"Examine the image closely and share its details"
"Write exhaustive depiction of the given image"
Table 12: The list of instructions for detailed image description.
by other captions. We then start from the noun-phrases with lowest remaining frequency, add the captions that contain this noun-phrase to the candidate pool. If the frequency of the noun-phrase larger than 100, we randomly choose a subset of size 100 out all its captions. This results in around 595K image-text pairs.
The comparison of noun-phrase statistics before and after filtering CC3M shown in Figure 7. The filtered dataset shows good coverage of concepts whose frequency is higher from 3, but with smaller number of image-text pairs.
10a
CC3M: 108182
Frequency
CC3M (Filtered): 31423
103
10a
10000
20000
30000
40000
50000
Unique noun-phrases (ordered by frequency in the descending order)
Figure 7: Comparison of noun-phrase statistics before and after filtering CC3M. The total number of unique noun-phrases are reported the legend.
F
Prompts
The prompt used to generate image-based conversation from ChatGPT/GPT-4 shown in Table 13.
21

---

messages = "roolee:ssyterr" "content"; Y"""You are Al visual assistant, and you are seeing single image. What you see provided with five sentences, describing the same image you looking Answer all questions as you are seeing the image.
Design conversation between you and a person asking about this photo. The answers should be in a tone that visual Al assistant is seeing the image and answering the question. Ask diverse questions and give corresponding answers.
Include questions asking about the visual content the image, including the object types, counting the objects, object actions, object locations, relative positions between objects, etc. Only include questions that have definite answers:
confidently; (2) one can determine confidently from the image that not the image. Do not any question that cannot be answered confidently.
(1) see the content in the image that the question asks about and
objects Also include complex questions that relevant to the content in the image, for example, asking about background knowledge
image, asking to discuss about events happening the image, etc. Again, do not ask about uncertain details. Provide detailed answers when answering complex questions. For example, give detailed examples or reasoning steps to make the content more convincing and well-organized. You can include multiple paragraphs if necessary." """)
for sample fewshot_ ssapleesss
lresages.appene (("role": (("ool "user", "content" : sample [' context '}})
messages append (("role": "assistant", "content" saattent"=smppee [ (rreponsse'] messages... append (("role": "user", "content"; "jinn'inniini..n.oa (query))
Table 13:
For each query, illustrate the prompt construction process for ChaiGPT/GPT-4 collect query [ r'resonnse' '] from query ['context'], '] using few-shot in-context-learning where examples are from fewshot_ samples, each example including input sample [6 context'] and output sample [ 'reppnnse" J.. Note that messages is the final prompt. In this example. we provide the prompt used to generate the conversation response, please see also see its in-context-leaming examples in Table 15 and Table 16 for details. We recommend readers to check out the codebase for the prompts to generated two other types of responses, including detailed decription and complex reasoning.
22

---

Context type 1: Captions
group of people standing outside black vehicle with various luggage. Luggage surrounds a vehicle in an underground parking area
People try to fit all of their luggage in an SUV.
The sport utility vehicle is parked in the public garage, being packed for trip Some people with luggage near van that transporting it. Context type 2: Boxes
person: [0.681, 0.242, 0.774, 0.694], person: [0.63, 0.222, 0.686, 0.516], person: [0.444, 0.233,
0.487, 0.34], backpack: [0.384, 0.696, 0.485, 0.914], backpack: [0.755, 0.413, 0.846, 0.692],
suitcase: (0.758, 0.413, 0.845, 0.69], suitcase: 10.1, 0.497, 0.173, 0.579), bicycle: [0.282, 0.363,
0.327, 0.442], car: [0.786, 0.25, 0.848, 0.322], car: [0.783, 0.27, 0.827, 0.335J, car: [0.86, 0.254, 0.891, 0.3]. car: [0.261, 0.101, 0.787, 0.626]
Response type I: conversation
Question: What type of vehicle featured in the image?
Answer: The image features black sport utility vehicle (SUV).
Question: Where is the vehicle parked?
Answer: The vehicle is parked in an underground parking area, likely in public garage.
Question: What the people in the image doing?
Answer: The people are trying to fit all of their luggage into SUV, likely preparing for trip.
Response type 2: detailed description
The image is an underground parking area with black sport utility vehicle (SUV) parked. There are three people in the scene, with one person standing closer to the left side of the vehicle, another person in the middle, and the third person the right side. They are all working together pack their luggage into SUV for a trip.
In the parking area, various lugeage items are scattered around the vehicle. There are two backpacks, one located near the left rear wheel and the other closer to the right side of the vehicle. Additionally, there are two suitcases, one the right side of the car and another further away near the center of the parking area. bicycle
also be Other cars are parked around the main SUV, with one car positioned hehind it slightly the left, another behind and slightly to the right, and the third car further behind on the right side.
the left side of the vehicle.
Response type 3: complex reasoning
Question: What challenges do these people face?
image, group people standing outside a black SUV parking area, surrounded by various pieces of luggage, including suitcases and backpacks. They are facing the challenge of fitting all their luggage into the black SUV. There are multiple suitcases and backpacks be packed, which suggests that the group has significant amount of belongings to accommodate. They might have to strategize and arrange the luggage efficiently ensure that everything fits properly into vehicle. Additionally, they need consider the comfort the passengers visihility while driving, so the placement of the luggage must not obstruct the driver's s view or make the passengers uncomfortable during the trip.
Answer:
Table 14:
One example to illustrate the cntruction-following data. The top block shows the contexts such as captions and boxes used to prompt GPT, and the bottom block shows the three types of responses. Note that the visual image is not used to prompt GPT, we only show it here reference.
23

---

Cuptions
There is movie theater that displays the show times ahove the doors. A rod hydrant deep snow.
The fire hydrant is the snow recendy plowed sidewalk.
This city has had very hard winter with snow.
A hotel for dogs the
winter.
Conversation
Question:
What color the lire hydrant in the image??
Answer:
hydrant image
Question:
Is the fire hydrant fully covered snow partially visible?
Answer:
The fire hydrant is partially visible, with significant portion of it covered by the deep snow.
Question:
What the condition of the sidewalk near the fire hydranc?
Answer:
The sidewalk near the fire hydrant has been recently plowed, relatively clear of compared to the suroannding
=
Question:
What does the image suggest about the winter scason in the city?
Answer:
The image suggests that the winter season in the city been quite harch and challenging. The significant snow accurmalation the image indicates that the city experienced heavy snowfall, which lead various difficulties disruptions for its residents.
For instance, the deep snow covering the fire hydrant could potentially hinder its accessibility in case of emergencies, which poses safety concerns. The recently plowed sidewalk implies that city's 's maintenance crews have been actively working to keep the walkways and safe for pedlestrias,, but the sheer anount of snowfall might make it difficult keep with the continuous snow removal efforts.
Furthermnoe,, such extreme winter conditions can impact tramsportaion, leading trafiic issues, delays in public transit, and slippery roads. also cause problems for challenges commating to work attending classes. Additionally, the heavy snow put strain on infrastructure, such rools and power lines, increasing the likelihood structural damage or power auluges.
accidents
schools, people might
In conclusion, the image of the red fire hydrant deep the snow and the recently plowed sidewalk suggest that the city has faced substantial snowfall that likely caused various challenges and disruptions
particulariy severe residents and infrusiructure.
Table One example used in in-context-lamming to construct visual conversation data.
24

---

Cuptions
A man skiing in the open snow covered hills making his way into snow.
A skier the foothills of huge mountain range.
A skier looks at mountains while standing near trail sign.
single person skiing
with lot hushes
Conversation
Question:
What the skier doing?
Answer:
The skiing in the snow covered making way through snow enjoying beautiful mountain scenery.
They are also standing near a trail sign, which indicates their location on the mountain.
Question:
Are there any other people or skiers visible the image?
Answer:
is only single person skiing image, they are surrounded by area bushes.
Question:
What kind of skiing the person doing? Cress-country or downhill?
Answer:
Based on what can see in the photo, it is not possible to definitively determine whether the skier doing cross-country or downhill skiing. However, given presence of trail sign and the skier engaged in cross-country skinng. Downhill skiing rypically takes place more steep slopes and involves ski lifts, whereas cross-country skiing often occurs on relatively flatter terriin with marked trails.
they are skiing the foothills, more likely that
Question:
Can you explain cross-country and downhill skiing more?
=
Answer:
Cross-country skiing and downhill skiing two distinct types of skiing, each with its own techniques, equipment, and temain preferences.
1. Cross Country Skiing:
Also known Nordic skiing, cross-country skiing involves traversing relatively flat terain gentle slopes using skis and poles. It is a popular rocecational and competitive sport that focuses on endurance, balance, and acrobic fitness. Cross-countty skiers use their own locomotion move across snow-covered temain, There two main tochniques cross-country skiing:
relying on or other forms assistance.
Classic: This technique involves skinng in parallel tracks, using forward- annard-and-aa backward gliding motion while keeping the skis parallel each other. The skier their poles propel themselves forward.
Skate skiing: This technique is similar to ice skating, where the skier pashes off ane ski at an angle and glides on the other. It requires smooth, groomed surface and typically faster than the classic technique.
cross-countty ski equipment consists of lighteeigtt skis, boots, bindings, and poles. The boots are more flexible than those used downhill skinng allowing better movemeni and control.
2. Downhill Skiing:
Also known as alpine skiing, downhill skiing involves descending slopes high speeds skis and for halance and control. This sport is more focused on speed, tochnique, and navigating challenging terrain, including steep slopes, moguls, and jumps.
Downhill skiing can be further categorized into several disciplines. such as slalom, giant slalom, super-G. and downhill racing. equipment.
discipline
rules, courses,
Downhill ski equipment includes heavier and stiffer skis, boots, bindings. and poles than those used in cross-country skiing. The hoots are more rigid to provide better support and control during high speed descents and sharp turns.
In summary, cross-country skiing is an cndurance-based sport that involves traveling across flat or gently sloping temainn while navigate steeper slopes challenging terrain. Both sports require
downhill skingg locused on speed technique specialized equipment and techniques, but they offer different experiences and challenges to participants.
Table 16: One example used in in-context-leaming to construct visual conversation data.
25


Reviews:
[{"review_id": "NLiZoWMlTX", "reviewer": "Reviewer_RuBm", "summary": "This paper introduced LLaVA, an effective visual instruction tuning method to turn Large Language Models (LLMs) into multi-modal LLMs. LLaVA is first pre-trained on image-text pairs to connect a visual encoder (CLIP) and a LLM (Vicuna). Then the authors utilize GPT4 to generate ~150K visual instruction data for training visual instruction-following models. LLaVA is evaluated on two diverse and challenging benchmarks, as well as a science question answering dataset. Overall, LLaVA is a very early attempt for enhancing LLMs with multi-modal capacity. I believe it will inspire a lot to the research community.", "questions": "See Weakness", "limitations": "Yes. The authors have addressed the limitations and potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Expand instruction tuning to the vision domain. Visual instruction tuning is a new research problem for vision-language models. It endows vision-language models with powerful comprehension and reasoning capabilities.\n- A new pipeline for visual instruction data generation. LLaVA takes image captioning or object detection results as GPT4’s input for visual instruction generation. This is an effective way to quickly generate a large amount of visual instruction data.\n- A strong visual instruction model LLaVA with available pretrained models and demos.\n- Multi-modal instruction-following benchmark.", "weaknesses": "- Data quality. The proposed visual instruction data is automatically generated by GPT4. But there seems to be a lack of validation of the data quality. For example, MiniGPT4 [1] will manually verify the correctness of each image description. High quality instruction data is also proved to be important for pure LLMs in LIMA [2].\n- Is 595K image-text data enough for vision-language alignment? Blip2 uses >100M image-text pairs for vision-language alignment, while LLaVA only uses 595K data from CC3M. Have the authors tried to use more pre-training data, and will the model be further improved? Besides, I noticed that LLaVA inputs 256 CLIP visual tokens into the LLM, which is much large than Blip2 and MiniGPT4 (~30 tokens). Such a design will make the training much slower. So, do we really need 256 tokens?\n- The LLM in LLaVA is fully fine-tuned in the second stage. Will this lead to degradation of LLM's ability? Are there verification results on traditional LM tasks?\n[1] Enhancing Vision-Language Understanding with Advanced Large Language Models\n[2] LIMA: Less Is More for Alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Data Quality**\n\nWe agree that high quality instruction data is critical and have taken measures to ensure the data quality.\n\nFirst, we create image descriptions directly from the well-established manually-annotated MSCOCO dataset, which contains bounding box and caption annotations (L103-L110). This ensures the quality of the visual groundness of our textural context input to GPT-4.\n\nSecond, we perform text-based filtering to remove invalid responses: (1) incomplete responses; (2) GPT refuses to provide the answer; (3) contain words that make the answer not sound like it is looking at the images (e.g. according to the captions). We’ll include the detailed list of keywords in the revised appendix.\n\nThird, we iterate and validate our prompts on a subset of around 1000 samples, to validate the visual groundness of the generated outputs using GPT-4, and find that GPT-4 consistently provides higher quality instruction-following data (L128-L130).\n\nFinally, we ablate using LLaVA-Bench-COCO on the combination of different types of generated instruction-following data (Table 4, L217-L227).\n\nNote that MiniGPT-4 is a concurrent work to LLaVA, while we are more than happy to discuss.\n\n> *From the MiniGPT-4 paper: ...we check if each generated image description **follows our desired format**, and also manually refine the generated captions by **eliminating redundant words or sentences** ....*\n\nAccording to the MiniGPT-4 paper, the manual check is only performed to correct **textual format** errors, without mentioning checking the visual groundness of the generated responses.\n\nFurthermore, the source image description of MiniGPT-4 is generated by the first-stage MiniGPT-4. Besides the textual errors that are mentioned in the paper, the correctness and the visual groundness of the generated descriptions of MiniGPT-4 is unclear.\n\nWe believe that our data is more large-scale, diverse, content-rich, and the quality is more controlled.\n\n**Q2. Is 595K image-text pairs enough for vision-language alignment?**\n\nSince CC3M only has around 2M images available to download from the Internet, we choose the BLIP-captioned LAION-CC-SBU dataset (which is the training dataset of BLIP2). Due to the limitation of both resources and time during the rebuttal period, we have tried two subsets: 600K samples and 6M samples. We ablate this with Vicuna-13B using the same schedule as described in the paper, and evaluate on LLaVA-Bench-In-the-Wild. As shown in the paper, when scaling up the pretraining dataset from 600K to 6M, the overall performance on LLaVA-Bench-In-the-Wild does not vary too much (66.8 vs 66.5).\n\n| Pretrain Samples | Conversation | Detail | Complex | All |\n|--|--|--|--|--|\n| 600K | 56.7±3.9 | 54.2±3.1 | 80.2±1.5 | 66.8±0.8 |\n| 6M | 55.1±3.9 | 56.3±2.6 | 79.6±2.9 | 66.5±0.5 |\n\nWe believe the fast alignment of LLaVA can mainly be attributed to two reasons. First, our vision encoder, CLIP, was pretrained with image-text contrastive loss, and its visual feature is thus already aligned to a text space. It is sufficient to re-align this to a different text space using a linear layer. Second, it is much easier and requires fewer samples to optimize the linear layer (5.2M parameters) than the QFormer from BLIP2, which contains 1.1B parameters, orders of magnitude more than LLaVA’s alignment-stage trainable parameters. We thank the reviewer for bringing up this topic, and will include this discussion in the revision.\n\n**Q3. LLaVA inputs 256 CLIP visual tokens into the LLM, which is much larger than Blip2 and MiniGPT4 (~30 tokens). Such a design will make the training much slower. So, do we really need 256 tokens?**\n\nWe are happy to compare with the concurrent work, MiniGPT-4.\n\n*First, will this make the training slower?*\n\nSince LLaVA uses 256 tokens, it is around 4-5x slower than MiniGPT-4 per training iteration. However, since we only need 600K samples to converge, the total pretraining cost of LLaVA is 4 hours on 8x A100s (Supp. L62). MiniGPT-4 pretrains with ~6M image-text pairs and requires training approximately 10 hours on 4x A100s (which roughly equates to 5 hours on 8x A100s). When considering the total training time, LLaVA is slightly faster.\n\n*Second, do we really need 256 tokens?*\n\nThis is an interesting research question open to discussion. Compressing 256 tokens to 32 tokens is a process of information compression. We find that this is detrimental in terms of OCR capability, which is an interesting emergent capability of LLaVA. For example, on a suite of 27 text recognition related academic datasets,  LLaVA consistently outperforms MiniGPT4 on 23 out of 27 datasets, despite LLaVA being trained with an order of magnitude smaller image-text training data. \nWe also qualitatively show in Fig. 2 of Rebuttal Supplementary, that such compression process may discard information that the user is curious about: LLaVA recognizes the website that the image comes from by reading the text from the watermark, while MiniGPT-4 fails.\n\nFurthermore, having finer patch-level features can allow the model to perform region-level reasoning easier, as the region-level information is better preserved and readily extractable for downstream models.\n\n**Q4. Does full-model finetuning lead to the degradation of LLM’s ability?**\n\nWe show that LLaVA and Vicuna are comparable on MTBench[2], and LLaVA is only slightly worse (-0.8%) on MMLU[1].\n\n| | MTBench | MMLU |\n|---|---|--|\n| Vicuna-13B | 6.57  | 55.8 |\n| LLaVA-13B | 6.63  | 55.0 |\n\nWe find this result encouraging, and this can be partially attributed to the inclusion of complex reasoning questions, and long-form answers in LLaVA-Instruct-158K, which helps maintain the language capabilities of LLaVA. We also show that on ScienceQA, it even slightly outperforms Vicuna, on text-only categories. See detail in R4 to reviewer mhaS.\n\n[1] Hendrycks, et al. \"Measuring massive multitask language understanding.\".\n[2] Zheng, et al. \"Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.\".\n"}, {"review_id": "mAndfZYui2", "reviewer": "Reviewer_MLCz", "summary": "This paper studies instruction tuning in the multimodal domain. Instruction tuning has recently drawn a lot of attractions in the large language model (LLM) field, and hence it is interesting and important to study similar capabilities in multimodal models. This paper is a pioneer work in this direction. It constructs the first instruction tuning dataset LLaVA and conducts comprehensive analysis. This is an important step towards general-purpose multimodal language model (MMLM) that can interact with humans using natural languages.", "questions": "What did you use 2e-3 lr for pretraining? This seems to be a very large value given the 128 batch size.", "limitations": "NA", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. To the best of my knowledge, this is the first work of multimodal instruction tuning. This is an important direction and hence the impact of this paper is huge.\n2. The dataset constructed in this paper is both useful and inspiring. It constructed an impactful first step for future studies in this field.\n3. The paper is well written and contains a lot of detailed studies.", "weaknesses": "I don't see any major weakness of the paper. There are some minor ones that can be improved but I understand they may go beyond the scope of this paper. For example, it will be more comprehensive to conduct ablations on the models used in this paper other than Vicuna etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Ablations on other LLMs other than Vicuna.**\n\nUntil the paper submission deadline, Vicuna is the most adopted open-source instruction-tuned LLM. There are other great instruction-tuned LLMs coming out after that, including MPT, LLaMA-2-Chat, etc.\n\nWe present initial studies using these other LLMs on LLaVA-Bench-In-the-Wild below. We also include the performance of the base LLM measured by MMLU and MTBench from the Vicuna leaderboard.\n\n| BaseLLM | MMLU | MTBench | LLaVA-Bench-In-the-Wild |\n|------------|------|---------|----------|\n| MPT-7B | 32      | 5.42 | 53.6 |\n| Vicuna-7B | 49.8    | 6.17 | 63.3 |\n| LLaMA-2-7B-Chat      | 45.8    | 6.27 | 63.2 |\n\nThis initial study exhibits a correlation of the capability of the base LLM and the performance of the resulting multimodal model performance. It will be interesting to dig further into the relationship between the base LLMs and multimodal models. It is worth mentioning that MPT-7B and Vicuna-7B is instruction-tuned with supervised finetuning, while LLaMA-2-7B-Chat is additionally finetuned with RLHF. It will also be interesting to see the influence of RLHF in terms of the multimodal capabilities.\n\nWe believe this initial study will be valuable to the research community to better understand the mechanism and capability of the multimodal models. We will release these results and corresponding model checkpoints to the public.\n\n**Q2. Why do we use a 2e-3 learning rate for pretraining?**\n\nSince we pretrain our model on a small 595K dataset, there are only around 4.5K iterations during training. Given the few steps the model is optimized for, we empirically find a larger 2e-3 learning rate is slightly better for fast convergence than lower learning rates like 2e-4.\n"}, {"review_id": "DovNQKhIqE", "reviewer": "Reviewer_A2dU", "summary": "This presents a multi-modal instruction following model and evaluates it. The model is trained by using a frozen vision encoder whose features are used an input to a LLM, which is fine-tuned. It is trained first on simple captioning tasks using a large amount of data, and then on a multi-modal instruction following dataset built by having a LLM generate tasks using the captions and bounding boxes of images.", "questions": "Would the data generation pipeline benefit from even more detail image descriptions? There are other annotation like region captions or visual narratives that could have also been used. \n\nI am also curious if the model can handle very low-level detailed questions, like \"What color is the rope the man is using?\" for Table 3.\n\n278: year -> layer", "limitations": "The authors have a detailed section in the appendix.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Presents an impressive model that seems to have advanced instruction following abilities for images and text.\n- Present a detailed set of comparison and evaluation using LLaVA-Bench, and the ScienceQA experiments. I also appreciated the re-use of a qualitative example from the GPT-4 paper.\n- Some interesting insights, such as using the second to last layer and getting a sense of the benefit of each kind of instruction-tuning task.", "weaknesses": "- I think it would be valuable to see if this same pipeline could work with an open source LLM, using OpenAI's closed LLMs are not ideal for scientific understanding and reproducibility. I in particular wonder if other LLMs can use the bounding boxes annotation of the images effectively, which I imagine in much harder than the using the captions.\n- Seeing an evaluation on standard vision/language benchmarks would have been interesting, such as zero-shot VQA.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Does the proposed pipeline work with an open-source LLM?**\n\nIn our preliminary study, we find that the capability of the teacher is crucial to the quality of the generated instruction-following data (L128-L130). Until the submission deadline, the largest Vicuna model was 13B. Just as the reviewer’s concern, its complex reasoning and spatial reasoning capability is still limited and is behind proprietary models including ChatGPT and GPT-4. However, the recently released LLaMA-2-70B-Chat appears to have narrowed the gap. Due to the large size of the model, it requires a huge amount of VRAM and has a slow inference speed. We conducted a preliminary study on around 200 samples for each category.\n\nSpecifically, we generate 200 samples for each category (conversation, detailed description, complex reasoning), using LLaMA-2-70B-Chat, ChatGPT, and GPT-4.\n\nAfter generating the response, we find that unlike previous open-source models, LLaMA-2-70B-Chat can start to follow complex instructions like creating multimodal instructions.\n\nHowever, it still fails in the conversation category, as we find that the LLaMA-2-70B-Chat is not correctly following the conversation format. This may be potentially fixed with more sophisticated prompt tuning. However, due to the limited rebuttal period, we do not evaluate on the conversation category. This is also one of the main limitations we find of LLaMA-2-70B-Chat.\n\nWe then quantitatively evaluate the generated instructions using GPT-4 as the judge: (1) the correctness of the answers generated, and (2) the complexity of the instructions generated for complex reasoning questions.\n\n|          | Correctness | Complexity |\n|----------|-------------|------------|\n| LLaMA-2-70B-Chat | 8.7         | 7.4        |\n| ChatGPT  | 9.5         | 9.2        |\n\nThese initial results are promising, and suggest that our pipeline can be potentially applied to open-source models as their capabilities are improved. We look for more comprehensive studies and deeper explorations for future research.\n\n**Q2. Zero-shot VQA**\n\nWe evaluate LLaVA-13B on VQA-v2 and OK-VQA.\n\nNote that for OKVQA, we use a slightly relaxed evaluation protocol for evaluation. Since LLaVA typically outputs a short sentence rather than one or two words, if the generated sentence by LLaVA contains the ground truth answer, we consider it a correct prediction.\nWe find that although LLaVA lags behinds on VQA-v2 (a task where answers can be directly derived from the image), LLaVA performs surprisingly well on OK-VQA (a task where answers require strong knowledge and reasoning) — LLaVA outperforms Flamingo-80B on zero-shot OK-VQA. This is because (1) LLaVA’s advantage lies in its capable LLM; (2) all of the instructions in LLaVA are prompting the model to output a complete sentence, and it thus struggles a bit on those standard benchmarks, which requires answers of one or two words.\n\nWe believe the latter issue can be alleviated by incorporating short-form answers into the instruction tuning data or improving the multimodal in-context learning capabilities, and we leave them to future work.\n\n| Models       | VQAv2 | OKVQA |\n|--------------|-------|-------|\n| PICa (in-context few-shot prompting GPT3-175B) [1]  | --  |48.0  |  \n| Flamingo-80B [2] | 56.3  | 50.6  |\n| LLaVA-13B    | 44.2  | 55.0  |\n\n[1] PICa: An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA\n\n[2] Flamingo: a visual language model for few-shot learning\n\n**Q3. Will the data generation pipeline benefit from even more detailed image descriptions?**\n\nThis is a great suggestion. As we show in Table 4, adding detailed image description to conversation data can improve the model’s capability. We believe that more detailed annotations can further improve the detailedness and the quality of the generated instruction dataset, ultimately resulting in an improved model. Annotations, such as region captions, visual narratives, or scene graphs, are definitely valuable for future research.\n\n**Q4. Can LLaVA handle questions about extremely low-level details?**\n\nCurrently, LLaVA is not able to handle low-level details such as the color of the rope, as the rope is currently very thin and is covering only a few pixels after center cropped and resized to 224x224. However, given that LLaVA is capable of correctly identifying the color of those slightly larger regions (e.g. the clothes that the man is ironing), we believe that scaling up the image resolution, and/or incorporating more detailed descriptions, as mentioned by the reviewer in the comment above, could unlock LLaVA’s low-level detail recognition/reasoning capabilities.\n\n> 278: year -> layer\n\nThanks for pointing out the typo. We’ll fix it in the revision.\n\n"}, {"review_id": "jYKBhHTDUz", "reviewer": "Reviewer_mhaS", "summary": "This paper introduces the first attempt to extend instruction-tuning paradigm to multimodal domain. This work has several major contributions: (a) the curation of the first vision-language instruction-following dataset by converting public image-text pairs into appropriate format using ChatGPT, resulting in over 100K+ multimodal instruction-following samples, (b) results indicating that a multimodal model (consisting of a CLIP visual encoder, a linear projection layer to convert visual tokens into language prompts, and a LLaMA language decoder) trained on this dataset can achieve robust multimodal chatting abilities. All assets used in their research including datasets and models are open-source. ", "questions": "I summarized my most concerned questions about this work:\n- Why did you not sample train/val splits for evaluating LLaVA's multimodal chatting abilities? Are there specific concerns?\n- Is it possible to extend LLaVA's evaluation to other VQA benchmarks (as reported by GPT4) such as VQA2.0 which has balanced language prior?\n- Why is the architecture design of LLaVA more superior than a dense captioning model + instruction-following LLM, if both are trained on the same dataset?\n- What is the sampling procedure of LLaVA? \n\nGiven that this paper presents a significant dataset contribution, I would be happy to revise my rating if the authors can address my above-mentioned weaknesses and questions.\n\nUpdated in Aug 17th: I have increased my rating based on the author's promise to revise the paper by including more discussion on more scientific evaluation metrics and benchmarks for MLLMs.", "limitations": "Yes, the authors discuss about limitations in supplemental material.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 4, "strengths": "There are two major novel contributions of this work: (1) it introduces one of the first large-scale instruction-following multimodal datasets by leveraging public image-text pairs, and (2) it releases all training code, pre-trained models, and evaluation benchmarks to the wider public. These assets (outlined in supplemental L64) are undeniably valuable to the multimodal research community. ", "weaknesses": "I have several major concerns about (a) the evaluation benchmarks and metrics, (b) the lack of simple baselines such as captioning-based approaches, (c) missing implementation details such as the sampling procedure.\n\nA: Issues about quantitative analysis for multi-modal chatting.\n\n- The paper uses rather small evaluation sets (L217-235) to construct the LLaVA-Bench, including 30 randomly selected COCO images and 24 in-the-wild images. Why is this subset is much smaller than the pre-training dataset with 100K+ multimodal instruction-following samples? And how do you select the 24 in-the-wild images? I couldn't find evidence in the current draft to suggest that these 24 images are not cherry-picked.\n\n- The evaluation is text-only and the authors use GPT4 to explicitly assign a score. While prior works such as Vicuna [1] also uses GPT4 to score their responses in a text-only fashion, it is unclear how robust is GPT4 for multimodal reasoning while doing text-only evaluation. For more robust quantitative analysis, I would encourage the authors to split the instruction-following datasets into train/val splits and also include the results of classic text-only scoring metrics. Small-scale human evaluation will also be beneficial.\n\nB: Language prior of ScienceQA benchmark.\n\n- I am shocked that a text-only (vision-blind) GPT4 can achieve as high as 82% accuracy on ScienceQA, suggesting that this particular VQA benchmark has severe language prior [2]. Even though prior works also adopt this benchmark in their evaluation, this makes it hard to interpret the progress achieve by LLaVA towards a truly “multimodal” instruction-following agent as this benchmark can be largely addressed by language prior information.\n- Is it possible to report zero-shot LLaVA performance on ScienceQA?\n\nC: Simple baselines such as dense captioning:\n\n- Even though the model architecture of LLaVA looks elegant as it only uses a linear projection to connect CLIP’s visual tokens to soft language prompts, I believe an even simpler baseline is to train a dense captioning model (using the existing rich descriptions generated by prompting ChatGPT with caption+bounding box information). During inference time, the dense captioner can turn an image into a rich textual description, which can be sent to an instruction-following text-only LLM (Vicuna/GPT4).\n\nD: Missing implementation details such as sampling.\n\n- The sampling procedure (e.g., top-k/nucleus sampling/beam search) can have profound impact on the quality of generated texts. However, the current draft does not discuss how to perform sampling for LLaVA. Also, when using GPT4 for text-only evaluation, the exact hyperparameters used such as temperature should also be reported.\n\nE: Generalization or bias?\n\n- Fig. 5 in appendix suggests that LLaVA is able to generalize to unseen domains, i.e., correctly identifying that the person holding a doge coin is Elon Musk, while Elon Musk does not appear in LLaVA’s training dataset. However, it is unclear whether this is a result of generalization or language bias of LLMs. Perhaps your model tends to answer “Elon Musk” when asking about the name of the person, or perhaps it tends to answer “Elon Musk” when there is a doge coin in the image.\n\n\nOne minor typo:\nL99: “and curate such a questions list” -> “to curate such a list of questions”\n\n\nFinally, I have an doubt about \"multimodal instruction-following\" (this is not a weakness but open to discussion): \n\n- Studies in NLP such as [3,9] have suggested that instruction-following is effective mostly because LLMs such as LLaMA are already capable foundation models, and therefore instruction-following can effectively align the model output with human interest. However, it is unclear whether multimodal foundation models such as CLIP (as used in LLaVA) is powerful enough. For example, a wide range of recent works and benchmarks [4,5,6,7,8] suggest that CLIP behaves like bag-of-words and do not have strong vision-language reasoning capabilities. As we do not yet have strong enough vision-language foundation models, it is unclear if the multimodal research community is ready to embrace the instruction-following paradigm. \n\n[1] Vicuna. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.\n\n[2] Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. Goyal et al. 2016.\n\n[3] LIMA: Less Is More for Alignment. Zhou et al. 2023.\n\n[4] When and why vision-language models behave like bags-of-words, and what to do about it? Yuksekgonul et al. 2022.\n\n[5] Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality. Thrush et al. 2022.\n\n[6] CREPE: Can Vision-Language Foundation Models Reason Compositionally? Ma et al. 2022.\n\n[7] Equivariant Similarity for Vision-Language Foundation Models. Wang et al. 2023.\n\n[8] Visio-Linguistic Reasoning with Multimodal Generative Pre-Training Scores. Lin et al. 2023.\n\n[9] The False Promise of Imitating Proprietary LLMs. Gudibande et al. 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Why do we use a small LLaVA-Bench-COCO split with only 30 images?**\n\nSince we divide the question into three categories, we have 90 questions for COCO and 60 questions for In-the-Wild. The amount of the questions in our test set is similar to Vicuna-Bench [1], which has 80 questions. The reason we consider small evaluation sets is due to the cost using GPT-4 evaluation, the current size allows more accessibility for e.g., university labs to report results.\n\n**Q2. How do we select the 24 in-the-wild images?**\n\nDifferent from exsiting benchmark that evaluates a single specific capablity of multimodal models, we are seeking for a benchmark, where each evaluation sample can measure multiple integrated capabilities of a model. Such capabilities include recognition, OCR, knowledge, language generation, spatial awareness, counting, etc.  To design a benchmark covering a wide range of capabilities, while keeping the evaluation cost affordable, we find 24 samples that require multiple multimodal capabilities. Besides, by using the description-based annotations for each image, we are able to create and extend more questions to improve the capability coverage.\n\n**Q3. Split to train/val splits**\n\nLLaVA-Bench-COCO follows the train/val split in machine learning. LLaVA-Instruct-158K uses COCO train 2014 images and annotations, while it samples 30 validation images from COCO val 2014 (L217). Both follow the same data generation pipeline (Sec. 3). It serves as validation to study model alignment and capabilities with consistent visual inputs (L219-L220). A small size is chosen, similar to minival split practice, for quick evaluation / iteration during development.\n\n**Q4. Language-prior in ScienceQA and zero-shot performance**\n\nScienceQA has three question modes: text, image, and no context (Table 7, TXT/IMG/NO).\n\nThe default evaluation pipeline of ScienceQA includes a \"random guess\" mechanism, which helps text-only GPT-4 but not LLaVA. Without this, GPT-4's IMG accuracy drops to 59%, suggesting a more balanced language prior.\n\nAs suggested by the reviewer, we provide two further experiments: (1) zero-shot performance of LLaVA and Vicuna; (2) Vicuna finetuned on ScienceQA.\n\nSince open-source models like Vicuna and LLaVA are still not good at following “format” instructions like \"conclude your answer with `The answer is`\", we use ChatGPT-3.5 to reformat the answer, following the recent practice in multimodal evaluation.\n\nWe show that zero-shot LLaVA outperforms both GPT-4 (+7.5%) and Vicuna (+10.3%) on IMG modality. Besides, zero-shot LLaVA consistently outperforms zero-shot Vicuna, in all categories including text-only questions.\nWe also show that the finetuned LLaVA outperforms the finetuned Vicuna in almost all categories, with an average performance gain of 5.2%.\n\n| ZeroShot| NAT | SOC| LAN| TXT| IMG| NO| G1-6 | G7-12 | AVG|\n|--|--|--|--|--|--|--|--|--|--|\n| GPT-4 | 77.44 | 64.23 | 86.45 | 74.73 | 59.05 | 90.38 | 78.49 | 74.36 | 77.01 |\n| Vicuna | 66.83 | 60.63 | 69.00 | 65.69 | 56.27 | 71.71 | 68.94 | 60.98 | 66.09 |\n| LLaVA  | 71.27 | 74.24 | 70.91 | 70.43 | 66.53 | 72.89 | 75.4 | 65.33 | 71.8 |\n\n| Finetuned | NAT | SOC| LAN| TXT| IMG| NO| G1-6 | G7-12 | AVG|\n|--|--|--|--|--|--|--|--|--|--|\n| Vicuna | 86.9 | 79.3 | 88.55 | 85.58| 76.85 | 91.43 | 85.68 | 85.83 | 85.73 |\n| LLaVA | 90.36 | 95.95 | 88 | 89.49| 88 | 90.66 | 90.93 | 90.9 | 90.92 |\n\nWe'll include these results and discussion in the revision.\n\n**Q5. Dense captioning model + LLM vs end-to-end**\n\nThere are two conceptual advantages.\n\n- Completeness of image representations. Dense captioning may not be able to capture all details of an image that the user instruction is concerned with. In contrast, an end-to-end multimodal model can be instruction-aware and only focus on the relevant visual contents via attention mechanism. See Rebuttal Fig. 2 for a qualitative example.\n- Single model. End-to-end models save computational resources and reduce complexity in model serving and request handling, which is beyond the scope of this paper.\n\n**Q6. Sampling Details**\nWe set LLaVA's temperature to 0.2, and tuning other hyper-parameters does not further improve the quality. Beam search improves LLaVA-13B output quality (67.3->69.8), while it is not trivially compatible with real-time UI like ChatGPT. We report all numbers without the beam search to make the evaluation consistent with the user interface. For GPT queries, we follow Alpaca to set the temperature and top_p to 1.0. We will include this in revision.\n\n**Q7. Elon Meme: Generalization or bias**\nThis is a great question that is worth further studying. We designed two sets of images in Rebuttal Fig. 1 to verify. We find this study intriguing and it further supports it being a form of generalization of our model.\n\n**Q8. Is CLIP powerful enough for multimodal instruction-following?**\n\nThanks for bringing up this interesting topic. \n\nWe agree with that instruction tuning plays a more important role in guiding capable foundation models to follow human intents, rather than adding new knowledge. In multimodal settings, we leverage the existing capabilities of two models: LLM for language knowledge and CLIP for image-text alignment, rather than making each individual of them stronger. LLaVA tuning largely aligns the capable foundation LLM to understand image-related human intent. Interestingly, new emerging properties occur when combining the two existing capabilities like OCR in the wild.\n\nGrid features or the pooled features on [CLS] token of CLIP on their own may lose spatial information (behave like bag-of-words). However, in LLaVA’s design, we feed the raw image patch features into the LLM, and the positional embeddings of LLM are incorporated into the visual token representations, implicitly maintaining the spatial information of the visual inputs.\n\n[1] Chiang, et al. \"Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.\"."}]

Author rebuttal:
We sincerely thank all the reviewers for their time and their thoughtful comments and questions. We are encouraged that the reviewers find that:

- Our work is a pioneer in the multimodal instruction tuning field (RuBm, MLCz, A2dU, mhaS). It will inspire a lot to the research community (RuBm) and have a huge impact on this field (MLCz).
- We have made significant contributions, including
    - an inspiring pipeline for multimodal instruction data generation (RuBm,MLCz)
    - one of the first large-scale vision-language instruction-following datasets (mhaS) and a multimodal instruction-following benchmark (RuBm)
    - LLaVA, a strong visual instruction model (RuBm) with elegant designs (mhaS) and impressive instruction following capabilities for images and text (A2dU).
    - fully open-sourced assets that are undeniably valuable to the multimodal research community (mhaS).
- The paper is well written (MLCz), contains some interesting insights (A2dU), and has a detailed set of comparisons and evaluations (MLCz,A2dU).

We attempted our best to address the questions as time allowed. We believe the comments & revisions have made the paper stronger and thank all the reviewers for their help. Please find individual responses to your questions below.

Meta-review: This paper introduces LLaVa, an multimodal large language model that visual instructionally fine-tuned.  LLaVA is first pre-trained on image-text pairwise data to align the image encoder and a pre-trained LLM. In the second step, the authors fine-tune the model using ~150k visual instruction data automatically generated by GPT4. All code and data are all open sourced to the research community.

The reviewers comment out this paper is one of the first work of multimodal instruction tuning, pioneering on the direction of tuning a pre-trained LLM for a better cross-modal alignment and understanding.  One major contribution is the 150k multimodal instruction tuning data. The idea of using GPT4 to generate instruction following data for multimodal is interesting and inspiring. The results are also strong and encouraging, indicating the approach of instruction tuning a LLM to accept multimodal information is a promising direction. Although one weakness, as reviewer A2dU pointed out,  is that the current dataset construction process is largely depend on the closed-source GPT4, and it still remains unclear how this can be fully reproduced using open sourced LLMs. The paper still deserves its merit to pioneering the approach and excellent experience to demonstrate the path is valid.

The AC carefully reads the submission, comments, and authors’ responses. The paper is also well written and structured well. Authors have been very engaged and answer all reviewers’ question well with reasonable thoughts and numbers. The AC generally agrees the reviewer’s comment and appreciates the fruitful discussions. The paper is an important step towards general purpose multimodal language model. The open sourced data and code will also benefit the community and motivate the research in this direction.

Therefore, the AC recommends to accept this paper.
Decision: Accept (oral)