Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning (2024)

Abstract: Despite vision-language models’ (VLMs) remarkable capabilities as versatile visual assistants, two substantial challenges persist within the existing VLM frameworks: (1) lacking task diversity in pretraining and visual instruction tuning, and (2) annotation error and bias in GPT-4 synthesized instruction tuning data. Both challenges lead to issues such as poor generalizability, hallucination, and catastrophic forgetting. To address these challenges, we construct VISION-FLAN, the most diverse publicly available visual instruction tuning dataset to date, comprising 187 diverse tasks and 1,664,261 instances sourced from academic datasets, and each task is accompanied by an expert-written instruction. In addition, we propose a two-stage instruction tuning framework, in which VLMs are firstly finetuned on VISION-FLAN and further tuned on GPT-4 synthesized data. We find this two-stage tuning framework significantly outperforms the traditional single-stage visual instruction tuning framework and achieves the state-of-the-art performance across a wide range of multi-modal evaluation benchmarks. Finally, we conduct in-depth analyses to understand visual instruction tuning and our findings reveal that: (1) GPT-4 synthesized data does not substantially enhance VLMs’ capabilities but rather modulates the model’s responses to human-preferred formats; (2) A minimal quantity (e.g., 1,000) of GPT-4 synthesized data can effectively align VLM responses with human-preference; (3) Visual instruction tuning mainly helps largelanguage models (LLMs) to understand visual features. Our dataset and models are publicly available at https://github.com/VT-NLP/ Vision-Flan.

heading: 1 Introduction
text: Recent vision-language models (VLMs) (Liu et al., 2023e; Li et al., 2023d; Dai et al., 2023), built upon pre-trained large-language models (LLMs) (Chiang et al., 2023; Gao et al., 2023) and pretrained
image encoders (Sun et al., 2023), have shown impressive capabilities as general visual assistants. Besides the unimodal encoders, the main ingredients of these VLM frameworks encompass: (1) a bridging module, such as the MLP layers in the LLaVA model (Liu et al., 2023e; Li et al., 2023d), that establishes connections between the pretrained image encoders and LLMs, (2) large-scale textimage pairs (Schuhmann et al., 2022) used for pre-training the bridging module, and (3) GPT-4 synthesized visual instruction tuning datasets (Liu et al., 2023e; Li et al., 2023b) to align the responses of VLMs with human preferences (i.e., following users’ instruction to generate detailed and helpful responses). Despite their notable successes, we identify two remaining challenges that merit further investigation.
Firstly, the data used in the pre-training stage is dominated by the image captioning task, which lacks diversity, resulting in limited generalizability of VLMs (Chen et al., 2023c; Zhang et al., 2023). For instance, the LLaVA model (Liu et al., 2023e) performs poorly on the optical character recognition (OCR) task due to the absence of instances related to text detection during pre-training (Zhang et al., 2023). Several recent studies address this problem by further fine-tuning VLMs on instruction tuning datasets covering more tasks (Zhang et al., 2023; Hu et al., 2023; Liu et al., 2023d) such as visual question answering and OCR but the coverage of the tasks is still limited.
Secondly, most of the existing visual instruction tuning datasets (Liu et al., 2023e; Li et al., 2023b; Yin et al., 2023) are synthetically generated via GPT-4 by repurposing text annotations such as captions or dense captions from existing computervision datasets to generate new tasks, such as visual dialogue, Complex VQA and detail captions. While they enable VLMs to generate fluent and detailed responses aligned with human preferences, the lack of task diversity, spurious co-occurring pat-
15271
terns between objects, and long-form outputs may cause severe hallucination (Liu et al., 2023c; Li et al., 2023g; Liu et al., 2023a; Zhou et al., 2023), and catastrophic forgetting – VLMs fail to maintain a similar classification performance on basic detection tasks, such as MNIST (LeCun, 1998) and CIFAR-10 (Krizhevsky et al., 2009), compared to the zero-shot performance of their vision encoders (Zhai et al., 2023).
To address both challenges, we introduce VISION-FLAN, the most diverse public-available visual instruction tuning dataset consisting of 187 tasks drawn from academic datasets, covering perception tasks such as object detection and OCR, domain-specific tasks such as image-quality classification and image-style classification, complex reasoning tasks such as graph understanding and geometric question answering, and many more. Each task in VISION-FLAN is accompanied by an expertwritten instruction. We show some sample tasks from VISION-FLAN in Figure 1 and provide the full list of tasks in Appendix J.
In addition, we introduce a two-stage instruction tuning framework. In the first stage, we utilize the pre-trained LLaVA model (Liu et al., 2023e) as our initial model, and finetune it on VISION-FLAN to gain diverse capabilities, resulting in the VISIONFLAN BASE model. However, due to the concise nature of target outputs in academic datasets, the
responses generated by VISION-FLAN BASE tend to be brief and not aligned with human preferences. Therefore, in the second stage, we further finetune VISION-FLAN BASE using a minimal amount of GPT-4 synthesized data. This step aims to adjust the model’s outputs to be more in line with human preferences, resulting in the VISION-FLAN CHAT model.
Our experimental results demonstrate that highquality human annotations from VISION-FLAN significantly enhance the capabilities of both VISIONFLAN BASE and VISION-FLAN CHAT while reducing the risk of hallucination and catastrophic forgetting. The two-stage instruction tuning framework enables VISION-FLAN CHAT to achieve better human preference alignment using much less GPT-4 synthesized data compared to the state-of-the-art VLMs. Finally, we perform extensive analysis to understand visual instruction tuning including the roles of human-labeled and GPT-4 synthesized data, and the impacts of various training strategies. Our investigation yields several key insights:
• Increasing the number of human-labeled tasks in visual instruction tuning can substantially enhance VLMs’ capabilities across extensive evaluation benchmarks. • GPT-4 synthesized data does not substantially enhance VLMs capabilities and yields marginal improvements in the VLMs’ perfor-
mance on comprehensive evaluation benchmarks, such as MME (Fu et al., 2023) and MM-Bench (Liu et al., 2023f).
• A minimal quantity (1,000) of GPT-4 synthesized data is sufficient to align VLMs’ responses with human preferences. Notably, increasing GPT-4 synthesized data does not correspond to a proportional enhancement in alignment and introduces hallucination and bias into the VLMs.
• Visual instruction tuning mainly enhances the ability of large-language models (LLMs) to process and understand visual features. The training of the bridging module, which maps visual features to the embedding space of LLMs, is predominantly achieved during the pre-training phase.

heading: 2 VISION-FLAN

heading: 2.1 Collection Pipeline
text: We carefully design an annotator selection process to identify qualified annotators, which involves 2 iterations of training and testing. More details of the selection process and compensation can be found in Appendix A.1. In the end, we hire 7 out of 21 candidates as our annotators and all of them are graduate students in computer science. To ensure the diversity and quality of the tasks in VISIONFLAN, we design a rigorous annotation pipeline with four major steps:
Existing dataset collection and pre-processing: Two expert researchers (i.e., senior Ph.D. students in the fields of natural language processing and computer vision) search online and identify highquality vision-language datasets. The datasets are then equally distributed to 7 annotators to download and preprocess the datasets. Each processed instance consists of an image, an instruction (the task definition from the original dataset with minor modifications), a text input if applicable, and a target output.
Creating new tasks: The two expert researchers and annotators also discuss potential new tasks that could be derived from the existing annotations. We derive new tasks by combining the annotations of two or more existing tasks on a dataset. For example, in the Concadia dataset (Kreiss et al., 2022), each instance consists of an image caption and a knowledge snippet related to the image. We propose a new task to predict both the caption and the
background knowledge given an image, which is a free-form generation task. The new target output is formed by concatenating the caption with the knowledge snippet. We also develop new tasks by creating more basic versions of the original tasks. For example, given the object detection annotations in MSCOCO (Lin et al., 2014), we propose an object selection task in which we provide a list of objects and ask the model to select the object that appears in the image (the negative options are created by sampling objects that appear in other images but not in the given image). The expert researchers and annotators manually solve 20 instances for each newly developed task. If the human predictions match the target outputs, this new task is considered valid.
Iteratively refining the task instructions and output templates: For existing tasks, we ask annotators to write instructions based on the original task definitions with minor modifications. For newly developed tasks, the annotators write instructions by discussing with the expert researchers. Once an annotator finishes writing a new instruction, one of the two expert researchers is randomly assigned to examine the instances and provide feedback for revising the instruction. This step iterates repeatedly until the instruction meets our requirements. We require the instruction to be clear, easy to understand, and can be correctly executed by a human. Each task together with its associated dataset and instruction is then added to the pool of candidate tasks for VISION-FLAN.
Verifying the quality of each task: From the candidate task pool, two expert researchers, including a native English speaker, work together to select the high-quality tasks where the instruction is fluent and effectively conveys the intended task and the task does not overlap with other tasks.
Based on these four steps, we finally collect 187 high-quality tasks, and for each task, we randomly sample 10,000 instances from its corresponding dataset. If a dataset contains less than 10,000 instances, we include all of them. We name the dataset as VISION-FLAN, consisting of 1,664,261 instances for 187 tasks in total. We include references to all the datasets used in VISION-FLAN in Appendix H and show an instance for each task in Appendix J.
Dataset Instances # Tasks # Source
LLaVA (Liu et al., 2023e) 150K 3 Synthetic LAMM (Yin et al., 2023) 196K 8 Synthetic VL-Qwen (Bai et al., 2023a) 350K Unknown Private M3IT (Li et al., 2023e) 2.4M 40 Synthetic mPlug-Owl (Ye et al., 2023) 150K 3 Synthetic Shikra (Chen et al., 2023a) 156K 4 Synthetic SVIT (Zhao et al., 2023) 4.2M 4 Synthetic MultiInstruct (Xu et al., 2023) 510K 62 Public VISION-FLAN (Ours) 1.6M 196 Public
Table 1: Comparison between VISION-FLAN and existing visual instruction tuning datasets.

heading: 2.2 Comparison with Existing Datasets
text: Table 1 presents a comparison between existing visual instruction tuning datasets and VISION-FLAN. For existing visual instruction tuning datasets, we directly adopt the numbers of tasks and instances reported in their original papers. The majority of these datasets are generated using proprietary language models, such as ChatGPT1 and GPT-42, and exhibit a narrow range of task diversity. VLQwen (Bai et al., 2023a) is a recently introduced large-scale dataset annotated by humans but remains inaccessible to the public. Although MultiInstruct (Xu et al., 2023) is based on publicly available datasets, it mainly focuses on visual grounding tasks and only contains 29 tasks that do not involve region-specific information. In contrast, VISIONFLAN encompasses a significantly more diverse array of tasks, offering a three-times increase compared to the number of tasks in MultiInstruct.
In Figure 2, we compare the task categories covered by VISION-FLAN and other datasets. Tasks within VISION-FLAN are first categorized into
1https://openai.com/blog/chatgpt 2https://openai.com/research/gpt-4
three primary groups: Question Answering, Classification, and Generation, and each of these primary groups is further divided into specific, fine-grained categories. For instance, within the Classification group, the General Object category involves classifying objects in images into various concepts, such as “fish”, “car”, and “dog”. Contrastingly, the Vehicle Model category demands the models to accurately identify specific car brands or models, like “Toyota” and “Camry”. The visualization in Figure 2 clearly demonstrates the superior diversity and volume of tasks in VISION-FLAN compared to existing datasets. We list tasks in each category in Appendix I.

heading: 3 VISION-FLAN Finetuning
text: Model Architecture We adopt the same VLM architecture as LLaVA (Liu et al., 2023d) and denote it as LLaVA-Architecture. As shown in Figure 3, it consists of a pre-trained vision encoder, a pre-trained large language model, and two layers of MLPs to connect them. In the vision-language pre-training phase of the LLaVAArchitecture, both the pre-trained vision encoder and large language model remain frozen, and only the MLP layers are trained on a large-scale image captioning dataset (Schuhmann et al., 2022). We leverage this pre-trained LLaVA model , without any visual instruction tuning, as our initial model and finetune it on VISION-FLAN. During visual instruction tuning, we finetune both the MLP layers and the language model while keeping the vision encoder frozen.
Two-stage Visual Instruction Tuning Contrary to prior approaches (Liu et al., 2023d; Dai et al.,
2023) that mix human-labeled data with GPT-4 synthesized data for visual instruction tuning, our study introduces a two-stage instruction tuning pipeline. As shown in Figure 3, in the first stage, we finetune the VLM on all 187 tasks of VISION-FLAN to acquire diverse capabilities and name the resulting model as VISION-FLAN BASE. However, due to the brevity of target outputs presented in academic datasets, the responses from VISION-FLAN BASE are not in human-preferred formats. Hence, we further finetune VISION-FLAN BASE on GPT-4 synthesized data to align the model’s outputs with human preference. We denote the yielded model as VISION-FLAN CHAT. This training framework requires minimal GPT-4 synthesized data while providing deep insights into the distinct contributions of human-labeled and GPT-4 synthesized data in visual instruction tuning.
Implementation Details We leverage LLaVAArchitecture with Vicuna-13B v1.5 (Chiang et al., 2023), CLIP-ViT-L-336px (Radford et al., 2021) and two layers of MLP as our VLM. For the firststage instruction tuning, we finetune the MLP layers and the language model on VISION-FLAN for 1 epoch with a learning rate 2e-5 and per device batch size 16 on 8 A100 GPUs. For the second-stage instruction tuning, we further finetune the MLP layers and the language model on 1,000 instances randomly sampled from the LLaVA dataset (Liu et al., 2023e) with learning rate 1e-5 and per device batch size 8 on 8 GPUs for 128 steps. In the following sections, we use LLaVA dataset and GPT-4 synthesized data interchangeably.

heading: 4 Experiment Setup
text: Evaluation Datasets We evaluate the models on several widely adopted multimodal evaluation benchmark datasets including multiple-choice
Evaluation Protocols For MMbench, MME, MM-Vet, LLaVA-Bench, POPE and MMMU, we strictly follow their official implementations of evaluation code to evaluate the performance of each model. For datasets that do not have official evaluation codes including CIFAR-10, CIFAR-100, MNIST, and miniImageNet, we leverage the stateof-the-art open-source LLM, Vicuna 1.5 13B, to perform the evaluation and report the averaged performance on these four datasets in the CF column in Table 2. More details of evaluation protocols can be found in Appendix C.
Baselines We compare our models with several recent state-of-the-art vision-language models, including BLIP-2 (Li et al., 2023d), InstructBLIP (Dai et al., 2023), Shikra (Chen et al., 2023a), LLaVA (Liu et al., 2023e), Qwen-VL, Qwen-VL-Chat (Bai et al., 2023b), and LLaVA1.5 (Liu et al., 2023d). The LLMs and image encoders used in all baselines are shown in Table 2. Details of baselines can be found in Appendix D.

heading: 5 Results and Analysis

heading: 5.1 Main Results
text: As demonstrated in Table 2, VISION-FLAN BASE achieves state-of-the-art performance on comprehensive evaluation benchmarks including MME, MM-Bench and MMMU, while reducing hallucination and catastrophic forgetting. However, we observe that VISION-FLAN BASE scores significantly lower on the LLaVA-Bench dataset in comparison to VLMs trained using GPT-4 synthesized data. We attribute this discrepancy to the conciseness and brevity of target outputs within academic datasets. As shown in Figure 1, VQA tasks frequently yield outputs comprising a single or a few words. Even outputs of many generation tasks are typically confined to one or two succinct sentences. Training on these tasks leads VISION-FLAN BASE to generate brief responses, which are not aligned with human

heading: 5.2 Effect of Human-Labeled and GPT-4 Synthesized Datasets
text: Effect of Task Diversity in VISION-FLAN Figure 4 illustrates the relationship between the number of tasks from VISION-FLAN employed during visual instruction tuning and the performance of VISION-FLAN BASE across four comprehen-
Effect of GPT-4 Synthesized Data on Comprehensive Evaluation Benchmarks Furthermore, we analyze if GPT-4 synthesized data can improve the model’s performance on comprehensive evaluation benchmarks and show the results in Figure 5. Further tuning VISION-FLAN BASE on GPT-4 synthesized data instances does not lead to performance improvement. Tuning pretrained LLaVA model on a small amount of GPT-4 synthesized data (100) can improve its performance on MME but further increasing the number of training instances does not lead to any improvement. We also observe a similar trend on the MM-Bench dataset and report the result in Appendix E.1. These observations are in line with recent findings in LLMs: GPT-4 synthesized data does not improve model’s

heading: GPT-4 Synthesized Data Causes Hallucination
text: and Bias Concurrent work (Liu et al., 2023c)

heading: 5.4 Effect of Newly Created Tasks

heading: 5.3 Single-stage Tuning on Mixed Data Vs. Two-stage Tuning

heading: 5.5 Contributions of Tasks from Different Task Groups
text: Understanding the contribution of each task to the models’ performance is crucial for visual instruction tuning. However, training a model on each task in VISION-FLAN can impose significant computational cost. Instead, we propose a group-level analysis of task contributions. As illustrated in Figure 2, tasks are categorized into three primary groups: Question Answering (QA), Classification, and Generation. We train three VISION-FLAN CHAT models on VISION-FLAN, each time excluding all tasks from one primary group, and show their performance in Table 6.
Model Name MME Pope LLaVA-Bench MM-Vet
VISION-FLAN CHAT 1537.8 86.1 78.3 38.0 w/o QA 1211.6 83.3 73.5 37.9 w/o Classification 1376.6 84.9 75.4 37.3 w/o Generation 1390.9 83.8 68.1 36.2
Table 6: Contributions of different tasks group to the performance of VISION-FLAN CHAT.
Our findings indicate that: (1) the inclusion of generation tasks markedly enhances the model’s capability in free-form generation tasks, as evidenced by performance on LLaVA-Bench; (2) QA tasks and classification tasks contribute significantly to model’s performance MME, and (3) synergistic interactions between different task groups lead to further performance enhancements.

heading: 5.6 What is Essentially Improved in VLMs during Visual Instruction Tuning
text: In LLaVA-Architecture, the MLP layers map the visual features from a vision encoder into the embedding space of LLMs. The LLMs then interpret the
LLM MLPs MM-Bench MME LLaVA-Bench Pope
✗ ✗ 45.0 936.3 32.4 51.9 ✗ ✓ 52.4 1107.3 39.1 83.3 ✓ ✗ 69.2 1495.5 39.3 85.6 ✓ ✓ 69.8 1537.8 38.5 85.9
Table 8: Results of replacing visual instruction tuned MLPs with pretrained MLPs. Gray rows show the performance of the original models and yellow rows show the performance after replacing instruction-tuned MLPs with pretrained MLPs.

heading: 6 Related Work
text: Instruction tuning (Wei et al., 2022) is first introduced in NLP and has been adapted to the visuallanguage domain. MultiInstruct (Xu et al., 2023) propose the first human-label multi-modal instruction tuning dataset for improving the zero-shot performance of pre-trained VLMs. LLaVA (Liu et al., 2023e) leverage GPT-4 to repurpose text annotations such as captions or dense captions from
existing computer-vision datasets to generate visual dialogues, Complex VQA and detail captions for visual instruction tuning. Following LLaVA, mPLUG-Owl (Ye et al., 2023), LAMM (Yin et al., 2023), MIMIC-IT (Li et al., 2023a) and MacawLLM (Lyu et al., 2023) leverage proprietary LLMs such as GPT-4 and ChatGPT to further extend the instruction tuning tasks into 3D-domain, multipleimages and videos, and increase the amount of training instances. MiniGPT-4 (Zhu et al., 2023) utilizes ChatGPT to refine output from the pretrained VLM itself. InstructBLIP (Dai et al., 2023) and LLaVA-1.5 (Liu et al., 2023d) mix the humanannotated and GPT4 synthesized datasets to enhance visual instruction tuning.
Several recent work explores different strategies to improve visual instruction tuning. StableLLaVA (Li et al., 2023f) and VPG-C (Li et al., 2023c) generate both images and texts using Stable Diffusion (Rombach et al., 2022) or Blended Diffusion (Avrahami et al., 2022) to alleviate domain bias and encourage VLMs attend to visual details. (Liu et al., 2023b) demonstrate the bias introduced by positive instructions and introduce negative instruction examples for improving robustness. Shikra (Chen et al., 2023a) incorporate visual grounding tasks in visual instruction tuning to improve the VLM’s referential capability. LLaVAR (Zhang et al., 2023) and BLIVA (Hu et al., 2023) leverage OCR tools and GPT-4 to generate tasks helping VLMs to understand text in images. (Lu et al., 2023) and SVIT (Zhao et al., 2023) empirically study the effect of scaling the size of VLMs and the size of GPT-4 synthesized dataset. Two concurrent works (Wang et al., 2023a; Chen et al., 2023b) directly prompt GPT-4V with images as input to generate visual instruction tuning data and achieve superior performance. Additional related work can be found in Appendix G.
Unlike all prior work, our work mainly focuses on scaling human-labeled tasks in visual instruction tuning to improve VLMs’ capabilities. Additionally, we perform extensive analysis to understand the characteristics of human-labeled and GPT-4 synthesized data and draw meaningful conclusions.

heading: 7 Conclusion
text: We construct VISION-FLAN, the most diverse public-available visual instruction tuning dataset, consisting of 187 diverse tasks and 1,664,261 instances collected from academic datasets, and each task is accompanied by an expert-written
instruction. We demonstrate that VLMs trained on VISION-FLAN with proposed two-stage tuning framework achieve state-of-the-art performance on comprehensive evaluation benchmarks. Additionally, we perform extensive analysis and reveal the distinct contributions of human-labeled and GPT-4 synthesized data in visual instruction tuning.

heading: 8 Limitations
text: All the tasks included in VISION-FLAN are in English, which confines the usage of our dataset and models to English speaking populations. Future work should extend VISION-FLAN with multilingual tasks. In addition, all the tasks in VISIONFLAN only consists of a single image. Many realworld vision-language tasks require the model to take multiple images as inputs, such as script generation (Qi et al., 2024) and multimodal entity linking (Yao et al., 2024). Thus, future work should explore vision-language tasks that involve multiple images or videos.
Our analysis mainly focuses on the GPT-4 synthesized visual instruction tuning dataset. Recently, as the GPT-4V 3 becomes publicly available, there are some concurrent works (Wang et al., 2023a; Chen et al., 2023b) prompting GPT-4V with images as inputs to generate visual instruction tuning data. Future work can analyze the effect of tuning VLMs on such datasets and identify the advantages and disadvantages.
In our experiments, we mainly focus on the LLaVA-Architecture (Liu et al., 2023e) due to its strong performance and high efficiency. However, there are other foundation architectures such as Qformer in BLIP2 (Li et al., 2023d) and Perceiver Resampler in Flamingo (Alayrac et al., 2022). More diverse VLM architectures can be explored in the future to draw more general conclusions.

heading: Acknowledgments
text: This research is based upon work supported by the U.S. DARPA ECOLE Program #HR001122S0052. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental
3https://openai.com/research/ gpt-4v-system-card
purposes notwithstanding any copyright annotation therein.

heading: A More Details on the Annotation Process of VISION-FLAN

heading: A.1 Annotator Selection
text: Due to the complexity of the annotation task, we carefully design a selection process to select qualified annotators. Specifically, at beginning, the authors send out emails looking for graduate students in computer science who are interested in NLP and multi-modal learning. A group of 21 graduate computer science students signed up for a tutorial section. In the tutorial section, two PhD students in NLP explain the requirements for writing instructions, downloading the dataset and processing raw datasets into a unified format. After the tutorial, each candidate is assigned with three datasets and they have totally three days to process the raw datasets and write instructions. In the end, each candidate submits their annotations and two PhD students provide feedback to each candidate. The candidates then have two days to modify their instructions or formats based on the feedback. After two days, the candidates submit their final version of annotations and two PhD students discuss the quality of the annotations case by case. In the end, 7 out of 21 students were selected as qualified annotators. The compensation is 15$ per hour.

heading: B Evaluation Datasets
text: We evaluate our models on several widely used multimodal evaluation benchmark datasets: (1) MMbench (Liu et al., 2023f) is a comprehensive evaluation benchmark measuring VLM’s capabilities from 20 dimensions. (2)MME (Fu et al., 2023) measures VLM’s perception and cognition capabilities based on 14 diverse tasks. (3) MM-Vet (Yu et al., 2023) focuses on measuring the integration
of various capabilities of VLMs, including OCR, recognition, knowledge, spatial awareness, math, and language generation. (4) LLaVA-Bench (Liu et al., 2023e) evaluates the instruction following and chat ability of VLMs in diverse daily-life visual tasks. (5) POPE (Li et al., 2023g) is an evaluation benchmark that probes object hallucination in VLMs. (6) MMMU (Yue et al., 2023) evaluates VLMs on multi-discipline tasks that require college-level subject knowledge and deliberate reasoning.
We also evaluate the newly proposed catastrophic forgetting problem (Zhai et al., 2023) of VLMs on 4 datasets: CIFAR-10 and CIFAR100 (Krizhevsky et al., 2009), MNIST (LeCun, 1998), and miniImageNet (Vinyals et al., 2016). We report the averaged performance of VLMs on the four benchmarks in the CF column of Table 2.

heading: C Evaluation Protocols
text: For MM-Bench, MME, MM-Vet, LLaVA-Bench, POPE and MMMU, we use their official implementations of evaluation code4 to evaluate the performance. Specifically, the evaluation scripts of MMbench and MM-Vet call GPT-4 API to evaluate the correctness of a prediction given the target output and produce a binary score (0 or 1). Similarly, the evaluation of LLaVA-Bench also leverages GPT-4, and in addition to the target outputs, the evaluation method considers detail descriptions of images. The evaluation results are scores indicating not only the correctness but the human-preference of the predictions. MME and POPE are binary classification tasks and their evaluation is based on string matching between the predictions and target labels.

heading: D Baselines
text: We compare our method with recent visionlanguage models. All the baselines listed below have similar architectures which consist of a pretrained LLM, a pretrained image encoder, and a bridging module that connects them. BLIP-2 (Li
4https://github.com/BradyFU/ Awesome-Multimodal-Large-Language-Models/ tree/Evaluation https://mmbench.opencompass.org.cn/leaderboard https://github.com/yuweihao/MM-Vet https://github.com/haotian-liu/LLaVA/blob/ main/docs/LLaVA_Bench.md https://github.com/RUCAIBox/POPE https://github.com/MMMU-Benchmark/MMMU
et al., 2023d) utilizes the Q-Former to bridge a pretrained image encoder with a pretrained LLM, and achieves strong zero-shot capabilities. InstructBLIP (Dai et al., 2023) is a visual-instructiontuned BLIP-2 (Li et al., 2023d) model. The instruction tuning dataset is a mixture of 13 academic datasets and the LLaVA (Liu et al., 2023e) dataset. Shikra (Chen et al., 2023a) focuses more on the object grounding capability and is instruction tuned on referential dialogue dataset and LLaVA dataset (Liu et al., 2023e), both of which are synthetically generated via GPT-4. LLaVA (Liu et al., 2023e) is the first VLM finetuned on GPT-4 synthesized visual instruction tuning dataset and achieves remarkable performance as a general-purpose visual chatbot. Qwen-VL and Qwen-VL-Chat (Bai et al., 2023b) are recently proposed VLMs based on Qwen (Bai et al., 2023a) language model and are trained on a large-scale (50 million instances) private visual instruction tuning dataset. LLaVA1.5 (Liu et al., 2023d) is a LLaVA model trained on a mixture of shareGPT 5, LLaVA (Liu et al., 2023e) and 8 academic image-text datasets.

heading: E Additional Results
text: E.1 Effect of GPT-4 synthesized data on comprehensive evaluation benchmarks
We show additional results in Figure 8 to support our claim that training on GPT-4 synthesized data can not effectively improve VLMs’ performance on comprehensive evaluation benchmarks. As we can observe, adding more training instances from the LLaVA dataset does not improve VISION-FLAN BASE’s performance on MM-Bench.

heading: E.2 Why VLMs Trained on VISION-FLAN are Better than State-of-the-Art VLMs?
text: In this section, we perform two case studies to explain why models trained on VISION-FLAN can perform better compared to state-of-the-art VLMs.

heading: E.2.1 Case Study on OCR
text: When we manually check the predictions of VISION-FLAN CHAT and compare them to other VLMs, the first trend we observe is that VISIONFLAN CHAT can better perform OCR as shown in Figure 10. To quantify this observation, we evaluate LLaVA, LLaVA 1.5 and our models on the challenging TextOCR dataset (Singh et al., 2021). We ask the VLMs to predict all the text on each
5https://sharegpt.com/
image and check the overlap between the target list of text pieces and the predictions. As shown in Figure 9, the recall of VISION-FLAN BASE and VISION-FLAN CHAT is much higher compared to LLaVA 1.5 while the averaged numbers of predicted tokens per response are similar.

heading: E.2.2 Case Study on Entity Recognition
text: We also spot that models trained on VISION-FLAN can better identify entities in an image while LLaVA 1.5 simply captions the appearance of entities in an image. A qualitative example is shown in Figure 11.
To compute quantitative results, we randomly sample 1,000 images with their captions from the WIT dataset (Srinivasan et al., 2021), in which the images are from Wikipedia pages and the captions commonly contain entities appearing in the im-
image.
LLaVA1.5 An advertisement for Telbruin, introducing their best call rates yet. The advertisement features a clock and several landmarks from around the world, including the Eiffel Tower, Big Ben, and the Space Needle. The advertisement claims that they are offering low rates as low as $0.25 per minute.
VisionFlan Chat The image features a red background and an advertisement for Telbru, a telecommunications company. The ad highlights the introduction of their best 098 call rates yet, which are as low as $0.25 per minute. The rates apply from 27th July 2014, and the ad encourages customers to visit their website or call their customer care for more information…

heading: F.1 The Bridging Module Can Be Shared Across LLMs with Same Architecture

heading: F Additional Analysis
text: can be shared across LLMs with identical architecture but are tuned on different text alignment datasets. As shown in Table 9, we take four different models including VISION-FLAN BASE w/ frozen LLM which is finetuned on VISION-FLAN but with LLMs kept frozen as a case study, and directly replace their LLMs (Vicuna v1.5) with offthe-shelf LLaMA 2 Chat model. During inference, we use the official prompting template of LLaMA 2 chat instead of Vicuna v1.5. The results demonstrate that MLPs can be shared between LLMs with the same architecture but trained on different alignment datasets. An interesting observation is that

heading: F.2 Discrepancy Between Evaluation Benchmarks
text: 38.7, respectively, compared to LLaVA 1.5, which scores 1531.3 and 66.7. Furthermore, this trend is also evident in Table 9. Upon substituting the LLMs in VISION-FLAN BASE and VISION-FLAN CHAT with off-the-shelf LLaMA 2 Chat, both models exhibit a notable decline in performance on MME and MM-Bench, while maintaining comparable performance on LLaVA-Bench. Our hypothesis posits that LLaVA-Bench does not require LLM’s strong understanding of the visual features, but rather relies on the language-prior of LLMs (Lin et al., 2023). Furthermore, the data synthesized by GPT-4 facilitates the model’s ability to generate long-form responses, aligning with the preferences of the evaluation metric, namely, GPT-4 itself.

heading: G Additional Related Work
text: Vision-Language Models. Previous works (Li et al., 2019; Chen et al., 2020; Tan and Bansal, 2019; Su et al., 2020; Wang et al., 2023b) mainly pretrain vision-language models (VLMs) from scratch with a unified masked-language modeling (MLM) objective (Devlin et al., 2019), which can impose significant training cost and inferior performance. Recently, a line of works proposes to build VLMs from the off-the-shelf visual encoders and LLMs by introducing a small amount of bridging parameters that maps visual features into the embedding space of LLMs. Flamingo (Alayrac et al., 2022) presents a VLM that is capable of processing interleaved image-text inputs and generating responses based on the visual content. It proposes Perceiver Resampler as the bridging module to connect the frozen LLM and visual encoder. OFA (Wang et al., 2022) proposes a sequence-tosequence learning framework that maps images to discrete visual tokens and feeds the discrete visual tokens into LLMs. BLIP-2 (Li et al., 2023d) introduces Q-Former to bridge pre-trained and frozen vision and language models, based on which, MiniGPT-4 (Zhu et al., 2023) further adds a linear projector to bridge the gap between the visual encoder and language model encoder. LLaVA (Liu et al., 2023e) introduces a projector to fuse visual information into a large language model and unfreezes language model during visual instruction tuning.
H Datasets Used in VISION-FLAN
Dataset & Reference Tasks CINIC-10 (Darlow et al., 2018) 1. animal recognition in low resolution image
2. shipping method recognition in low resolution image 3. transportation option recognition in low resolution image 4. animal presence classification in low resolution image 5. object shipping object presence in low resolution image
MSCOCO (Lin et al., 2014) 1. multiple choice VQA 2. short image captioning 3. appliance recognition 4. furniture recognition 5. kitchen object recognition 6. vehicle recognition 7. animal recognition 8. sports object recognition 9. image text matching 10. image text selection
FairFace (Kärkkäinen and Joo, 2021) 1. human age classification 2. human gender classification 3. human race classification
IconQA (Lu et al., 2021b) 1. abstract diagram understanding 2. fill in blank in abstract diagram understanding
ImageNet-A (Hendrycks et al., 2021b) 1. object recognition of natural adversarial examples
ImageNet-C (Hendrycks and Dietterich, 2019) 1. blur type classification 2. coarse-grained image corruption classification 3. weather type classification 4. fine-grained image corruption classification
InfographicVQA (Mathew et al., 2022) 1. VQA 2. document level VQA
SemArt (Garcia and Vogiatzis, 2018) 1. painting time frame recognition 2. painting type recognition 3. painting school recognition 4. painting technique recognition 5. detailed image description
Set5 (Bevilacqua et al., 2012) 1. object recognition in low resolution image TextCaps (Sidorov et al., 2020) 1. image captioning with reading comprehension VisDial (Das et al., 2019) 1. visual dialogue with short context
2. visual dialogue with medium context 3. visual dialogue with long context 4. visual dialogue with very long context
STL-10 (Coates et al., 2011) 1. object recognition Places365 (Zhou et al., 2018) 1. scene classification Office-31 (Saenko et al., 2010) 1. image domain and office object classification
2. office object recognition
Dataset & Reference Tasks LSUN (Yu et al., 2015) 1. scene classification FGVC-Aircraft (Maji et al., 2013) 1. aircraft family classification
2. aircraft manufacturer classification 3. aircraft variant classification 4. aircraft model classification
DeepFashion (Liu et al., 2016) 1. cloth texture classification CUB-200-2011 (Wah et al., 2011) 1. bird species recognition CLEVR (Johnson et al., 2017) 1. VQA in 3D rendered images
2. question answer matching 3. visual dialogue in 3D rendered images 4. VQA in 3D rendered images with multiple questions
CLEVR-CoGenT (Johnson et al., 2017) 1. VQA in 3D rendered images 2. question answer matching 3. VQA in 3D rendered images with multiple questions
A-OKVQA (Schwenk et al., 2022) 1. rationales generation 2. answer rationale generation 3. outside knowledge VQA
AI2D (Kembhavi et al., 2016) 1. diagram VQA AID (Xia et al., 2017) 1. aerial scene classification Caltech-256 (Griffin et al., 2007) 1. object recognition CoVA (Kumar et al., 2022) 1. webpage recognition DeepWeeds (Olsen et al., 2018) 1. weed species recognition ExDark (Loh and Chan, 2019) 1. object recognition in low light environments FFHQ-Text (Zhou and Shimada, 2021) 1. facial attribute textual descriptions generation FlickrLogos-27 (Kalantidis et al., 2011) 1. logo recognition FoodLogoDet-1500 (Hou et al., 2021) 1. food logo recognition ImageNet-R (Hendrycks et al., 2021a) 1. object recognition in diverse image domain
2. image style classification ImageNet-Sketch (Wang et al., 2019) 1. object recognition in sketch JHU-CROWD++ (Sindagi et al., 2019) 1. scene classification MNIST-M (Ganin et al., 2017) 1. number recognition MVTecAD (Bergmann et al., 2021) 1. object anomaly detection
2. industrial item recognition
Dataset & Reference Tasks NABirds (Horn et al., 2015) 1. bird species recognition in north America
2. bird body parts detection Road-Anomaly (Lis et al., 2019) 1. road anomaly detection SCUT-CTW1500 (Liu et al., 2017) 1. curve text detection in the wild Total-Text (Chng et al., 2020) 1. scene text detection and recognition VisDA-2017 (Peng et al., 2017) 1. object recognition in 3D rendered image
2. multiple choice object recognition in 3D rendered image
Yoga-82 (Verma et al., 2020) 1. yoga pose recognition Caltech101 (Fei-Fei et al., 2004) 1. object recognition
2. living organism classification Cars (Krause et al., 2013) 1. car brand maker and year classification
2. car brand classification Core50 (Lomonaco and Maltoni, 2017) 1. object recognition NUS-WIDE (Chua et al., 2009) 1. animal presence classification ObjectNet (Barbu et al., 2019) 1. object recognition Places205 (Zhou et al., 2014) 1. indoor outdoor classification 300w (Sagonas et al., 2016) 1. indoor outdoor classification Yahoo (Farhadi et al., 2009) 1. object recognition LFW (Huang et al., 2007) 1. celebrity recognition model-vs-human (Geirhos et al., 2019) 1. image-style classification Office-Home (Venkateswara et al., 2017) 1. object recognition Winoground (Thrush et al., 2022) 1. image caption matching ConceptualCaptions (Sharma et al., 2018) 1. conceptual image captioning KVQA+image question answer (Shah et al., 2019) 1. knowledge-aware VQA 2. visual entity recognition
MemeCap (Hwang and Shwartz, 2023) 1. meme understanding PlotQA (Methani et al., 2020) 1. VQA over scientific plots SentiCap (Mathews et al., 2016) 1. image captioning conditioned on sentiment VQA-E (Li et al., 2018) 1. VQA
2. short image captioning VQG (Mostafazadeh et al., 2016) 1. visual question generation
2. short image captioning
Dataset & Reference Tasks WIT (Srinivasan et al., 2021) 1. background knowledge extraction WikiArt (Tan et al., 2019) 1. artist genre style recognition VQA-RAD (Lau et al., 2019) 1. VQA in radiology VOC2007 (Everingham et al., 2010) 1. multiple object recognition VizWiz (Gurari et al., 2020) 1. answering visual questions from blind people
2. captioning image taken by blind people 3. quality issue classification of image taken by blind people
ViQuAE (Lerner et al., 2022) 1. knowledge based VQA about entities ST-VQA (Biten et al., 2019) 1. scene text VQA Stanford Dogs (Khosla et al., 2011) 1. dog species classification Sketch (Eitz et al., 2012) 1. living organism classification in sketch
2. object recongnition in sketch RAVEN (Zhang et al., 2019) 1. relational and analogical visual reasoning PICKAPIC (Kirstain et al., 2023) 1. image prompt generation PACS (Li et al., 2017) 1. object recognition in art painting
2. object recognition in cartoon 3. object recognition in photograph 4. dog image style classification 5. elephant image style classification 6. giraffe image style classification 7. guitar image style classification 8. horse image style classification 9. house image style classification 10. person image style classification
NOCAPS (Agrawal et al., 2019) 1. multiple short captions generation Localized Narratives (Pont-Tuset et al., 2020) 1. COCO detailed image captioning
2. flickr30k detailed image captioning 3. open images detailed image captioning 4. ade20k detailed image captioning
INATURALIST (Horn et al., 2018) 1. class classification 2. family classification 3. genus classification 4. Latin English name classification 5. order classification 6. phylum classification 7. supercategory classification
HICO (Chao et al., 2015) 1. human activity detection GEOMETRY3K (Lu et al., 2021a) 1. geometry question answering FUNSD (Guillaume Jaume, 2019) 1. text detection in noisy scanned documents FLICKR30K (Plummer et al., 2017) 1. multiple captions generation DVQA (Kafle et al., 2018) 1. chart question answering DTD (Cimpoi et al., 2014) 1. coarse grained texture classification
2. multiple texture detection
Dataset & Reference Tasks DOMAIN NET (Peng et al., 2019) 1. object recognition in clip art
2. object recognition in infograph 3. object recognition in painting 4. object recognition in quickdraw 5. object recognition in real image 6. image style classification
DOCVQA (Mathew et al., 2020) 1. document level VQA DAQUAR (Malinowski and Fritz, 2014) 1. VQA CONCADIA (Kreiss et al., 2022) 1. caption with background knowledge
2. short image captioning Visual7W (Zhu et al., 2016) 1. VQA object attribute VQAv2 (Goyal et al., 2017) 1. general VQA
2. question image matching Visual Genome(Krishna et al., 2017) 1. spatial relationship question answering OK-VQA(Marino et al., 2019) 1. outside knowledge VQA ScienceQA (Lu et al., 2022) 1. VQA
2. explanation generation OCR-VQA (Mishra et al., 2019) 1. VQA by reading text in image wikiHow-image (Yang et al., 2021) 1. next step generation
2. image text step ordering 3. immediate next step selection 4. text image step ordering
SciCap (Hsu et al., 2021) 1. figure captioning LAD (Zhao et al., 2019) 1. detailed object description generation Dark Zurich (Sakaridis et al., 2019) 1. time of the day classification RAF-DB (Li and Deng, 2019) 1. human emotion detection GQA (Hudson and Manning, 2019) 1. spatial relationship question answering VQA (Antol et al., 2015) 1. color
2. activity recognition 3. counting 4. object presence 5. object recognition 6. positional reasoning 7. scene recognition 8. sentiment understanding 9. sport recognition 10. utility affordance
Multimodal Factual Checking (Yao et al., 2023) 1. multimodal factual checking
I Task Categories in VISION-FLAN
Category Tasks Perception 1. CLEVR-CoGenT VQA in 3D rendered images
2. CLEVR-CoGenT question answer matching 3. CLEVR-CoGenT VQA in 3D rendered images with multiple questions 4. CLEVR VQA in 3D rendered images with multiple questions 5. GQA spatial relationship question answering 6. VQA color 7. VQA activity recognition 8. VQA counting 9. VQA object presence 10. VQA object recognition 11. VQA positional reasoning 12. VQA scene recognition 13. VQA sentiment understanding 14. VQA sport recognition 15. VQA utility affordance 16. VQA-E VQA 17. VQAv2 general VQA 18. Visual Genome spatial relationship question answering 19. CLEVR question answer matching 20. VizWiz answering visual questions from blind people 21. DAQUAR VQA 22. MSCOCO multiple choice VQA 23. Visual7W VQA object attribute 24. CLEVR VQA in 3D rendered images
Outside Knowledge 1. KVQA knowledge aware VQA 2. VIQUAE knowledge based VQA about entities 3. VQARAD VQA in radiology 4. OK-VQA outside knowledge VQA 5. A-OKVQA outside knowledge VQA
Reasoning 1. GEOMETRY3K geometry question answering 2. IconQA abstract diagram understanding 3. IconQA fill in blank in abstract diagram understanding 4. InfographicVQA VQA 5. InfographicVQA document level VQA 6. ScienceQA VQA 7. AI2D diagram VQA
OCR 1. DOCVQA document level VQA 2. DVQA chart question answering 3. PlotQA VQA over scientific plots 4. OCR-VQA VQA by reading text in image 5. ST-VQA scene text VQA
Category Tasks Document-Level OCR 1. FUNSD text detection in noisy scanned docu-
ments 2. SCUT-CTW1500 curve text detection in the wild 3. Total-Text scene text detection and recognition
Phrase-Level OCR 1. CoVA webpage recognition 2. FlickrLogos-27 logo recognition 3. FoodLogoDet-1500 food logo recognition
Knowledge Extraction 1. CONCADIA caption with background knowledge 2. KVQA visual entity recognition 3. WIT background knowledge extraction
Semantic Art Understanding 1. Semart painting time frame recognition 2. Semart painting type recognition 3. Semart painting school recognition 4. Semart painting technique recognition 5. Semart detailed image description 6. WikiArt artist genre style recognition
Visual Dialogue 1. CLEVR visual dialogue in 3D rendered images 2. Visdial visual dialogue with short context 3. Visdial visual dialogue with medium context 4. Visdial visual dialogue with long context 5. Visdial visual dialogue with very long context
Rational and Script Generation 1. ScienceQA explanation generation 2. A-OKVQA rationales generation 3. A-OKVQA answer rationale generation 4. MemeCap meme understanding 5. wikiHow-image next step generation 6. VQG visual question generation
Coarse-grained Captioning 1. ConceptualCaptions conceptual image captioning 2. FLICKR30K multiple captions generation 3. NOCAPS multiple short captions generation 4. PICKAPIC image prompt generation 5. VizWiz captioning image taken by blind people 6. VQA-E short image captioning 7. VQG short image captioning 8. MSCOCO short image captioning 9. CONCADIA short image captioning
Category Tasks Fine-grained Captioning 1. LAD detailed object description generation
2. FFHQ-Text facial attribute textual descriptions generation 3. Localized Narratives COCO detailed image captioning 4. Localized Narratives flickr30k detailed image captioning 5. Localized Narratives open images detailed image captioning 6. Localized Narratives ade20k detailed image captioning 7. SciCap figure captioning 8. SentiCap image captioning conditioned on sentiment 9. TextCaps image captioning with reading comprehension
Scene Classification 1. 300w indoor outdoor classification 2. AID aerial scene classification 3. Dark-Zurich time of the day classification 4. JHU-CROWD scene classification 5. LSUN scene classification 6. Places205 indoor outdoor classification 7. places365 scene classification
Animal Classification 1. CUB-200-2011 bird species recognition 2. DeepWeeds weed species recognition 3. INATURALIST class classification 4. INATURALIST family classification 5. INATURALIST genus classification 6. INATURALIST Latin English name classification 7. INATURALIST order classification 8. INATURALIST phylum classification 9. INATURALIST supercategory classification 10. NABirds bird species recognition in north America 11. NUS-WIDE animal presence classification 12. STANFORD DOGS dog species classification 13. NABirds bird body parts detection
Category Tasks Vehicle Classification 1. Cars car brand maker and year classification
2. Cars car brand classification 3. FGVC-Aircraft aircraft family classification 4. FGVC-Aircraft aircraft manufacturer classification 5. FGVC-Aircraft aircraft variant classification 6. FGVC-Aircraft aircraft model classification
Human Activity 1. HICO human activity detection 2. RAF-DB human emotion detection 3. Yoga-82 yoga pose recognition
Facial Recognition 1. LFW celebrity recognition 2. Fairface human age classification 3. Fairface human gender classification 4. Fairface human race classification
Anomaly Detection 1. Road-Anomaly road anomaly detection 2. MVTecAD object anomaly detection
General Object 1. Caltech-256 object recognition 2. Caltech101 object recognition 3. Caltech101 living organism classification 4. Core50 object recognition 5. ImageNet-A object recognition of natural adversarial examples 6. MNIST-M number recognition 7. MVTecAD industrial item recognition 8. ObjectNet object recognition 9. Office-Home object recognition 10. Office-31 image domain and office object classification 11. Office-31 office object recognition 12. STL-10 object recognition 13. Set5 object recognition in low resolution image 14. VOC2007 multiple object recognition 15. MSCOCO appliance recognition 16. MSCOCO furniture recognition 17. MSCOCO kitchen object recognition 18. MSCOCO vehicle recognition 19. MSCOCO animal recognition 20. MSCOCO sports object recognition 21. Yahoo object recognition
Category Tasks Complex Reasoning 1. RAVEN relational and analogical visual rea-
soning 2. Multimodal Factual Checking multimodal factual checking 3. wikiHow-image image text step ordering 4. wikiHow-image immediate next step selection 5. wikiHow-image text image step ordering
Image Text Matching 1. MSCOCO image text matching 2. Winoground image caption matching 3. MSCOCO image text selection 4. MSCOCO question image matching
General Object Classification in Special Image Domain 1. DOMAIN NET object recognition in clip art 2. DOMAIN NET object recognition in infograph 3. DOMAIN NET object recognition in painting 4. DOMAIN NET object recognition in quickdraw 5. DOMAIN NET object recognition in real image 6. ExDark object recognition in low light environments 7. ImageNet-R object recognition in diverse image domain 8. ImageNet-Sketch object recognition in sketch 9. PACS object recognition in art painting 10. PACS object recognition in cartoon 11. PACS object recognition in photograph 12. SKETCH living organism classification in sketch 13. SKETCH object recognition in sketch 14. Cinic-10 animal recognition in low resolution image 15. Cinic-10 shipping method recognition in low resolution image 16. Cinic-10 transportation option recognition in low resolution image 17. Cinic-10 animal presence classification in low resolution image 18. Cinic-10 object shipping object presence in low resolution image 19. VisDA-2017 object recognition in 3D rendered image 20. VisDA-2017 multiple choice object recognition in 3D rendered image
Category Tasks Image-Style Classification 1. DOMAIN-NET image style classification
2. ImageNet-R image style classification 3. PACS dog image style classification 4. PACS elephant image style classification 5. PACS giraffe image style classification 6. PACS guitar image style classification 7. PACS horse image style classification 8. PACS house image style classification 9. PACS person image style classification 10. Model-vs-human image style classification
Image Quality Classification 1. ImageNet-C blur type classification 2. ImageNet-C coarse-grained image corruption classification 3. ImageNet-C weather type classification 4. ImageNet-C fine-grained image corruption classification 5. VizWiz quality issue classification of image taken by blind people
Texture Classification 1. DTD coarse grained texture classification 2. DTD multiple texture detection 3. DeepFashion cloth texture classification
J VISION-FLAN Tasks
J.1 Generation Tasks

heading: Task: FoodLogoDet-1500_food_logo_recognition

heading: Task: FlickrLogos-27_logo_recognition

heading: Task: CoVA_webpage_recognition

heading: Task: FUNSD_text_detection_in_noisy_scanned_documents

heading: Task: SCUT-CTW1500_curve_text_detection_in_the_wild

heading: Task: Total-Text_scene_text_detection_and_recognition

heading: Task: WIKIART_artist_genre_style_recognition

heading: Task: Semart_painting_type_recognition

heading: Task: Semart_painting_timeframe_recognition

heading: Task: WIT_background_knowledge_extraction

heading: Task: KVQA_visual_entity_recognition

heading: Task: CONCADIA_caption_with_background_knowledge

heading: Task: Semart_detailed_image_description

heading: Task: Semart_painting_technique_recognition

heading: Task: Semart_painting_school_recognition

heading: Task: CLEVR_visual_dialogue_in_3D_rendered_images

heading: Task: Wikihow_next_step_generation

heading: Task: MemeCap_meme_understanding

heading: Task: A-OKVQA_answer_rationale_generation

heading: Task: A-OKVQA_rationales_generation

heading: Task: ScienceQA_explanation_generation

heading: Task: PICKAPIC_image_prompt_generation

heading: Task: CONCADIA _short_image_captioning

heading: Task: MSCOCO_short_image_captioning

heading: Task: NOCAPS_multiple_short_captions_generation

heading: Task: FLICKR30K_multiple_captions_generation

heading: Task: ConceptualCaptions_conceptual_image_captioning

heading: Task: textcaps_image_captioning_with_reading_comprehension

heading: Task: SentiCap_image_captioning_conditioned_on_ sentiment

heading: Task: SciCap+figure_captioning

heading: Task: LOC-NARRATIVES_open_images_detailed_image_captioning

heading: Task: LOC-NARRATIVES_flickr30k_detailed_image_captioning

heading: Task: LOC-NARRATIVES_COCO_detailed_image_captioning

heading: Task: FFHQ-Text_facial_attribute_textual_descriptions_generation
text: J.2 Classification Tasks

heading: Task: JHU-CROWD_scene_classification

heading: Task: Dark-Zurich_time_of_the_day_classification

heading: Task: AID+aerial_scene_classification

heading: Task: Places205_indoor_outdoor_classification

heading: Task: LSUN_scene_classification

heading: Task: DeepWeeds_weed_species_recognition

heading: Task: CUB-200-2011_bird_species_recognition

heading: Task: places365_scene_classification

heading: Task: FGVC-Aircraft_aircraft_variant_classification

heading: Task: FGVC-Aircraft_aircraft_ manufacturer_classification

heading: Task: FGVC-Aircraft_aircraft_family_classification

heading: Task: STANFORD_DOGS_dog_species_classification

heading: Task: NUS-WIDE_animal_presence_classification

heading: Task: NABirds_bird_body_parts_detection

heading: Task: NABirds_bird_species_recognition_in_north_America

heading: Task: Yoga-82_yoga_pose_recognition

heading: Task: RAF_DB_human_emotion_detection

heading: Task: HICO_human_activity_detection

heading: Task: Cars_car_brand_classification

heading: Task: FGVC-Aircraft_aircraft_model_classification

heading: Task: Cars_car_brand_maker_and_year_classification

heading: Task: LFW_human_face_recognition

heading: Task: Fairface_human_race _classification

heading: Task: Fairface_human_gender_classification

heading: Task: Fairface_human_age_classification

heading: Task: Caltech101_living_ organism_classification

heading: Task: Caltech101_object_recognition

heading: Task: Caltech-256_object_recognition

heading: Task: MVTecAD_object_anomaly_detection

heading: Task: Road-Anomaly_road_anomaly_detection

heading: Task: MVTecAD_industrial_item_recognition

heading: Task: Office-Home_object_recognition

heading: Task: MNIST-M_number_recognition

heading: Task: Core50_object_recognition

heading: Task: STL-10_object_recognition

heading: Task: Office-31_office_object_recognition

heading: Task: Office-31_image_domain_and_office_object_classification

heading: Task: ObjectNet_object_recognition

heading: Task: MSCOCO_vehicle_recognition

heading: Task: MSCOCO_kitchen_object_recognition

heading: Task: MSCOCO_furniture_recognition

heading: Task: MSCOCO_appliance_recognition

heading: Task: Set5_object_recognition_in_low_resolution_image

heading: Task: Wikihow_text_image_step_ordering

heading: Task: Wikihow_immediate_next_step_selection

heading: Task: Wikihow_image_text_step_ordering

heading: Task: MSCOCO_sports_object_recognition

heading: Task: MSCOCO_animal_recognition

heading: Task: image_text_selection

heading: Task: Winoground+image_caption_matching

heading: Task: image_text_matching

heading: Task: RAVEN_relational_and_analogical_visual_reasoning

heading: Task: multimodal_factual_checking

heading: Task: ExDark_object_recognition_in_low_light_environments

heading: Task: DOMAIN-NET_object_recognition_in_real_image

heading: Task: DOMAIN-NET_object_recognition_in_quickdraw

heading: Task: DOMAIN-NET_object_recognition_in_painting

heading: Task: DOMAIN-NET_object_recognition_in_infograph

heading: Task: DOMAIN-NET_object_recognition_in_clip_art

heading: Task: question_image_matching

heading: Task: SKETCH_living_organism_classification_in_sketch

heading: Task: PACS_object_recognition_in_photograph

heading: Task: PACS_object_recognition_in_cartoon

heading: Task: PACS_object_recognition_in_art_painting

heading: Task: Cinic-10+object_shipping_object_presence_in_low_resolution_image

heading: Task: Cinic-10_animal_presence_classification_in_low_resolution_image

heading: Task: Cinic-10_transportation_option_recognition_in_low_resolution_image

heading: Task: Cinic-10_shipping_method_recognition_in_low_resolution_image

heading: Task: Cinic-10_animal_recognition_in_low_resolution_image

heading: Task: SKETCH_object_recongnition_in_sketch

heading: Task: PACS_dog_image_style_classification

heading: Task: DOMAIN-NET_image_style_classification

heading: Task: Model-vs-human_image_style_classification

heading: Task: PACS_person_image_style_classification

heading: Task: PACS_house_image_style_classification

heading: Task: PACS_horse_image_style_classification

heading: Task: PACS_guitar_image_style_classification

heading: Task: PACS_giraffe_image_style_classification

heading: Task: PACS_elephant_image_style_classification

heading: Task: DTD_multiple_texture_detection

heading: Task: DeepFashion_cloth_texture_classification

heading: Task: DTD+coarse_grained_texture_classification
text: J.3 VQA Tasks

heading: Task: MSCOCO_multiple_choice_VQA

heading: Task: GQA_spatial_relationship_question_answering

heading: Task: CLEVR-CoGenT_VQA_ in_3D_rendered_images_with_multiple_questions

heading: Task: CLEVR_VQA_in_3D_rendered_images

heading: Task: DAQUAR_VQA

heading: Task: CLEVR-CoGenT_question_answer_matching

heading: Task: KVQA_world_knowledge_enabled_VQA

heading: Task: CLEVR_ VQA_ in_3D_rendered_images_with_multiple_questions

heading: Task: CLEVR-question_answer_matching

heading: Task: GEOMETRY3K_geometry_question_answering

heading: Task: A-OK-VQA _outside_knowledge_VQA

heading: Task: OK-VQA_outside_knowledge_VQA

heading: Task: OCR-VQA_VQA_by_reading_text_in_image

heading: Task: DVQA_chart_question_answering

heading: Task: DOCVQA_document_level_VQA

heading: Task: AI2D_diagram_VQA

heading: Task: ScienceQA_VQA

heading: Future Work
text: Future work should extend VISION-FLAN with multilingual tasks. In addition, all the tasks in VISIONFLAN only consists of a single image. Many realworld vision-language tasks require the model to take multiple images as inputs, such as script generation (Qi et al. , 2024) and multimodal entity linking (Yao et al. , 2024). Thus, future work should explore vision-language tasks that involve multiple images or videos. Our analysis mainly focuses on the GPT-4 synthesized visual instruction tuning dataset. Recently, as the GPT-4V 3 becomes publicly available, there are some concurrent works (Wang et al. , 2023a; Chen et al. , 2023b) prompting GPT-4V with images as inputs to generate visual instruction tuning data. Future work can analyze the effect of tuning VLMs on such datasets and identify the advantages and disadvantages. In our experiments, we mainly focus on the LLaVA-Architecture (Liu et al. , 2023e) due to its strong performance and high efficiency. However, there are other foundation architectures such as Qformer in BLIP2 (Li et al. , 2023d) and Perceiver Resampler in Flamingo (Alayrac et al. , 2022)