MULTIINSTRUCT: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning
Year: 2023

Abstract:
Instruction tuning, a new learning paradigm that fine-tunes pre-trained language models on tasks specified through instructions, has shown promising zero-shot performance on various natural language processing tasks. However, it has yet to be explored for vision and multimodal tasks. In this work, we introduce MULTIINSTRUCT, the first multimodal instruction tuning benchmark dataset that consists of 62 diverse multimodal tasks in a unified seq-toseq format covering 10 broad categories. The tasks are derived from 21 existing open-source datasets and each task is equipped with 5 expertwritten instructions. We take OFA (Wang et al., 2022a) as the base pre-trained model for multimodal instruction tuning, and to further improve its zero-shot performance, we explore multiple transfer learning strategies to leverage the large-scale NATURAL INSTRUCTIONS dataset (Mishra et al., 2022). Experimental results demonstrate strong zero-shot performance on various unseen multimodal tasks and the benefit of transfer learning from a text-only instruction dataset. We also design a new evaluation metric – Sensitivity, to evaluate how sensitive the model is to the variety of instructions. Our results indicate that fine-tuning the model on a diverse set of tasks and instructions leads to a reduced sensitivity to variations in instructions for each task1.

heading: 1 Introduction
text: With the advances in large-scale pre-trained language models (PLMs), recent studies have explored various efficient learning paradigms (Brown et al., 2020; Liu et al., 2021; Wei et al., 2021; Xie et al., 2021) to generalize PLMs to new tasks without task-specific tuning. Among these, instruction
∗ Zhiyang Xu and Ying Shen contributed equally to this work.
1The dataset, source code, and model checkpoints are publicly available at https://github.com/VT-NLP/ MultiInstruct.
tuning (Wei et al., 2021) has achieved significant success in zero-shot learning on natural language processing tasks. By fine-tuning a PLM on tasks described through instructions, instruction tuning allows the model to learn to understand and follow the instructions to perform predictions on unseen tasks. Recent advancement in multimodal pretraining (Wang et al., 2022a; Alayrac et al., 2022; Bao et al., 2022; Wang et al., 2022c) has shown the potential of jointly interpreting text and images in a shared semantic space, which further leads us to ask: can the instruction tuning be leveraged to improve the generalizability of Vision-Language pretrained models on multi-modal and vision tasks?
In this work, we propose MULTIINSTRUCT, the first benchmark dataset for multimodal instruction tuning with 62 diverse tasks from 10 broad categories, including Visual Question Answering (Goyal et al., 2017; Suhr et al., 2017), Commonsense Reasoning (Zellers et al., 2019; Xie et al., 2019), Visual Relationship Understanding (Krishna et al., 2017) and so on. We equipped each task with 5 instructions that are written by two experts in natural language processing. As shown in Figure 1, we formulate all the tasks into a unified sequence-to-sequence format in which the input text, images, instructions, and bounding boxes are represented in the same token space.
We use OFA (Wang et al., 2022a)2, a unified model that is pre-trained on a diverse set of multimodal and unimodal tasks in a single Transformerbased sequence-to-sequence framework, as the base pre-trained multimodal language model, and fine-tune it on MULTIINSTRUCT. To utilize NATURAL INSTRUCTIONS (Mishra et al., 2022), a largescale text-only instruction tuning dataset, we further explore two transfer learning strategies, in-
2We use OFA as it was the largest and most powerful open-source multimodal pre-trained model available at the time of our research while other stronger models didn’t have publicly available checkpoints at that time.
11445
cluding Mixed Instruction Tuning and Sequential Instruction Tuning. Experimental results demonstrate strong zero-shot performance on various unseen multimodal tasks with instruction tuning and the potential of further improving it by leveraging large-scale text-only instruction datasets.
As suggested by previous studies (Webson and Pavlick, 2022; Liu et al., 2022b), PLMs are highly sensitive toward the wording and length of instructions. Thus, we propose a new metric – Sensitivity, which measures how sensitive the model is toward the variety of instructions for the same task. Experimental results demonstrate that (1) instruction tuning significantly reduces the sensitivity of OFA to the varying wording of instructions. The more tuning tasks and instructions for each task are introduced, the lower sensitivity tends to be achieved, and (2) transferring from a larger text-only instruction dataset can also significantly reduces the sensitivity of OFA.

heading: 2 Related Work
text: Multimodal Pretraining Multimodal pretraining (Tan and Bansal, 2019; Cho et al., 2021; Singh et al., 2022; Alayrac et al., 2022; Wang et al., 2022a; Li et al., 2022b,a) has significantly advanced the vision-language tasks. Several recent studies (Cho et al., 2021; Wang et al., 2022a,c; Lu et al., 2022) also started to build a unified pre-training framework to handle a diverse set of cross-modal and unimodal tasks. Among them, VLT5 (Cho et al., 2021) tackles vision-and-language tasks with a unified text-generation objective conditioned on multimodal inputs, while OFA (Wang
et al., 2022a) further extends it to image generation tasks by using a unified vocabulary for all text and visual tokens. BEIT-3 (Wang et al., 2022c) utilizes a novel shared Multiway Transformer network with a shared self-attention module to align different modalities and provide deep fusion. Building on the success of multimodal pretraining, our work focuses on improving the generalization and zeroshot performance on various unseen multimodal tasks through instruction tuning.
Efficient Language Model Tuning To improve the generalizability and adaptivity of large-scale pre-trained language models, various efficient language model tuning strategies have been proposed recently. Prompt tuning (Liu et al., 2021; Li and Liang, 2021; Han et al., 2022; Wang et al., 2022b; Sanh et al., 2022) aims to learn a task-specific prompt by reformulating the downstream tasks to the format that the model was initially trained on and has shown competitive performance across various natural language processing applications. As a special form of prompt tuning, in-context learning (Xie et al., 2021; Min et al., 2021) takes one or a few examples as the prompt to demonstrate the task. Instruction tuning (Wei et al., 2021) is another simple yet effective strategy to improve the generalizability of large language models. NATURAL INSTRUCTIONS (Mishra et al., 2022) is a meta-dataset containing diverse tasks with human-authored definitions, things to avoid, and demonstrations. It has shown effectiveness in improving the generalizability of language models even when the size is relatively small (e.g., BART_base) (Mishra et al.,
2022; Wang et al., 2022d). InstructDial (Gupta et al., 2022) applies instruction tuning to the dialogue domain and shows significant zero-shot performance on unseen dialogue tasks. While these studies have been successful in text-only domains, it has not yet been extensively explored for vision or multimodal tasks.

heading: 3 MULTIINSTRUCT

heading: 3.1 Multimodal Task and Data Collection
text: The MULTIINSTRUCT dataset is designed to cover a wide range of multimodal tasks that require reasoning among regions, images, and text. These tasks are meant to teach machine learning models to perform various tasks such as object recognition, visual relationship understanding, text-image grounding, and so on by following instructions so that they can perform zero-shot prediction on unseen tasks. To build MULTIINSTRUCT, we first collect 34 tasks from the existing studies in visual and multimodal learning, covering Visual Question Answering (Goyal et al., 2017; Krishna et al., 2017; Zhu et al., 2016; Hudson and Manning, 2019; Singh et al., 2019; Marino et al., 2019), Commonsense Reasoning (Suhr et al., 2017; Liu et al., 2022a; Zellers et al., 2019; Xie et al., 2019), Region Understanding (Krishna et al., 2017), Image Understanding (Kafle and Kanan, 2017; Chiu et al., 2020), Grounded Generation (Krishna et al., 2017; Yu et al., 2016; Lin et al., 2014), Image-Text Matching (Lin et al., 2014; Goyal et al., 2017), Grounded Matching (Krishna et al., 2017; Veit et al., 2016; Yu et al., 2016), Visual Relationship (Krishna et al., 2017; Pham et al., 2021), Temporal Ordering tasks that are created from WikiHow3, and Miscellaneous (Yao et al., 2022; Kiela et al., 2020; Das et al., 2017; Lin et al., 2014; Veit et al., 2016; Alam et al., 2022). Each of the 34 tasks can be found with one or multiple open-source datasets, which are incorporated into MULTIINSTRUCT. Details of each task and their corresponding datasets are shown in Tables 7 to 9 in Appendix.
For each of these tasks, we further examine the possibility of deriving new tasks based on the input and output of the original task to augment the task repository. For example, Visual Grounding requires the model to generate a caption for a given region in the image. We derive two additional tasks from it: Grounded Caption Selection, which is a simpler task that requires the model to select the
3https://www.wikihow.com.
corresponding caption from multiple candidates for the given region, and Visual Grounding Selection, which requires the model to select the corresponding region from the provided candidate regions based on a given caption. Compared with Visual Grounding, these two new tasks require different skills based on distinct input and output information. In this way, we further derived 28 new tasks from the 34 existing tasks. We divide all 62 tasks into 10 broad categories as shown in Figure 2.
For the existing tasks, we use their available open-source datasets to create instances (i.e., input and output pairs) while for each new task, we create its instances by extracting the necessary information from instances of existing tasks or reformulating them. Each new task is created with 5,000 to 5M instances. We split the 62 tasks into training and evaluation based on the following criteria: (1) we take the tasks that are similar to the pre-training tasks of OFA (Wang et al., 2022a) for training; and (2) we select the challenging multimodal tasks that do not overlap with the training tasks for evaluation. Table 5 and Table 6 in Appendix A show the detailed statistics for the training and evaluation tasks in MULTIINSTRUCT and Tables 7 to 9 show their corresponding datasets.

heading: 3.2 Task Instruction Creation
text: We first provide a definition for “instruction” used in MULTIINSTRUCT. An instruction is defined with a template that describes how the task should be performed and contains an arbitrary number of placeholders, including <TEXT>, <REGION> and <OPTION>, for the input information from the original task. For example, in the instruction of the Grounded Captioning task, “Generate a caption for <REGION>”, <REGION> is the placeholder for region-specific information. Note that the placeholder <OPTION> is only used in classification tasks and for some tasks, the input may also include an image that is not included in the instruction and will be fed as a separate input to the model. Figure 1 provides several instruction examples for the tasks included in MULTIINSTRUCT.
To produce high-quality instructions that accurately convey the intended tasks, we employ an iterative annotation process involving two expert annotators who have a thorough understanding of the task and the dataset. Step 1: each annotator first writes 2-3 instructions for each task by giving them the specific goals of
this task, the format of input data, and 10 example instances randomly sampled from the dataset. The information about the dataset is obtained from the dataset’s README file or the publication that introduced the dataset. For newly derived tasks, we provide annotators with task descriptions along with 10 constructed example instances. Step 2: to guarantee the quality of the instructions and that they effectively convey the intended tasks, we have each annotator review the instructions created by their peers, checking if they can clearly understand and identify the intended task by just reading the instruction. If any issues are identified, the reviewing annotator provides suggestions and works with the original annotator to revise the instructions. Step 3: to ensure the consistency and avoid conflicts or repetition among instructions from different annotators, we have both annotators review the sets of instructions together, identifying any discrepancies or inconsistencies. If any are found, the annotators collaborate to resolve them and create a final set of instructions that accurately and clearly describe the task. In this way, each task will be created with 5 high-quality instructions. Step 4: we repeat steps 1-3 to create 5 instructions for each of the training and evaluation tasks. Finally, both annotators review each task and its instructions and filter out the task that is not repre-
sentative or overlaps with other tasks.

heading: 3.3 Multimodal Instruction Formatting
text: To unify the processing of various input/output data types, we follow the method from OFA (Wang et al., 2022a), which involves representing images, text, and bounding box coordinates as tokens in a unified vocabulary. Specifically, we apply bytepair encoding (BPE) (Sennrich et al., 2016) to encode the text input. For the target image, we apply VQ-GAN (Esser et al., 2021) to generate discrete image tokens through image quantization. To represent regions or bounding boxes of an image, we discretize the four corner coordinates into location tokens such as "<bin_242> <bin_180> <bin_736> <bin_475>" where each location token "<bin_NUM>" represents a quantized coordinate obtained by dividing the image into 1,000 bins. This approach allows us to convert different types of input into a unified vocabulary.
All tasks in MULTIINSTRUCT can then be formulated as natural language sequence-to-sequence generation problems, where the input includes: (1) an image (if there is no input image, a black picture is used as the input); and (2) an instruction where the placeholders such as <TEXT>, <REGION> or <OPTION> are filled with specific information of each input instance. Notably, for the <OPTION> of the instructions for classification tasks, we intro-
duce two special tokens for this field: “[Options]” to mark the beginning of the option field and “||||” to delimit the given options. We concatenate all the options with “||||” in the option field and the model will directly generate one option from them. Figure 1 provides several examples of the formulated input and illustrates how the original data input is combined with the instruction in the MULTIINSTRUCT.

heading: 4 Problem Setup and Models

heading: 4.1 Problem Setup
text: We follow the same instruction tuning setting as the previous study (Wei et al., 2021) and mainly evaluate the zero-shot learning capabilities of the finetuned large language models. Specifically, given a pre-trained multimodal language model M , we aim to finetune it on a collection of instruction tasks T . Each task t ∈ T is associated with a number of training instances Dt = {(It, xtj , ytj) ∈ It×X t×Yt}Nj=1, where xtj denotes the input text, image, region, and options if provided, ytj denotes the output of each instance, and It represents the set of five task instructions written by experts. The input information from xtj will be used to fill in the placeholders in the instruction.
We use OFA (Wang et al., 2022a) as the pretrained multimodal model due to its unified architecture and flexible input-output modalities. We finetune it on our MULTIINSTRUCT dataset to demonstrate the effectiveness of instruction tuning. Specifically, we use the transformer-based encoder of OFA to encode the instruction along with all necessary information and an optional image, and predict the output with the transformer-based decoder. Given that the training dataset contains many tasks, we mix all the training instances from these tasks and randomly shuffle them. For each instance, we also randomly sample an instruction template for each batch-based training. Note that, though some of the training tasks in MULTIINSTRUCT are similar to the pre-training tasks of OFA4, we ensure that the evaluation tasks in MULTIINSTRUCT do not overlap with either the pre-training tasks in OFA nor the training tasks in MULTIINSTRUCT.
4Table 10 in Appendix lists the multimodal tasks and dataset used in OFA pre-training.

heading: 4.2 Transfer Learning from NATURAL
text: INSTRUCTIONS
We notice that the scale of NATURAL INSTRUCTIONS (Mishra et al., 2022) is significantly larger than MULTIINSTRUCT, indicating the potential of transferring the instruction learning capability from the larger set of natural language tasks to multimodal tasks. We take 832 English tasks in NATURAL INSTRUCTIONS and explore several simple transfer-learning strategies:
Mixed Instruction Tuning (OFAMixedInstruct) We combine the instances of NATURAL INSTRUCTIONS and MULTIINSTRUCT and randomly shuffle them before finetuning OFA with instructions. Note that, each task in NATURAL INSTRUCTIONS is just associated with one instruction while for each instance from MULTIINSTRUCT, we always randomly sample one instruction from the five instructions for each instance of training.
Sequential Instruction Tuning (OFASeqInstruct) Inspired by the Pre-Finetuning approach discussed in Aghajanyan et al. (2021), we propose a twostage sequential instruction tuning strategy where we first fine-tune OFA on the NATURAL INSTRUCTIONS dataset to encourage the model to follow instructions to perform language-only tasks, and then further fine-tune it on MULTIINSTRUCT to adapt the instruction learning capability to multimodal tasks. To maximize the effectiveness of the NATURAL INSTRUCTIONS dataset, we use all instances in English-language tasks to tune the model in the first training stage.

heading: 5 Experimental Setup
text: Evaluation Metrics We report the accuracy for classification tasks and ROUGE-L (Lin, 2004) for all generation tasks. For the region classification task, we compute the Intersection over Union (IoU) between the generated region and all regions in the options, select the option with the highest IoU as the prediction, and compute accuracy based on this prediction. If the predicted region has no intersection with any of the regions in the options, we treat this prediction as incorrect. For classification tasks where the answer is not a single-word binary classification, we also report ROUGE-L scores following Mishra et al. (2022), which treats all tasks as text generation problems. For each task, we conduct five experiments by evaluating the model using one of the five instructions in each experiment. We re-
port the mean and maximum performance and the standard deviation of the performance across all five experiments. We also compute the aggregated performance for each model based on the mean of the model’s performance on all multimodal and NLP unseen tasks. We use Rouge-L as the evaluation metric for most tasks and accuracy for tasks that only have accuracy as a metric.
In addition, as instruction tuning mainly relies on the instructions to guide the model to perform prediction on various unseen multimodal tasks, we further propose to evaluate how sensitive the model is to the variety of human-written instructions in the same task, which has not been discussed in previous instruction tuning studies but is necessary to understand the effectiveness of instruction tuning. We thus further design a new metric as follows:
Sensitivity refers to the model’s capability of consistently producing the same results, regardless of slight variations in the wording of instructions, as long as the intended task remains the same. Specifically, for each task t ∈ T , given its associated instances with task instructions: Dt = {(It, xtj , ytj) ∈ It × X t × Yt}Nj=1, we formally define sensitivity as:
Et∈T
[ σi∈It [ E(x,y)∈Dt [L(fθ(i, x), y)] ]
µi∈It [ E(x,y)∈Dt [L(fθ(i, x), y)]
] ]
where L denotes the evaluation metric such as accuracy or ROUGE-L, fθ(·) represents the multimodal instruction-tuned model. The standard deviation and mean of the model’s performance across all instructions are denoted by σi∈It [·] and µi∈It [·], respectively.
Evaluation datasets We evaluate the models on nine unseen multimodal tasks: Text VQA (Singh et al., 2019), Grounded VQA (Zhu et al., 2016), Commonsense VQA (Zellers et al., 2019), Visual Entailment (Xie et al., 2019), Visual Spatial Reasoning (Liu et al., 2022a), Natural Language for Visual Reasoning (NLVR) (Suhr et al., 2017), Visual Text Extraction (Kiela et al., 2020), Visual Dialogue (Das et al., 2017), and Disaster Type Classification (Alam et al., 2022). These tasks belong to three task groups: Commonsense Reasoning, VQA, and Miscellaneous as shown in Figure 2. Tasks in the Commonsense Reasoning group have no overlap with any training task groups. Tasks in Miscellaneous do not share similarities with other tasks in the group. Although Text VQA and Grounded
VQA belong to the VQA task group, they require additional skills such as extracting text from images or generating regions, making them fundamentally different from other tasks in VQA. In addition to multimodal tasks, we also evaluate the model on 20 NLP tasks collected from the test split of NATURAL INSTRUCTIONS.
Approaches for Comparison We denote the OFA finetuned on MULTIINSTRUCT as OFAMultiInstruct, and compare it with the original pre-trained OFA5, OFATaskName which is fine-tuned on MULTIINSTRUCT but uses the task name instead of instruction to guide the model to make predictions, and several approaches that leverage the large-scale NATURAL INSTRUCTIONS dataset, including OFANaturalInstruct which only fine-tunes OFA on NATURAL INSTRUCTIONS with instruction tuning, OFAMixedInstruct and OFASeqInstruct that are specified in Section 4.2.
More details regarding the evaluation datasets, baseline approaches and training details can be found in Appendix B.

heading: 6 Results and Discussion

heading: 6.1 Effectiveness of Instruction Tuning on
text: MULTIINSTRUCT
We evaluate the zero-shot performance of various approaches on all the unseen evaluation tasks, as shown in Table 1 and 2. Our results indicate that OFAMultiInstruct significantly improves the model’s zero-short performance over the original pre-trained OFA model across all unseen tasks and metrics, demonstrating the effectiveness of multimodal instruction tuning on MULTIINSTRUCT. As seen in Table 2, OFA achieves extremely low (nearly zero) zero-shot performance on the Grounded VQA task, which requires the model to generate region-specific tokens in order to answer the question. By examining the generated results, we find that OFA, without instruction tuning, failed to follow the instruction and produce results that contain region tokens. However, by fine-tuning OFA on MULTIINSTRUCT, the model is able to better interpret and follow the instructions to properly generate the expected output. Additionally, OFAMultiInstruct outperforms OFATaskName on all unseen tasks, particularly on the Grounded VQA task, where OFATaskName achieves nearly zero per-
5https://ofa-beijing.oss-cn-beijing. aliyuncs.com/checkpoints/ofa_large.pt
formance. This suggests that the performance gain of OFAMultiInstruct mainly comes from instructions rather than multi-task training.

heading: 6.2 Impact of Transfer Learning from
text: NATURAL INSTRUCTIONS
One key question in multimodal instruction tuning is how to effectively leverage the large-scale textonly NATURAL INSTRUCTIONS dataset to enhance the zero-shot performance on multimodal tasks. We observe that only fine-tuning OFA on NATURAL INSTRUCTIONS actually degrades the model’s zero-shot performance on almost all multimodal tasks, as shown by comparing OFANaturalInstruct and OFA in Table 1 and 2. One potential reason for this decline in performance is that during fine-tuning on the text-only dataset, the model learns to focus more on text tokens and attend less to image tokens. To verify this assumption, we compare the attention of text tokens on image tokens between OFANaturalInstruct and other methods and observe that text tokens attend much less to image tokens after fine-tuning on the NATURAL INSTRUCTIONS dataset. The detailed explanations and analysis can be found in Appendix C.
Another observation is that although our transfer learning methods do not lead to significant performance gains over OFAMixedInstruct, both OFASeqInstruct and OFAMixedInstruct achieve lower standard deviation on 6 out of 9 unseen multimodal tasks compared with OFAMultiInstruct, demonstrating
the potential benefits of the much larger text-only instruction datasets to multimodal instruction tuning.

heading: 6.3 Impact of Increasing Multimodal Instruction Task Clusters
text: To evaluate the impact of the number of tasks clusters for instruction tuning, we start with the task groups shown in Figure 2 and group them into five larger clusters: (1) Img Und (VQA + Image Understanding), (2) Grounding (Grounded Matching + Grounded Generation), (3) MISC, ITM (Temporal Ordering + Miscellaneous + Image Text Matching), (4) Relation (Visual Relationship),
(5) Region (Region Understanding), together with (6) NLP, a collection of NLP tasks from NATURAL INSTRUCTIONS. We measure the change in both the aggregated performance and sensitivity of OFAMixedInstruct as we gradually add the task clusters for training.
As we increase the number of task clusters, we observe an improvement in both the mean and maximum aggregated performance and a decrease in sensitivity, as shown in Figure 3. Note that low sensitivity indicates that the model can produce consistent results despite variations in the wording of instructions. These results suggest that increasing the number of task clusters improves the model’s performance on unseen tasks and leads to more consistent outputs. The results also support the effectiveness of our proposed MULTIINSTRUCT dataset.

heading: 6.4 Effect of Diverse Instructions on Instruction Tuning
text: We hypothesize that using a diverse set of instructions for each task during multimodal instruction tuning can improve the model’s zero-shot performance on unseen tasks and reduce its sensitivity to variation in the instructions. To test this hypothesis, we train an OFA model on MULTIINSTRUCT with a single fixed instruction template per task and compare its performance with OFA finetuned on 5 different instructions. As shown in Table 3, OFA finetuned on 5 instructions achieves much higher aggregated performance on all evaluation tasks and shows lower sensitivity. These results demonstrate the effectiveness of increasing the diversity of instructions and suggest that future work could explore crowd-sourcing or automatic generation strategies to create even more diverse instructions for instruction tuning.

heading: 6.5 Effect of Fine-tuning Strategies on Model
text: Sensitivity
In Section 6.3 and 6.4, we have shown that the more tasks and instructions used for instruction
tuning, the lower sensitivity the model will achieve toward the variations in instructions for each task. We further investigate the impact of fine-tuning and transfer learning strategies on model sensitivity. Figure 4 shows the averaged sensitivity of each model across all multimodal unseen tasks. The original OFA exhibits significantly higher sensitivity to variations in instructions compared to models fine-tuned on instruction datasets, indicating that multimodal instruction tuning significantly improves the model’s capability on interpreting instructions, even with varying wordings. In addition, by transferring the large-scale NATURAL INSTRUCTIONS dataset to MULTIINSTRUCT, sensitivity is also reduced by a large margin, highlighting the benefit of fine-tuning the model on a larger instruction dataset, regardless of different formats and modalities.

heading: 7 Zero-Shot Performance on NLP Tasks
text: So far, our focus has been on evaluating the zeroshot performance of multimodal tasks. In this section, we investigate the effect of multimodal instruction tuning on the performance of text-only tasks. To do this, we evaluate all our approaches on 20 natural language processing (NLP) tasks from the default test split in NATURAL INSTRUCTIONS6. The detailed task list can be found in Appendix B.2.
As shown in Table 4, OFAMultiInstruct outperforms OFA, despite the instruction tuning dataset and the unseen dataset are in different modalities. This suggests that multimodal instruction tuning can help improve the zero-shot performance on NLP tasks. In addition, we observe that OFANaturalInstruct achieves the best performance on NLP tasks and OFAMixedInstruct is more effective in preserving the zero-shot capability gained from NATURAL INSTRUCTIONS on NLP tasks compared
6https://github.com/allenai/ natural-instructions
to OFASeqInstruct. Based on the results in Tables 1, 2 and 4, we conclude that OFAMixedInstruct is able to achieve overall best aggregated performance on all multimodal and NLP tasks and shows much lower sensitivity towards variations in the wording of instructions, making it the most promising approach.

heading: 8 Conclusion
text: We present a new large-scale multi-modal instruction tuning benchmark dataset – MULTIINSTRUCT, which covers a wide variety of vision and multimodal tasks while each task is associated with multiple expert-written instructions. By finetuning OFA (Wang et al., 2022a), a recently state-ofthe-art multimodal pre-trained language model, on MULTIINSTRUCT with instruction tuning, its zeroshot performance on various unseen multimodal tasks is significantly improved. We also explore several transfer learning techniques to leverage the much larger text-only NATURAL INSTRUCTIONS dataset and demonstrate its benefit. Moreover, we design a new evaluation metric Sensitivity to assess the model’s sensitivity towards the variations in the wording of instructions. Results show that the model becomes less sensitive to these variations after being fine-tuned on a variety of tasks and instructions.
Limitations
Limitations of Data Collection Our proposed dataset only targets English language tasks. Future work should explore multimodal instruction tuning in a more diverse language setting and augment our MULTIINSTRUCT with multi-multilingual tasks. In addition, our current dataset mainly focuses on vision-language tasks. Datasets from more diverse modalities should be considered such as audio (Panayotov et al., 2015; Gemmeke et al., 2017; You et al., 2022) and video (Soomro et al., 2012; Ionescu et al., 2014). While we have built a novel
multimodal instruction dataset containing 62 tasks, the number of tasks and associated instructions remains limited. To address this, future research could consider utilizing crowd-sourcing or automatic generation and augmentation techniques to increase the variety of instructions available.
Limitations of Experiments and Evaluation Our work is the first to explore instruction tuning on multimodal tasks and shows improved performance compared to baseline methods. However, there is still room for improvement, specifically in utilizing text-only instruction datasets. Future research could explore alternative architectures and stronger vision-language pre-trained models, or develop additional training loss functions to better utilize these unimodal instruction datasets. Additionally, we only used OFA as the baseline model as it was the largest open-source multimodal pretrained model available when we conducted this research. As more and stronger multimodal pretrained models being publicly available, it would be interesting to conduct a thorough comparison between models with different sizes. Finally, we take the first step to define sensitivity as a metric to evaluate the robustness of the models on understanding and following human-written instructions, which can be a potential standard metric for all the following instruction-tuning studies. However, it’s only based on the variation of model performance across different instructions for the same task. In the future, we will consider more broad factors, e.g., the model’s capability to understand different instructions for different tasks (Inter-task sensitivity), to further improve the sensitivity metric for instruction tuning.

heading: Acknowledgments
text: This research is based upon work supported by the U.S. DARPA KMASS Program # HR001121S0034. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

heading: A Tasks Defined in MULTIINSTRUCT
text: Table 5 shows the distribution of input and output modalities for both training and evaluation tasks in MULTIINSTRUCT, and Table 6 shows the detailed statistics for all the training and evaluation tasks separately. Tables 7 to 9 provide a comprehensive list of the 62 tasks included in MULTIINSTRUCT, along with one example of instruction for each task.

heading: B More Details for Experimental Setup
text: B.1 Multimodal Evaluation Datasets Text VQA (Singh et al., 2019) requires models to read and reason about the text in an image to answer questions based on them.
Grounded VQA (Zhu et al., 2016) requires models to answer the questions about an image, with the answers being specific visual regions within the image.
Commonsense VQA (Zellers et al., 2019) requires the model to answer a multiple-choice question that requires commonsense reasoning about an image. Both the question and answers are presented in a combination of natural language and references to specific image regions within the image.
Visual Entailment (Xie et al., 2019) requires the model to determine whether the image semantically entails the text.
Natural Language for Visual Reasoning (NLVR) (Suhr et al., 2017) requires the model to answer a question that requires visual and set-theoretic reasoning on a synthetic image.
Visual Text Extraction is a new task derived from Hateful Memes (Kiela et al., 2020) dataset. This task requires the model to extract the text that appears in the image.
Visual Dialogue (Das et al., 2017) requires the model to answer a question given an image and a dialogue history.
Disaster Type Classification (Alam et al., 2022) requires the model to determine the disaster type based on the image.
B.2 NLP Evaluation Tasks
Below are the task names of the 20 NLP tasks that we used to test the zero-shot performance of all the methods. The 20 NLP tasks are from the default test split of the NATURAL INSTRUCTIONS dataset. During testing, we leverage the ’Definition’ of the task as an instruction and prepend it with each input.
task1624_disfl_qa_question_yesno_classification, task133_winowhy_reason_plausibility_detection, task569_recipe_nlg_text_generation, task1631_openpi_answer_generation, task957_e2e_nlg_text_generation_generate, task1386_anli_r2_entailment, task393_plausible_result_generation, task670_ambigqa_question_generation, task890_gcwd_classification, task1534_daily_dialog_question_classification,
task1388_cb_entailment, task190_snli_classification, task1533_daily_dialog_formal_classification, task1598_nyc_long_text_generation, task199_mnli_classification, task1439_doqa_cooking_isanswerable, task1409_dart_text_generation, task1529_scitail1.1_classification, task648_answer_generation, task050_multirc_answerability
B.3 Approaches for Comparison
OFA (Wang et al., 2022a) denotes the original pre-trained OFA model without any fine-tuning. Here, we use OFA-large8 which contains 472M parameters and was trained on 8 tasks shown in Table 10. As reported in Wang et al. (2022a), OFA has demonstrated certain zero-shot capability on unseen multimodal tasks.
OFATaskName is finetuned on MULTIINSTRUCT but it does not use the instructions we created for the tasks. Instead, we prepend the task name to each input and use a semicolon to separate the task name and the input. For a fair comparison, we still keep the two special tokens “[Options]” and “||||” for the option field.
OFAMultiInstruct only fine-tunes OFA on our newly introduced MULTIINSTRUCT dataset with instruction tuning.
OFANaturalInstruct only fine-tunes OFA on the large-scale NATURAL INSTRUCTIONS dataset (Mishra et al., 2022; Wang et al., 2022d) with instruction tuning. To ensure a fair comparison, we evaluate this baseline on instruction templates that removed all specific tokens, including “[Options]” and “||||”, since the model being tested has not been exposed to these specific tokens during instruction-tuning. We want to ensure that the evaluation is not biased in favor of models that have seen these tokens during training.
OFAMixedInstruct fine-tunes OFA on the mix of the large-scale NATURAL INSTRUCTIONS (Mishra et al., 2022; Wang et al., 2022d) and MULTIINSTRUCT dataset with instruction tuning.
OFASeqInstruct sequentially fine-tunes OFA on the large-scale NATURAL INSTRUCTIONS (Mishra
8https://ofa-beijing.oss-cn-beijing. aliyuncs.com/checkpoints/ofa_large.pt
et al., 2022; Wang et al., 2022d) and MULTIINSTRUCT dataset with instruction tuning.
B.4 Training Details
We set the maximum length of input tokens to 1024 and the maximum target length to 512. For image preprocessing, we strictly follow the process in the OFA. Please refer to the original paper for more details. We train the models on 8 Nvidia A100 GPUs with a batch size 8 per GPU, a learning rate of 1e-05, and float16 enabled for 3 epochs for all the setups and datasets. We run all the experiments once.

heading: C Attention Analysis
text: In Section 6.1, we have demonstrated that finetuning OFA with NATURAL INSTRUCTIONS alone results in a decline in its zero-shot performance. In this section, we examine one possible reason for this decline by examining if fine-tuning the model on a text-only instruction dataset causes it to give less attention to image inputs.
To understand this, we conduct an analysis of the self-attention layers within the OFA encoder. The OFA encoder comprises 12 selfattention layers, each with 16 attention heads. We denote the input to self-attention layer l as h(l) = [x(l)1 , . . . , x (l) p , . . . x (l) L ], where L is the length of sequence. The input h(0) = [x
(0) 1 , . . . , x (0) I , x (0) I+1, . . . x (0) I+T ] to the first selfattention layer is actually the concatenation of image embeddings and text embeddings, where I , T is the length of image and text embeddings respectively. For ease of understanding and simplicity, we have altered the naming conventions and refer to xlp, p = [1, ..., I] as image states and xlp, p = [I + 1, ..., I + T ] as text states.
For each self-attention layer, we first compute the attention given to the image states in relation to text states for each attention head. Specifically, for each text state as the query, we sum its attention scores on image states (i.e. the attention scores where the text state is the query and image states are the keys). We then compute the text-to-image attention across all text states. Finally, we average the text-to-image across all attention heads. This results in a text-to-image attention score for each self-attention layer.
Figure 5 illustrates the results of text-to-image attention scores on three unseen multimodal tasks: Text VQA, Visual Entailment, and Visual Text
Extraction. The results on all three unseen tasks show that, in all self-attention layers of the OFA encoder, OFANaturalInstruct has significantly lower text-to-image attention scores compared to other models. This decrease is particularly pronounced in the first two self-attention layers. This suggests that fine-tuning the model on a text-only instruction dataset leads to a reduction in the attention paid to image inputs, which may explain the decline in zero-shot performance.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
section 9
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? abstract, section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? section 5, 6, 7, appendix B
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? appendix B.4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? appendix B.4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 5
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 5
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? section 3
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? section 3
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? We don’t have paid participants involved
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? The data is publically available
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? The data is collected from publicly available benchmark datasets and there is no potential ethical issue.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Future work should explore multimodal instruction tuning in a more diverse language setting and augment our MULTIINSTRUCT with multi-multilingual tasks. In addition, our current dataset mainly focuses on vision-language tasks. Datasets from more diverse modalities should be considered such as audio (Panayotov et al. , 2015; Gemmeke et al. , 2017; You et al. , 2022) and video (Soomro et al. , 2012; Ionescu et al. , 2014). While we have built a novel
multimodal instruction dataset containing 62 tasks, the number of tasks and associated instructions remains limited. To address this, future research could consider utilizing crowd-sourcing or automatic generation and augmentation techniques to increase the variety of instructions available. Limitations of Experiments and Evaluation Our work is the first to explore instruction tuning on multimodal tasks and shows improved performance compared to baseline methods. However, there is still room for improvement, specifically in utilizing text-only instruction datasets. Future research could explore alternative architectures and stronger vision-language pre-trained models, or develop additional training loss functions to better utilize these unimodal instruction datasets. Additionally, we only used OFA as the baseline model as it was the largest open-source multimodal pretrained model available when we conducted this research. As more and stronger multimodal pretrained models being publicly available, it would be interesting to conduct a thorough comparison between models with different sizes. Finally, we take the first step to define sensitivity as a metric to evaluate the robustness of the models on understanding and following human-written instructions, which can be a potential standard metric for all the following instruction-tuning studies. However, it’s only based on the variation of model performance across different instructions for the same task. In the future, we will consider more broad factors, e. g. , the model’s capability to understand different instructions for different tasks (Inter-task sensitivity), to further improve the sensitivity metric for instruction tuning