MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning
Authors: Haozhe Zhao
Zefan Cai
Shuzheng Si
Xiaojian Ma
Kaikai An
Liang Chen
Zixuan Liu
Sheng Wang
Wenjuan Han
Baobao Chang
Venue: ICLR 2024 poster (2024)
Keywords: multi-modal in-context learning; multi-modal instruction tuning; vision-language model
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Since the resurgence of deep learning, vision-language models (VLMs) enhanced by large language models (LLMs) have grown exponentially in popularity. 
However, while LLMs can utilize extensive background knowledge and task information with in-context learning, most VLMs still struggle with understanding complex multi-modal prompts with multiple images, making VLMs less effective in downstream vision-language tasks.
In this paper, we address the limitation above by 1) introducing vision-language Model with **M**ulti-**M**odal **I**n-**C**ontext **L**earning(MMICL), a new approach to allow the VLM to deal with multi-modal inputs efficiently; 2) proposing a novel context scheme to augment the in-context learning ability of the VLM; 3) constructing the Multi-modal In-Context Learning (MIC) dataset, designed to enhance the VLM's ability to understand complex multi-modal prompts.
Our experiments confirm that MMICL achieves new state-of-the-art zero-shot performance on a wide range of general vision-language tasks, especially for complex benchmarks, including MME and MMBench. Our analysis demonstrates that MMICL effectively tackles the challenge of complex multi-modal prompt understanding and emerges the impressive ICL ability. Furthermore, we observe that MMICL successfully alleviates language bias in VLMs, a common issue for VLMs that often leads to hallucination when faced with extensive textual context.
Our code, dataset, dataset tool, and model are available at https://github.com/PKUnlp-icler/MIC.

Published as a conference paper at ICLR 2024
MMICL: EMPOWERING WITH MULTI-MODAL IN-CONTEXT LEARNING
MODEL
Haozhe Zhao Zefan Cai*! Shuzheng Si*! Xiaojian Ma², Kaikai An
Liang Chen', Zixuan Liu', Sheng Wang", Wenjuan Han"s, 55 Baobbo Chang
NNational Key Laboratory for Multimedia Information Processing, Peking University
School of Software and Microelectronics, Peking University, China
3 Nattional Key Laboratory of General Artificial Intelligence. BIGAI
*Paul G. Allen School of Computer Science and Engineering, University of Washington
'Beijing Jiaotong University
mi mimazhee53600ggaall 1, com, zef zefncai@gmail com, sishuzhengêsstu pku. edu. cn
https:/// thub com hwtp//KKnnlppppp http://knnnnn-iccee///EE.
ABSTRACT
Since the resurgence of deep learning, vision-language models (VLMs) enhanced by large language models (LLMs) have grown exponentially in popularity. How- ever, while LLMS can utilize extensive background knowledge and task infor- mation with in-context learning. most VLMs still struggle with understanding complex multi-modal prompts with multiple images, making VLMs less effective in downstream vision-language tasks. this paper, we address the limitation above by 1) introducing vision-language Model with Multi-Modal In-Context Learning(MMICI)
new approach allow the VLM to deal with multi-modal inputs efficiently; 2) proposing novel context scheme to augment the in-context learning ability of the VLM; 3) constructing the Multi-modal In In-Context Learn- ing (MIC) dataset, designed to enhance the VLM's ability to understand com- plex multi-modal prompts. Our experiments confirm that MMICL achieves new late-of-the-arr zero-shot performance wide range of general vision-language tasks, especially for complex benchmarks, including MME and MMBench. Our analysis demonstrates that MMICL effectively tackles the challenge of complex multi-modal prompt understanding and emerges the impressive ICL. ability. Fur- thermore, we observe that MMICL successfully alleviates language bias in VLMs, a common issue for VLMs that often leads to ballucination when faced with ex- tensive textual context. Our code, dataset, dataset tool, and model available
INTRODUCTION
General-purpose vision Dision-language pre traitrdined models (VLMs) have made significant advancements Li ct al. 2022] 2023d Zhu et al. 2023] Li et al., (023b). Recent VLMs mostly augment a large
language model (LLM) with visual encoder and exhibit impressive zero-shot capacities in various visual tasks. However, unlike LLMs that can extract rich background knowledge and task information from the prompt with in-context learning (ICL), most VLMs still struggle to understand complex multi-modal prompts that include multiple images. Previous studies (Li et al. 2023d b) primarily focus handling the user queries with single image rather than multi-modal prompts with interleaved multiple images and text. Although some VLMs like Flamingo (Alayrac et al. (2022) and Kosmos-1 (Huang al., 2023b) can handle user queries with multiple images, their pre- training data can not provide more sophisticated multi-modal prompts than interleaved image and text crawled from the web (Awadalla et al. 2023). Hence, there is gap between the prompts used pre-training these VLMs and the user queries in real-world scenarios. which always contain multiple images and
"qqual contribution.
Prooecct lead.
"Correspnnding author.

---

Published as a conference paper at ICLR 2024
Please deseribe the information of
Carefully analyae given images and snswer the question:
Tell there a honse in
especially the relationship
What diffcrences does
and
not passible to from the
imaae a horae is in the image.
have?
A man scen walliing down stroct with large orange sittnng on the man's head, and scome to be enjoying the walk.
Here some examples
Sporty car road vs, off road jeep with mountinous terrain the background on the ofher hand.
The is horac
there horse
What similaritiss the first sccond image have??
Please tike a closer look the two images and explain the connection. between them.
Therefore, please refer
They has similarity of vehicle One is on read and the other has mounainous terrain with capped mountains the foragound.
and shape
and image?
hone
The baby crying as he booke the cup.
Yes, there is horse the and standing on grassy
(8)
What huppens the man after hitting the bull?
He falls the ground in front of green grassy with and shrubbery surnounding
spot where he the golf
(4)
These inage illustone the growth plases tree, please describe the comenee of each image carefully.
The image just garmirating, the image only imsge luxusiant, and the image 3 is growing plant.
Figure 1: Examples of vision-language dialogue generated by MMICL typically contain prompts with interleaved images and text. MMICL understands spatial (a), logical (b), and temporal (e) relationships among images. MMICL can also grasp text-to-image references as (c).(d) and (l).
more sophisticated text. Specifically, these VLMs may suffer from the following three limitations, which makes VLMs less effective in downstream vision-language tasks.
Previous studies rarely attempt to address the issue of text-to-image reference in the multi- modal prompts. However, there are often intricate referential relationships between the text and images in user queries, with different words mentioning different images. For example, the user may ask a specific question about multiple images [ and Fig. (f) or use multiple images as exemplars to ask the question only about a specific Image(Fig. dd). However, the training data used in previous studies (Li et al., 2023d Alayrac et al.) 2022; Huang et al. 2023a) are crawled from the web and may lack explicit rext-to-image references. VLMs, thus might fail to handle user queries involving intricate text-to-image references.
Hard to Understand Text-to-Image Reference:
Hard to Understand the Relationships between Multiple Images: There often spatial, temporal, and logical relationships between multiple images, and correctly understanding them allows the model to handle user queries better. However, the pre-training data used by previous VLMs (Alayrac) eet al. 2022) are collected from the internet, lacking close connections among images, especially when these images are far apart on the same webpage. hampers the ability of VLMs understand the intricate relationships among the images and further limits their reasoning ability.
Hard to Learn from In-Context Multi-Modal Demonstrations: Previous studies have shown that pretrained LLMs can benefit from few in- co-context demonstrations (Brown et al. 2020 Dong al. 2023). However, the ICL ability of current VLMs is rather limited, specifically: 1) VLMs
like BL.P-2 (Li al. 2023d), LLaVA ([.i et al.) 2023b) only support multi-modal prompts with single image, hampering their abilities to use multiple multi-modal demonstrations to enhance their performance during the inference; 2) Althhh VLMs such Flamingo (Alayrac et al. (022) support multi-image inputs during pretraining and emerge ICL abilitics, their context schemes fail to provide text-image references and closely related images. It inhibits them from offering sophisticated enough prompts to the VLMs, thereby limiting the effectiveness of their ICL ability. Besides, the lack of further supervised instruction tuning hinders their effectiveness across downstream tasks.
In this paper, to address the aforementioned limitations 1) We present MMICL, a new approach to allow VLMs efficiently deal with multi-modal inputs, including relationships among multiple
2

---

Published as a conference paper at ICLR 2024
LLM
LLM
cubbed
TeM
andel
VPG
VJG
Inagz Dedimaion
VPG
VPG
VLMs Focused single image
(b) VLMs with few-shot ability
(c) MMICL
Comparison of different VLM architectures: VLMs focused single image, VLMs with few-shot ability, and MMICL with equal treatment of image and text representation.
Figure 2:
images and text-to-image references. 2) We propose novel context scheme in which incorporating an extra image declaration section, along with the inclusion of image proxy tokens, enhances the ICL ability of the VI.M. 3) We construct multi-modal in-context learning dataset in accordance with the proposed scheme. The dataset is adapted from a range of existing datasets and can be used to provide support for the training of more capable VLMs.
Our experiments show that MMICL achieves new state-of-the-ar performance on various of vision- language benchmarks including MME (Fu et al. 2023) and MMBench (Liu al. 2023c) [ Com- prehensive examinations of the three limitations we aim to address reveal that MMICL exhibits exceptional ability in understanding text-to-image references (13- (13-points improvement on the vision- language compositionality benchmark, Winoground (Thrush et al. 2022a)) and intricate relationships among images(12 (12-points improvement on the multi-image reasoning benchmark, RAVEN (Huang et al.] ). Moreover, MMICL demonstrates impressive multi-modal ICL. performance across var- ious tasks. We also observe that MMICL efficiently mitigates the language bias, which often causes VLMs to ignore visual contents when facing extensive textual contexts, leading to hallucinations.
2 MMICL
2.1 MODEL ARCHITECTURE
Most VLMs utilize Visual- Prommpt Generators (VPG) (e.g., Resampler (Alayrac et al., 2022). Q- former (Li et al. 2023d)) to extract visual embeddings from the image features encoded by vision backbones and use visual embeddings to help LLMs understand visual inputs. The model architecture shown in Fig. a belongs to VLMs that focus on prompts with single image, such as Blip-2 (Li al., 2023d). which always places the image at the top of the entire input and can not handle the inputs with multiple images. In Fig. b,, VLMs with few-shot ability, such as Flamingo (Alayrac) ct al.) 2022), encode images into image embeddings with fixed number of visual tokens and inserts new gated cross-attention layers into the LLM to inject visual features. Different from previous work, MMICL shown in Fig. 2 treats image and text representations equally and establishes the reference between image and text via image declaration. It enables users to have the flexibility input multiple images and text in any desired order, with no restrictions on the quantity or placement of images in contexts. As shown in Fig. 5 each given image is encoded by a vision encoder (e.g.. ViT (Radford) ee al. 2021)) to get the image representation. Then, we use the Q-former as the VPG to encode images into embeddings understandable by the language model. We utilize fully connected layer as the projection layer to convert each visual embedding to the same dimension as the text embedding of the LLM. Finally, we combine the visual and text embeddings into an interleaved style and feed them into the LLM. This design is natural extension of the original attention mechanism in LLMs. We set weights for mapping query and value vectors in the attention layer of LLM as learnable to better adapt to multi-modal prompts with multiple images. More details are presented in Appendix FF
2.2 THE DESIGN OF CONTEXT SCHEME OF MMICI
In this section, we outline the design of the Context Scheme for MMICL. The proposed scheme is devised to proficiently transform the interleaved image-text data into the training context for MMICL.
Results of MMICI. submitted August 28th, 2023.
3

---

Published as a conference paper at ICLR 2024
Original YL Task
Image Declaration
(b) Multi-modal Data with Interconnected Images
Visual Question Ausweriag
Carefully analyze image J: [[MG]]
Carefullly analyze images answer question.
to answerr the question.
image 0: [fMG.]
image I: [[MG]]
the and womenn are quarrelligg?
Q: men wumen quarrellingg Yes
quarrelling with image [PMGG]]
Answer: Yes
(c) Unified
Format
Q: The image is [1M6g]
Carefully amalyze the
Caplasning
image is |/MG)1
image 0 ginerate concise and accurate descripsion that sucuralely represetts olyccds, pappl, soonry preseul. Az An aimplae flying the sky;
Carefully analyze imuge tn generase a concise and accurate description that occarstely represents the objects, people, and scenery present.
An airplane flying the sky.
Q: The image j [iM6]]
Carefullly anulyze the
image to generate cockise and accurati description that accurately represents the objects, people, soenery present.
Machine Annotation
Manual Annotation
[MMG]
Image Proxy
Figure 3: Context scheme for MMICI, which seamlessly transforms the interleaved image-text data into training context unified format.
2.2.1 IMAGE DECLARATION
Users may use textual descriptions to refer particular images in their queries. Such reference can provide information about the visual content mentioned in the text to the VLM, allowing it to learn alignment between two modalities. To precisely link text and image, templates for each image in mixed inputs, as shown in Fig. 3 Firstly, we allocate a unique image proxy ([MG to reference the visual embedding of image j. which provides unique identifier for VLMs to index and distinguish between visual and text embeddings. Then, we utilize natural language prompts to establish references between text and image. Incorporating the explicit -xt-tto-mmaee reference in the image declaration assists the model in correlating the text with the appropriate image. Meanwhile, the image declaration, maintained as textual content, can also preserve the flexibility to appear at any position within the prompt. Each instance I, follows the structure, where the X; symbolizes the set of image decorations that can be placed anywhere within the instance L, q: and a, denote the question with instruction and corresponding answer, respectively.
form image declaration
I = (X,, q: a;)
2.2.2 MULTI-MODAL DATA WITH INTERCONNECTED IMAGES
To incorporate abundant multi-image information within the context scheme of MMICL, we generate interconnected multi-image data that includes spatial, logical, and temporal relationships. It aids MMICL in understanding the intricate relationships among images in user queries. Specifically, we derive frames from videos to build multi-image data. The frames extracted from video inherently sustain close temporal and spatial relations, which infuse spatial and temporal correlation information among images into the context scheme. Besides, we build multi-image data from images depicting multiple object interactions. We detect the objects within the image and generate bounding boxes for each object. We acquire multiple sub-images of different objects by cropping the image according to bounding boxes. We then replace the textual references to these objects with their corresponding cropped images, thus forming interleaved multi-modal data with logical and causal interconnected images, as delineated in Fig. 3 and Fig.
4 Each instance I, comprises a question-answer text pair along with K images, where the X, X; represents the image declaration for the k-th image.
I = (fx), } qi a4)
(2)
2.2.3 UNIFIED MULTI-MODAL IN-CONTEXT FORMAT FOR DIFFERENT TASKS
We propose a design for producing multi-modal in-context learning data for different tasks to enrich the context scheme of MMICL. It aims to improve the instruction-aware ability of VLM and expand its abilities for proficient mudti- in-context learning. Specifically, we start by crafting diverse instructions for each task and generate different templates for the task utilizing these instructions. We then fill in the randomly selected template with the original task to assemble data equipped with instructions Appendix G Moreover, we convert the data into a multi-moda in-context format by
4

---

Published as a conference paper at ICLR 2024
Annotation from Existing Datasets
Step3: Image Declaration
Raw Image:
Image4 Image6
Task Descrip tion:
Image5
Image3
Infoc
Carefully anulyze images to the question.
Image Description:
Image
Amage7
Image 0: [LMG0]
Image L:[MMGI]
Image 2: [IMG2]
Bounding Box Description:
Bounding Box for Image 1: [869, 384, 1261,
Image 3: [LMG3]
Image [MMG4]
...
Bounding Box for Image- [163, 338,
Ques tion:
Why is imuge inside of cage?
Image 5: [MMG5]
Image 6: [IMG6]
Options:
Choice Image 7 is about to attack image image 2. swarm of butterflies there.
Choice Because there
Choice Image 3 has pet bird in the cage.
Image 7: [IMG7]
Choice 3: Image 7 is in the so that can't fly away.
Step4: Data Construction
Answer:
Choice 3
Input of Data Format:
Processed Instuction:
Stepı: Raw Information Preparation
Conduct a meticulous examination of the visual details within
the images, employing critical analysis artention
Raw Instruct Fion:Now you need the question based previous images. Dataset Description: Image comprehension task requiring understanding of objects and relations.
Ques tion:
Why image? 7
inside of cage?
Options:
Choice O: Image
is about to attack image and imago
Step2: Chatgpt Refinement
Choice I: Because there swarm butterflies there.
Choice Image has pet bird cage.
Comprehensive Instruction:
Choice 3:
Image
the cage that it can't 't fly away.
Conduct a meticulous examination the visual details within the images, employing critical analysis and attention nuanced elements, in order accurately and comprehensively address the posed question.
Output Data Format: answer is Choice
Figure 4: Illustration of automatic data construction pipeline for multi-model data with interconnected images. The automatic construction is based on the existing annotation of VCR dataset (Zellers et al. 2019) without human involvement. ChaiGPT is used for instruction refinement.
constructing few-shot exemplars generated by sampling instances from the data. These exemplars combined with the input instance produce the multi-modal in-context data. In this way, can transform all tasks into unified multi-modal in-context format, as illustrated in Fig. 3 This method facilitates amassing an extensive amount of high-quality data from different tasks, enriching the context scheme of MMICL with an abundant diversity of multi-modal in-context data teeming with diverse instructions. Ultimately, this improves the model's ability to follow instructions and multi-modal in-context learning ability. Each instance I, comprises N exemplars.
I, ((P1,-.. PN). 9: a:)
(3)
Each exemplar P3 (X,, qy ay), X, denotes the image declaration of the j-th exemplar. q, and ay denote the question and answer for the j-th exemplar, respectively.
2.3 MULTIMODALITY IN-CONTEXT LEARNING (MIC) DATASET CONSTRUCTION
To help VLMs understand the complex prompts, construct MIC dataset by gathering data from public data resources and converting them based on the context scheme. It has three key aspects: 1) image declaration, 2) multi-modal data with closely related images, and 3) multi-modal in-context data for different tasks. Training set of MIC comes from 16 datasets across 8 categories, while the test set comes from 18 datasets across 10 categories.
Our dataset is automatically constructed based existing datasets. Firstly, we created an image declaration for each instance in all datasets to produce datasets with explicit text-to-image reference. Secondly. created an instruction template for each dataset and asked Chatgpt to rewrite instructions, filling in the data from the existing datasets to obtain a dataset with diverse instruction formats. Finally, used those datasets with instructions to construct the MIC dataset according proposed
5

---

Published as a conference paper at ICLR 2024
quaneeling
Encoder
Queties
Q-Forner
Projection
trained LLM
Yes, image is quarrelling with image
Stagel
Stage II
Pretrnining
Context Tuning
Visiza Encoder
Visioa Eincoer
Untreezze
hg
Text embedding
+
DOG
Image Proxy
Projection
Projection
Freeze
Visinn Prompt
Vivaal embedding
Pre-traieed 1:M
Presmiinnt L.
Generator
Figure 5:
Illustration of MMICL architecture and training paradigm. The upper part denotes the overview of model architecture, and the bottom denotes the pipeline of two-stage training paradigm.
context scheme. For the example presented in Fig- 3 and Fig. (e.g.. two people quarreling with each other), we constructed the data based existing annotations (i.e., bounding boxes and the relation between bounding boxes) provided by the VCR dataset (Zellers et al. (019). Additionally, also constructed in-context learning dataset by sampling examples from the original dataset. We also extracted eight frames per video from video datasets to generate the multi-modal data with interconnected images. Details presented at Appendix D
We convert all data into vision-language Q&A format to create high-quality multi-modal training data and accumulate 5.8M samples in MIC dataset. Due resource constraints, we approximately 10% of the data with the sampling strategy described in Appendix [F to finetune MMICL. It is anticipated that larger model trained all of our data would yield more promising result.
2.4 TRAINING PARADIGM
Stage I: Pretraining. This stage aims to assist the model aligning the image and text embeddings. During this stage, both the vision encoder and the LLM remain frozen. The VPG (i.e., Q-Former) and projection layer are trained to learn visual embeddings that can be interpreted by the LLM.
Stage I:: Multi-Modal In-Context Tuning. In this stage, we aim to address the aforementioned limitations and take our model a step further by extending it to multi-modal in-context learning. Specifically, aim make the model understand the intricate referential relationships between the text and images and the complex relationships among multiple images and ultimately acquire a proficient multi-moda. in-context learning ability. Therefore, perform multi-modal In-Context Tuning on MIC dataset. During the stage II, we freeze the image encoder, Q-former, and LLM while jointly training the projection layer and query and value vectors. Details be found in Appendix
EXPERIMENT
3.1 EXPERIMENTAL SETUP
Evaluation Setup. We aim to develop general-purpose VLMS that can generally adapt diverse, challenging multi-modal prompts. Therefore, we evaluate our models in several vision-language benchmarks, including tasks that involve images and vidcos. The metrics used in these benchmarks and further details are shown in Appendix
Models and Baselines. We provide two versions of MMICL: (1) MMICL (FLAN-T5) which uses BLIP-2 (Li et al. 2023d) as the backbone and (2) MMICL (Instruct-FLAN
which uses Instruct-
BLIP (Dai al. (023) the backbone. We also adopt XL and XXL of FLANTS (Chung et al. (022) model for both versions. We compare MMICL with following strong baselines: Flamingo (Alayrac
al. 2022 KOSMOS-1 (Huang al. 2023a), B1.IP-2-FLAN-TS InstructBI.IP-FL..

---

Published as a conference paper at ICLR 2024
Cugnition
Perception
Medel
Medel Sine
Code.
Color
Poster
Total Avg.
LLeVA
138
57.14
50.000
57.50
50:00
50.000
55,00
5000
50:00
49:00
MiniGPT-4
13B
59.20
45.00
0.00
40.00
68.33
55.00
43.33
75.00
57.50
41.84
71.75
54.00
60.50
51. 51.85
Modal GPT
98
49.29
02.50
00.00
55.00
61.67
55.00
58.33
08.33
$2.50
57.82
73.82
68.00
09.75
29.50
62.97
VLn 60
60
39.29
45,00
50.00
47.50
85.00
50.00
JB.33
55.00
42.50
63.99
53:24
146.25
83.75
75.25
63.36
7M
64.29
50,00
77.50
$7.50
70.00
35.00
63.33
73.35
77.50
84.01
53.53
141.75
64.75
77.25
74.27
LIVIN
13B
87.14
65.00
47.50
50,00
185.00
88.31
63.33
75.00
107.5
50
79
47.35
136.75
93.50
87.25
86.65
LLaMA Adupter V2
7B
91.43
62.50
50.00
55.00
120.00
50.00
48.33
75.00
00
99.66
148.50
69.75
57.26
@PLLIGGOO!
78
66,00
92.00
50,00
55,00
InstrucIBLIP
66.67
BLIP-2
12:18
110:00
40.00
75.00
160100
135.00
73.33
148,33
110.00
141.84
145.25
138,30
136.50
112.13
Lynx
7B
110.71
17.50
42.50
45.00
195.00
151.67
90.00
170.00
77.30
124.83
118.24
168.50
162.00
119.50
113.50
GIT2
5.18
09.29
50.00
07.51
45.80
190100
118.33
96.67
158.53
65.00
112.59
145 8M
158.50
140.50
146.25
113.85
Otter
9M
106.43
72.50
57. 50
70.00
195.00
86:33
86.67
113.33
72.50
138.78
172.65
158.75
137.25
129.00
114.19
Cheetor
73
98.57
77.50
57.50
87.50
180000
95.67
30,00
116.67
100.00
147.28
164. 12
156.00
145.73
113.50
115.79
L LRV-Insimntton
78
100.71
70,00
55.00
72.50
165.00
611.67
86.67
165.00
110.00
150.04
112.5
147
08
160.53
101
116.29
BLIVA
12.18
136.43
57.50
77.50
60,00
180.00
138.33
$1.67
150,00
87.50
155,10
140 58
151.90
89,50
133.25
119.23
136.43
$2.50
132.50
77.50
170000
156.87
146.26
153.75
135.50
Table 1: Evaluation results on the MME. Top two scores are highlighted and underlined, respectively.
plants suncunding lighthulls
Do you agnoe following image
cf
C2: A lightbilb surrounding some plants
Q1: the Caption monches the imagel
Q2: the Caption manches the image2?
Conzzett
Comeett
Correct?
Q3: the Caption2 monches the image ?
Caption2
image??
Answer: P[Yes(Q)
XGg 10)
KG 1o
Illustration of two complex vision language reasoning tasks: Winoground (Thrush et al., 2022b) (Left) and RAVEN (Zhang et al. 2019) (Right).
Figure 6:
Shikra (Chen et al. 2023a), Otter (Li et al. 2023a). Ying-VLM (Li al. 2023e). The details of MMICL and baselines are shown in Appendix H and Appendix O
3.2 GENERAL PERFORMANCE EVALUATIONS
We evaluate the general performance of MMICL on both MME (Fu et al. 2023) and MMBench (Lu al. (202c)) benchmarks MME evaluates VLMs with 14 sub-tasks that encompass cognition and perception abilities. Results in Table D show that MMICL can achieve the best average scores com- pared with current VLMs on cognition and perception tasks. MMICL also demonstrates outstanding performance and significantly surpasses other VLMs on the MMBench benchmark, which thoroughly evaluates the diverse skills of VLMs. The detailed results are presented in Table 22 See Appendix ! and for MMICL's evaluation detail and comparisons with other VLMSs
3.3
PERFORMANCE PROB
3.3.1 UNDERSTANDING TEXT-TO-IMAGE REFERENCE
The Winoground Thrush et al. 2022b pro- poses a task of correctly matching two given images and captions, depicted in the left of Fig. 6 The challenge lies in the fact that both captions consist of the exact same words, albeit in different order. VLMs must compare both images and texts to discern their subtle differ- ences and capture the implicit reference between them. Therefore, we select the Winoground to evaluate whether VL.Ms understand the text-to- image reference, MMICL is given two images and two captions in each prompt during evaluation. Results in Table demonstrate that MMICL captures the referential relationship between image and text, surpassing previous baselines.
Table 2: Results on Winoground across text, image
and group score metrics.
Model
Text
Image
Group
MTurk Human
89.50
88,50
85,50
VQ2
Yarom al.
2023
47.00
42.20
30.50
PALI
[Chen et al. 2022
46.50
38.00
28.75
Blip-2 (Lr Bl al. 2023d
44.00
26.00
23.50
GPT4-V
Wi et at.
2023
69.25
46,25
39.25
MMICL (FLAN-15XXXL)
45.00
45.00
43.00
All the reported performance for the baseline methods is from the leaderboard of MME Fu et al. 2023 and MMBench (Luu al. 2023ch. We report the result of MMICL with SarructRlip-FLAANS--XXX backhone.
7

---

Published as a conference paper at ICLR 2024
Model
Hickkr 30K WebSRC VQAv2
Momes
VixWiz
Flamingo:15 3B
Alavrac et al
2024
(wlo ICL cxample)
60,60
49.20
53.70
28,90
Flamingo-38 3B
Alayrac
(w/ ICL examples
72.00
53.20
53.60
34.00
Flamingo-98
Alavzze et al
M22
(w/o ICL example)
61.50
51.80
57.00
28.80
Hamingo-98
Alayyae al
(w/ ICL, examples
72.60
56.30
62.70
34.90
KOSMOS-1
Huine et
21
(wlo example)
67.30
3.80
63,90
KOSMOS-1
Toaing eI 01
20236
(w/ ICL cxamples (4)
75.30
51,80
35.30
cxample
BLIP-2
Li e5 al 2023d
(FLANTSXXL)
64,51
12:25
58.79
60.00
25.55
1P-2
(PLANII-XXI)
60.74
10.10
62.25
22.50
Instraall.. IP
Dai et al. 2023
(FLANNT-XL))
77.16
10.00
36.77
58.54
32.08
Instruc:LLIP
(Daret
(FLANTS-XXL)
73.13
11.50
63.69
61.70
15.11
JCL cxample Evalustion
MMICL (FLAN- TS-XL) (wio ICL example)
83.47
12.55
62.17
60.28
25.04
MMICI. ANN T5-XI.) (w/ ICI, examples (4))
83.84
12.30
60.80
50.17
MMICI (FL AN- TS-XXL) twfo example)
85.03
18.85
69.99
60.32
39.34
MMICL (FLAN--T----OOOA
examples
18.70
MMICL (lastruct-FLANN-TS
(wlo ICL esample)
82.68
14.75
69.13
61.12
29.92
MMICL (lastruct-LAN (5-X)) ICL examples (4)
88.31
14,80
69.16
61.12
33.16
MMICL (Iestruct-FLAN) TS XS-XXL) (wlo ICL example)
73.97
17.05
70.30
62.23
24:15
MMICL (Instruct-FLANN XXL) (wi ICL examples (4))
88.79
19.65
70.56
64.60
50.28
Table 4: Main results illustrating the multi-modal in-context learning ability of MMICL across
various vision-language tasks") All metrics employed in the evaluations introduced in Table 25
Model
Model Size
Average Performance
Don't 't Require Infomation
Require Infomation
Performance Gap
Random Guess
35.50
35,80
34,90
Ying-VLM eL 2023e
13.68
55.70
66.60
44,90
21.70
InstrnerRI. IP (aa er al 2023
12.18
71.30
82.00
60:70
21,30
Otter et 20230
9B
63.10
70.90
55.70
15,20
Shikra (Chen nl. 2123a
7.2R
45,80
$2.90
39.
13,60
81.70
Table 5: Zero-shot performance of different VLMs on ScienceQA-IMG dataset in different split. MMICL outperforms other VLMs by successfully alleviating language bias.
3.3.2 UNDERSTANDING COMPLEX IMAGE-TO-IMANE RELATIONSHIP
RAVEN (Zhang et al. 2019 Huang et al.
Table 3: Zero-shot generalization on Raven IQ test.
2023a) test is widely used evaluate the nonver- bal reasoning ability of VLMSs. Each instance has or images as inputs and candidate im- ages with unique answer, and the goal is to predict the right image as shown the right of Fig. 6 It requires visual and logical skills to understand the relationships among images. We conduct zero-shot experiments the Raven test evaluate VLM's ability to understand image- to-image relationships. The result in Pable3]showe that MMICL achieves 12 points improvement compared KOSMOS-1. It indicates that MMICL is able capture the complex image-to-image relationships and conduct nonverbal visual reasoning tasks.
Model
Accuracy
Random Choice
17
InstructBlip Dai
2023
10.00
Otter (Li et al. 2023a
22.00
KOSMOS- (Huang et al. 2023a
22.00
MMICL I-
34.00
3.4 LEARNING FROM IN-CONTEXT MULTI-MODAL DEMONSTRATIONS
As shown in Table we evaluate the multi-modal in-context learning ability of MMICL across various lasion-language tasks. MMICL outperforms other VLMs on both the held-in and held-out datasets and achieves the state-of-the-art sew-shot performance. For example, few-shot evaluation (4-shot) MMICL on the VizWiz benchmark outperforms the baseline Flamingo-9B (Alayrac et al., 2022) and KOSMOS-1 (Huang al. 2023b) by 15.38 and 14.98 points, respectively. Since VizWiz has never been exposed in the training data, this superior suggests the ability of MMICL to generalize new tasks with a few exemplars. The few-shot performance of Flickr30K decreases with examples given because the captions examples may provide noise for the VLM to finish the task(i.e., in-context exemplars generally do not provide hints for models perform image captioning tasks).
"The anomalous score of InstructBlip the Vizwiz dataset results from correct outputs not heing calculated by the VQA accuracy metric due the exact match failure.
8

---

which Published as a conference paper at ICLR 2024
TeonQA text VisDial
IconQA img
Rongard
Stage
Slage (Blip- 2-FLA--SSSAAAAA XL)
61.62
45.44
35.43
48.42
52.75
Stage (BBip-2-FLANIS--
63.18
50.08
36.48
48.42
59.20
Stage (InstrucBBLIP FLANTS- XL)
35.36
53.15
Stage II-N:c/BBBPPPFANS--XXXX
65.06
51.39
36.09
45.10
63.35
Stage1 + Stuge
Stage 1+ Stage (BLIP-2-FLAN-TS--L)
62.85
47.23
35.76
51.24
56.95
Stage Stuge II SOLI--2-FLAN-SS--XA
64.73
50.55
37.00
34.93
68.05
Stage Stage (JnsrucBLLP-FLLNN-TT
70.54
52.55
36.87
47.27
74.20
Stage Stage (IastucBLII--FLN-TTSAA
66.45
52.00
37.98
60.85
67.20
Table 6: Ablation study on Training Paradigm across five datasets: VSR (Liu et al. (022). IconQA- text (Lu et al. 2021), VisDial (Das et al. 2017), IconQA-img, and Bongard- -Ho (Jiang et al. (022).
Model
MMESnerattio
MMECcttttiinn leon-QA
NIVR2
Raven Winnground
w/o context scheme
1238.99
316.79
52.80
56.65
8.00
6.00
w/o image declaration
1170.87
341.07
47.15
61.00
18.00
3.00
w/o in-context format
1141.02
345,36
51.95
62.63
28.00
20.00
-//o interrelated images
1207.70
333.21
54.35
59.60
16.00
25.75
MMICL
1303.59
370.71
58.12
72.45
32.00
38,75
Table 7: Ablation study on Context Scheme across different tasks. We only report the overall perception and cognition score of MME and the Group score of Winoground.
3.5 HALLUCINATION AND LANGUAGE BIAS OF VLMS
Current VLMs have significant visual hallucinations (Li al. 2023f preventing VLMs from benefiting from multi-modal ICL. Especially when dcaling with complex prompts with multiple images (e.g,, multi- modall chain of thoughts (Zhang al. 2023b)). VLMs often overlook visual content when facing extensive text. This language bias reduces their efficiency in answering questions that require both images and text. ScienceQA-IMG (Lu et al. (022) a challenging task that requires a model to use both modalities to answer the question. We manually split the dataset into two groups:
questions needing images to answer and those not. Details are presented Appendix Q Extensive experiments in Table ddemonsrattttt that MMICL effectively mitigates language bias as it performs equally well in both groups. We also examined object hallucination in MMICL in Appendix L shows impressive performance.
3.6 ABLATION STUDY
We conduct an ablation study on various tasks to evaluate the effect of multi-modal in-context tuning. Table 6 displays significant enhancement due to multi-modal in-context tuning. It can be observed across all types and sizes of models, especially for tasks that involve multiple images. This indicates that with the help of Stage II, MMICL can handle complex multi-modal prompts and accomplish challenging tasks with multiple images. Result in Appendix Klaso confirms this point with the outstanding performance of MMICL across video datasets.
Ablation Study on Training Paradigm:
Ablation Study on Context Scheme: We conducted ablation experiments using the InstructBLIP. FLANT5-XL. the backbone model in various settings verify the effectiveness of the context scheme, aiming at pinpointing the exact source driving improvements in certain capabilities of our model. As shown in Table 7 the superiority of MMICL is driven by the collective impact of our design sernents-remooing any component cannot guarantee the superior performance of our model. Each component of our design significantly contributes different aspects of model. Details and further analysis are available Appendix
4 CONCLUSION
In this paper, we highlight the limitations VLMs handling the complex multi-modal prompts with multiple images, which makes VLMs less effective in downstream vision-language tasks. We introduce MMICL to address the aforementioned limitations and take our model a step further by extending it to multi-modal in-context learning. This breakthrough enables VLMs to better understand complex multi-m modal prompts. Furthermore, MMICL sets a new -fate-rf-th performance on the general VLM benchmarks and complex multi-modal reasoning benchmarks.

---

Published as a conference paper at ICLR 2024
5 ACKNOWLEDGEMENT
We extend our heartfelt gratitude to the anonymous reviewers whose dedication and insightful feedback have significantly enhanced the caliber of this paper. Their constructive critiques and valuable suggestions were instrumental in refining our work. Additionally, we are deeply appreciative of the Program Chairs and Area Chairs for their meticulous handling of our submission and for their comprehensive and invaluable feedback. Their guidance has been pivotal in elevating the quality of our research.
This work is supported by the National Science Foundation of China under Grant No.6193601 and 61876004.
REFERENCES
Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruy Batra, and Devi Parikh. Vqa: Visual question answering, 2016.
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In ICCV, pp. 9948-8957, 2019.
Cean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikot aj Binkkwski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 23716-23736. Curran Associates, Inc., 2022. URL htt.:httpss/prrcceedigee neurips wwtpeeei....p.pppeeeeeeeeeeeeeee http:////.....eeee.0600/67771071
pdf
Kaikai An, Ce Zheng, Bofei Gao, Haozhe Zhao, and Baobao Chang. Coarse-to-fine dual encoders are better frame identification learners. In Houda Bouamor, Juan Pino. and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 13455-13466, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653//1/2023 madings-emnlp.899. URL https:://t//stttttttt orw.oooooooooogg2/9000 otg/ http:///.orrrr//2...iniinn---eee findings-emnlp. 897
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Yitzhak Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training large autoregressive vision-language models. ArXiv, abs/2308.01390 2023.
URL https://app. htw.:app...smmaattnsssolaaaal.am wtt.aaaaiaaccco.ooooooooooroaaaa
Jeffrey P Bigham, Chandrika Jayant, Hanjie Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. pp. 333-342, 2010.
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawa- har, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision. pp. 4291-4301. 2019.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter. Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Yucheng Cai, Wentao Ma, Yuchuan Wu, Shuzheng Si, Yuan Shao, Zhijian Ou, and Yongbin Li. Unipem: Universal pre-trained conversation model with task-aware automatic prompt, 2023.
10

---

Published as a conference paper at ICLR 2024
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2021.
David Chen and William B Dolan, Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for compulational linguisties: human language technologies, pp. 190-200, 2011.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's 's referential dialogue magic, 2023a.
Liang Chen, Shuming Ma. Dongdong Zhang, Furu Wei, and Baobao Chang. On the pareto front of multilingual neural machine translation. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine ), Advances in Neural Information Processing Systems, volume 36, pp. 33188- 33201. Curran Associates, Inc., 2023b. URL httsspppper/22///lle/////// ott.r///..0.011eeo0o.oooooo000e0
cc/paper .pff
Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang. Tianyu Liu, and Baobao Chang. Towards end-to-end embodied decision making via multi-modal large language model:
Explorations with gpt4-vision and beyond, 2023c.
Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, and Baobao Chang. Pca-bench: Evaluating multimodal large language models in perceetion-cooniiionn-ccm
chain, 2024a
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai. Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: soug-and-pay inference accelcration for large vision- language models, 2024b.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.0677 2022.
Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. WebSRC: dataset for web-based structural reading comprehension. In Proceedings of the 202 I Conference on Empirical Methods in Natural Language Processing, pp. 4173-4185, Online and Punta Cana, Dominican Republic, November 2021a. Association for Computational Linguistics. doi: 10.18653/112021..oooo 343
URL httttttss///claatttt
/rgg/222.1 emnlp-main
Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, and Kai Yu. Websre: dataset for web-based structural reading comprehension. arXiv preprint arXiv:2101.09465 2021b.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. UtL|https
//mssys.org//lgg// 2023-03-2220000 -3- 00023-00330--iiio 30-vicuna/
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai. Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instraction-finetuee language models, 2022.
Maria Cipollone, Catherine C Schifter, and Rick A Moffat. Minecraft creative tool: A case study. International Journal of Game-Based Learning (IJBL), 4(2): 2014.
Wenliang Dai, Junnan Li. Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purper purpose langual models with instruction tuning. arXiv preprint 2rXiiv2305.06500 2023.
11

---

Published as a conference paper at ICLR 2024
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 326 2366335,211
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey on in-context learning, 2023.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint arXiv:2103.10360, 2021.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19358-19369 June 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin. Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: comprehensive evaluation benchmark for multimodal large language models. arXiv preprint 2rX1v:2306.13394, 2023.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng. Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Alama-adapted v2: Parameter-efficcint visual instruction model. arXiv preprint arXiv:2304. / 15100, 2023.
Tianyu Gao, Adam Fisch, and Dangi Chen. Making pre-trained language models better few-shot learners. arXiv preprint 2rXiv:2012.15723, 2020.
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Mullmodal-gpt:: A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.04790 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role image understanding in visual question answering. In CVPR, July 2017.
Helan Hu, Shuzheng Si, Haozhe Zhao, Shuang Zeng, Kaikai An, Zefan Cai, and Baobao Chang. Distantly-supervised named entity recognition with uncertainty-aware teacher learning and student- student collaborative learning, 2023a.
Wenbo Hu, Yifan Xu, Li, W Li, Z Chen, and Z Tu. Bliva: A simple multimodal Imm for better handling of text-rich visual questions. arXiv preprint arXiv:2308. 09936, 2023b.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302. 14045, 2023a.
Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302. 14045, 2023b.
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Huaizu Jiang, Xiaojian Ma, Weili Nie, Zhiding Yu, Yuke Zhu, and Anima Anandkumar. Bongard- hoi: Benchmarking shw-shot visual reasoning for human--bbjed interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19056-19065, 2022.
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in Neural Information Processing Systems, 33:2611-2624, 2020.
12

---

Published as a conference paper at ICLR 2024
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
Po-Nien Kung and Nanyun Peng. Do models really learn to follow instructions? an empirical study of instruction tuning. arXiv preprini arXiv:2305.11388 2023.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning, 2023a.
Chunyuan Li. Cliff Wong, Sheng Zhang, Naoto Usuyama. Haotian Liu, Jianwei Yang, Tristan
Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and-ving assistant for biomedicine in one day. arXiv preprint arXiv:2306.00033 2023b.
Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Hanwang Zhang, Wei Ji, Wenqiao Zhang, Tat- Seng Chua, Siliang Tang, and Yueting Zhuang. Empowering vision-language models to follow interleaved vision-language instructions. arXiv preprint arXiv:2308.04152 2023c.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp.
PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: 2: Bootstrapping togguae-imapg pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023d.
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, and Qi Liu, M A large-scale dataset towards multi-modal multilingual instruction tuning, 2023e.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models, 2023f.
Yunshui Li, Binyuan Hui, ZhiChao Yin, Min Yang, Fei Huang, and Yongbin Li. PaCE: Unified multi- modal dialogue pre-training with progressive and compositional experts. In Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13402-13416, Toronto, Canada, July 2023g, Association for Computational Linguistics. doi: http://///2....ooooooo....
10.18653/v 10t.6////1//22.... lol-lolg..745555 7749. URL httpthttp:///cllt /tpp/tttp///////nnooloogg.. .og/ 2023
749
Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang. Min Yang, Lei Zhang, Shuzheng Si, Junhao Liu, Tongliang Liu, Fei Huang, and Yongbin Li. One shot learning as instruction data prospector for large language models, 2024.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Fangyu Liu, Guy Emerson, and Nigel Collier. arXiv:2205.00330 00363, 2022.
Visual spatial reasoning. arXiv preprint
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. /4565, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li. Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2023c.
Yuliang Liu, Xiangru Tang, Zefan Cai, Junjie Lu, Yichi Zhang, Yanjun Shao, Zexuan Deng, Helan Hu, Zengxian Yang, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Zhengliang Li, Liang Chen, Yiming Zong, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark Gerstein. MI-bench: Large language models leverage open-source libraries for machine learning tasks, 2023d.
13

---

Published as a conference paper at ICLR 2024
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang. Zhou Yu, Xiaodan Liang, and Song-Chin Chuu Zhu. Iconqu: new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.1322 2021.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient iision-language instruction tuning for large language models. arXiv preprint 1.Xi::3305.15023, 2023.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. arXiv preprint wwrXiv:2110.55444 2021.
Ivona Najdenkoska, Xiantong Zhen, and Marcel Worring. Meta learning to bridge vision and language models for multimodal few-shot learning. arXiv preprint 13444423002.4794, 2023.
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using mil- lion captioned photographs. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinherger (eds.), Advances Neural Information Processing Systems, volume 24. Curran Asso- ciates, Inc., 2011. URL /ttpo///...tt/////////eoeeeeeenn 5c0gobsee33666f1b888877788883333
Junting Pan, Ziyi Lin, Yuying Ge, Xiatian Zhu, Renrui Zhang, Yi Wang, Yu Qiao, and Hongsheng Li. Retrieving-to-ansee Zero-shot video question answering with frozen large language models, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 9748-8763, PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking. Storage and Analysis, pp. 1-16. IEEF, 2020.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong Hc. Deepspeed: System op- timizations enable training deep learning models with over 100 billion parameters. In Pro- ceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, pp. 3505-3506. New York, NY, USA, 2020. Association for Com- puting Machinery. ISBN 9781 9781450379984.
doi: 10. 10.1145/3394486644007700 URL https
//doi.org/10. 0ttpr///o.11453334466533 3406703
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-2551 2015.
Babak Salch and Ahmed Elgammal. Large-scale classification of fine-art paintings: Learning the right metric on the right feature. arXiv preprint 1505 2015.
14

---

Published as a conference paper at ICLR 2024
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2556-2565, Mclbourne, Australia, July 2018, Association for Computational Linguistics. doi: 10.18653/v1//133 1238. URL httptss///cclaathoo... .
118-1238
Shuzheng Si, Shuang Zeng, and Baobao Chang. Mining clues from incomplete utterance: A query-enhanced network for incomplete utterance rewriting. In Marine Carpuat, Cathe-Ce de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the
North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4839-4847, Seattle, United States, July 2022a. Association for Computational Linguistics. doi: 10.11653/v1100000 naacl-main.3566 URL https::/aclaanthloog/ org.orgg//02 2022 naacl main. 356
Shuzheng Si, Shuang Zeng, Jiaxing Lin, and Baobao Chang. SCL-RAI: Span Span-based contrastive learning with retrieval augmented inference for unlabeled entity problem in NER. In Nicoletta Calzolari, Chu-Ren Huang. Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum- Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Scokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (eds.), Proceedings of the 29th International Conference on Computational Linguistics, pp. 2313-2318 Gyeongju, Republic of Korea, October 2022b. International Committee /rg/ 2022. www.ng-l..nncc 1.20
on Computational Linguisties. URL pattps://aclanthol//
Shuzheng Si, Zefan Cai, Shuang Zeng, Guoqiang Feng. Jiaxing Lin, and Baobao Chang. SANTA: Separate strategies for inaccurate and incomplete annotation noise in distantly -sspeii-supee named entity recognition. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 3883-3896, Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10. 18653/81/20233
166.:////22.....nnn---a....222. findings-acl.239. URL
org/ 2023. findings-ache 239
Shuzheng Si, Wentao Ma, Haoyu Gao, Yuchuan Wu, Ting-En Lin, Yinpei Dai, Hangyu Li, Rui Yan, Fei Huang. and Yongbin Li. Spokenwoz: A large-scale speech-text benchmark for spoken task- oriented dialogue agents. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36. pp. 39088-391 18. Curran Associates, Inc., 2023b. URL htlphtttss//pprceednng..ee
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read, In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 8317-8326. 2019.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv: 1.00491, 2018
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Can- dace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
9238-5248, 2022a.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Can- dace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2022b.
15

---

Published as a conference paper at ICLR 2024
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200- 212.0-21.2200 2021.
Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint 2tt1:22005.11000 2022a.
Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. DiffusionDB: A large-scale prompt gallery dataset for text-to-image generative models. arXiv:2210. 14896 [cs], 2022b. URL https://arxii.. wtgg/bs////..ooooo 2210. 14896
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew Dai, and Quoc V. Le. Finetuned language models are zero-shot learners, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick VON Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin I. Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 38-45, Online, October 2020, Association for Computational Linguistics. URL http:s//wwww.c.w.....///tttttooo
Yifan Wu, Pengchuan Zhang, Wenhan Xiong, Barlas Oguz, James C. Gee, and Yixin Nie. The role of chain-of- th-uugh in complex vision-language reasoning task, 2023.
Junbin Xiao, Xindi Shang. Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9777-9788 2021.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2016. Zhiyang Xu, Ying Shen, and Lifu Huang. Multilnstruct: Improving multi-modal zero-shot Iearn- ing via instruction tuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers). pp. 11445-11465, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653//1/2223.add. hwwp://22...alloo.66444.444 641. URL
acl- 641
Antoine Yang, Antoine Miech, Josef Sivic. Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF International
Conference on Computer Vision, pp. 1686-1697, 2021.
Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roce Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation. arXiv preprint vrXiv:2305.10400 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14111 14177844..44...... 2023.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2, 2014.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara I. Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 69-85. Springer, 2016.
Weihao Yu, Zhengyuan Yang, Linjie Li. Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint 2rXiv:2308.0244 2023.
16

---

Published as a conference paper at ICLR 2024
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and patter recognition, pp. 6720-6731 2019.
Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, and Tao Kong. What matters in training gpt4-style language model with multimodal inputs? arXiv preprint arXiv:2307.0244:
2023.
Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, and Tat-Seng Chua. Transfer visual prompt generator across 1mms. CoRR, abs/23045.01278, 2023a. URL http::://do..rg///
48550/arXiv. 2305.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5317-5327, 2019.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal Mfain-of-thougee reasoning in language models, 2023b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
17

---

Published as a conference paper at ICLR 2024
A RELATED WORK
A.1
VISION-LANGUAGE PRETRAINING
Model
Iulti-Image Inputs Multi-modal Instruction Tuning Text-to-Image Reference
Flamingo
X
Meta learner
X
BLIP-2
X
X
X
LLAVA
X
X
MiniGPT-4
X
A
InstrucIBLIP
X
Shikra
X
Kosmos-1
Otter
MMICL
Table 8: Summary of Vision-Language Pre-Trained Models.
Our work is inspired by recent vision-language pre-training works (Zhu et al. 20033] Liu et al. 2023b Li 2022 2023d), which have been proven effective for aligning visual inputs and frozen LLMs to obtain cross-modal generalization ability.
BLIP-2 BLIP-2 [Lietal, 2023d) bridges the modality gap with lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from frozen image encoder. The second stage bootstraps vision-to- to-langugg generative learning from frozen language model.
InstructBLIP InstructBLIP (Dai et al. 2023) performs vision-languse instruction tuning based on the pre-trained BLIP-2 models with converted multi molti-modal datasets and the LLaVA (Liu et al. 2023b) dataset generated by GPT-4.
MiniGPT-4 MiniGPT-4 (huu et al. 2023 alignn CLIP visual encoder with frozen Vincuna Chi ang et al. 2023 with an artificially collected dialog dataset
Shikra Shikra (Chen et al. 2023a), VLM which can handle spatial coordinate inputs and outputs natural language. It makes Shikra excel referential dialogue and general vision- lsanuag tasks, resulting in outstanding performance.
However, there is still less work focusing on VLMs with multi-image inputs.
Flamingo Flamingo (Tsimpoukelli et al. 2021) achieves multi-visual inputs based self-attention for images but performs poorly in downstream tasks. Flamingo supports Few-Shot Learning (FSL) in VLM via ICL by leveraging its robust capability to handle multi-visual inputs and uses cross-attention instead of self-attention to get better performance. However, still suffers from the unableness to explicitly point images, so they introduce hacky cross-attention mask.
Kosmos-1 Kosmos-1 (Huang et al., 2023a), is trained from scratch on billion-scale multi-moda corpora, including interleaved text-image web page data, image-text caption, and language-only instruction tuning data. can multi-modal Few-Shot Learning and Chain-of-Thought processes, thereby achieving formidable performance.
Otter Otter (Li et al. 2023a). an open-source implementation of flamingo and trained with multi- modal instruction in-context tuning data.
(2023) uses meta-learning objective to train an adapter that aggregates multiple image features the original VLM and adapter become better few-shot learner.
Meta learner Najdenkoska et al.
Recent VLMs (Zhu et al.
2023 Liu et al. 2023b ei al. 2022) Alayrac et al. 2022 Dai et al, 2023
Chen et al. 2024b) have been proven effective for aligning visual inputs and frozen L.Mss obtain
18

---

Published as a conference paper at ICLR 2024
cross-modal generalization ability. However, previous works overlooked multi-image VLMs, mainly focusing on handling single-image prompts. [simpoukelli et al. (202I) supports multi-image inputs using self--ttention for images but performs poorly in downstream tasks. Although Flamingo (Alayrac et al. 2022) supports Few-Shot Learning VLMS and uses cross-attention to capture text-image re relationships, still suffers from exact reference to specific images.
A.2 MULTI-MODAL INSTRUCTION TUNING
Pre-trained language models have achieved great success in both natural language understanding
et al. 2022b; 2023a An al. 2023
Chen al. 2023b) and natural language generation (Si et al. 2022a; 2023b: Cai et al. 2023 Liu et al. 2023d). Recently, instruction
Hu al.
2023a
tuning (Kung & Peng 2023
al., 2024 Wei et al. 2022) has attracted more and more attention
in order to enable LLMs to follow natural language instructions and complete real-world tasks. However, multi-modal instruction tuning still requires further exploration. Multinntruct (Xu et al.) 2023) introduces instruction tuning to enhance the performance of VLMs in cnstruction-ollovviin ability. Due to the architectural design, Multinstruct still struggles with complex contexts containing multiple images. Otter (Li et al. 2023a) fine-tunes Openflamingo (Awadalla et al.) (023) to augment its instruction comprehension capabilities. However, Otter's ' dataset lacks text-to-image references and interconnected image-to saage-to-image image data. This limitation hinders its capability to handle complex contexts that involve visual-textual relationships.
A.3 IN-CONTEXT LEARNING
It has been well-explored enable ICL in pre-trained language models (PLM). MetalCL (Min et al., 202T) proposes meta- meta--raining framework for few-shot learning to tune a PLM to do in-context learning large set of training tasks. LM-BFF (Gao et al. 2020) studies few-shot fine-tuning of PL.Ms. However, ICI. in VL. is still less explored. Recent works VLM mainly focus on zero-shot evaluation with single image input.
B MULTI-MODAL ICL DATA
We construct two training datasets, text-image interleaved data and in-context learning data, for the text-image relationship challenge and image-image relationship challenge, respectively. In this section, we will cover the data resources.
C DATA RESOURCE
The data resource used in constructing the MIC dataset is displayed in Fig. Our training dataset comes from 8 task categories and 16 datasets.
Image Captioning aims to produce descriptions the given images according to different needs. Our training dataset includes MS COCO (Lin et al. 2014). DiffusionDB (Wang al. (022b). and Flickr 30K (Young et al. 2014).
Knowledgeable Visual Question Answering (KVQA) requires the model to make use of commonsense knowledge outside the input image to answer questions. Our training dataset includes OK-VQA
(Marino et al. 2019).
Image Question Answering (IQA) requires the model to answer the questions based on image correctly. Our training dataset includes VQAv2 (Goyal et al.] (017). ST-VQA (Biten et al. (201)). Text-VQA (Singh et al. 2019), WikiART (Salch Elgammal 2015) and RefCOCO (Yu et al.
2016).
Video Question Answering (VideoQA) requires the model to answer questions based on the video correctly. We extract eight frames per video as visual inputs for Video QA tasks. Our training dataset includes MSRVTTQA (Xu et al. 2016).
Video Captioning requires the model to give the caption based on the video. We extract eight frames per video as visual inputs for Video Captioning tasks. Our training dataset includes MSRVTT (Xu
al. 2016).
19

---

Published as a conference paper at ICLR 2024
#samples Val
Task
Dataset
Used
License
Train
Test
MS COCO Lin et al. 2014
Yes
413,952
202,496
Custom MIT
DiffusionDB Wang et al. 2022b Flickr
Yes
19,963
0
Captioning
Young et al. 2014
Yes
144,896
4,864
4,864
Custom CC BY 2.0
NoCaps 7
Agrawal et al 2019
Yes
0
45,000
Classification
Minilmage
Russakovsky al.
20155
Yes
38,400
9,600
12,000
CCO 1.0
VQA v2
Goyal et al. 2017
Yes
592,998
26, 16,
25,747
CC BY 4.0
ST-VQA (Biten et al. 2019
Yes
78,222
0
4,070
CC BY 4.0
VQA
TQA [Sngh et
20T9
Yes
34,602
0
CC BY 4,0
NLVR2 Suhr et al. 2018
Yes
86,373
6,982
6,967
CC BY 4.0
RefCOCO Yu et al.
2016
Yes
141,968
0
Custom
KVQA
OK-VQA (Marino et al.
2019
Yes 9,009
5,046
CC BY 4.0
GQA (Hudson Manning
2019
Yes
943,000
132,062
12,578
CC BY 4.0
Reasoning
VCK (Zellers et |0,
2019))
Yes 118,156
14,472
5,000
Custom
Winoground (Thrish et al
2022a
No
0
400
Custom
MSRVTT-QA (Xu et al.
2016
Yes 158.581
12,278
72.821
MIT
video
MSRVTT Xu et 2016
Yes 130,260
9,940
59,800
MIT
WikiART (Salch & Elgammal 2015
Yes
13,000
5,500
Custom CC BY 4.0
Others
IL.AVA-Insttuc--Too (I.iu et al. 2023b
Yes
236,564
0
Table 9: Detailed task descriptions and statistics of our instruction tuning tasks, including all datasets in all types of tasks. The column "Used" indicates whether we use this dataset in the multi-modal in-context tuning stage.
Image Captioning
Video Question Captioning
Visual Reasoning
Fen-Shot Image
Classification
Flickr30k
MERVIT
GQA
Minitmagenet
COCD Capsion
Visual Commonserse Reasoning
Video Question Answering
HatefulMemms
Diffusionds
MSRVTT QA
Language
Bongard-HOP HOI
Nocaps
Reasoning V2
IVQA
Knowledge Question Amswering
Visual Vinuvl Ressoning
Nonverbal Reasnning
IQ Test
OKVQA
NexiQA- MuhtieeeChoice
Wineground
A-OKVQA
ScienosQA
NoxrQA-
Domain
Multi Mutt--mmage- choice
lusqAA-
Visual Dialog
Inscoruct
Image Question Answering
Multi-toxt-chan
lcon@A-
Visual Dialog
VQAu2
STVQA
TextVQ4
Web Page Question Amsering
OOD Generalization
Wikiart
RefcOCO
ViaWiz
Minecraft
Figure 7: Illustration of the data resource used to construct MIC dataset. It consists of 11 tasks and 33 different datasets. The held-in datasets are indicated by white and the held-out datasets are indicated by yellow.
Visual Reasoning requires the model to correctly perform image reasoning and answer questions. Our training dataset includes GQA (Hudson & Manning 2019). VCR (Zellers al. 2019) and NLVR2 (Suhr et al. 2018).
Image Classification involves classifying an image based given set of candidate labels. Our training dataset includes Minilmage (Russakovsky et al. 2015).
Visual Dialog requires the model hold meaningful dialog about visual content with humans in natural, conversational language. Our training dataset includes InnAVA-Instruut-115555 (Liu et al. 2023b).
Our testing dataset comes from 10 task categories and 18 datasets.
Image Captioning includes the Nocaps (Agrawal al. 2019) dataset.
20

---

Published as a conference paper at ICLR 2024
Knowledgeable Visual Question Answering (KVQA) includes the ScienceQA (u et al. 2022 and A-OKVQA (Schwenk et al. 2022) datasets.
Image Question Answering (IQA) includes the VizWiz (Bigham al.
2010 dataset.
Visual Reasoning includes the Winoground (Thrush et al. 2022b), VSR (Liu et al. 2022) and
IconQA (Lu et al., (021)) dataset. Winoground proposes task of matching two given images and two captions correctly. The challenge of this task is that both captions contain a completely identical set of words, only in a different order. VSR deseribes the spatial relation of two individual objects in the image, and a VLM needs to judge whether the caption correctly describes the image (True) or not (False). The IconQA dataset has two sub-datasets: image question answering with multiple text choice and image question answering with multiple image choice.
Web Page Question Answering (Web QA) includes the Websre (Chen al. 2021a Huang al. 2023a) datasets. The model must answer questions based on the web image and the optional extracted texts. We sampled 2000 instances from Websre for the evaluation. To align with KOSMOS-1 (Huang
et al. 2023a), we only use the web image as input.
Video Question Answering (VideoQA) includes the iVQA Yang et al. 2021). MVSD (Chen & Dolan
2011), and NexIQA (Xiao et al., 2021) dateset. The NexIQA dataset has two sub-datasets: video
question answering with multiple choice and open-domain video question answering.
Few-shot Image Classification includes the HatefulMemes (Kiela et al., (020) and Bonard- (Tingg jet al. 2022) dataset. HatefulMemes requires the model to determine if a meme is hateful based on the image and explanation provided. Bonard-HOI is the benchmark for evaluating the model's ability Few-Shot Visual Reasoning for Human -bject Interactions. It provides few-shot examples with challenging negatives, where positive and negative images only differ in action labels. The model is then asked whether the final image is positive or negative. We sampled 2000 instances from Bonard-HOI for the evaluation.
Nonverbal Reasoning includes the Raven IQ test (Huang et al. 2023a). Each instance in the Raven IQ test has 3 or 8 images as inputs and six candidate images with unique correct completion, and the goal is to predict the next image from the candidates.
Visual Dialog includes the visual dialog dataset (Das et al. 2017). We the question of the final dialogue as the question for instance and take all preceding dialogues as the context to perform open-domain image question answering.
OOD Generalization includes the Minecraft dataset that we construct using Minecraft (Cipollone al. 2014) game which requires the VLM identify whether an animal (i.e., cow, llama, chicken, donkey, and so on) is present in picture.
More detailed task descriptions and statistics about the datasets are shown in Table
D DATA CONSTRUCTION
Firstly, create an image declaration per instance in all datasets to generate datasets with explicit text-to-image reference. We then have annotators scrutinize every dataset's 's samples and provide task instructions. This practice aids in gaining a comprehensive understanding of the task and helps craft high-quality templates.
Next, we employ ChatGP1 to rewrite the instructions to describe the key characteristics of each task accurately. After ChatGPT generates the instructions, we undergo a manual review to guarantee the high quality of the instructions.
We select ten suitable templates matching as candidates, then merge the original dataset's input into a randomly chosen template. We assemble demonstrations for each instance from the dataset by selecting a small amount of data and arranging them sequentially. These demonstrations are integrated with the input instance to generate multi-modal contextual data")
'We use the g-t-3.5-tturbo version of ChatGPT. "Except for the video datasets, dataset, and LLLVVa dataset.
21

---

Published as a conference paper at ICLR 2024
We construct multi-in tintti--mmeeeee data by extracting eight frames per video from MSRVTT Xu al. 2016
and MSRVTTQA (Xu et al. 2016) datasets. We also crop images from the VCR (Zellers et al. 2019) dataset using object bounding boxes to produce intertwined multi-modal data with closely re related images. We convert all data into a vision-language Q&A format to create high-quality multi-modal training data and accumulate 5.8M samples in MIC dataset. Due to resource constraints, approximately 10% of the data with the sampling strategy described in Appendix F to finetune MMICL. It anticipated that larger model trained on all of our data would yield more promising result.
Algorithm Image Declaration
Require: Interleaved multi-modal input: X, containing visual embedding: V (Ui, ......} and text embed- ding H = (/1, h.,...), ... where 2, represents the image embedding and h, represents the span between the image embeddings.
Ensure:
Interleaved multi-modal input with image declaration: X I: for each interleaved multi-modal input X do
n 4- number images X
3: Initialize image proxy tokens
4: for each image in X do
5: Ref. "image i is [IMG.]ee "image [LMG.] vz"
end
7: R - {Refi, Ref2....)
Replace v; in X with Refa & [Refi, hn ft,
9: end for
D.1 MULTI-MODAL DATA WITH INTERCONNECTED IMAGES
The data construction framework for Multi-modal Data with Interconnected Images as Sec. 2.2.2 developed through an automated process that leverages existing dataset annotations (i.e. Visual Commonsense Reasoning dataset (VCR) (Zellers al. 2019)). This approach obviates the necessity for supplementary manual annotation efforts.
The original annotations of VCR dataset comprise images, bounding boxes, and set of questions
and answers that specifically reference these bounding boxes as the left part of Fig. g This structured annotation approach facilitates comprehensive understanding of the image context. Leveraging the existing annotations, we develop an automated workflow, including extracting sub-images from raw images based on bounding boxes and employing ChatGPT to formulate a variety of instructions. Initially, we manually craft a set of instruction templates, which, alongside the dataset description, are inputted into CharGPT. Subsequently, ChatGPT asked to expand the templates, thereby yielding a diverse set of instruction templates.
D.2 UNIFIED MULTI-MODAL IN-CONTEXT FORMAT FOR DIFFERENT TASKS
The data construction framework for the multi-modal in-context format data Sec. 2.2.3 is developed through an automated process that leverages existing dataset annotations. The used dataset presented Appendix F
Through selective sampling within the dataset, we have formulated in-context examples for each respective task. These examples encompass range of elements, including images, questions, and answers. ChatGPT is used to formulate a variety instructions. The formulated instruction templates presented in Appendix G
D.3 MULTI-MODAL DATA WITH VIDEO FRAMES
The data construction framework for Video Data is developed through automated process that leverages existing dataset annotations. The used dataset are MSRVTT (Xu et al.) (016) and MSRVT- TQA (Xu et al. 2016) datasets as Appendix
22

---

Published as a conference paper at ICLR 2024
Annotation from Existing Datasets
Step3: Image Declaration
Raw Image:
Image4 Imagee
Task Description:
Image5 Image3
Infoc
Carefally analyze images to answer the question.
Image Descriptions tion:
Image
Emogee
Image 0: [IMG0)
Image I:[MGGI]
Image [IMG2]
Bounding Box Description:
Bounding Box for Image 1: 1869, 384, 1261, 794
Image 3: [IMG3]
Image 4: [JMG4]
Bounding Box for Image 7: [163, 481, 338, 696
Question:
Why image inside of enge?
Image 5: [IMGS]
[JMG6]
Options:
Choice 0: Image 7 is about to atlack image and image 2.
swarm of butterflies there, Choice Image 3 bas pet bird the cage.
Choice Because there
Image 7: [IMG7]
Choice Image 7 is the cage
can't away.
Step4: Data Construction
Answer:
Choice 3
Input Data Format:
Processed Instuction:
Stepı: Raw Information Preparation
Conduct meticulous examination of the visual details within the images, employing critical analysis and attention to...
Raw Instruct Nion:Now you need answer the question based on previous imuges. Dataset Description: Image comprehension task requiring
Ques tion:
Why image"
inside of cage?
Options:
understanding of objects and relations.
Choice Image about attuck image and image
Stepe: Chatgpt Refinement
Choice I:
Because there is a swarm of butterflies there.
Choice 2: Image has pet bird the cage. cage so that it can't fly away.
Comprehensive Instruction:
Choice Image
Conduct a meticulous examination of the visual details within the images, employing critical analysis and attention to nuanced elements, order to accurately and comprehensively address the posed question.
Output of Data Format: The answer is Choice 3.
Figure 8: Illustration of automatic data construction pipeline for multi model data with interconnected images. The automatic construction is based on existing annotation of VCR dataset (Zellers et al.. 2019) without human involvement. ChatGPT is used for instruction refinement.
E MODEL STRUCTURE
As shown in Fig. I1 MMICL treats the image and language representations equally and combines them into interleaved image-text representations, similar to the original input. Each given image is encoded by a vision encoder (c.g., ViT (Radford et al., 2021 Fang et al. (023)) to get the vision representation of the image. Then, use the Q-former as the VPG to extract the visual embedding. We utilize a fully connected layer as the projection layer to convert each visual embedding to the same dimension the text embedding of the LLMs. This alignment helps the LLM to understand the images. Our approach treats the visual and text embedding equally, enabling flexible combination of visual and textual content. Finally, we combine the visual embeddings of multiple images with text embeddings in an interleaved style and then feed them into the LLM. We set the weights for mapping query and value vectors in the attention layer of LLM as learnable better adapt to the modal context with multiple images. During the pre-training, we freeze the image encoder, Q-former, and the backbone L.M while jointly training the language projection and the query and value vectors of the LLM.
F DATA BALANCE
Previous studies have shown that the data balance of training data could significantly influence the model performance (Dai et al. 2023) Mixing the training data of each dataset uniformly could
23

---

Published as a conference paper at ICLR 2024
Annotation from Existing Datasets
Data Construction
Input of Data Format:
Image:
Image:
In-Context Example:
Example Image 0: [IMG0]
Carefully analyze the
image 0 to generate conciso and accurate answer.
Q: What this bird called?
A: The answer parrot.
Carefully analyze the answer. Q: What color is the helmet in the middle imuge?
Example Image [IMGI]
Ques diuestion:
image generate concise
Ques tion:
What color the helmet
What this bird callod?
the middle of the imuge?
A: The answer light blue.
Answer:
Answer:
Example Image [JMG2] image 2 to generale concise and accurate answer.
Carefully analyze the
parrot.
light blue.
Q: Is it an indoors or outdoors scene?
Image:
Image:
A; The answer indoors.
Comprehensive Instruction: Respond the inquiry by drawing upon illustrative examples for clarification and support.
Image 3: [TMG3]
Carefully analyze the image to
Ques tion:
Ques there napkins under
generate concise and accurate answer.
Is an indoors outdoors scene?
Q: Are there napkins under the utensil?
the ulensil?
A:
Answer: indoors.
Answer:
Output of Data Format:
yes.
The answer is yes.
Figure 9: Illustration of automatic data construction pipeline for multi-model data with in-context examples. The automatic construction is based existing annotation without human involvement. ChatGPT is used for instruction refinement. Data source can found at Appendix instruction templates are presented in Appendix G
The formulated
Annotation from Existing Datasets
Data Construction
Image0
Imagel
Input of Data Format:
Task Description:
Carefully analyze series images and give caption
Image Description:
Image2
Image3
Image 0: [IMGO]
Image [IMGI]
Image 2: [IMG2]
Image 3: [IMG3]
Image4
Image5
Image 4: [IMG4]
Image 5: [IMG5]
Image 6: [IMG6]
Image 7: [IMG7]
Image6
Image7
Comprehensive Instruction:
Create descriptive captions that accuratcly reflect the content and context provided images.
Sote mik and ato
Output of Data Format:
The summarization of images can be: in a kitchen woman adds different ingredients into the pot and stirs it.
Caption: in kitchen adds different
ingredients into pot
Figure 10: Illustration of automatic data construction pipeline for multi-model data with video. The automatic construction is based on existing annotation without human involvement. ChatGPT used for instruction refinement.
24

---

Published as a conference paper at ICLR 2024
Imeeteeved Pronopts with muztiple images
Insouctona
Image D [1SCC))
image [MG11
Questions
Projection layer
Imiage Eacoder
Tokenizer Embedding
hy
A2
Rg
Large Language Model Layer
Value 8
Key
Quety
Text Embodding
Output
Add &ormal& FFN
Large Language Model Layer
Visua Prompts
Add & Normal
Freere
Language Response
Figure 11: Illustration the MMICL structure.
cause the model to overfit smaller datasets and underfit larger datasets, causing poor performance. In order alleviate this problem, employ sampling strategy to sample datasets with probabilities proportional to the square root the number of training samples following Dai et al. (0023). Formally, given D datasets with N. training samples (N1, N2,..., ND), the probability Pa of data samples being selected from dataset during training is as follows.
VNa
Pd
E VN.
G INSTRUCTION TEMPLATE FOR DATA CONSTRUCTION
As Sec. 2.2.3 the constructions of MIC require carefully designed templates. The instruction templates for each task are presented in this section. The templates for tasks MSCOCO, Flick30k, Nocaps, and Diffusiondb are presented in Table TO The templates for tasks Minilmagenet are presented in Table [T] The templates for tasks VQAv2, S-VQA, WikiART, and RefCOCO are presented in Table 13
The templates for task OKVQA are presented Table 14 The templates
for task MSRVTT are presented in Table 15 The templates for tasks MSRVTTQA and MSVD are
presented in Table 16
H EXPERIMENT DETAILS
Following Chung et 20222. we use FLANT5-XL FLANT5-XXL (Chung al. 2022 as the backbone LLMs. In Stage I, we set the vision encoder and language model to be frozen and utilize the images from Coco et al. 2014), CC3rmaharma al.) I 2018). Visual Genome(Krishna et al.] (017). CC12M (cannCCiag et al.) 2021). SBU et al.] 2011) and LAION 400M data (ata(SShuhman et al. 2021), as well as the captions generate by ILIP large (Li et al. 2022 to perform feature alignment training the Q-former. We keep the other part of the VLM frozen and jointly train the Q-former and projection layer. The Q-former's 's visual embedding output occurs in 32-token size of the backbone language model. To benefit from BLIP-2's significant visual representation extraction ability, we integrate its powerful vision encoder to initialize the Q-former and projection layer. [ In Stage II, train the model for three epochs with lower learning of le The weights of mapping query and value vectors in the attention layer of LLMs are learnable in this stage better adapt the multi-modal prompts with multiple images. In this stage, we freeze the visual encoder, Q-former, and the backbone LLM and jointly train the projection layer, the query vectors, and the value vectors of the LLM.
IIn practice, we checkpoint of BIIP-2 and InstructBlip as the hackbone for MMICI, Stage I is skipped.
25

---

Published as a conference paper at ICLR 2024
Templates of Image Captioning (MSCOCO, Flick30k,, Nocaps, Diffusiondh)
(1) Carefully analyze image [[MGO] (image) generate concise and accurate description that accurately represents the objects, people, and scenery present
(2) Use clear and voncise language that accurately describes content imuge D: [IMGO] [imuge). J.
(3) Your caption should provide sufficient information ahout image O: [MMG0] [imuge] so that someone who has not seen the image can understand it.
(4) image is [IMGO] [image]. Be specific and detailed your description image 0. but also try capture the of image 0 in
succinet way.
image describe what contained this photo. Your caption should be than few sentences and should be grammaticcaly correct and free of spelling errors.
image [IMG]] (image). Based
(6) Include information in your caption that is specific image D: [IMG0] [image] and avoid using generic ambiguous descriptions.
(7) image 0 [TMGO] [image). J. Rased on the image Ol give caption about this image. Think about what message or story image O is conveying, and try lo capture that in your image caplion.
(8) Based the image give caption about this image. Your caption should provide enough detail about image 0: [TMGO] [image]
give the viewer a sonse of what is happoning in the image.
(9) Give caption about this image. Avoid using overly complex language or jargon in your caption of image [IMGO] (image) that might confuse the viewer.
(10) Be creative in your approach to captioning image [IMGO] [image] and try to convey unique perspective or story.
Table 10: Instruction templates used for transforming datasets into instruction tuning data. (I) (image) ) denotes image embedding encoded by image encoder, image embedding will be concatenated with language embedding as input. <imagej> denotes image token to exact reference the j-th image in an instance as described in Sec. 2.2.1
Templates of Image Classification (Minilmagenet,, etc)
(1) image is [MMG]]
[image]. Please ideatify the object concept depicted in image
(2) imuge is [MGG]]
(image). What is the main subject of image O??
(3) image is [MMG0]
[image]. Can you recognize and lahel the object shown in image
(4) image is [IMG0]
(image). Identify the category or class to which image belongs.
(5) image
[IMG]]
(image]. Based on the visual content, determine
image O represents.
(6) image 0 is [IMG0]
[image]. What is the name or label of the item captured in image 0?
(7) image is [IMG]]
[image). Please provide a description or ideatification of the subject in image
(8) image
[MG0]
[image). From the visual
determine the object entity depicted in image
(9) image is [IMGO]
[image]. Can you recognize and name the primary element shown in image O??
(10) image [TM0]] [image). Identify the object concept that best describes what is depicted in image
Table 11: Instruction templates used for transforming datasets into instruction tuning data. (I) (image) ) denotes image embedding encoded by image encoder, image embedding will be concatenated with language embedding as input. <imagej> denotes image token to exact reference the j-th image in an instance as described in Sec.
2.2.1 2.2.1
Templates of Knowledge Visual Question Answering (OK-VAQ)
(1) Look at image labeled [IMGO] [inuge) carefully and read question: question. Try understamd what is being asked before selecting answer.
(2) image is [IMGO] (image). Consider all of the information in image 0 labeled [IMG0] when answering question. Look objects, colors, shapes, and other details that may be relevant to question: question Answer:
(3) image is [IMGO] (image). Read each answer choice carefully and answers question : question based on the information provided image O.
(4) image [IMG]] (imuge). Given the picture [LMGO], pay atlention wording of question and answer the tollowing question:
question Answer:
(5) Read the question carefully and look image labeled [TMG0] [image). Use your intuition and common sense when answering the
question: question
(6) Consider all of the information in image 0 labeled [MMG]] (image) when answering the question: question
Take your time when answering each question. Don't rush through the questions, and make sure you have carefully considered all of the
information provided in image 0 labeled [IMGO] (image) and the question before making your selection. Question: question Answer:
based (9) Carefully examine image labeled [IMG0] (image) hefore answering the question. Questionzquesion Answer:
(8) Make sure your answers
information presented the imuge [IMGO] finage). J- Question:qustiin Answer:
(10) Please refer to image 0: [IMG0) [image) when answering the following questions: question Answer:
Table 12: Instruction templates for task OKVQA.
26

---

Published as a conference paper at ICLR 2024
Templates of Image Question Answering (VQAV2, ST.VQA. WIKIART, ReCOOC, etc)
VQAv2
(1) image is [[MGO] (image). For the question, carefully examine the image and use your knowledge determine the correct answer.
Question: question Answer:
(2) image is [TMG0] [image]. Given the picture [TMGO], pay attention the wording of question and answer the following question:
question Answer:
(3) Read the question carefully and look image labeled [IMGO] [image). J. Use your intuition and common sense when answering the
question: question
(4) Answer each question based on the information presented in image O: [IMG0] (image). Given the picture [IMG0), what the answer to
the question: question Answer:
(5) Please refer to image O: [IMGO] (image) when answering the following questions: question Answer:
(6) Questions is related image O: [TMG0] [image]. J. Please analyze the image and provide the correct answer for the question: question
(7) Read the question carcfully and look at image 0 labeled [IMGO] [image). Use your intuition and common sense when answering the
question: question
(8) Consider all of the information in image 0 labeled [IMG0] (image) when answering the question: question
(9) Take your time when answering each queation. Don't I rush through the questions, and make sure you have caretully considered all of the
information provided in image O labeled [IMGU] [inuge] and the question before making your selection. Questiou: question Answer:
(10) Use the image D: [IMGO] [image] as visual aid to help you answer the questions accurately. Question:uestion Answer:
ST-VQA
(1) Answer each question based on the information presented in image D: [IMG0] (image). Given the picture [IMG0], what the answer
the question; question Answer:
(2) Please refer image O: [MGG]] (image] when amswering the following questions: question Answer:
(3) Questions is related to image 0: [TMG0] [image] j. Please analyze the image and provide the correct answer for the question: question
(4) For cach question, use the image O: [IMGO] [image] as a reference to answer the question: question
(5) Make sure your answers based on the information presented the image U: [IMG0] (image), ). and any OCR text associated with it.
Quesiomuuessionn Answer:
(6) Answer the question accurately as possible using the information provided in the image O: [TMGO] (image), and any OCR text
associated with it. Question Answer:
(7) Please ensure that you are answering the question hased information presented in the image [IMGO] [image) S Question:question Answer:
(8) The image 0: [IMG)) (image) is the primary source of information for answering the questions. Please refer to carefully when answering question: question Answer:
(9) Pay close atlention to the details in image O: [IMG0] (image], as they may provide important information for answering the questions. Question=suussion Answer:
(10) Use the image 0: [IMGO] [image] as a visual aid to help you understand the context and answer the questions accurately. Ques- tionqqestton Answer:
WikiART
(1) image O is [JMGO] [image]. Please provide information about the artss, genre, and style of this artwork.
(2) image is [IMG0]
[image]. I would like to know artist's 's name, the genre, and the specific style depicted in this painting.
(3) image is [MMG0] [image). Could you identify the artistic geure, the artist, and the style portrayed this artwork?
(4) imuge is [MMG]] [image). J. In this painting. which genre does belong to, who the artist, and what is the presoomimnt style?
(5) image is [IMGO] [image]. Tell ahout the artist, genre, and style associated with this particular artwork.
(6) image 0 [IMG0] [image]. This piece of art seems intriguing. Can you provide details about the genre, the artist, and the style it represents?
(7) image is [IMG0] [image]. Identify the genre, artist, and style of this captivating artwork, please.
(8) image is [IMGO] [image). I'm curions learn about the artist's name, the genre, and the distinetive style showcased in this artwork.
(9) image is [LMG]] [image]. Could you ealighten about the genre, artist, and the artistic style that characterizes this beautiful piece??
(10) image is [IMG)]
(imuge). In terms of genre, artist, and style, what information can you provide regarding this fascinating artwork?
RefCOCO
(1) image 0 is [IMGO] [image]. Given image create descriptive caption that accuratlly represents the content of the image, including the item located in the [quadrant] of the image.
(2) Use your knowledge the imuge 0 and the (quudrant] location to generate detailed and accurate caption that captures the essence the scene. Keep in mind that image is [MMG0] (image).
(3) image is [IMGO] (image]. When writing your caption, sure to include specific details about the item located in the (quadrant) the image such its sizc, shape, color, and position.
(4) Think about the intended audience for your caption and use appropriate language and tone. Consider the context of the image: [IMGD]
[image] I and the [quidrant] location when creating your caption, and make sure that accurately reflects the content of the image.
(5) Your caption should be concise and to the point, while still capturing the essence of the image 0 and the item located in the (quadrant) of image. Avaid including irrelevant information in your caption that detracts from the main content of image. Remember that image 0 is [MMG0] (image].
(6) image 0 is [IMGO] [image]. Cheek your caption for accuracy and grammatical errors before submitting. Be creative in your approach to captioning the image and the item located the (quadrant).
(7) image O is [IMG]] (image). Given imagee 0, describe the item in the (sumdrant) of the image.
(8) image is [IMGO] [image]. Using image O. provide caption for the object located in the [quadrant] of the image.
(9) For image 0: [IMG0] [image], describe the object in the (quadrant) of the image.
(10) Given the image 0: [MGG]] (image). Generate description for item located in [quadrant] of the image.
(11) image is [MG0]] [image). Using the provided image 0, describe the object located the [qundrant] of the image.
Table 13: Instruction templates for tasks VQAv2, ST-VQA, WikiART, and RefCOCO.
27

---

Published as a conference paper at ICLR 2024
Templates of Video Question Captioning (MSRVTT)
(1) image is [IMGO] (image). image is [[MGI] [image], image 2 is [IMG2] [image].. image 3 is [MMG3] (image). image 4 is [IMG4]
(image). image 5 [IMG5] [image]. image [IMG6] (image). ). image 7 is [IMG77 [image]. Watch the images carefully and write detailed description of what sec.
(2) image 0 is ([MG0) (image). image is [IMG1] [image). image is [IMG2) [image). image is [MMG3] (image). image is [IMG4]
(image). ). image 5 is [IMG5] (image]. image is [IMG6] (image]. image [MMG7] [image]. After viewing the images, provide a summary of the main events or key points depicted.
(3) image 0 is [[MG0] [image]. image is [[MGI] [image]. image is [IMG2] [image]] image is [IMG3] (image). image is [TMG4]
(image). image 5 is [IMG5] [image]. image is [IMG6] [image]] image is [IMG7] {image). Pay close attention to the details in the
images and provide accurate description to the images hased what you see.
(4) image 0 is [IMG0] (image), image is [[MGI] [image], image is [IMG2] [image]. image 3 is [MGG]] [image]. image 4 is [TMG4]
(image). ). image 5 is [IMG5] [image). ). image is [[MG6] [image]. image [IMG7] (imago]. Utilize your comprehension skills to describe the cuntext and events depicted in the images.
(5) image 0 is [IMG0] (image). J. image is [IMGI] [image]. image 2 is [IMG2] [image]. ). image 3 is [IMG3] (image). image is [IMG4] (image). ). image is [IMG5] (image). image [[MG6] [image], image 7 [IMG7] [image]. Reflect on the images's narrative structure and identify any storytclling tochniques or narrative devioes used. Write detailed description of what you sec.
(6) image 0 is [IMG0] (image]. image is [TMGI] [image]. image is [IMG2] [image). image is [MMG]] [image). image is [TMG4] (image). image 5 is [IMG5] (image). image is [IMG6] [image)] image 7 is [JMG7] [image]. Consider both the explicit and implicit information conveyed in the images to provide comprehensive description of the images.
Table 14: Instruction templates for task MSRVTT.
Templates of Video Question Answering (MSRVTT QA, MSVD. etc)
(1) image is [LMG]] [image]. J. image is [IMG1] [image]. image is [MMG2) [image]. image is [MM33 (image) J. image [IMGI
(image]. J. image is [IMG5] [imuge], J. image 6 [IMG6] (image). J. image 7 is [IMG7] (image). Watch the provided images carefully and answer following questions
on your understanding the images content. Qusetion: [question]. J. Answer:
(2) image 0 is [IMGO) [image). image is [IMGI] [image]. image is [IMG2) [image). image 3 is [IMG3] (image). image 4 is [IMG4]
[image]. J. image
[LMG5] [image). image 6 is [MGG6) (image). ). image is [IMG7] [image). Carefully analyze the visual elements of the smages and answer the questions based your observations. Qusetion: [uuestion]. Answer:
(3) imago 0 is [IMG]] (image). image is [IMG1] [image]. image 2 is [IMG2] [image). image 3 is [IMG3] (image) ). image 4 is [IMG4]
(image). image
[IMG5] [imugee). ). image
[IM66] [imuge]. imuge is [MM77] [imuge]. ). Pay close atlention to details in the
images and provide accurate answers to the questions based on what you see. Qusetion: [question]. Answer:
(4) image 0 is [IMG0] [image). ). image is [[MGI] |image] I. image is [IMG2] [image] ). image is [[MG]] [image). image [TMG4]
(image). image 5 is [JMG5] [image]. 3. image 6 is [[MG6] [image]. image is [IMG7] (image]. Utilize your comprehension skills
answer the questions based on the context and events depicted in the images. Qusetion: [question]. Answer:
(5) image is (IMG0) [image]. image is [IMGI] [image]] image is [MMG2] [image]. iimage is [IMG3] (image). ). image is [IMG4]
(image). image 5 is [MMG5] [image]. image is [TM66] [image]. image 7 is [TMG7] (image). Consider the relationships hetween the
images frames, scenes, and the provided questions formulate accurate answers. Qusetion: (question), ). Answer:
(6) image 0 is [TMG0) (image). ). image is [TMG]] [image]. image is [IMG2] [image]. image 3 is [TMG]] [image). ). image 4 is [MMG4]
(image). ). image 5 is [IMG5] [image)) J. image is [[MG] [image]. ). image [IM77] (image). Use your knowlodgs of the images's s content answer the questions by recalling specific details and events. Qusction: [question]. Answer:
(7) image O is [IMG0] (image). J. image is [IMGI] [image]] image 2 is [IMG2] [image]] imaae 3 is [IMG3] [image J. inage is [IMG4]
(image). image 5 is [IMG5] (image). image 6 is [TMG66 (image). image 7 is [IMG7] [image]. Make Ingical inferences based the
information presented in the images to answer the questions with reasoned explanations. Qusction: [question). ). Answer:
(8) image is [IMGO] (image). J. image is [IMGI] [image]. image is [IMG2] [image). image [IMG3] (image). ). image is [IMG4]
(image), image 5 [MMG5] [image]] image 6 is [IMG6) [image]. image 7 is [MMG]] [image]. While answering the questions, consider both the explicit and implicit information conveyod in the images to provide comprchensive responses. Qusction: [question]. Answer: (9) image 0 is [LMG]] (image]. J. image is [IMGII [image]. image is JMMG2| [image]. image is [MM33] [image), image is JIMG4] (image). J. image 5 is [MGG5] (image). J. image 6 is [JMG6] [image]. image 7 is [IMG7] (image) ). Formalate your answers by considering the temporal context of the images and the chronological order of events. Qusetion: [question]. Answer:
image6is
(10) image 0 is [IMGO] (image]. | image is [IMGI] (image). image 2 (IMG2) (imags). image 3 [IMG3] [image]. imago is [IMG4]
(imuge). image is [IMG5] [image). I. image 6 is [IMG6] [image]. image is [TMG7] [image]. into account the emotions, actions, and interactions of the characters in the images when answering the questions. Qusction: [question]. J. Answer:
Table 15: Instruction templates for task MSRVTT QA and MSVD.
28

---

Published as a conference paper at ICLR 2024
Templates Visual Reasoning (GQA, VCR, NLVR v2, etc)
GQA
(1) image is [IMGO] (image). For the question, carefully examine the image and use your knowledge to determine the correct answer.
Question: [question] Answer:
(2) image 0 [IMG0] [image)]. Given the picture (IMGO), pay atteation the wording of question and answer the following question:
(question] Answer:
(3) Read the question carefully and look image labeled [IMGO] [image]. J. Use your intuition and common when answering the
question: [question]
(4) Consider of the information in image O labeled [IMGB] (imuge] when unswering question: [question]
(5) The image 0: [TMGG] (image) is the primary source of information for answering the questions. Please refer to it carefully when answering question: [question) Answer:
(6) Pay close atiention to the details in image O: [IMG0] (image). as they may provide important information for answering the questions.
Question:
Quesion:[qqustioel) (question] Answer:
(7) image is [IMGO] [image]. Make your answer relevant to the question and the image O. Question: (question) Answer:
(8) image O is [IMG0] [image). ). Do not provide answers based on assumptions or personal opinions; only use the information presented in
image and the question. Question: qquestion) Answer:
(9) Look at image 0 labeled [IMGO] [image] carefully and read question: (question). Try to understand what is heing asked before selecting
an answer.
(10) image O [IMGU] (image). Consider all of the information in image labeled [LMGO] when answering question. Look objects.
colors, shapes, and other details that may be relevant to question: [question] | Answer:
VCR
(1) [prumpt]. Given the options below, based on the photo [IMGO], select the must suitable answer for the following question: [question). Options: [options]
answers question. [prompt]. Given the images, select the best option that answers the question from the available answer choices. Question: [question] Optione [options] Answer:
(2) Pleasse the question and answer choices carefully. Select option
(3) Choose the answer that best fits the description or action in the image. pprompt). Consider the scene depicted in the images, choose the answer that best fits the description action in the image from the available answer choices. Question: [question] Options: [options] Answer:
(4) [prompt]. Examine the detaill in the pictures and them inform your answer the question. Choose the best answer from the available options. Question: [question] Options: [options) Answer:
(5) Look closely
images think about what is happening in the scene. [prompt]. Given the pictures, carefully examine the images and select the hest answer that describes what is happening in the scene from the available answer choices. Question: [question] Options:
[options) Answer:
(6) Consider all of the details in the image and the wording of the question before making your selection. (prompt). Given the pictures. consider all of the details in the image and the worling of the question before selecting the best unswer choice from the available options. [question] I Options: (options] Answer:
Question:
(7) Remember to use your common sense and reasoning skills to choose the best answer. [prompt). Think about the imagos, use your common sense and reasoning skills to select hest answer choice from available options. Question: [question] Options: [options] Answer:
(8) [prompt). Select the answer that most closely matches the description or action in images, based on the available options. Given the picture [LMG0), select the answer choice that most closely matches the description action the inage from the available options. [question] Options: [aptions] Answer:
Question:
to the question. based the available information. [prompt] Given the images, select the option that provides the most accurate and complete answer to the question from the availablc answer choices. Question:
(9) Choose the option that provides the accurate and complete
[question| I Options: [options] Answer:
(10) (prompt). Use the information in the images to help you make the best choice from the available answer options for the question Question: [question] Options: (options) ) Answer:
NLVR v2
(1) image is [JMGO] (image]. Given the picture [IMG0), answer the following question: [question] Is this correct? True or False. Answer:
(2) For the question: (question). ). carefully examine imuge 0: [IMG0] (image) and use your knowledge to deternine if the statement is True
False.
(3) Please refer to image O: [IMG0] (image) when answering the question: [question] Is this correct? True or False, Answer:
(4) Remember to consider both the question and information presented image 0: [IMG0] (image] when answering the True or False
question: [question]
(5) image is [IMG0] [imago]. Answer the question: [question) | based on the information presented the image 0 and determine the
statement True False.
(6) Carefully examine the image O: [IMGO] (image) and use your knowledge to determine whether the siatement is True or Falee Question: (question)
(?) Remember that the answer to cach question either True or Falsc, so make sure you choose the correct option based on the information
presented image O: [IMG0] (image]. Question: [question]
(8) Make sure your answers are based on the information presented in the image O: [IMG0] (image). Question: (quessioon Is this
corect/True or False. Answer:
(9) Carefully examine image labeled [IMGO] (imuge) before amswering the question. Dneestion/qqustiion True False? Answer:
Table 16: Instruction templates for tasks GQV, VCR and NLVR v2.
29

---

Published as a conference paper at ICLR 2024
Commonsense Reasoning
Numerical Calculation
Text Translation
Code Reasoning
Model
Avg.
MiniGPT-4 Zhu
2023
$9.29
45.00
0.00
40.00
36.07
VisualGL Duu el al
2021
39.29
45.00
50.00
47.50
45.45
LLaVA (Liu et al.
2023b
57.14
50.00
57.50
50.00
53.66
Lynx (Zeng at.
2023
110.71
17.50
42.50
45.00
53.93
MultiModal-GiP
Gong al.
2023
49.29
62.50
60.00
55.00
56.70
LLaMA-Adapter
(Gao et a
2023
81.43
62.50
50.00
55.00
62.23
VPGTrans (Zhang ct al. 202.5a
64.29
50.00
77.50
57.50
62.32
LaVIN [Luo et al.
2023
87.14
65.00
47.50
50.00
62.41
GIT2 Wang
2122a
99.29
50.00
67.50
45.00
65.45
mPLUG-OWl
al.
2023
78.57
60.00
80.00
57.50
69.02
BLIP-2 [i et al 2023d
110.00
40.00
65.00
75.00
72.50
Instruc(BLIP (Dai et al.
2023)
129.29
40.00
65.00
57.50
72.95
Otter (Li ct al.
2025a)
106.43
72.50
57.50
70.00
76.61
Cheetor (Li
2023c
98.57
77.50
57.50
87.50
78.02
LRV-Instraction
Liu ct al.
2023a
100.71
70.00
85.00
72.50
82.05
BLIVA (Hu et al.
2023b)
136.43
57.50
77.50
60.00
82.86
MMICL
136.43
82.50
132.50
77.50
107.23
Table 17:
Evaluation of cognition. In the MME benchmark, each image will have questions, with answers restricted to 'yes' or "no". The evaluation metrics for this benchmark include ACC and ACC+ ACC refers the accuracy calculated for each question, while ACC+ represents the accuracy for each image, where both questions must be answered correctly. The Avg. metric denotes the average value across all numbers. is important methods are obtained from the MME benchmark (Fu et al. 2023). We use the FLAN-T5-XXL. version of MMICL to evaluate the performance.
that all the reported figures for the baseline
All experiments are conducted with 6 NVIDIA A40 GPUs with the zero2-offload (Rajbhandari et al. 2020) of Deepspeed (Rasley et al.) 2020) with the trainer of huggingface transformers (Wolf et al. 2020). The batch size is 10 and for MMICL (FLAN-T5-X) and MMICL (FLAN-T5-XXL), respectively. The largest MMICL (FLAN-TT--XXL) requires about two days for the Stage II.
1
MME BENCHMARK
Model
Existen.
Count
Pos.
Color
OCR
Poster
Cele.
Scene
Land.
Art.
Avg.
LLaVA
50.00
50.00
50.00
55.00
50.00
50.00
48.82
50.00
50.00
49.00
50.28
MiniGPT-4
68.33
55.00
43.33
75.00
57.50
41.84
54.41
71.75
54.00
60.50
58.17
MultiModal-GG
61.67
55.00
58.33
68.33
82.50
57.82
73.82
68.00
69.75
59.50
65.47
VisualGLM-6B
85.00
50.00
48.33
55.00
42.50
65.99
53.24
146.25
83.75
75.25
70.53
VPGTrans
70.00
85.00
63.33
73.33
77.50
84.01
53.53
141.75
64.75
77.25
79.05
LaVIN
185.00
88.33
63.33
75.00
107.50
79.59
47.35
136.75
93.50
87.25
96.36
I. V22
120.00
50.00
48.33
75.00
125.00
99.66
86.
148.50
150.25
69.75
97.27
mPLUG-OWl
120.00
50.00
50.00
55.00
65.00
136.05
100.29
135.50
159.25
96.25
96.73
InstructBLIP
185.00
143.33
66.67
153.33
72.50
123.81
101.18
153.00
79.75
134.25
121.28
BLIP-2
160.00
135.00
73.33
148.33
110.00
141.84
105.59
145.25
138.00
136.50
129.38
Lynx
195.00
151.67
90.00
170.00
77.50
124.83
118.24
164.50
162.00
119.50
137.32
GIT2
190.00
118.33
96.67
158.33
65.00
112.59
145.88
158.50
140.50
146.25
133.21
Otter
195.00
88.33
86.67
113.33
72.50
138.78
172.65
158.75
137.25
129.00
129.23
Cheetor
180.00
96.67
80.00
116.67
100.00
147.28
164.12
156.00
145.73
113.50
130.00
LRV-Instruction
165.00
111.67
86.67
165.00
110.00
139.04
112.65
147.98
160.53
101.25
129.98
BLIVA
180.00
138.33
81.67
180.00
87.50
155.10
140.88
151.50
89.50
133.25
133.77
MMICL
170.00
160.00
81.67
156.67
100.00 146.26
141.76
153.75
136.13
135.50
138.17
Table 18: Evaluation of coarse-grained and fine-grained recognition and OCR. The settings are the same as Table I7 It is important to note that all the reported figures for the baseline methods are obtained from the MME benchmark (Fu et al. 2023). We use the FLAN-T5-X0 XXXL version of MMICL to evaluate the performance.
30

---

Published as a conference paper at ICLR 2024
Existence ACC
Count ACC
Position ACC
Color
OCR
Model
Avg.
ACC+
ACC+
ACC+
ACC
ACC+
ACC
ACC+
BLIP-2
73.33
75.00
60.00
56.67
16.67
81.67
66.67
70.00
40.00
62.67
LLaVA
50.00
0.00
50.00
0.00
50.00
0.00
51.67
3.33
50.00
0.00
25.49
MiniGPT-4
75.00
60.00
66.67
56.67
56.67
33.33
71.67
53.33
62.50
35,00
57.08
mPLUG-Ow/
73.33
46.67
50.00
0.00
50.00
0.00
51.67
3.33
55.00
10.00
34,00
LLaMA-Adapees
76.67
56.67
58.33
43.33
55.00
16.67
57.50
15.00
38.92
VisualGLM-66
61.67
23.33
50.00
0.00
48.33
0.00
51.67
3.33
42.50
0.00
28.08
Otter
53.33
6.67
50.00
0.00
50.00
0.00
51.67
3.33
50.00
0.00
26.50
MultimodalGGPT
46.67
10.00
51.67
6.67
45.00
13.33
55.00
13.33
57.50
25.00
32,42
PandaGPT
56.67
13.33
50.00
0.00
50.00
0.00
50.00
0.00
50.00
0.00
27.00
MMICL
90.00
80.00
86.67
73.33
55.00
26.67
88.33
73.33
60.00
40.00
67.33
Table 19:
Fine-graines result of MME benchmark
Poster ACC
Celebrity ACC:
Scene ACC
Landmark ACC
Artwork ACC
Model
Avg.
ACC+
ACC+
ACC+
ACC+
ACC+
BLIP-2
79.25
62.59
58.53
37.06
81.25
64.00
79.00
59.00
76.50
60.00
66.72
LLaVA
50.00
0.00
48.82
0.00
50.00
0.00
50.00
0.00
49.00
0.00
24.78
MiniGPT-4
49.32
19.73
58.82
24.71
68.25
45.50
59.75
30.50
56.25
27.00
44.00
mPLUG-Owl
77.89
57.14
66.18
34.12
78.00
57.50
86.25
73.00
63.25
33.00
62.63
ALaMA-Adapter-V
52.72
10.88
55.00
21.18
68.75
44.50
53.00
9.00
52.50
14.50
38.2
InstructBLIP
74.15
49.66
67.06
34.12
84.00
69.00
59.75
20.00
76.75
57.50
59.20
VLsu
54.42
12.24
50.88
2.35
81.75
64.50
59.75
24.00
55.75
20.00
42.56
Otter
45.24
0.00
50.00
0.00
55.00
14.50
52.00
4.50
48.00
5.50
27.47
Multimodal GPT
45.24
17.01
49.12
24.12
50.50
17.50
50.50
23.00
46.00
12.00
33,50
PandaGPT
56.80
19.73
46.47
10.59
72.50
45.50
56.25
13.50
50.25
1.00
37.26
MMICL
81.63
64.63
79.41
62.35
83.75
70.00
76.96
59.16
76.50
59.00
71.04
Table 20: Fine-grained result of MME benchmark
MME comprehensively evaluates VLMs with 14 sub-tasks that encompass perception and cognition abilities. Other than OCR, perception ability includes the recognition of coarse-grained and fine- grained objects. The former identifies the existence, count, position, and color of objects. The latter recognizes movie posters, celebrities, scenes, landmarks, and artworks. The cognition includes commonsense reasoning, numerical calculation, text translation, and code reasoning.
MME evaluates a wide range of multi-modal abilities. The compared baselines include LLaVA (Liu MultiModal-GPP (Gong al., (023), VisualGPM
et al. 2023bb, MiniGPT-4 (Zhu al.| 2003).
6B (Du et al. 202T), VPGTrans (Zhang et al.
2023a) LaVIN (Luo et al. 2023). 3, mPl mPLUG-OWl (Ye
et al. 2023). A-MA-Adapter V2 (Gao
2023),
InstructBI.PP (Dai et al. (023). Otter (Li et al.
2023a), BLIP-2 (Li et al., 2023d). L.V-IInstrcctin (Liu et al.
2023a), 3 Cheetor (Li et al. 2023c).
GIT2 (Wang et al. (2022a), Lynx (Zeng al.) 2023). BLIVA Hu al. 2023b). We also provide
more detail evaluation results for MMICL at Table 18 Table 19 Table 20 and Table 2] Results
show that MMICL can achieve the best average scores comparisons with current VLMs.
J MMBENCH BENCHMARK
2023c) is thoughtfully designed benchmark that thoroughly evaluates the diverse skills of vision-language models. The results of all different VLMs from the test set are presented in Table 22
MMBench (Liu et al.
K UNDERSTANDING MULTIPLE IMAGES IN THE MULTI-MODAL PROMPT
Videos contain temporal information compared to static images. We test MMICL across different video-languages tasks to evaluate whether the MMICL is able to support the multiple images in the complex prompts. The result is present in Table 23 Our model, MMICI, achieved
31

---

Published as a conference paper at ICLR 2024
Common.
Reason. ACC+
Numerical Calculation
Text Translation
Code Reason.
Model
Avg.
ACC
ACC
ACC+
ACC
ACC+
ACC
ACC
BLIP-2
68.57
41.43
40.00
0.00
$5.00
10.00
55.00
20.00
36.25
LLaVA
49.29
11.43
50.00
0.00
52.50
5.00
50.00
0.00
27.27
MiniGPT-4
58.57
34.29
47.50
20.00
42.50
15.00
67.50
45.00
41.30
mPLUG-Owl
59.29
24.29
50.00
10.00
60.00
20.00
47.50
10.00
35.14
LLaMA-AGO Ada. V2
54.29
14.29
52.50
5,00
52.50
5.00
52.50
10.00
30.76
InstructBLIP
75.00
54.29
35.00
5.00
55.00
10.00
47.50
0.00
35.22
VLsu
45.71
12.86
45.00
0.00
55.00
10.00
50.00
0.00
27.32
Otter
48.57
10.00
47.50
10.00
55.00
10.00
50.00
0.00
28.88
MuT
45.71
5.71
50.00
20.00
50.00
5.00
45.00
10.00
28.93
PandaGPT
56.43
17.14
50.00
0.00
52.50
5.00
47.50
0.00
28.67
MMICL
76.43
60.00
47.50
35.00
72.50
60.00
47.50
30.00
53.62
Table 21:
Fine-grained result of MME benchmark
Method
Language Model
Vision Model
Overall
LR
AR
RR
FP-S
FP-C
CP
MMGPT
LLaMA-7B
CLIP VIT-L/14
16.0
1.1
20.7
MiniGPT-4
Vincuna-7B
EVA-G
12.0
13.6
32.9
8.9
28.8
11.2
28.3
PandaGPT
Vincuna- 13B
ImageBind ViT-H/14
30.6
15.3
41.5
22.0
20.3
20.4
47.9
VLsualGLM
ChatGLM-6B
EVA-CLIP
33.5
11.4
48.8
27.7
35.8
17.6
41.5
InstructBLIP
Vincuna-72
EVA-G
21.6
22.5
33.0
24.4
41.1
LLaVA
LLaMA-7B
CLIP VIT-L/14
36.2
15.9
28.6
41.8
20.0
G2PT
LLaMA-7B
VIT-G
39.8
14.8
46.7
31.5
41.8
34.4
49.8
Otter-I
LLaMA-7B
CLIP VIT-L/14
48.3
22.2
63.3
39.4
46.8
36.4
60.6
Shikra
Vincuna-7B
CLIP VIT-L/14
60.2
33.5
69.6
53.1
61.8
50.4
71.7
LMEye
Flan-XL
CLIP VIT-L/14
36.9
73.0
55.4
68.0
mPLUG-Owl
LLaMA-7B
CLIP VIT-L/14
62.3
37.5
75.4
56.8
67.3
52.4
67.2
JiuTian
FLANT5-XXL
EVA-G
64.7
46.6
76.5
66.7
66.5
51.6
68.7
MMICL
FLAN-T5-XXL
EVA-G
65.24
44.32
77.85
64.78
66.5
53.6
70.64
Table 22: Evaluation of MM benchmark dev set. All the reported performance for the baseline methods is from the leaderboard of MM benchmark (Liu et al. 2023ch. We use the FLAN-T5-XXL version of MMICL to evaluate the performance.
significant improvement of 10.86, 4.53, and 2.45 points for MSVD-QA (Chen & Dolan (011). NEXT-QA (Xiao et al., 2021), and iVQA (Yang et al. 2021) respectively, when compared to the strongest baselines. important to note that our training dataset did not include any videos. This indicates that MMICL effectively enhances the model's 's ability to understand temporal information videos.
OBJECT HALLUCINATION EVALUATION
We test the following VLMs on the POPE benchmark to evaluate their object hallucination perfor-
mance: MMICL, Shikra (Chen al. 2023a InstructBLIP (Dai et al. (023). MiniGPT-4 (Zhu et al.
2023), LLaVA (Liu et al. 2023b), MM-GPT
Gong et al. 2023) and mPLUU-OWl Ye et al. 2023)
The result is present in the Table
M DETAILS FOR EVALUATION
this Section. we provide details for evaluation in our experiments as Sec.
M.1 EVALUATION METRICS
We provide evaluation metrics Table 25
32

---

Published as a conference paper at ICLR 2024
NExT QA Multi-choice
Model
MSVD QA
iVQA
Flamingo-3B Alayrac al.
2022
(Zero-Shot)
27.50
32.70
Flamingo-3B Alayrac et al.
2022
(4-Shot)
33.00
35.20
Flamingo-91 Alayrac
al.
2022
(Zero-Shot)
30.20
35.20
Flamingo-99 9B Alarac et al. 2022
(4-Shot)
36.20
37.70
Flamingo-80B [laarrac et al.
2022
(Zero-Shot)
35.60
40.70
Flamingo-80B Allayrac et al.
2022
(4-Shot)
41.70
44.10
R2A (Pan et al.
2023)
37.00
29.30
BLIP-2 Li et al.
2023d
(FLANTS-XL))
33.70
61.73
37.30
BLIP-2
[I et al.
2023d
(FLANTS-XXL)
34.40
61.97
49.38
InstructBLIIP (Dai al.
2023 (FLANTS-XL)
43.40
36.10
25.18
InstructBLIP (Da et al.
2023) (FLANTS-XXL)
44.30
64.27
36.15
MMICL (FLAN-TS-XL)
47.31
66.17
41.68
MMICL (FLAN-T5-XXL)
55.16
64.67
41.13
MMICL (InstructFFLAN-TT FELA-------IIII -L)
53.68
65.33
49.28
MMICL (Instruct-FLAN-T55 XXLL
52.19
68.80
51.83
Table 23: Results of MMICL compared with other VLMs across different video-languages tasks. For Blip-2 and Instructblip, We concatenate the visual embeddings of all frames and place them on the top of the textual prompts following Dai al. (2023).
Table 24: Performance result of different VLMs the POPE benchmark
Models
Dataset
Metric
MMICL
Shikra
InstructBLIP
MiniGPT-4
LLaVA
MM-GPT
mPLUG-OWl
Accuracy
87.29
86.90
88.57
79.67
50.37
50.10
$3.97
Precision
94.63
94.40
84,09
78.24
50.19
50.05
52.07
Random
Recall
79.87
79.27
95.13
82.20
99.13
100.00
99.60
F1-Score
86.62
86.19
89.27
80.17
66.64
66.71
68.39
Yes
43.51
43.26
56.57
52.53
98.77
99.90
95.63
Accuracy
82.70
83,97
82.77
69.73
49.87
$0.00
$0.90
Precision
85.11
87.55
76.27
65.86
49.93
50.00
50.46
Popular
Recall
79.27
79.20
95.13
81.93
99.27
100.00
99.40
F1-Score
82.08
83.16
84.66
73.02
66.44
66.67
66.94
Yes
46.57
45.23
62.37
62.20
99.40
100.00
98.57
Accuracy
80.97
83.10
72.10
65.17
49.70
50.00
$0.67
Precision
81.88
85.60
61.19
49.85
50.00
50.34
Adversarial
Recall
79.53
79.60
95.13
82.93
99.07
100.00
99.33
F1-Score
80.69
82.49
77.32
70.42
66.32
66.67
66.82
Yes
48.57
46.50
73.03
67.77
99.37
100.00
98.67
M.2 VQA TOOLS
use the same VQA Tools as the original VQA paper (Agrawal et al. 2016) and use in all metrics using the VQA accuracy.
N ABLATION STUDY
N. ABLATION STUDY CONTEXT SCHEME
We conduct ablation experiments using the InstructBLIP-LLANTE as the backbone model the following settings verify the effectiveness of our proposed context scheme:
33

---

Published as a conference paper at ICLR 2024
Dataset
Metries
MSVD Chen & Dolan 2011
Top-1 Acc.
IVQA Yang el al. 2021
IVQA Acc.
NEXT-QA-muliilee-e
Xiao el al.
2021
Top-1 Acc.
NEXT-QAA-penddmmin (Xiao al.
WUPS Score.
Hateful Memes (Kiela ct al.
2020)
AUC Score
WebSRC (Chen al.
2021b
Exact Match
VSR (Liu 2022
Top-1 Acc.
* *VQAv2 Goyal
2017
VQA Acc.
VizWiz BBgham et al.
2010h
VQA Acc.
IconQA- (Lu al
2021
Top-1 Acc.
IconQA-img (Lu
2021
Top-1 Acc.
ScienceQA-IMG (Lu al.
2022
Top-1 Acc.
Bongard-HOl (Jiang ct al.
2022
Top-1 Acc.
VisDial (Das ct al. 2017)
Exact Match
NoCaps (Agrawal ct al. 2019)
Cider Score
A-OKVQA (Agrawal
2019
Top-1 Acc.
*Flickr (Young el al. 2014
Cider Score
Winoground Thrush cl al.
2022b
Winoground mertic.
Raven IQ Test (Huang et a.
2023a
Top-1 Acc.
Minecraft
Top-1 Acc.
Table 25: Summary of the evaluation datasets and metrics. These datasets are used to validate the general design of MMICL. The datasets marked with are the hold-in datasets, where their training set is used in training the MMICL.
Winoground
Image-score
Text-score
Group-score
w/o context scheme w/o image declaration
12.25
17.25
6,00
4.00
22.25
3.00
w/o in-context format
31.25
25.75
20.00
w/o interrelated images
36.50
31.25
25.75
MMICL
46.00
39.25
38.75
Table 26: Ablation study on Context Scheme for different scores on Winoground.
w/o context scheme: This model trained only with instructions using all datasets from stage 2 of MMICL, without any additional design in context format and model architecture. It aims to ablate the contribution of the multi-modal instruction tuning.
w/o image declaration: This model trained without image declaration but includes multi-modal data with interconnected images and in-context data. It utilizes all datasets used in stage 2 of MMICL and intends to evaluate the impact of image declaration design.
w/o in-context format: This model, devoid of in-context format data, comprises of multi- modal data with related images and image declaration. uses all datasets found in stage of MMICL with a single image-text pair and aims to assess the effectiveness of multi-modal in- in-context data design.
w/o interrelated images: This version the model trained without any interconnected images. Its training pipeline contains multi modiil in-context data and image declaration. It utilizes all datasets used in stage 2 MMICL, excluding the vidco/VCR datasets. aims evaluate the contribution of the design which incorporates multi-modal data with interrelated images
We use the equivalent amount of data as in the stage II of MMICL for the ablation models. We employ the ablation study on both the general (MMM),, Winoground (Thrush et al., 2022b) and multi-image datasets(IconQAA img (Lu et al.) 2021) & NLVR2 (Suhr et al. 2018), Raven IQ- test (Huang et al. 2023a)) to comprehensively evaluate the sources of the observed performance improvement in MMICI
34

---

Published as a conference paper at ICLR 2024
The ablation results in aafirm that the superiority of MMICL is driven by the collective impact of our design elements- removing any component cannot guarantee superior performance of our model. Each component of our design significantly contributes to different aspects of our model.
Ablation of Context Scheme: Our improved results demonstrate the effectiveness of our context scheme design rather than merely aggregating more pretraining data into the pipeline. The substantial improvements across all tasks when compared the w/o context scheme further substantiate its effectiveness. Lacking the context scheme and our model architecture supporting the scheme, the model performed poorly across most datasets, except performance on the single image task focusing on specific object perception (such as Existence, Count). This is presumably due to the alignment of the additional single image-text pair training with the task's evaluation format.
Ablation of Interrelated Images: The exclusion of this design leads to a significant decrease in multi-image understanding ability, evidenced in the NLVR2 and Raven datasets. However, multi- modal in-context training pipeline still enabled the model to retain the nulti-image understanding ability, achieving an advantage over datasets with multiple images (IconQA img, NLVR2, Raven) compared the w/o context scheme. At the same time, the design of multi-modal in-context data still improves the model on single image understanding over the w/o in-context format, particularly in understanding the task about specific object perception (Perception), but performed poorly in more abstract tasks involving recognition and Dognition(Cognition) When compared to the w/o in-context format, it demonstrates a superior understanding of paired image and text data, evident in the performance in Winoground. This suggests that multi modal data with interrelated images provide some assistance in understanding multiple images. However, without the help of multi-modal ICL data, will not display superior performance. It aligns with our initial insight.
Ablation of In-context Format: Excluding this design revealed that even with the introduction
of extra datasets, performance improvement on the single image tasks was not achieved. Compared to other ablation settings, despite the w/o in-context format receiving a noticeable improvement in tasks related to comprehension and inference(Cognition) compared to other ablation settings, we observed a significant drop in the task about specific object perception (Perception). Training with multiple images enhanced its ability to better understand abstract logic information within a single image and to perform better in downstream tasks (Cognninn).. However, the model lost its ability to analyze and recognize specific objects in single image, leading to a significant decrease in the perception score of MME We found out that, by benefiting from learning on interrelated image data, its understanding ability on multiple images has not significantly declined, exhibiting its superiority over other settings. The performance in the MME also confirms this, indicating that multi-modan in-context data offers significant advantages for comprehending multiple images and enhancing the model's 's ability to understand abstract information in the signal image.
The removal of image declaration led to a stark decline in performance in understanding complex image-text relationships (Winoground) due to lack of explicit modeling between image-text references. This phenomenon was also observed in tasks that involved understanding multiple image relations(Raven, IconQA img, NLVR2). However, it can be observed lacking interrelated images harms the model's 's multi-image understanding more than missing image declarations, given the relatively better performance of model without image declarations. This indicates the importance of image declaration in handling the multi-image and image-text relationships.
Ablation of Image Declaration:
Ablation of In-context Learning Ability: The ablation results in Table 27 clarify the significance of the proposed context scheme for boosting the in-context learning ability of MMICL. The effectiveness of MMICL in improving the in-context learning ability of the VLM attributed to the integration of all three designs in the context scheme, not merely the in-context learning format in the designed scheme. Notably, ICL-only achieves the most substantial enhancements compared to the other settings in the Vizwiz dataset. This highlights the efficacy of tuning with in-context format data to augment the model's 's comprehension of in-context instances. However, the modest advancements on the Flickr dataset indicate that, in the absence of multi--maae instruction tuning, the model finds it challenging to effectively leverage in-context examples for enhancing its performance in tasks without fixed output formats, such image captioning.
35

---

Published as a conference paper at ICLR 2024
Task
ICL example
MMICL
w/o context scheme
w/o in-context format
w/o interrelated images
w/o ICL example
27.10
28.13
15,90
23.43
VizWiz
ICL example(S)
42.66
36.23
28.33
37.13
15.56
8.10
12.43
13.70
w/o ICL example
83.94
77.84
65,33
76.88
Flickr
w ICL example(5)
88.11
79.38
65.95
77.09
4.17
1.54
0.62
0.18
Table 27: An Ablation Study of the Contribution of the Context Scheme to the Model's In-Context Learning Ability
O BASELINES
We primarily compare MMICL with recently proposed powerful multi-modal approaches, including:
Baselines
(1) Flamingo (Alayrac al. 2022) where VLM is trained large-scale multi-modal- web corpora containing arbitrarily interleaved text and images;
(2) KOSMOS-1 (Huang al. 2023a) which is trained from scratch on web-scale multi-modal corpora;
(3) BLIP-2-FLAN-TS (Li et al.
2023d) where an instruction-tumed Flan-T5 (Chung et al. 2022 is
connected with powerful visual encoder to perform a series of multi-modal tasks;
(4) InstruccBLIP-FLAN- (Dai et al. 2023 recently proposed instruction tuning enhanced multi-modal agents with FLAN-T5 with converted multi-modal datasets and the LLaVA (Liu et al. 2023b dataset generated by GPT-4 OOpenAl 2023);
(5) Shikra (Chen et al. 2023a), a VLM that can handle spatial coordinate inputs and outputs in natural language without the need for extra vocabularies or external plugin models. All inputs and outputs of Shikra are natural language form.
(6) Otter (Li et al. 2023a), open-source implementation of flamingo (Alayrac al. 2022 By utilizing multi-modal instruction in-context tuning data, Otter fine-tunes Openflamingo to augment its instruction comprehension capabilities while maintaining its ability to learn in context;
(7) Ying- VLM (Li et al. 2023e). a VLM model trained on Multi-Modal multilingual instruction tnning dataset, showcasing its potential answer complex questions requiring world knowledge, generalize to unseen video tasks, and comprehend unseen instructions in Chinese
P OOD GENERALIZATION TO UNSEEN DOMAIN
Method
Shot
Top-1 Acc.
MiniGPT-4 (Vincuna-77 7B)
Zero-Shot
35.10
MiniGPT-4 (Vincuna- 13B)
Zero-Shot
48.40
MMICL (PLAN-TS-XXT
Zero-Shot
55.41
MMICI (FLAN-T5-XI)
4-Shot
64.05
MMICL (FLANNTT-XX-
8-Shot 65.41
Table 28; Results of generalization of MMICL to unseen domain in Minecraft. Results show that MMICL is able to generalize to unseen domains and tasks given few examples.
In an unseen challenging domain with limited exemplars, analyzing regular patterns, reasoning, and learning new knowledge (OOD Generalization unseen domain) great way to test multi-modal ICL ability.
We construct task using Minecraft (Chen al. 2024a 2023c Cipollone al. 2014), which requires the VLM to identify whether an animal (i.c., cow, llama, chicken, donkey, and so on) is present in case (d) of Fig.
36

---

Published as a conference paper at ICLR 2024
Model
rec
OCr
know
gen
spat
math
InstruciBlip(FlantS-XX MMICL((nstructBiipp- Flant5-XXL.
17.10
8.40
5.50
2.60
8.00
3.80
14.10
29.90
14.00
17.40
11.20
18.10
3.80
24.80
Table 29: Performance metrics for different models.
We collect 550 cases and transfer the task to a vision-to-text
task to evaluate
the performance OOD generalization of MMICL. The results are shown Table 28 Results demonstrate that MMICL is able to generalize to the Minecraft domain even if the images extremely different compared to the images used by training at Stage and Sec. 2.4
Q DETAILS OF SCIENCEQA
In the test set of Science-QA. each item consists of a question and several choices. Out of 4221 items, we sampled 2017 instances that contain multimodal information incorporating both text and image, a subset we refer to as ScienceQA-IMG, while the remainder only includes text. However, that not all questions within these 2017 items necessarily require answers based on image information. The standard for annotation is whether understanding of the image sample is essential to answer the questions in the test set.
observe
To illustrate this, provide examples of questions and their corresponding choices:
Question: Which animal is also adapted to be camouflaged among dead leaves? Choices: [ "strawberry poison frog". "Surinam horned frog" ]:
Question: Which property do these three objects have common? Choices: 'slippery", 'opaque"].
["shiny',
Clearly, the first question be answered without reference to image, while the second not. As such, ScienceQA-IMe can be divided into two groups: questions that require images for answers (1012) and those that do not (1005).
R FREE-FORM ANSWERING EVALUATION
MM-VET Yu et al. 2023 is an evaluation benchmark that examines VLMs on complex multi-modal tasks. It requires free-form answers and is evaluated by GPT-4, providing different perspective the model's capabilities.
It's worth noting that compared the Flant5 backbone's instructblip model in table 29 MMICL, with the identical backbone, shows significant improvement on MM-VET. This demonstrates that our proposed method prevents the model from overfitting a specific data structure and substan- tially enhances the model's 's free-form answering capability. Meanwhile, we can observe that the performance of MMICL on MM-VET is relatively low, which is due to MMICL using the Flant5 backbone model. As '5 is mainly fine-tuned on NLP benchmarks containing many multi-choice QA and classification datasets, it tends to output shorter content during free-form question answering, which is disadvantageous when evaluated with GPT4.
PERFORMANCE COMPARISON ON THE MULTI-IMAGE DATASETS
We evaluated the MMICL model against other models with multi-modal instruction tuning on various multi-image datasets, including the Raven-IQ dataset and random sample of 2000 instances from NIvr2 and (conQA-imm datasets. The query prompts for different methods are available in Table ?? Our results in Table 30 demonstrate that our approach consistently outperforms others.
The minimum hourly wage in the United States is near $7.5, which can be found at httpss//www. worker On average, annotating example takes 30 seconds. Consequently, the cost per annotator amounts to
gov
$262.5
37

---

Published as a conference paper at ICLR 2024
Method
NLVR2
lconQA-img
Raven
InstructionBlip
53.95
45.10
10.00
OTTER
47.2
34.10
22.0
MMICL
66.6
60.85
34.00
Table 30; Performance comparison of different models on multi multi--maae databases
Method
Tasks
Query prompt
NLVR2
<image><imageSCrrefflly read the statement and evaluate the associated images, based on your analysis of the images and statement, can you confirm the statement: (Question) is correct? True or False?
IconQA-img
<image>...<imaee>> Use the images as visual aid to help you answer the questions accurately. Given the options blow, based on the first image. from the following images, select the most suitable image for the question: (Question). Options: [Options), J. Select the right image. sinag>>..iinageusser Here are images. Consider the relationships of the images to infer the pattern and make the correct decision. Is the last image follows the pattem. Yes or No?
InstructionBlip
Raven
Winoground
<image><<image> caption 0 is (Caption 0)-Am caption I is (Caption J.Am Given images and captions, determine image match caption 1? Yes or no?
NLVR2
<image><<iage>> Carefully read the statement and evaluate the as- sociated images, based on your analysis of the two images and statement, you confirm if the statement: (Question) is correct? True or False? GPT:<answer>
IconQA-img
images as a visual answer the questions accurately. Given the options blow, based on the first image, from the following images. select the most suitable image for the question: (Question). Options: (Options). Seleet the right image. GPT:<answer>>
<image>... <inmage>>useer
help you
Otter
Raven
Here are images. Consider the relationships of the images to infer the pattern and make the correct decision. Is the last image follows the pattern. Yes or No? GPT:canswer>>
Winoground
<image>><imag>> caption 0 is (Caption 0)-An caption 1 is (Caption 1|An Given images and captions, determine image match caption 1? Yes or no? Ganseanser>>
NLVR2
The image is [TMGO] [image) J-yp The image (imgge)
The image 0 is the on the left and the image is on the right.) y Given the image 0 and image 1 visual aid to answer the following question: [Question] is correct? True False?
IconQA-img
The image 0 is [IMGO][image] J. 1/m.... The image [IMG]] (image) J-An Given the options blow, based on the photo [IMGO]. select the most suit- able image for the following question: (Question). Options: {Options). Select the right image.
MMICL
Raven
Here are images. y image [IMGO] Consider relationships images to infer the pattern and make correct decision whether the image O: [IMGO] follows the pattern. Yes or No? image 0 is [TMG0] (image].yy J-yn image is [imaae(maae)...
[IMG]][
Winoground
caption is (Caption 0J. v2 caption is [Caption 1) Given the images and captions, determine if image matches caption Yes or no? Questions is related to image O: [imag]( Please analyze the image and pay attention to the objects in the image for the question: Question)
POPE
Table 31: Query Prompt of different methods on different tasks
38

---

Published as a conference paper at ICLR 2024
T LIMITATIONS
T.I POTENTIAL OVERFITTING:
We mitigate the risk of overfitting specific data structures by expanding the instruction templates with ChatGPT, ensuring diversity in the MIC dataset. The result in Table Sable28demonstrateer the generalization ability of MMICL to an unseen domain in Minecraft. This out-of-domain understanding ability demonstrates that MMICL does not overfit a specific data structure.
T.2 CONTEXT LENGTH CONSTRAINTS:
The context length of the backbone language model imposes restrictions on the number of images that can be included in the input. This limitation renders MMICL unable accommodate number of images. Consequently, during MIC tuning, we integrate to eight images per instance. Notably, MMICL demonstrates robust generalization capabilities beyond this constraint. For instance, in the context of video datasets, MMICL maintains excellent performance with imputs of to 12/16 frames. We attribute this impressive generalization the utilization relative position embeddings in the architecture, employed by Flant5 (Chung et al. 2022).
unlimited
T.3 MODEL EXPLORATION ON DECODER-ONLY ARCHITECTURES:
We acknowledge that our exploration has primarily focused on the T5 (Raffel et al. 2023 series models, and the effectiveness of our proposed approach with decoder-only architectures has not been comprehensively investigated.
39


Reviews:
[{"review_id": "YuJIyuxROi", "reviewer": "Reviewer_7yvn", "summary": "# Summary\nThe paper introduces MMICL (Multi-modal In-Context Learning), a novel approach designed to enhance Vision-Language Models (VLMs) in understanding complex multi-modal prompts, which include multiple images and text. The authors highlight that while Large Language Models (LLMs) excel in in-context learning from text prompts, VLMs lag behind, especially when dealing with prompts that involve intricate text-to-image references and relationships among multiple images. To overcome these limitations, the paper proposes a new context scheme that includes an image declaration section and image proxy tokens to improve the in-context learning capabilities of VLMs. Additionally, the authors have constructed a new dataset tailored to train VLMs on complex multi-modal prompts. The experimental results suggest that MMICL sets a new state-of-the-art in zero-shot performance on various vision-language tasks and benchmarks, demonstrating a significant improvement in understanding text-to-image references and relationships between images. MMICL also shows a reduction in language bias, which often leads VLMs to overlook visual content.", "questions": "Most of my considerations are at the Weaknesses part. The authors may refer it and consider to address these questions.\n\n1. Why using interleaved image-text data for instruction tuning an VLM would be considered beneficial? In terms of the model performance, is there any experimental comparison to support this claim?\n\n2. Is there any quantitative results to support the models ability in free-form answering (evaluated by GPT-4)?\n\n3. Why the MMICL's design is better for image-text interleaved chatting than Flamingo-based or other architectures?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "# Strengths\n\n1. **Enhanced Multi-modal Understanding:** MMICL's approach to handling complex prompts with multiple images and text could significantly improve VLMs' performance on downstream tasks.\n\n2. **State-of-the-Art Performance:** The paper reports new benchmarks in zero-shot performance on vision-language tasks, indicating a substantial advancement over existing models. On MME, MMICL seems to achieve the best average scores compared with current VLMs on cognition and perception tasks, indicating a strong performance. On MMBench, they also achieved SOTA performance which demonstrate the prominent ability of MMICL.\n\n3. **Reduction in Language Bias:** MMICL reduces the chances of VLMs ignoring visual content, which is crucial for accurate multi-modal reasoning. Their experiments provided promising results.", "weaknesses": "# Weaknesses\n\nAlthough I can spot many advantages in MMICL and I truly believe its a wonderful model. This paper also did a lot of experiments to demonstrate its ability. But here I should address few weaknesses and the authors should better clarify it to make the work's claims more sound.\n\n1. The paper emphasizes the use of interleaved image-text data pairs and in-context learning data for training, suggesting this approach is beneficial for understanding complex multi-modal prompts. However, without experimental comparisons, it's unclear if interleaved pairs offer a substantial advantage over single image-text pairs. It's possible that similar results could be achieved without the interleaved structure. The authors should clarify whether the interleaved structure is a key contributor to performance improvements or if it primarily serves to enhance demonstrations and applicability to real-world scenarios.\n\n2. The paper's focus on MME and MMBench, which involve fixed-format questions, might not fully represent the model's ability to handle freeform answers. Benchmarks like MM-VET[1], which require freeform answers and are evaluated by GPT-4, could provide a different perspective on the model's capabilities. A more diverse set of benchmarks, including those requiring freeform answers, would offer a more comprehensive evaluation of the model's performance and generalizability.\n\n3. The paper does not fully discuss why MMICL's architecture is superior to the Flamingo or other paradigms[2] or other existing methods, which leaves the comparison incomplete. Many claims and designs in Section 2.1/2.2 seems just the author's considerations and are lacking sufficient reasons. Although conducting more comparative experiments may be challenging, the author should at least provide more explanations in these aspects to make the arguments in this paper more substantial. Instead of presenting a final result by combining all designs together, the author should provide further explanations to justify their choices.\n\nFor the dataset and training methodology to be validated, the authors should ideally show that the interleaved image-text pairs lead to better model performance than traditional single image-text pairs, across a range of tasks that include both fixed-format and freeform response requirements. Additionally, the paper should discuss any limitations of the proposed methods, such as potential overfitting to a specific data structure or benchmark, to provide a balanced and transparent evaluation of the approach.\n\n[1] Yu, Weihao, et al. \"Mm-vet: Evaluating large multimodal models for integrated capabilities.\" arXiv preprint arXiv:2308.02490 (2023).\n\n[2] Yao, Zhewei, et al. \"DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention.\" arXiv preprint arXiv:2309.14327 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Jm49THxs0", "reviewer": "Reviewer_iVLP", "summary": "The paper explores a pioneering context schema, designed to bolster the VLM's proficiency in effectively handling multimodal inputs and facilitating in-context learning. This method mainly aims to construct the Text-Image Interleaved Dataset (MIC dataset) with the objective of enhancing the VLM's in-context learning capabilities. There are three distinct formats in the MIC dataset, including single image format, multiple interconnected images format, and in-context format, which respectively are tailored for different instances of various vision-language tasks. In experiments, the authors demonstrate that the proposed method (MMICL) outperforms baselines on a wide range of general vision-language tasks, especially on MME and MMBench benchmarks. Moreover, MMICL also shows in-context learning ability on some traditional vision-language tasks in the few-shot setting.", "questions": "+ In Table 4, I observe that MMICL with different backbones exhibits significant performance differences in certain tasks. For instance, the performance gap between MMICL (Instruct-FLAN-T5-XL) and MMICL (Instruct-FLAN-T5-XXL) on Flickr 30k is as large as 34.6. A similar trend can be seen in the VizWiz dataset, with InstructBLIP (Flan-XL) scoring 32.08, while InstructBLIP (Flan-XXL) drops to 15.11. This phenomenon is absent in InstructBLIP. What could be the reason for this? \n+ In Table 4, MMICL (FLAN-T5-XL) shows an improvement of approximately 25.13 in the 4-shot setting compared to 0-shot, whereas MMICL (FLAN-T5-XXL) only improves by 3.82. It's perplexing that increasing the backbone's size leads to a significant decline in few-shot performance. What could possibly be the underlying reason for this?\n+ How many visual tokens does each image take up? Does the input of multiple images take up a lot of LLM's input context length?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "+ This paper offers a novel perspective on how to construct image-text interleaved instruction datasets. It highlights how VLM (Vision-Language Model) can simultaneously acquire in-context learning ability during the visual instruction tuning process.\n+ The MIC dataset proposed by this paper covers a wide range of vision-language tasks, including Image Captioning, Video Caption, VQA, VideoQA, Visual Reasoning, Visual Dialogue, and Image Classification.\n+ The experiments conducted on a wide range of general vision-language tasks are very comprehensive. The average results on the MME and MMBench benchmarks demonstrate the impressive zero-shot performance of MMICL compared with other baselines.", "weaknesses": "+ The few-shot performance improvement of MMICL appears to be somewhat limited. For instance, in Table 4, except for the VizWiz dataset, in some cases (e.g., FLAN-T5-XXL and Instruct-FLAN-T5-XL), MMICL does not demonstrate significant improvements on other test datasets when adding in-context examples.\n+ The lack of exploration of various interleaved image-text data formats for VLM's in-context learning ability is evident. It is unclear whether only the in-context format (equation 3) can enhance in-context learning ability or if the multiple interconnected images format (equation 2) can also play a certain role.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VE3JLbWSZB", "reviewer": "Reviewer_d1dY", "summary": "The paper introduces a context learning scheme for multimodal large language models and constructs a instruction dataset. It demonstrates the model's comprehension ability when dealing with complex multi-modal prompts and uncovers its potential for in-context learning. Overall, this work leans more towards a technical report, but it is comprehensive.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Compared to open-/flamingo, kosmos, the model can more easily handle multiple image inputs and their relationships.\n\n2. Through prompt engineering, it unifies more tasks.\n\n3. More instruction tuning data has been obtained through chatgpt.", "weaknesses": "The synthetic dataset, training paradigms, and model structure have not undergone an ablation study, so the specific gain is unclear.\n\nThis work has expanded further on the data level compared to works such as InstrutBlip and LLaVA, and has achieved better results, which I am not surprised by. However, its design is slightly complex, and it contains a lot of prompt engineering. I believe this work is above the borderline, but it may not be classified as a 'good' paper here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gpo1pqAAjm", "reviewer": "Reviewer_QUmJ", "summary": "The author proposed a new kind of in-context learning, called multi-modal in-context learning, to deal with multi-modal inputs. To train model, the author proposed a novel context scheme and constructed the Multi-modal In-Context Learning (MIC) dataset. Experiments are conducted to show the advantages on several well known datasets.", "questions": "See the above section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The targeting problem is interesting.\n\n2. Improvements are achieved on several well-known datasets.\n\n3. Extensive experimental results are provided.", "weaknesses": "1. It seems that the explicit meaning of MMICL is not mentioned in this paper. Some explanations might be helpful.\n\n2. The experiments are conducted on T5 family models, which are encoder-decoder architecture models. Why not use the decoder-only models?\n\n3. Model sizes should be included in some tables. Otherwise, it is not easy to compare the competing models.\n\n4. It seems that bigger model does not always have better performance. Is there any explanations for that?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GUfks2Zefp", "reviewer": "Reviewer_eWon", "summary": "The paper introduces an interleaved image-text dataset built upon public VQA/captioning/Reasoning datasets such as Flickr/VQAv2/MSRVTT. It proposes several schemes to convert existing datasets into interleaved-image-text formats. The proposed dataset is used to train a VQA model that can take multiple images as input to answer questions (following user instructions). It shows state-of-the-art results on recent VQA benchmarks such as MME/MMBench, and demonstrated ability to improve downstream task performance using few-shot image-text samples. However, the writing of this paper is extremely bad, consisting of numerous typos, grammatical mistakes, and sentences that are difficult to understand. There are lots of missing details regarding dataset curation and evaluation protocols.", "questions": "I think the current writing really hurts the paper. In addition to improve the writing quality, it is unclear what the key innovation / scientific insight is. The authors should ablate the use of training data in order to highlight the advantage of including interleaved image-text samples, which seems extremely costly to obtain.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The proposed dataset with 5.8M samples could be useful for the multimodal community. The proposed MMICL model demonstrates top-tier performance on recent VQA benchmarks (MME/MMBench) compared to earlier models such as LLaVA and MiniGPT-4.", "weaknesses": "First, I find it **extremely hard to follow the methodology** in this paper. For example, as dataset curation is one of the most important contributions of this work, I am not able to understand what efforts the authors put in order to gather a massive 5.8M interleaved-image-text datasets. For instance, Could you explain how you manage to collect the sample as shown in Figure 3(b) and 4? In these two figures, it is shown two people quarreling with each other; this requires the original dataset to not just have bounding boxes for person1 and person2, but also the action relation (quarreling) between these two bounding boxes. I would appreciate if the authors could point us to the actual dataset used to gather this particular example, and explain if extra (and perhaps costly) human annotation is required to get such samples.\n\nThe authors should also provide more visual examples from each dataset and more detailed description on how you convert them into instruction-following formats. For example, the current Section 2.3 is very hard to understand. It says *“Next, we employ ChatGPT to rewrite the instructions to describe the key characteristics of each task accurately”* — what does it mean by *“key characteristics of each task”* and how did ChatGPT come up with them?\n\nThere are **lots of missing details** regarding model evaluation.  \n\n- For one example, in Table 4, how did the authors select the few-shot samples? Will different few-shot samples affect the performance? No variance/std is provided in Table4.\n\n- How did you manually split ScienceQA-IMG into two groups that require images or not? Did you hire human annotators for this task? What rationales did the annotators employ?\n\n- Table 8 shows that Flickr/VQAv2 are used in your training set. Then how could Table 4 claim that your model achieves good “zero-shot” and “few-shot” performance on Flickr/VQAv2?\n\n- Table 8 attempted to show the licenses for all training datasets, but almost half are labeled with “Unknown” license. This is counter-intuitive because some widely used datasets such as Flickr have well-documented licenses listed on their websites. \n\n- Do you use the same datasets during both stage-1 and stage-2 pre-training?\n\n- As you show on Table 8, Winoground only has 800 samples. How could there be an image score of 44.99?\n\nFinally, it is unclear if the performance boost really comes from having more interleaved-image-text data, because the current paper does not perform ablation on the selection of training data.\n\nThe paper has way too many typos. To list a few:\n- Figure 1: missing a white space “MMICLtypically”.\n- Page 2: fall should be fail? “their context schemes fall to”\n- Page 3: missing a period “multiple images In Fig. 2.b”\n- Figure 3: missing a period “unified format”\n- Page 4: missing a white space “MMICLin”\n- Page 8: hinds or hints “provide hinds”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)