What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration
Authors: Libo Qin
Qiguang Chen
Hao Fei
Zhi Chen
Min Li
Wanxiang Che
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-modal In-Context Learning
Demonstration Retrieval
Demonstration Ordering
In-Context Learning
Primary area: natural_language_processing

Abstract: Recently, rapid advancements in Multi-Modal In-Context Learning (MM-ICL) have achieved notable success, which is capable of achieving superior performance across various tasks without requiring additional parameter tuning. However, the underlying rules for the effectiveness of MM-ICL remain under-explored. To fill this gap, this work aims to investigate the research question: "_What factors affect the performance of MM-ICL?_" To this end, we investigate extensive experiments on the three core steps of MM-ICL including demonstration retrieval, demonstration ordering, and prompt construction using 6 vision large language models and 20 strategies. Our findings highlight (1) the necessity of a multi-modal retriever for demonstration retrieval, (2) the importance of intra-demonstration ordering over inter-demonstration ordering, and (3) the enhancement of task comprehension through introductory instructions in prompts. We hope this study can serve as a foundational guide for optimizing MM-ICL strategies in future research.

What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration
Libo Qin Qiguang Chen Hao Fei
Zhi Chen Min Li Wanxiang Che
School of Computer Science and Engineering, Central South University
t Research Center for Social Computing and Information Retrieval
Harbin Institute of Technology o Tsinghua University * ByteDance lbqin@csu. edu. cn, ggchen@ir.. www..eu..cc... cn
Abstract
Recently, rapid advancements in Multi-Modal In-Context Learning (MM-ICL) have achieved notable success, which is capable of achieving superior performance across various tasks without requiring additional parameter tuning. However, the underlying rules for the effectiveness this gap, this work aims investigate the research question: "What factors affect the performance of MM-ICL," To this end, we investigate extensive experiments on the three core steps of MM-ICL including demonstration retrieval, demonstration ordering, and prompt construction using 6 vision large language models and 20 strategies. Our findings highlight (1) the necessity of multi-modal retriever for demonstration retrieval, (2) the importance of intra-demone denrraddmmttttttion ordering over interddemonstrrtion ordering, and (3) the enhancement of task comprehension through introductory instructions in prompts. We hope this study can serve as a foundational guide for optimizing MM-ICL. strategies future research.
MM-ICL remain under-explored. To fill
Introduction
Recently, Large Language Models (LLMs) have demonstrated remarkable advancements, showcasing proficiency in wide range of tasks [Zhao et al.. 2023a, Qin et al., 2023, 2024, Hu et al., 2023, Pan et al., 2023]. Notably, advanced LLMS exhibit the emergence of novel capabilities such as In-Context Learning (ICL) [Wei et al., 2022a, Dong et al., 2022, Zhuang et al., 2023], which optimize task performance by incorporating demonstrations into input prompts [Giannou et al., 2023, Li et al., 2023d, Wies et al., 2023, Zhou et al., 2022]. In partieular, multi-modal in-contextlleaming (MM-ICL) capable utilizing multi-modas demonstrations to quickly adapt to the downstream task without parameter tuning [Yin et al., 2023, He al., 2023, Zhang al., 2024, Li and Lu, 2024].
In the literature, series works emerge to enhance MM-ICL. Specifically, Gong et al. [2023] man- ually create general template with multiple images and corresponding responses during instruction- tuning (IT) stage to improve MM-ICL. Tsimpoukelli et al. [2021], Li et al. [2023b], Doveh et al. [2024] and Zhao et al. [2024] develop task-specific MM-ICL templates during the IT stage, further extending its capabilities across more domains. Li et al. [2023a] introduce OtterHD, adapting MM- ICL for sighddffnition image tasks. Furthermore, Sun al. [2023] and Tian al. [2024] explore the potential of MM-ICL the image generation tasks. Jin et al. [2024] provide compelling evidence for the effectiveness of MM-ICL in comprehending game instructions. Zong et al. [2024] and Shukor et al. [2024] develop fine-grained benchmarks and evaluate the MM-ICL in classification tasks.
While significant progress has been witnessed in MM-ICI, the existing work still mainly focuses on how to optimize MM-ICI, ignoring the underlying factors that influence its effectiveness and
Equal Contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Multimodal Validation Set (V&)
Demonstration Set (C2)
Demonstration Ordered List (Ex)
Prompt (P)
Demonstration Retrieval
Demonstration Ordering
Prompt Construction
Instruction-
Samplexx
Sample: *s,
Sample: Sxg
Sample: Ax,
Samplexxa
Sample: Xeg
Sample: *x;
Semple: Xx,
Sample: Xx,
Sampleucy
Sample: Ax,
Sample:
Figure 1: The whole process of prompting creation for multi-mone in-contett-leaningg
performance. Such gap impedes comprehensive understanding of the mechanisms and performance determinants of MM-ICL, thereby limiting further exploration and research in this field. Motivated by this, this paper aims systematically investigate the research question: What factors affect the performance of MM-ICL?, hoping to offer a unified view and guideline for researchers to build better MM-ICL. Specifically, as illustrated in Figure 1, the MM-ICL process comprises three steps: demonstration retrieval, demonstration ordering, and prompt construction. Therefore, We systematically investigate the following sub-questions: (a) how select multi-mode demonstrations (Sec. 3.1); (b) how to order multi-moda demonstrations (Sec. 3.2); and (c) how to construct MM-ICL prompts (Sec. 3.3) to this end. To achieve this, we conduct detailed experiments on MM-ICL using 20 strategies across 4 tasks with 6 representative vision large language models (VLLMs).
Through extensive investigations, the main findings
follows:
Multi-modal alignment is the bottleneck for MM-ICL. Our analysis confirms that, on average, multi-modal retrieval methods outperform single-modal ones. Furthermore, multi- modal alignment in VLLMs has greater impact on MM-ICL effectiveness than parameter size, identifying alignment as the key limitation in both backbone structure and demonstration quality.
Intra-demonstration ordering holds greater importance than inter-demonstration ordering. Our investigation first indicates that the intra-demonstration ordering, particularly the ordering of modalities, greatly influences model performance more than demonstration arrangement.
Introductory instruction guides better task understanding for MM-ICL. To construct comprehensive MM-ICL prompt, it essential to include introductory instructions preceding the demonstrations. This approach consistently enhances the performance of MM-ICL campared with summative instruction placed after demonstrations, and intra-demonstration instruction.
2 Background
In this work, we formally present the prompt building process for MM-ICL. As depicted in Figure 1, the process of prompt building for MM-ICL involves three sequential stages:
(1) Demonstration Retrieval:
The core MM-ICL requires retrieval obtain demonstrations that can help MM-ICL. Formally, given validation dataset V,, {I1, 22,,.. ................................ Jn each multi-modal
sample 2; includes textual input 1art, visual input 1. and output O,, For a specific test query q.
this step aims to identify subset of relevant demonstrations C₁ {xx,, 1 where Va5 E Vn-
(2) Demonstration Ordering: Researches [Iuu et al., 2022b, Wu et al., 2023, Xiang et al., 2024] show that LLMs are highly sensitive the order of demonstrations, Thus, arranging these demonstrations effectively is crucial for MM-ICL. After retrieving relevant demonstrations, we must rearrange the sequence Lu | which will be used to construct the prompt.
(3) Prompt Construction:
Previous research indicates that using delimiters and instructions can significantly enhance textual ICL capabilities [Min et al., 2022, Qin al., 2023]. Therefore, the final core step is to transform the ordered demonstrations into structured prompt P, incorporating delimiters and instructions optimize MM-ICL.
3
What Factors Affect Multi-modal In-Context Learning?
3.1 Exploration of MM-ICL Demonstration Retrieval
The efficacy of ICL heavily depends on the quality the retrieved demonstrations C, which provide essential prior knowledge for MM-ICL. As illustrated in Figure 2, the retrieval process encompasses

---

Au
Samplesh
Sampleshy
She
Samplesh,
Samplesky
Query
Sampleck,
Sampleshy
Textual Encoder
Visual Encoder
Multi-modal Encoder
Query:
Sample:
Sample: Xx
Query: y
Sample: x
Sample: Xx
Query:
Sample: x
Sample: Xx
(a) Sample Representation
Sampla:2:
Samples2ar
Samples2:
SamplesQa
Samples2
Samples2
Cosine Distance
L2 Distance
Semantic Coverage
Quarysky
Sample:
Sample: fx
Querysha
Sample: ig
Sample: Rx
Query.Ah
Sample: h
Sample: Bag
(b) Sample Comparison
Sampleuxys
aa
Samplexees
Samplex
Samplexees
Samplecs.
Domain Selection
Image
Style Selection
Modality Distance
Query:
Sample::
Samples@
Query:
Sampleg
Sample:Q.v
Query
Sample
Sample.2
(c) Sample Selection
Figure 2: The demonstration retrieval process for MM-ICL.
three key steps: (1) Sample Representation, (2) Sample Comparison, and (3) Sample Selection. In this section, we conduct systematic analysis how various strategies for sample representation, comparison, and selection affect MM-ICL task performance.
Sample Representation. It involves defining encoder (Encoder(-)) map each input sample Tj E V and user query g into a shared representation space:
he = Encoder(zz).
Specifically, we evaluate various encoder architectures modalities, focusing on the impact of visual encoder (Encoder,is), text encoder (Encoder:zt), and multi-modal encoder SEncodermuttl on model performance.
Sample Comparison. After deriving the representations, we employ a metric M to evaluate the quality Q, of the sample h2 in comparison to the query representation hg and the dataset samples hy:
2 MM h2))
Specifically, we explore various comparison metries, including cosine similarity Moge [Liu et al., 2022a], L2 similarity Mr2 [Liu et al., 2022a], and semantic diversity Maio [Li and Qiu, 2023a], to assess sample quality and understand the correlation with model performance.
Sample Selection. After quality assessments, we apply a selection criterion S to identify the most advantageous samples ,,, for inclusion in the demonstration set
C= {May [ay S 2,),5 S k).
Sample selection is guided by factors such as domain information [He al., 2023], demonstration style [Agrawal et al., 2023], and token distance [Liu et al., 2022a]. Specifically, we systematically examine samples from both in-domain and out-of-domain collections. And we also assess the impact of image style on the selected demonstrations. Further, we investigate the token distance between modalities to understand its effects on sample selection for MM-ICL.
3.2 Exploration of MM-ICL Demonstration Ordering
Following Lu et al. [2022b] and Wu et al. [2023], the order of the demonstration set significantly impacts MM-ICL performance. As shown in Figure 3, this section explores two key aspects:
Intra-demonstration Ordering. The sequence within a demonstration, especially modalities (e.g.. text and image), is an important component that might affect the MM-ICL capabilities. Therefore,

---

Demonstration Ordered List
Textual Imput: pnat
Visual imputs/
Textual output: O
Sample: S
Samplexx,1
Samplex
Samplexx,3
dimosriimnnnnoation Ordering
Inter cnter-demonstraatin Ordering
Visual input./"'s
+ Textual input./"
① Textual output:
Samplexx
Samplexx,
O
Samplexx
Figure 3: The demonstration ordering process for MM-ICL.
we introduce intra-demonstrraioo ordering permutation (IOP) define this sequence:
L [IOP (z=,): IOP (00)) I0P(xx.))
We conduct systematic exploration of various IOP configurations, including rext-image-tee (IOPtre), text-text-image (IOP*t)), and image-text-text (IOPRt)). These order analyses aim to evaluate the impact of different modal sequences the model's performance.
demonstemonttoation Ordering. The sequence in which demonstrations are organized within also is the key component that might impact the performance of MM-ICI. Formally, we define sample ordering permutation o to specify the arrangement:
L = [To Toz..... Zox ITo, C),
(5)
where Tay represents the j-th demonstration in the ordered demonstration list.
3.3 Exploration of MM-ICL Prompt Construction
VLLMs are highly sensitive to input instructions [Kojima et al., 2022, Qin et al., 2023]. Inspired by this, to enhance task comprehension,
incorporate different instructions explore the performance influence for MM-ICL. Formally, we construct instruction methods I(.) that describe the task and position them within the prompt. The prompt construction process is:
P [(6(2.)) ), 6(zo.), 6(xaz)
(6)
Specifically, as shown in Figure 4, we explore three instruction categories to bolster MM-ICI, process:
Introductory Instruction (I,ntro) refers to the initial guidance that offers an overview of the task prior to any demonstrations. As shown in Figure 4 (a), this instruction, denoted as Tintros is positioned the start of the ordered demonstration sequence, L.
(a) Introductory Instruction
(h) Summative Instruction
Instruction: I
Ordered List: C
Ordered List: &
Instruction: Z
Instruction Injection
Instruction Injection
Ordered List: &
Ordered List: &
(c) Intra- Instruction
Ordered
Input: 1
Instruction: I
Output: 0
Input: 12
Instruction: 7
Output: Og
L isst (C)
Instruction Injection
Ordered List (C)
Input: 4
Output: 0,
Input: 1
Output: Oz
Figure 4: The process of instruction injection for MM-ICL prompt construction involves three key elements. The Introductory Instruction provides an overview instruction of the task before demonstrations. The Summative Instruction summarizes after the examples, guiding the model to apply the learned concepts to practical problems. The Intra-demonstration Instruction embeds task-specific guidance within each demonstration, enabling VLLMMs to grasp task requirements during learning. Further details and additional prompts are provided in Appendix C.3.

---

Summative Instruction (Zeum) offers a summary after the examples, guiding the model to apply the learned concepts real-world problems. As shown in Figure (b), this instruction Z is added at the end of the demonstration list L.
Intra-demonstratioo Instruction (Inntra) embeds task instructions within each demonstration, helping VLLMs understand the task requirements during the learning process. As shown in Figure 4 (c), this instruction I is included within each demonstration 2; in the list L.
4 Experimental Setup
Following the setting of Li et al. [2023c], we systematically explore tasks, including image-caption. visual question answering (VQA), image classification, and chain-of-thought reasoning, which come from M [Li et al., 2023c] and M [Chen et al., 2024b] (as shown in Tables 2), providing a universal paradigm can help researchers conduct unified and fairer comparisons and studies within unified framework. In order to evaluate the MM-ICL performance accurately, we use two indicators for each task. Following Zhang al. [2019], Li et al. [2023b], and Zong et al. [2024], we CIDER [Vedantam et al., 2015] and BertScore [Zhang et al., 2019] as image-caption metrics. Since M includes various VQA tasks with free-form answers, inspired by the success of free-form and precise answer hybrid evaluation in machine reading comprehension, following Rajpurkar et al. [2016], Zhang al. [2019], we adapt Token-F1 [Rajpurkar al., 2016] and BertScore as visual question answering (VQA) metrics (The correlation analysis of the indicators and accuracy as shown Table 3). Following Li et al. [2023c.b]. we use accuracy and F1 score as indicators of image classification. Following Lu et al. [2022a], Golovneva et al. [2022] and Qin et al. [2023], we use accuracy and reasoning alignment score [Golovneva et al., 2022] (RAS) as indicators of reasoning.
To ensure rigorous experimental control, we established a baseline using a multi-modal encoder for data representation and cosine similarity for sample comparison, limiting retrieval to within the same task. This baseline ranks samples based on similarity, with a delimiter and a 3-shot setting (see Appendix A for details). In addition, all open source models complete inference A100 80G. For all experiments, we select top-p from (0.95, } and adjust the temperature parameter within [0, 1]. Among them, temperature is the main error variable in this work.
5 Empirical Analysis of Factors Affecting MM-ICL
5.1 Empirical Analysis of MM-ICL Demonstration Retrieval
5.1.1 Sample Representation
Multi-modal alignment is the bottleneck for MM-ICL in both backbones and demonstrations. To evaluate the impact of semantic representation in different modalities for MM-ICL, assessed three distinct encoders: RoBERTa [Liu et al., 2019] as a textual encoder for Textual Retriever, CLIP-Vision Encoder [Radford et al., 2021] for Visual Retriever, and BridgeTower [Xu et al., 2023] as multi-modal encoder for Multi -Modal Retriever. As illustrated in Table 1, multi-modal retrieval consistently outperforms zero-shot, randomly selected, and single-modality methods, high- lighting the advantages of multi-modal semantic learning for MM-ICL. What's 's more, as shown in Table 1, our results reveal that increasing model parameters from 8 billion to over 100 billion does not significantly enhance performance, suggesting that beyond parameter size, multi-modal context under- standing and alignment are more crucial for MM-ICL than model scaling. Our analysis demonstrates that multi-modal alignment is the critical factor in both the backbone and demonstrations.
Current multi-modal encoders still lack modeling of multi-modal logic. Actually, multi-modal retrieval attains better performance in many scenarios like Image Caption and VQA. However, our experiments show that textual retrieval works well for classification and reasoning tasks. Based on the qualitative analysis, we observe that due to the semantic richness of the labels and rationales, textual retrieval can obtain more similar samples. However, the current multi-modal retrieval struggles with complex text semantics, often favoring image similarity. This aligns with recent work [Tong et al., 2023, 2024, Fei al., 2024c], which valuable for future exploration.
Multi-modal context diminishes the necessity careful demonstration selection. As shown in Table 1, adding relevant demonstrations slightly improves performance, but the gains less
5

---

Caption
VQA
Classification
Reasoning
AVG
CIDER
BERTScore
BERTScore
Acc
OpenFlaminge (9B) [Awadalla 2023]
Zero-shot
1.84
81.18
2.78
76.17
15.17
3.63
16.53
85.13
35.30
Fez- (Randon)
8,23
56.63
12.63
67.37
13.11
5.11
21.35
86.53
33.87
Textual Retriever
13.39
74.22
21.80
13.67
12.04
25.63
87.71
40.52
Visual Retriever
6.33
53.88
12.82
68.76
13.67
10.87
23.10
87.36
34.60
Kulti-Modal Retriever
13.47
85.01
7.85
79.05
19.66
10.10
24.96
88.11
41.03
Otter (91) [L. et al., 2023b]
Zero-ahot
2.86
86.42
20,90
24.34
10.85
82.67
43.76
Feu-shot (Randon)
3.50
86.62
20.95
87.76
25.66
10.28
34.23
83.67
44.08
Textual Retriever
3,89
86.62
20.40
87.89
25. 25.28
8.47
32.88
81.93
43.42
Retriever
3.50
86.51
12.44
32.21
43.97
Nulti Modil Retriever
3.77
86.57
18.80
87.56
28.65
11.83
35.92
83.74
44.60
Qwen- VI. (TOB) [Bai et al., 20233
Zero shot
13.57
87.65
24.96
85.09
50.19
54,28
48.40
90.87
$6.87
Fes-shot (Randon)
28.52
53.50
Textual Retriever
21.58
88.07
26.
85.62
49.44
53,04
46.04
90.72
57.69
+ Visual Retriever
30.81
88.56
28.79
86.23
59.74
54,43
46.88
91.30
60.84
41.51
89.03
30.20
86.78
59.36
53.17
46.21
62.22
GPT4V (>100B) [OpenAl: 2023]
Zero shot
5.15
85.43
20.01
84.77
61.42
59.07
54.64
92.46
57.87
Few-shot (Randon)
6.37
85.95
24.43
85.42
60.11
60.81
54.30
92.54
58.74
Textual
86.02
31.81
87.02
62.55
$1.40
55,99
59.57
Vieual Retriever
9,36
86.26
32.47
86.96
63.30
57.79
59.87
93.19
61.15
Kulti-Modal Retriever
16.55
86.77
32,92
86.87
62.55
59.97
60.88
93. 10
62.45
IDEFICS2 (8B) [Laurençon et al., 2024b]
88.59
26.88
Few-shot (Randon)
39.68
88,88
30.82
87.59
61.61
53.39
51.94
89.52
62.93
+ Textual Retriever
35.95
88.45
31.66
87.58
61.99
67.13
45.03
88.98
63.34
46.61
89.55
32,05
87.92
64.04
62.51
51.26
89.83
65.47
Multi-Modal Retriever
52.55
89.66
33.65
88.17
65.54
63.86
51.43
89.57
66.80
Gemini-Pro (> 100B) [Google, 2023]
Zero-shot
14.05
87.07
26.93
85.78
68.20
66.10
55.14 14
90.72
61.75
Feu-shot (Randon)
21.21
88.13
32,99
86.81
63.67
69.75
55.65
63.75
Textual Retriever
15.79
87.75
34.96
87.18
69.10
72.31
53.29
91.57
63.99
Visual Retriever
21.35
87.98
44.74
89.33
65.73
64.18
53.29
91.92
64.81
Multi-Modal Retriever
35.64
88.67
45.47
89.61
65.17
70.51
58.01
92.17
68.16
Table 1: Performance comparison of retrievers utilizing different modal representations, where Few-shot (Random) refers to MM-ICL methods in which the demonstrations are randomly selected from the development set.
significant compared to text-only ICL scenarios. Specifically, retrieved demonstrations yield an average performance boost of 3.84%, compared to random demonstrations. In contrast, text-only scenarios show performance increases of over 10% with carefully selected samples [Shi et al., 2023]. Furthermore, the model remains unaffected by irrelevant samples, and the performance of almost all models is higher than zero-shot. This indicates that multi-modal context significantly reduces the need for careful demonstration selection, unlike in text-only scenarios.
VLLMs learn semantic representations instead of token pattern representations for MM-ICL. As depicted by Agrawal al. [2023], textual ICL primarily learns token patterns (e.g., similar output formats, reasoning paths) among demonstration outputs. To investigate whether VLLMs rely on repetitive token patterns, utilize the average BLEU score across demonstration outputs as representation of token repetition. Figure shows that only the image captioning task exhibits a positive correlation. In contrast, other tasks show a decline BLEU scores exceed 30%. This underscores that MM-ICL primarily learns semantic rather than token pattern representations for effective performance.
5.1.2 Sample Comparison
To further analyze the influencing factors of MM-ICL sample retrieval, this study employs similarity and diversity metrics, which help assess how MM-ICL processes sample similarities and differences, enhancing understanding of its mechanisms. See Appendix B for more details and results.

---

Distance
Diversity Retriever
Retriever
Capliun Reasoning
130
CIDER
CIDER
Performance (%%
RAS
BERTScore
100
80
Acc.
60
40
BERTSOore
BERTSOore
20
Accunicy
Accunwy
40
(AVG: 53.9 68 16)
(AVG: 66.5
68L 16)
BLEU
[8] impact of different similarit
(b) impact utilization diversity metrics.
metrics.
Figure 5: The impact of token pattern representation in Gemini-Pro.
Figure 6: The impact of different sample comparison methodologies in Gemini-Pro.
Cosine similarity matters for sample comparison. Following Liu et al. [2022b], we compare two representative similarity metries, cosine similarity and L2 similarity. As shown in Figure 6 (a), cosine similarity, which measures the directional semantic alignment, emerges as the superior metric in MM-ICL than L2 similarity. Supported by Deza al. [2009] and Steck et [2024], it indicates that MM-ICL prioritizes semantic directional consistency over complete semantic aligmment.
Diversity does not show significant influence for sample comparison. He et al. [2023), Li and Qiu [2023b] have shown that demonstrations with better diversity can effectively improve textual ICL. To explore whether it exists in MM-ICL, following Li and Qiu [2023b], we ultilize the "diversity retriever", which selects the top-10 samples and further chooses the best 3 samples based on semantic diversity to obtain more diverse MM-ICL. As demonstrated Figure (b), although diversity significantly enhances performance in text-based ICL, our experiments show limited improvement in MM-ICL tasks. This suggests that diversity may not directly correlate with better MM-ICL.
5.1.3 Sample Selection
Domain interval matters for sample selection. Prior research highlights the critical role of domain relevance in enhancing ICI. performance. Inspired by this, we employ the multi-modal retriever to select samples from both in-domain and out-of-domain pools. Figure (a) shows nearly 4% performance drop when out-of-domain demonstrations are included, underscoring the necessity of in-domain demonstrations for optimal MM-ICL.
crucial factor in sample selection. Although stylistic similarity in text samples is known to bolster ICL, its effect on the visual modality remains ambiguous. Utilizing CLIP for image classification,
Visual style
investigate the impact of stylistic coherence in multi-modal samples on MM-ICI. performance. As depicted in Figure (b). significant enhancements are observed solely in VQA task, while captioning and classification show minimal effects and reasoning tasks decline. This indicates that diverse visual styles are not crucial in general MM-ICI..
Token distances between modalities need be considered for different tasks to improve sample selection. For textual ICL, excessive token distance between samples can impede performance [Liu
Classification
Reasoming
VQV
Caption
CIDER
80
Performance (9))
RAS
BERTScore
70
80
50
Acc.
Token Fi
40
30
20
BERTScore
Accuracy
20
500
540
(AVG: 6LG9 16)
The number of images with same type (b) The impact of visual style sample selection on perfomance.
The number of Textual Tokens
(a) The impact whether the sample in-domain on performance.
(c) The impact of token distance between demonstrotions in sample solection.
Figure 7: The impact of sample selection average score performance in Gemini- Prro.

---

et al., 2022a]. We extend this inquiry to MM-ICL, analyzing how token distance across modalities influences results. Specifically, during the sample selection process, we considered the impact of the average token distance between two images on the model within the entire prompt of MM-ICI.. As illustrated Figure (c), the effect of token distance varies by task, typically showing an initial performance increase followed by decline as distance grows, particularly in non-captioning tasks. This highlights the task-dependent nature of optimal token distance in MM-ICL.
5.2 Empirical Analysis of MM-ICL Demonstration Ordering
detonstemonttration ordering significantly impacts performance. Within the demonstration, organizing the ordering, especially the relationship between modalities is a crucial topic. We investigate this by arranging inputs and outputs across modalities using three methods: text input=>text output-timage input (Text-Image), text input input->text outpur (Text-Image- Text), and image -teu input->tex output (Image- Text). As shown in Figure 8 (a), positioning the image at the start significantly enhances model performance. This suggests that presenting visual information first improves multi-modal comprehension, thereby boosting its learning abilities.
Inter-demonstration ordering demonstrates minimal impacts. Following Lu et al. [2022c], investigate how the order of demonstration presentation influences model efficacy. We explore various strategies: random rearrangement. a "similar-last" approach where samples similar the query are shown last, and "similar-firy approach where similar samples are presented first. Figure 8 (b) illustrates that inter-demonstration ordering has a negligible impact on MM-ICL performance. This suggests the order-robustness, with the presentation sequence having minimal effect.
Image-Text
Text-image
Text Test-Image-Ten
Best Performance
70
66.7
66.5
6a.2 60.4 59.2
60
549 56.7
44.6
40
36.0
36.0
Gemini Pro
Qwen-VL (a) The effect of the order of modals within demonstration average performance
IDEFICS2
OpenFlamingo
Similar-Pirst Similar-Last O Random @ Best Performance
08.2
67.6
70
6144 61.5
60
50
44.6 45.0 44-3
37-7
38.1
30
Gemini Pro
Qwen-VL (b) The effect of the order of demonstrations on average performance
IDEFICS2
Otter
OpenFlamingo
Figure 8: The impact of demonstration ordering performance.
5.3 Empirical Analysis of MM-ICL Prompt Construction
Introductory Instruction is consistently effective for better MM-ICL. To investigate the impact of inserting task-related instructions within prompts, we conduct the following experiment on three categories of instruction: Introductory Instruction, Sunmative Instruction, and Intru-demonstration Instruction. As depicted Figure 9, our analysis indicates that introductory instructions stably
No-Instruction O Introductory Instruction O Summative Instruction Intra-demoosstration Instruction O Best Performance
70
68.5
68.4
60.8
66.2 64,3
62.2 63.0
60
50
44.6
44.0
435
41.0
435
40
30
Gemini Pro
IDEFICS2
Qwen-VL
Otter
OpenFlaminge
Figure 9: The impact of injecting instruction into demonstrations model average score performance.

---

Centel
w/o Delimiter
w/ Deliiter
velo Deliciter
w/ Deliniter
Rreusing
CIDER
CIDER
RAS
NorTSem
RAS
BERTScore
Token
500
50
40
49
TIKT Scare
Accuracy
Accuracy
(AVG: 64.76 08.10)
(AVG: (3% 19- 06,00)
numbbrr derustatrattone
The sumberr demoratrration
(e) Gemini
(b) IDEFICS2
(a) The impoct of the number demon- strrions avarage score performance.
(b) The impoct of number of demonn-
strations on different task performames.
The impact of inserting delimiter into the input and output demonstration on model performance. See Figure 16 for more results.
Figure 11:
Figure 10: The impact of the number of demon- strations on performance.
enhance model performance. In contrast, other instructions generally decrease performance. This finding suggests that introductory instructions facilitate targeted contextual learning and more effective semantic comprehension in demonstrations, We show more prompts and details in Appendix C.3.
MM-ICL affected by the number of demonstrations depending on the task. Contrary to traditional text-based ICL, where performance improves with more samples, findings in Figure 10 (a) suggest that MM-ICL does not experience significant gains from more demonstrations. To further understand the reason behind, we analysis the performance different tasks. As shown in Figure 10 (b), increasing the number demonstrations enhances performance in caption and VQA tasks, a trend also reported in prior studies [Alayrac al., 2022, Laurençon al., 2024a, Shukor et al., 2024]. However, performance declines when demonstrations exceed three across all tested VL.MM. In more complex reasoning tasks, such as multi-step multi-modal chain-of-thouge reasoning, additional demonstrations do not yield effective improvements, aligning with the findings of Chen et al. [2024b]. and Fei et al. [2024a].
Moreover, we attribute it to the following reasons for this limitation: (1) Cognitive Overload: For complex tasks, understanding numerous demonstrations can overwhelm the model, impeding its ability to process and integrate information effectively [Chen et al., 2024a]. (1) Complexity of Reasoning Tasks: In reasoning tasks, the performance improvement from more demonstrations is often less pronounced than when using diverse retrievers. This suggests that reasoning tasks require sophisticated integration of information, where quality outweighs quantity. See Appendix C. for more detailed description.
The importance of delimiter lessens by text-image interleaved demonstrations. Previous re- search suggests that specific delimiters for input and output data can demonstrably influence textual ICL capabilities [Min al., 2022]. Therefore, to examine their necessity (see Appendix C.2 for details). As shown in Figure 11, the resulting minor performance decline suggests that while these delimiters are less critical in MM-ICL, the modality switch inherent to MM-ICI. may serve as an implicit delimiter, compensating for the absence of explicit delimiters.
utilize ablation experiments to omit these delimiters
6 Related Work
Recent advancements in vision large language models (VLLMs) have achieved great success in various vision-language tasks [Yin al., 2023. Wu et al., 2024a,b, Wang al., 2024, Fei al., 2024b]. Initially, VLLMs lack Multi-modal In-context Learning (MM-ICL) capabilities. To address this, researchers explore incorporating MM-ICL directly into the training phase. This involves constructing training samples with multi-modal interleaved data by manual and general templates, which unlock the MM-ICL capability [Alayrac et al., 2022, Awadalla al., 2023]. Building on this, Li et al. [2023b], Doveh et al. [2024] and Zhao et al. [2024] extend the MM-ICL to construct a series of task-specific templates, which improves generalization for MM-ICL. Further, Li al. [2023a] introduce OtterH-D and adapt the former process for high-definition images. The potential of MM-ICL. is further explored in scene text recognition, image generation, and game instructions [7hao al., 2023b, Sun et al., 2023, Jin et al., 2024].

---

Recognizing the effectiveness of MM-ICL, researches shift towards prompt optimization. These methods focus on directly optimizing multi-modal prompts to understand the task and generate the expected output, without parameter adjustments [Gong et al., 2023, Tsimpoukelli et al., 2021, Li al., 2023b]. This approach has significantly improved performance in visual reasoning tasks [Yang et al., 2022, Zheng et al., 2023]. Another approach involves textualizing visual information to enable VLLMs to leverage their background knowledge through in-context learning, further enhancing visual reasoning [Yang et al., 2023, Lu et al., 2024, Gupta and Kembhavi, 2023, Shen et al., 2024]. In addition, in order to better explore the MM-ICL, Zong et al. [2024] and Shukor et al. [2024] also provide a dataset to test the MM-ICL capabiliities of the multi-modal classification. Furthermore, Shukor et al. [2024] take the first step to conduct an instruction modification exploration for MM-ICL.
Meanwhile, Baldassini et al. [2024]. Chen et al. [2024c] pioneer the first naive multi-modal retrieval exploration to enhance MM-ICL. Different from the existing work, our study mainly focuses on a systematic exploration of the effectiveness of key factors influencing the effectiveness of MM-ICL in unified perspective. To this end, we conduct detailed analysis and exploration on VLLMs and 20 factors across 4 tasks, aiming to provide systematic and practical guidance for future research.
7 Discussion
Broader Impacts. Our work is the first systematically explore the factors influencing MM-ICL. We aim to enhance the understanding of MM-ICL mechanisms and guide future developments in this field. Additionally, our findings could foster more comprehensive comprehension of MM-ICL within the community. For social impact, this research may influence the creation of more effective multi-modal large language models and relevant applications.
Limitations & Future Work. Due to time and cost constraints, this work is limited to the explo- ration of image and text modalities. In future research, can extend our exploration to video modal ICL and multi-lingual MM-ICL scenarios. Another limitation of this work involves the insufficient consideration of certain image instructions, such grounding the inclusion of additional arrows. These aspects often require more complex human input and are not adequately supported by most current models.
Conclusion
This study is the first to systematically explore MM-ICL by identifying key performance determinants. Our experiments with 6 models and 20 factors across 4 tasks show that multi-modal retrieval significantly outperforms single-modal approaches and the intraddemosttration ordering critically influences learning efficacy. Additionally, incorporating task-specific instructions into prompts enhances model performance. We hope these findings will refine our understanding of MM-ICL mechanisms and guide more effective developments and future research in this evolving field.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (NSFC) via grant 62306342, 62236004, 62441603 and 62476073. This work was also sponsored by the Excellent Young Scientists Fund in Hunan Province (20241J4070) and the Science and Technology Innovation Program of Hunan Province under Grant 2024RC3024. We are grateful for resources from the High Performance Computing Center of Central South University, and the CCF-Zhipu.Al Large Model Innovation Fund NN0.CCT-2hipu202240). Libo Qin is the corresponding author.
References
Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. In- context examples selection for machine translation. In Findings of the Association for Computa- tional Linguistics: ACL 2023, pages 8857-8873, 2023.
Cean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for fhw-shot learning. Advances neural information processing systems, 35:23716-23736. 2022.
10

---

Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308. /2966, 2023.
Folco Bertini Baldassini, Mustafa Shukor, Matthieu Cord, Laure Soulier, and Benjamin Piwowarski. What makes multimodal in in-context learning work? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1539-1550, 2024.
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawa- har, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291-4301, 2019.
Qiguang Chen, Libo Qin, Jiaqi Wang, Jinxuan Zhou, and Wanxiang Che. Unlocking the boundaries of thought: A reasoning granularity framework to quantify and optimize chain-of tfann-of-thougt. arXiv preprint arXiv:2410.0569. 2024a.
Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M.COT: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought, In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8199-8221, Bangkok, Thailand, August 2024b. Association for Computational Linguisties. doi: 10.18663///222222aa---loogg URL https: /aaaaaalaaatooooocoo/o longo 446.
Shuo Chen, Zhen Han, Bailan He, Mark Buckley, Philip Torr, Volker Tresp, and Jindong Gu. Understanding and improving in-context learning vision-language models. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024c. URL attps://openreviie net/ faet/fol pest/forumm forum?dd-sSta WSS2SWFo3o..
Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00335, 2015.
Elena Deza, Michel Marie Deza, Michel Marie Deza, and Elena Deza. Encyclopedia of distances. Springer, 2009.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022.
Sivan Doveh, Shaked Perek, M Jehanzeb Mirza, Amit Alfassy, Assaf Arbelle, Shimon Ullman, and Leonid Karlinsky. Towards multimodal in-context learning for vision language models. arXiv preprint arXhv:2403.12733 2024.
Zhengfang Duanmu, Wentao Liu, Zhongling Wang, and Zhou Wang. Quantifying visual image quality: A bayesian view. Annual Review of Vision Science, 7(1::337-464, 2021.
Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu. Video-of-thought: Step-by-step video reasoning from perception to cognition. In Proceedings of the International Conference on Machine Learning, 2024a.
Hao Fei, Shenggiong Wu, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Vitron: A unified pixel-level vision 1lm for understanding, generating, segmenting, editing. 2024b.
Hao Fei, Shengqiong Wu, Meishan Zhang, Min Zhang, Tat-Seng Chua, and Shuicheng Yan. Enhanc- ing video-language representations with structural spatio-temporal alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024c.
Angeliki Giannou, Shashank Rajput, Yong Sohn, Kangwook Lee, Jason Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers. In Proc. of ICML, 2023.
11

---

Olga Golovneva, Moya Peng Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarande and Asli Celikyilmaz. Roscoe: suite of metries for scoring step-by-step reasoning. In The Eleventh International Conference on Learning Represeniations, 2022.
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. -ulti A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.04790, 2023.
Google. Gemini: A family of highly capable multimodal models, 2023. URL https://storage. pdf.
googleapis com/ dewpmindddinnddd nd
hwwppmiiiidmddaaaa/aee media/ /tteeppiinneeediiaaggaaiiaaaaaaa
Yash Goyal, Tejas Khot, Douglas Summers- -taa, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913, 2017.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 14953-14962, 2023.
Jiabang He, Lei Wang, Yi Hu, Ning Liu, Hui Liu, Xing Xu, and Heng Tao Shen. Icl-d3ie: In-context learning with diverse demonstrations updating for document information extraction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19485-19494, 2023.
Mengkang Hu, Yao Mu, Xinmiao Chelsey Yu, Mingyu Ding, Shiguang Wu, Wenqi Shao, Qiguang Chen, Bin Wang, Yu Qiao, and Ping Luo. Tree-planner: Efficient close-loop task planning with large language models. In The Twelfth International Conference Learning Representations, 2023.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7700-6709, 2019.
Yonggang Jin, Ge Zhang, Hao Zhao, Tianyu Zheng. Jiawei Guo, Liuyu Xiang, Shawn Yue, Stephen W Huang, Wenhu Chen, Zhaofeng He, et al. Read to play (r2-play): Decision transformer with multimodal game instruction. arXiv preprint arXiv:2402.04154 2024.
Maxime Kayser, Oana-Maria Camburu, Leonard Salewski. Cornelius Emde, Virginie Do. Zeynep Akata, and Thomas Lukasiewicz. e-vil: A dataset and benchmark for natural language explanations in vasion-language tasks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1244-125. 1254, 2021.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-22213 2022.
Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. hierarchical approach for generating descriptive image paragraphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 317-325, 2017.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, al. Obelics: An open web-scale filtered dataset interleaved image-text documents. Advances in Neural Information Processing Systems, 36, 2024a.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint vrXiv:2405.022466 2024b.
Bo Li, Peiyuan Zhang, Jingkang Yang, Yuanhan Zhang, Fanyi Pu, and Ziwei Liu. Otterhd: A high-resolution multi-modality model. arXiv preprint trxiv:2311.042// 2023a.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023b.
12

---

Jian Li and Weiheng Lu. A survey on benchmarks of multimodal large language models. arXiv preprint arXiv:2408.086.3 2024.
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. M 3 it: A large-scale dataset towards multi-moda multilingual instruction tuning. arXiv preprint arXiv:2306.04388 04387, 2023c.
Xiaonan Li and Xipeng Qiu. MoT: Memory-of-thought enables ChatGPT to self-improve. In Proc. of EMNLP, 2023a.
Xiaonan Li and Xipeng Qiu. Finding support examples for in-context learning. In Findings of the Association for Compulational Linguisties: EMNLP 2023, pages 6219-6235, 2023b.
Yingcong Li, M. Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning, 2023d.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Eneko Agirre, Marianna Apidianaki, and Ivan Vulić, editors, Proceedings of Deep Learning Inside Out (DeeLlO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100-114, Dublin, Ireland and Online, May 2022a. Association for Computational Linguistics. doi: 10.18653/v1/ 2022.deelio- URL https: : http:///aaaaaaollaaeeeee org/ 2022. deelio-1.10.
Jachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop Knowledge Extraction and Integration for Deep Learning Architectures, pages 100-114, 2022b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. / 1692, 2019.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-We Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022a.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-pla compositional reasoning with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), Dublin, Ireland, May 2022b. Association for Computational Linguistics. doi: pages wwwp:///222...llooooooo... 1o0..55...556 URL https: htap::////aaaahhhooggoo . org/ 22g/2022.co acl- acl-long. 556. 10. 10.18653/v1/2022
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086-8098 2022c.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cyf conference on computer vision and pattern recognition, pages 3195-3204 2019.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200-2209, 2021.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proc. of EMNLP, 2022.
13

---

Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Oer-vqa: Visual question answering by reading text images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947-952. IEEE, 2019.
OpenAI: Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report, 2023.
Wenbo Pan, Qiguang Chen, Xiao Xu, Wanxiang Che, and Libo Qin. A preliminary evaluation of chatgpt for zero-shot dialogue understanding. arXiv preprint arXiv:2304.04225, 2023.
Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. Cross-lingual prompting: Improving zero-shot thain-of-thought reasoning across languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2695-2709, 2023.
Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S Yu. Large language models meet nip: A survey, arXiv preprint arXiv:2405. 128899 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages
2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-2523 2015.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146-162. Springer, 2022.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36, 2024.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210-31227. PMLR, 2023.
Mustafa Shukor, Alexandre Rame, Corentin Dancette, and Matthieu Cord. Beyond task per- formance: evaluating and reducing the flaws of large multimodal models with in-context- learning. In The Twelfth International Conference on Learning Representations, 2024. URL //op:r///ppeeeviett net pest/forumt forumriddiiiiiiooo iidiiii@kmaaaa...........
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision-ECCe 2020: 16th European Conference, Glasgow, UK, August 23 23-28, 2020, Proceedings, Part 1l 16, pages 742-758. Springer, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
Harald Steck, Chaitanya Ekanadham, and Nathan Kallus. Is cosine-similarity of embeddings really about similarity? arXiv preprint 2:Xiv:2403.054400 2024.
Quan Sun, Yufeng Cui, Xiaosong Zhang. Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312.13286, 2023.
14

---

Changyao Tian, Xizhou Zhu, Yuwen Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Yuntao Chen, Lewei Lu, Tong Lu, Jie Zhou, et al. Mm-interleavve: Interleaved image-text generative modeling via multi-modal feature synchronizer. arXiv preprint arXiv:2401. 10208, 2024.
Shengbang Tong, Erik Jones, and Jacob Steinhardt. Mass-producing failures of multimodal systems with language models. In Thirty-seventh Conference on Neurul Information Processing Systems, 2023. URL https: http///ppppeeeeeeee/m/e/ee
10qSOh.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9568 9568-9578, 2024.
Maria Tsimpoukelli, Jacob L. Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-212, 2021.
Ramakrishna Vedantam, C Lawrence Ziticck, and Devi Parikh, Cider: Consensus-basee image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images, arXiv preprint arXiv: 1601.07 140, 2016.
Peng Wang, Yongheng Zhang, Hao Fei, Qiguang Chen, Yukai Wang. Jiasheng Si, Wenpeng Lu, Min Li, and Libo Qin. S3 agent: Unlocking the power of vllm for zero-shot multi-modal sarcasm detection. ACM Transactions on Multimedia Computing, Communications and Applications, 2024.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Proc. of NeurlPS, 2022b.
Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning, 2023.
Shenggiong Wu, Hao Fei, Xiangtai Li, Jiayi Ji, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Towards semantic equivalence of tokenization in multimodal Ilm. arXiv preprint 47X1v22406.05127, 2024a.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal 11m. In Proceedings of the International Conference on Machine Learning, 2024b.
Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long 2023.
Papers), pages
Yanzheng Xiang, Hanqi Yan, Lin Gui, and Yulan He. Addressing order sensitivity of in-context demonstration examples in causal language models. arXiv preprint arXiv:2402.15637 2024.
Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, Wanxiang Che, and Nan Duan. Bridgetower: Building bridges between encoders in vision-language representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10637-10647, 2023.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3081-3089, 2022.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXivv2303.1138/ 2023.
15

---

Barry Menglong Yao, Aditya Shah, Lichao Sun, Jee-Hee Cho, and Lifu Huang. End-to-end multi- modal fact-checking and explanation generation: A challenging dataset and models. In Proceedings of the 46th International ACM SIGIR Conference Research and Development in Information Retrieval, pages 2733-2743, 2023.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen, A survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2019.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? Advances in Neural Information Processing Systems, 36, 2024.
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang. Wenjuan Han, and Baobao Chang. MMICI: Empowering vision-language model with multi- modal in-context learning. In The Twelfth International Conference on Learning Representations, 2024. URL httttp:://openeeviee www./eeenootiieeo 1dt.oooonndoooouaiiiii.e
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjic Zhang. Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models, 2023a.
Zhen Zhao, Can Huang, Binghong Wu, Chunhui Lin, Hao Liu, Zhizhong Zhang, Xin Tan, Jingqun Tang, and Yuan Xie. Multi-modal in-context learning makes an ego-evolving scene text recognizer. arXiv preprint arXiv:2311.13120 2023b.
Ge Zheng, Bin Yang. Jiajin Tang, Hong- Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of- thought prompting for multimodal reasoning in language models. Advances in Neural Information Processing Systems, (6:5168-5199) 2023.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe Le, et al. Least-to-movt most prompting enables complex reasoning in large language models. In The Eleventh International Conference Learning Representations, 2022.
Ziyu Zhuang, Qiguang Chen, Longxuan Ma, Mingda Li, Yi Han, Yushan Qian, Haopeng Bai, Weinan Zhang, and Ting Liu. Through the lens of core competency: Survey on evaluation of large language models. In The 22nd Chinese National Conference on Computational Linguistics: Frontier Forum, page 88, 2023.
Yongshuo Zong, Ondrej Bohdal, and Timothy Hospedales. VI-icl bench: The devil in the details of benchmarking multimodal in-context learning, arXiv preprint urXiv:2403. 11.4001131644 13164, 2024.
16

---

Appendix
The Implement Details for Standard Baseline
To ensure rigorous control of experimental variables, we establish a standard baseline for our study. This baseline utilizes a multi-modal encoder for data representation and cosine similarity for sample comparison, with retrieval restricted to the same task. The following sections provide detailed insights into the implementation this baseline.
A.1 Demonstration Retrieval Implementation for Baseline
Text-modal Encoder for Sample Representation We employ RoBERTa [Liu et al., 2019] as multi-modal encoder to represent the data unified embedding space. is worth noting that since the Image Caption task only contains general task instruction, we will splice the field tags contained in the meta data in M3IT before the insrtuction for comprehensive text representation.
Multi-modal Encoder for Sample Representation We employ BridgeTower [Xu et al., 2023] as a
encoder to represent the data in a unified embedding space. This encoder integrates both visual and textual information, prioritizing single modality to capture the rich semantic content present in images.
Cosine Similarity for Sample Comparison To compare samples effectively, we use cosine sim- ilarity, a metric that measures the cosine of the angle between two non-zero vectors in a multi- dimensional space. This choice is motivated by its effectiveness in capturing the similarity between high-dimensional vectors, which are typical outputs of our multi-modal encoder. Specifically, we compute the cosine similarity between the query and each candidate sample, which is given by:
hg h4
cosine(hg. h4) | ||]
where hg and h, are the embedding vectors of the query q and candidate sample 24, respectively.
In-domain and Top-k Retrieval for Sample Selection To ensure the relevance and accuracy of the retrieval process, for sample selection, we first confine retrieval the same task and domain. This means that comparisons and rankings are conducted exclusively among samples within the same task and domain category, ensuring the contextual appropriateness of the retrieved results.
Datasct
Category
COCO Caption [Chen et al., 2015]
IC
TextCaps (Sidorov el al., 2020]
Paragraph Captioning [Krause et al., 2017]
COCO Text |Viit et ul., 2016]
CLS
ImageNet Image Classification (Rariiiattioonsssssssskkk.. et al., 2015]
CLS
IQA [Duanmu al., 2021]
COCO-ITM [Chen al., 2015]
CL.S
c-SNLI-VE [Kayser al., 2021]
CLS
Mocheg (Yao al., 2023]
CLS
VQA-v2 [Goyal al., 2017]
VQA
Doc [Mathew
2021]
VQA
OCR-VQA [Mishra et al., 2019]
VQA
ST-VQA [Biten et al., 2019]
VQA
Rext-VQA [Singh et al., 2019]
VQA
GQA [Hudson and Manning, 2019]
VQA
OKVQA [Marino et al., 2019]
VQA
A-OKVQA [Schwenk al., 2022]
VQA
ScienceQA [Lu al., 2022a]
C [Chen et al., 2024b]
R
Table 2: Dataset in MT and MCCoT, where IC: Image Captioning, CLS: Classification, VQA:
Visual Question Answering, Thoought Reasoning (with NL rationale). Due to the cost, for each task, we evenly sampled 500 items according to the sub-dataset.
17

---

In addition, for sample selection, samples are ranked according to their cosine similarity scores. Higher similarity scores indicate a closer alignment with the query sample, enabling the efficient identification of the most relevant samples. This ranking process involves two main steps: (1) Sorting: Candidate samples are sorted descending order based on their cosine similarity scores relative to the query. (2) Selection: Subsequently, the top-k ranked samples are selected based on their relevance as determined by the similarity scores.
A.2 Demonstration Ordering Implementation for Baseline
By default, utilize the methodology for ordering demonstrations within our baseline model. By default, we adopt text-after-image (Text-Image) approach for intra-demonstration sorting. This means that, within single demonstration, textual information is positioned after the corresponding image. This ordering is chosen based on preliminary findings suggesting that such a sequence aids in better contextual understanding and retention of the demonstrated information.
Furthermore, for the ordering of inter-demonstration sequences, we employ a similarity- bassed method. This method ranks demonstrations according their similarity the query, with more similar demonstrations placed higher in the order. The similarity is determined using a metric that assesses the alignment of key features between the query and the demonstrations. This approach ensures that the most relevant and contextually aligned demonstrations are prioritized, potentially enhancing the model's performance and the user's comprehension.
A.3 Prompt Construction Implementation for Baseline
To ensure consistency and comparability in our baseline, we introduce both delimiter and 3-shot setting (following Wei et al. [2022b]. Qin et al. [2023]). The delimiter serves to clearly demarcate different segments of the input data, preventing any potential confusion overlap between distinct portions of the input. This clear separation is crucial for model to accurately process and understand the structure of the data receives.
The 3-shot setting, on the other hand, involves providing three examples for each task within the prompt. This approach is designed to stabilize the learning process by presenting the model with sufficient contextual information. By offering three examples, we strike balance between providing enough context to guide the model's understanding and avoiding the cognitive overload that might occur with too many examples. This setting not only enhances the model's s performance but also ensures more robust and reliable learning process.
A.4 Baseline Prompt
In the context of using Vision-and-Languaa Large Models (VLLMss, it is essential to carefully structure the input prompts ensure accurate processing. The prompt format typically used is illustrated below:
[REQUETT] % Shot 1
Visual Input I
<Textual Inpul Tizl>
[RESPONSE]
<Textual Output IPiz>
[REQUEST] % Shot 2
<Visual Input Iyis> <Textual Input Tlat
RESPONSE]
<Textual Output INiE>
[REQUEST] % Shot
Visual Input Tyit> <Textual Input Tiat
18

---

[RESPONSE]
<Textual Output Izi=>
[REQUEST] % User Query
<Visual Input I <Textual Input T
where any gray text following the percent sign (%) is treated as a comment. These comments are not processed part of the primary input but serve to provide additional context or instructions within the coding environment. This convention helps in maintaining the clarity and functionality of the given prompting.
In conclusion, the standard baseline established here integrates a multi-modal encoder, cosine similarity, and task-specific retrieval with foeus visual modalities. ranks samples based on similarity and employs a delimiter with a 3-shot setting to ensure robust and consistent performance across different tasks.
The Implement Details for Sample Comparison
B.1
Metrie Calculation
Cosine Similarity (Moos) Compute the cosine similarity between by and hy using the formula:
hg hg Meos h2) | |1Il,
L2 Similarity (M2 Calculate the L2 similarity by computing the negative Euclidean distance between h2 and hy:
Mz h2) =- -1y h
(8)
Since Euclidean distance measures dissimilarity, we use the negative value to represent similarity, where higher value indicates greater similarity.
Semantic Diversity (Maie) Semantic diversity assessed by evaluating the differences in the distributional properties of h, and h3- This assessment involves analyzing the variance in how these properties are distributed across different samples. To determine the presence of semantic diversity within Multi-Modal In-Context Learning (MM-ICL), we adopt the methodology proposed by Li and Qiu [2023b]. Specifically, we employ the "diversity retriever." designed to enhance the diversity of the selected samples. The diversity retriever operates by first selecting the top 10 samples based on a preliminary measure of relevance. From these top 10 samples, it then identifies the 3 samples that exhibit the highest semantic diversity. This two-step process ensures that the final selection of samples for MM-ICL is not only relevant but also diverse in terms of their semantic content.
B.2 Comparison and Analysis
Comparing the results obtained using different metrics (Meos ML2, Mare) provides a compre- hensive understanding of their effectiveness and suitability for specific applications. It is essential to analyze the trade-offss associated with each metric and interpret the results to draw meaningful conclusions about sample quality and relevance.
As shown in Figure 12, cosine similarity, which measures directional semantic alignment, emerges the superior metric in MM-ICL compared to L2 similarity. This observation is supported by the findings of Deza et al. [2009] and Steck et al. [2024], who highlight that MM-ICL prioritizes semantic directional consistency over complete semantic alignment. Cosine similarity's 's ability to capture the nuances of directional alignment allows for more precise interpretations of semantic relationships within the data, making particularly effective for MM-ICL tasks.
In contrast, Figure 13 illustrates that while diversity, as measured by Mann. enhances performance in text-based in-context learning, our experiments reveal limited improvement in MM-ICL tasks. This finding suggests that diversity may not directly correlate with better performance MM-ICI. The limited impact of diversity on MM-ICL performance could be attributed to the specific nature of
19

---

Discarcce
Cosine Distance
La Distance
Cosine Distance
La Distance
Cosine Distance
CIDER
CIDER
CIDER
RFRTSoore
Apc.
ACC.
Token Pt
F1
HERTScnCe
F1
BERTScure
Ft
BERTSoore
Accuracy
Accuracy
Accuracy
(AVG: 53:9 68.16)
(AVG: 6361 6688)
(AVG: 5863 6222
(a) Gemini-Pro
(b) IDEFICS2
(c) Qwen-VL
Distance T Cosine Distance
12 Distance
Cosine Distamce
RAS
BERTScore
RAS
BEKTScore
Acc.
Token
Acc.
Token P1
BERTScore
BERTScore
Accuracy
Accuracy
(AVG: 37.75 44.60)
(AVG: 37.2 41(3)
(d) Olter
Flaminna
Figure 12: The impact of the different similarity metrics.
multi-modal data, where the interplay between diffferent modalities requires nuanced approach than simply maximizing diversity.
Further analysis of these metrics reveals the inherent trade-offs between them. For instance, while cosine similarity offers advantages in maintaining semantic directional consistency, may not capture the full extent of semantic similarity that 1,2 similarity can provide. On the other hand, L2 similarity, though comprehensive
measuring complete alignment, might lack the precision needed for tasks that rely heavily on directional semantic cues. Similarly, while diversity is beneficial in certain contexts, its role in MM-ICL needs to be reconsidered, potentially focusing on optimizing other aspects of sample quality.
In summary, the evaluation Meoss M222 and Maie underscores the importance of selecting appropriate metrics based on the specific requirements of the task. Understanding the trade-offs and cppeext-specific effectiveness these metries is crucial for optimizing performance in multi-modal in-context learning applications.
Exploration of MM-ICL Prompt Construction
C.1 The Implement Details for Demonstration Sampling
To examine the effect of demonstration sample quantity model performance, as shown in Figure 14, we select subset of k' demonstrations from the demonstration list L to the prompt, where is the number of retrieved demonstrations. Formally, the prompt construction process defined as:
P = I(ō(z 0(z.g)....,,66 ,(xxz,))
We systematically evaluate the influence of varying on MM-ICL performance.
C.2 The Implement Details for Delimiter Injection
To distinctly separate inputs and outputs within demonstrations z,, shown in Figure 15, leverage special delimiter markers. Delimiters like [Request] and [Response] are strategically placed before the inputs and outputs, respectively. Formally, delimiter injection function 8 maps inputs and outputs to the prompting sequences:
8(x..) [Request] 1 @ [Response] + O.,
(10)
20

---

Diversity Retriever
Similer Retriever
Diversity Retriever Similar Retriever
Diversity Retriever
Similer Retriever
BAS
BERTSOore
KAS
BERISSore
RAS
BERIScore
Act.
Token F1
Acc.
Token
Acc
Token Fi
BERTSare
BERTSOOE
BERTSore
Accuracy
Accuracy
Accuracy (AVG:
(AVG: 06.55
(AVG: 65.24 66.00)
62.2
(a) Gemini-Pro
(b) IDEFICS2
(c) Qwen-VL
Diversity
Simillar Retriever
Diversity
Sinilar Retriever
CIDER
CIDER
HERTScore
BERTScore
Acc.
RERTScore
F3
BERTSCore
Accuracy
Accuracy
(AVG: 43.18 44.60) (d) Otter
(AVG: 30 12 4103)
(e) OpenFlamingo
Figure 13: The impact of the utilization on diversity metrics.
Validation Dataset (V)
Prompt (P)
Sample: X1
Sample: Xj1
Demonstration
Ordered List: L
Sampling
Sample: Xy2
Sample: X
A dditional Sample: Xj1
Prompt (P)
A dditional Sample: X/2
Ordered List: L
Figure 14: The demonstration sampling process for MM-ICL prompt construction.
where I, and O denotes the input and output for the sample z,, respectively. In addition, represents
string concatenation operation.
Prompting
Ordered Li)t
[Request]
Input: x
[eesponse]
Output:
[Request]
Inpul: X2
[eeponnse]
Output: y/2
Delimiter Injection
Ordered List
Input: X2
Output: yı
Input: X2
Output: y2
Figure 15: The delimiter injection process for MM-ICL prompt construction.
C.3 The Implement Details for Instruction Injection
Visual Language Models (VLLMss) are known to be highly sensitive to input instructions, as demon- strated by Kojima et al. [2022] and Qin et al. [2023]. Inspired by this observation, aim to enhance
21

---

w/o Delimiter
Delimiter
w/o Delimiter
w/ Delimiter
w/o Delimiter
Delimiter
CIDER
RAS
BERTScoe
RERTSoorE
RAS
BERTScore
Adc.
Token F1
Acc
Token Ft
AdC.
Token
BERTScore
BERTSCore
Accuracy
Accurccy
Accuracy
(AVG: 64.76 - 68
(AVG: 65.19 66.00
(AVG: 61.76 68.16)
(a) Gemini
(b) IDEFICS2
(c) GPT4-V
w/o Delimiter
Delimiter
w/o Delimiter
w/ Delimiter
w/o Delimiter
w/ Delimiter
CIDER
CIDER
CIDER
BERTScore
BERTScore
BERTScore
100
50
100
Aoc.
Token F1
Acc.
Token F1
ACC.
Token Fi
F1
BERTScore
BERTScore
BERTScore
Accuracy
Accuracy
Accuracy
(AVG: 57.29 62.22
(AVG: 4441 4160)
(AVG: 38 41003)
(b) Qwen-VL
(c) Otter
(d) OpenFlamingo
Figure 16: The impact of inserting delimiter into the input and output of demonstration on model performance.
task comprehension Multi-Modal In-Context Learning (MM-ICL) by incorporating various instruc- tions to explore their influence on performance. Formally, we develop instruction methods, denoted as I(), which describe the task and are integrated into the prompt construction process. The prompt P is constructed as follows:
P= I
(11)
where 6(F.,) represents the transformation of the i-th demonstration example.
Specifically, we have designed distinct instructions tailored to different types of tasks, ensuring clarity and appropriateness for each unique context. For image captioning tasks, the prompt is:
Please provide caption for the image following the structure of the provided example.
In this context, the objective is to generate descriptive captions that accurately reflect the content and context of the image. For Visual Question Answering (VQA) tasks, our prompt is:
Examine the image and answer the question by closely following the structure shown in the example provided.
The VQA tasks require the model to analyze visual content and respond to specific queries. By following the example, can produce answers that are precise and directly related to the visual stimuli. For image classification tasks, the prompt is:
Carefully review the image and categorize it based on the options provided in [REQUEST].
following the classification format illustrated in the example.
Image classification involves categorizing images into predefined classes based on visual content. The provided example demonstrates the expected classification format. For chain-of of-thouht reasoning tasks, the prompt is:
22

---

Carefully review the given image and the associated text. Utilize the reasoning format illustrated in the provided examples, breaking down your thought process. Ensure that each reasoning step is explicitly connected to observable details in the image text, and articulate your conclusion in a clear and logical manner.
Mlain-of-thoughh reasoning tasks require information. The prompt encourages users to break down their reasoning process into clear, logical steps, each supported by specific details from the image text.
complex interaction between visual and textual
Furthermore, we explore three categories of instructions to enhance the MM-ICL, process:
Introductory Instruction (Iintro) This instruction provides overview of the task before present- ing any demonstrations. As depicted in Figure 4 (a), the introductory instruction Tintro is positioned at the beginning of the ordered demonstration list L. This setup aims to set the context for the subsequent examples. Specifically, the overall prompt template follows:
<Instruction Tintro>
[DEMONSTRATIONS]
[REQUEST] % Shot
<Visual Input I
<Textual Input T
[RESPONSE]
<Textual Output I
[QUERY] [REQUEST] % User Query
<Visual Input Tuis> <Textual Input T1at
Summative Instruction (Isum) This instruction offers summary after the examples, guiding the
model to apply the learned concepts to real-world problems. As shown Figure 4 (b), the summative instruction I is added at the end of the demonstration list C. This helps in reinforcing the learning objectives and expected outcomes. Specifically, the overall prompt template is as follows:
<Instruction Tintro>
IDEMONSTRATIONS]
[REQUEST] % Shot
<Visual Input I
<Textual Input Tt
[RESPONSE]
<Textual Output T
In summary, <Instruction Isum
[QUERY]
[REQUEST] % User Query
Visuual Input I <Textual Input Tiat >
Intra-demonstration Instruction (Inntra) This instruction embeds task instructions within each example, assisting the model in understanding the task requirements during the learning process. As
23

---

OKVQA [Marino et al., 2019]
VQA-v2 [Goyal et al., 2017]
Model
Accuracy
BERTScore
Token FI
Accuracy
BERTScore
Token F1
OpenFlamingo [Awadalla al., 2023]
40.28
78.10
17.45
53.33
83.34
25.67
GPT4V (OpenAl: al., 2023]
54.28
85.97
25.23
69.69
84,89
29.18
IDEFICS2 [Laurençon et al., 2U24b]
55.32
87.61
27.81
71.28
87.98
35.46
Table 3: The correlation analysis of the indicators and reproducted accuracy. The results are obtained by testing on subset of the test
illustrated in Figure 4 (c), the intra-demonstration instruction I is included within each demonstration xy in the list C. This method ensures that the task instructions are continuously reinforced throughout the learning process. Specifically, the overall prompt template is follows:
DEMONSTRATOOSS)
[REQUEST] % Shot
<Visual Input I
<Textual Input I <Instruction Tintre?
[RESPONSE]
<Textual Output I
[QUERY] [REQUEST] % User Query
<Visual Input T <Textual Input That> <Instruction Iintra>
By systematically incorporating these instruction categories into the MM-ICL framework, we aim investigate their impact on model performance and task comprehension.
D Prompt Robust
In preliminary experiments, observed that variations in prompts do significantly alter the overall conclusions. Specifically, we employed multiple profpts-difffring in instructions and delimiters- while maintaining equivalent semantic content but varying linguistic expression. As demonstrated in Table 4, the influence of these different prompts on the results is minimal. This suggests that our findings are robust to changes prompt formulation, thereby supporting the reliability of the experimental outcomes.
Caption
VQA
Classification
Reasoning
AVG
CIDER
BERTScore
Token F1
BERTScore
Acc
F1
Acc
RAS
P1
12.03
85.85
22.53
86.67
59.93
54.62
59.52
92.04
59.15
P2
14.01
86.77
23.59
86.00
58.53
53.61
61.85
91.86
59.53
P3
13.91
86.92
24.70
87.63
59.74
52.14
61.89
93.05
60.00
P4
14.44
86.48
23.14
87.77
60.23
50.48
60.54
92.27
59.42
Table 4: Performance across different prompts (i.e., Pl, P2, P3 and P4).
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have present main claims and outline the paper's contributions and scope in lines 6-13 the Abstract and 44-54 of the Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
Our paper does not involve any proofs or assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: As shown in Section 3, Section and Appendix, we have provided detailed descriptions and analyses of the experimental setups for all our investigations.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [No]
Justification: The code for exploratory prompt work generally does not need to be released, and readers can easily use the prompts we report directly reproduce the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
Policyy) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips "nips.ccpppli wwtppcpppcgcppeeccceeccccc gewideee/oeeeiieeeeeeeeeeeeeeeee wwt.SSuumissioopoooooooo ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: As detailed in Section 4, we have thoroughly described the experimental setups for all our explorations.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars are shown Figure and Figure with an explanation of the error variables provided in Section 4. However, we do not report error bars for all tasks due to the high costs associated with human annotation and computational resource consumption.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: As detailed in Section 4, we outline the specific model compute resources provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: https://neut http:://eeriipppeeeee http://eerrrs.ccpuul..eeee htt.aiic/Ethissuuieeeee
Answer: [Yes]
Justification:
We are convinced that we comply with NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have addressed the broader impacts of our work Section 7. Additionally, research is primarily empirical exploration and poses additional social risks, have not included a discussion on potential harmfulness.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwoee.hooo..... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
29

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "L31srORlAF", "reviewer": "Reviewer_Pm2y", "summary": "The paper investigates the underlying factors that influence the effectiveness of Multi-Modal In-Context Learning (MM-ICL). The authors conducted extensive experiments on three core steps of MM-ICL: demonstration retrieval, demonstration ordering, and prompt construction using six vision large language models and 20 strategies. Their findings highlight the necessity of a multi-modal retriever for demonstration retrieval, the importance of intra-demonstration ordering, and the enhancement of task comprehension through introductory instructions in prompts. This study aims to provide a foundational guide for optimizing MM-ICL strategies in future research.", "questions": "In addition to the list of weaknesses above, here are some questions for the authors to address:\n\n1.For multi-modal demonstration retrieval, why do you use a multi-modal encoder to calculate the similarity instead of using the product of text and vision single modality similarity? Exploring this alternative approach might offer additional insights into multi-modal retrieval.\n\n2.For the OpenAI GPT-4V model, how do you reconstruct the order of the text and the image? As far as I know, the API does not provide a way to structure the order of text and images.", "limitations": "Limitations are discussed in the paper. and the authors do not foresee any negative societal impact.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The article is well-written, and the authors have thoroughly addressed several important concepts that reviewers might raise. The paper effectively covers a broad range of factors affecting MM-ICL. The authors have conducted experiments across multiple models and strategies, providing a diverse set of data points, which is valuable for identifying trends and patterns.", "weaknesses": "Some weaknesses are listed below: \n\n1. The results of the experiments seem apparent and intuitive. For example, the multi-modal retriever performs better than the single modality retriever because it incorporates more information. Additionally, intra-demonstration ordering is crucial as it affects the structure of the input sample and how the model processes the input. As a result, the findings do not seem particularly inspiring.\n\n2. Some qualitative analysis is missing. It would be beneficial to compare how different in-context examples affect the answers.\n\n3. Considering that the margin of improvement in some metrics is quite small, certain designs of in-context examples may not be as important. For instance, as shown in Table 1, in most scenarios, a single textual retriever is sufficient. Therefore, using a multi-modal retriever, which may consume more computational resources, is unnecessary.\n\n4.\tIs it appropriate to combine all tasks to provide a universal paradigm for constructing MM-ICL strategies? For example, in classification tasks, the multi-modal retriever does not seem to perform the best.\n\nThere are also some minor mistakes in the paper:\n\n1. It is more common to use the term \"Large Vision-Language Models (LVLMs)\" instead of \"Vision Large Language Models (VLLMs),\" which you use throughout the paper (e.g., line 43).\n\n2.The reference for the model Qwen-VL in Table 1 is incorrect; the model is not from OpenAI.\n\n3.In Table 1, the Otter Model results are not highlighted in bold.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your acknowledgment and interest in our work! We sincerely appreciate your thorough and insightful comments on our work, and we will address each of your main concerns below:\n\n---\n**Q1:** The results of the experiments seem apparent and intuitive.\n\n**R1:** Thanks for your constructive feedback. We sincerely believe that our work is a meaningful step in the systematic exploration of the MM-ICL and provides some insightful observations.\n\n1. Previous studies mainly considered the inter-demonstration ordering. To our knowledge, we are the first to explore the intra-demonstration ordering and find that this ordering is significantly more important than inter-demonstration ordering, hoping to provide insightful guidance in the future.\n2. Additionally, we thoroughly explored how to effectively construct MM-ICL prompts by investigating Introductory Instruction, Summative Instruction, and Intra-demonstration Instruction, which is less explored in the previous research. \n3. In addition, we provide more insights on how to insert separators, optimize domain selection, and refine distance metrics, offering guidance on best practices for constructing MM-ICL.\n\nWe will follow your suggestion to add more discussion in the next version.\n\n---\n**Q2:** Some qualitative analysis is missing. It would be beneficial to compare how different in-context examples affect the answers.\n\n**R2:** Thanks for your insightful suggestion. Actually, the Exploration of MM-ICL Demonstration Ordering and Exploration of MM-ICL Prompt Construction can reflect how different in-context examples affect the answers. We will add more discussion and add more qualitative analysis for better understanding in the next version.\n\n---\n**Q3:** As shown in Table 1, in most scenarios, a single textual retriever is sufficient. Therefore, using a multi-modal retriever, which may consume more computational resources, is unnecessary.\n\n**R3:** Thank you for your insightful feedback. This is an interesting research question. Actually, multi-modal retrieval attains better performance in many scenarios like Image Caption and VQA. However, our experiments show that textual retrieval works well for classification and reasoning tasks.\n\nBased on the qualitative analysis, we observe that due to the semantic richness of the labels and rationales, textual retrieval can obtain more similar samples. However, the current multi-modal retrieval struggles with complex text semantics, often favoring image similarity. This aligns with recent work [1,2], which is valuable for future exploration.\n\nIn the future, we can consider how to integrate the strengths of text and visual modalities for better performance. \n\n[1] Tong et al. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. CVPR 2024.\n\n[2] Tong et al. Massproducing failures of multimodal systems with language. NeurIPS 2023.\n\n---\n**Q4:** Is it appropriate to combine all tasks to provide a universal paradigm for constructing MM-ICL strategies? \n\n**R4:** Thanks for your constructive feedback. We sincerely think that providing a universal paradigm can help researchers conduct unified and fairer comparisons and studies within a unified framework. For example, in each task of this unified paradigm, researchers can explore how to improve a type of task targetedly to achieve better MM-ICL performance. We will add more discussion in the next version.\n\n---\n**Q5:** Some minor mistakes in the paper.\n\n**R5:** Thanks for your kind mention. We will follow your suggestions to correct these issues one by one (e.g., modifying terminology, highlighting results, and fixing incorrect citation).\n\n---\n**Q6:** For multi-modal demonstration retrieval, why do you use a multi-modal encoder to calculate the similarity instead of using the product of text and vision single modality similarity?\n\n**R6:** Thanks for your insightful feedback. In our experiment, as shown in Table 1 below, we found that the performance is far inferior to cosine similarity, which is also consistent with our conclusion. We attribute it to the fact that the model is more expected to obtain a semantic direction similarity rather than a distance similarity.\n\nWe will add more discussion in the next version.\n\n\n|      | Caption | Caption | VQA | VQA | Classification | Classification | Reasoning | Reasoning | |\n| :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: |\n| | CIDER | BERTScore | Token F1 | BERTScore | Acc | F1 | Accuracy | RAS | AVG |\n| Dot | 10.41 | 86.76 | 18.15 | 84.41 | 58.75 | 40.47 | 52.14 | 91.12 | 55.28 |\n| L2 | 2.58 | 85.3 | 20.85 | 84.67 | 57.98 | 48.95 | 54.96 | 91.5 | 55.85 |\n| Cos | 13.91  | 86.92  | 24.70 | 87.63 | 59.74 | 52.14 | 61.89  | 93.05 | 60.00 |\n||\n\nTable 1: Performance of different similarity metrics on GPT4o.\n\n---\n**Q7:** For the OpenAI GPT-4V model, how do you reconstruct the order of the text and the image? \n\n**R7:** Thanks for your valuable feedback. GPT4V API provides a content list that allows for controlling the order of text and images. An example is as follows: \n\n```json\n{\n  \"model\": \"gpt-4-vision-preview\",\n  \"messages\": [\n    {\n      \"role\": \"user\",\n      \"content\": [\n        {\n          \"type\": \"text\",\n          \"text\": \"What’s in this image?\"\n        },\n        {\n          \"type\": \"image_url\",\n          \"image_url\": {\n            \"url\": f\"data:image/jpeg;base64,{base64_image}\"\n          },\n        }\n        {\n          \"type\": \"text\",\n          \"text\": \"What’s in this image?\"\n        },\n        {\n          \"type\": \"image_url\",\n          \"image_url\": {\n            \"url\": f\"data:image/jpeg;base64,{base64_image}\"\n          },\n\n        }\n      ]\n    }\n  ],\n  \"max_tokens\": 300\n}\n```\n\nWe will add more details in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VsSro0fvZz", "reviewer": "Reviewer_Wrbn", "summary": "The work presents an in-depth analysis of the factors influencing the performance of Multi-modal In-Context Learning (MM-ICL). The authors systematically investigate the core steps of MM-ICL, including demonstration retrieval, ordering, and prompt construction. Utilizing six vision large language models and a variety of strategies, the study uncovers the significance of multi-modal alignment, the importance of intra-demonstration ordering, and the role of introductory instructions in enhancing task comprehension.", "questions": "1. In Section 5.3, can you provide more analysis on why the number of demonstrations does not significantly impact MM-ICL?\n2. In your experiments, how did you handle multiple image inputs?\n3. Will different prompts affect the exploration experimental results?", "limitations": "This work presented some limitations in the submitted paper including: (1) extending the exploration to video modal ICL and (2) multi-lingual multi-modal ICL scenarios.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Comprehensive Analysis: The paper offers a thorough examination of the factors affecting MM-ICL, which is a significant contribution to the field.\n2. Clear Structure: The paper is well-organized, making it easy to follow the research question, methodology, findings, and conclusions.\n3. Potential Impact: The findings in this work can provide a foundational guide for optimizing MM-ICL strategies in future research.", "weaknesses": "1. The captions in the paper can be enriched to help readers gain a better understanding.\n2. In Section 5.3, can you provide more analysis on why the number of demonstrations does not significantly impact MM-ICL?\n3. The experimental section needs to be supplemented with a description of the LLM backbone.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your acknowledgment and interest in our work! We sincerely appreciate your thorough and insightful comments on our work, and we will address each of your main concerns below:\n\n---\n**Q1:** The captions in the paper can be enriched to help readers gain a better understanding.\n\n**R1:** Thanks for your constructive suggestion. We will follow your suggestion to add more detailed captions in the next version.\n\n---\n**Q2:** In Section 5.3, can you provide more analysis on why the number of demonstrations does not significantly impact MM-ICL?\n\n**R2:** Thanks for your insightful feedback. We find that in complex reasoning tasks such as multi-step multi-modal chain-of-thought reasoning, more demonstrations will not lead to better performance, which is consistent with the observation [1]. We think that the following might be reasons why more demonstrations may not improve performance in these tasks:\n1. **Cognitive Overload:** For complex tasks, understanding complex demonstrations is difficult. More demonstrations can overwhelm the model, making it harder to process and integrate information effectively.\n2. **Complexity of Reasoning Tasks:** We observe that for reasoning tasks, the performance improvement brought by the number of demonstrations is not even as good as using different retrievers. It shows that reasoning tasks require sophisticated integration of information, where quality trumps quantity.\n\nWe will add more discussion in the next version.\n\n[1] Chen et al. M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought. ACL 2024.\n\n---\n**Q3:** The experimental section needs to be supplemented with a description of the LLM backbone.\n\n**R3:** Thanks for your constructive suggestion. We will add a detailed description of each LLM backbone used in the next version according to your suggestion.\n\n---\n**Q4:** In your experiments, how did you handle multiple image inputs?\n\n**R4:** Thanks for your insightful comment. There are two main categories for handling multiple image inputs:\n- **Tokenizer-based LVLM:** It supports directly tokenizing the images and then directly concating them with textual tokens in an interlaced sequence. \n- **Encoding-based LVLM:** The soft encoding of images is concated at the embedding layer to form an interlaced sequence of images and text.\n\n---\n**Q5:** Will different prompts affect the exploration experimental results?\n\n**R5:** Thank you for your insightful feedback. In our preliminary experiments, we found that different prompts do not affect the overall conclusions. For example, we used different prompts (including instructions and delimiters) with the same semantics but different expressions. The results are shown in Table 2 below, and it can be seen that the impact of different prompts is not very large. We will add more discussion in the next version.\n\n|      | Caption | Caption | VQA | VQA | Classification | Classification | Reasoning | Reasoning | |\n| :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: |\n| | CIDER | BERTScore | Token F1 | BERTScore | Acc | F1 | Accuracy | RAS | AVG |\n| P1 | 12.03 | 85.85 | 22.53 | 86.67 | 59.93 | 54.62 | 59.52 | 92.04 | 59.15 |\n| P2 | 14.01 | 86.77 | 23.59 | 86.00 | 58.53 | 53.61 | 61.85 | 91.86 | 59.53 |\n| P3 | 13.91  | 86.92  | 24.70 | 87.63 | 59.74 | 52.14 | 61.89  | 93.05 | 60.00 |\n| P4 | 14.44 | 86.48 | 23.14 | 87.77 | 60.23 | 50.48 | 60.54 | 92.27 | 59.42 |\n||\n\nTable 2: Performance across different prompts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v5kVRBVPHS", "reviewer": "Reviewer_jbhA", "summary": "This work explores an interesting research question: “What factors affect the performance of MM-ICL?” To this end, they conduct comprehensive experiments on the three fundamental steps of MM-ICL: demonstration retrieval, demonstration ordering, and prompt construction. In addition, they explore 20 strategies across 4 tasks with 6 representative vision large language models (VLLMs).", "questions": "See above comments.", "limitations": "See above comments.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "（1）This work explores the research question “What factors affect the performance of MM-ICL?” which is an important topic in MM-ICL literature.\n\n（2）This work observes some interesting findings, including (i) multi-modal alignment is the bottleneck for MM-ICL; (ii) intra-demonstration ordering holds greater importance than inter-demonstration ordering; (iii) introductory instruction guides better task understanding for MM-ICL, which can offer a unified view and guideline for researchers to build better MM-ICL.\n\n（3）This work conducts very detailed experiments by exploring 20 strategies across 4 tasks with 6 representative vision large language models (VLLMs), which is impressive and encouraging.", "weaknesses": "Overall, I have no major issues. There are a few points that can be improved: \n(1) The 'Exploration of MM-ICL Prompt Construction' section can be explained more clearly. \n\n(2) Some parts of the appendix, such as the implementation details of the baseline, can be moved to the main text. \n\n(3) The 'Limitations' section can include some specific limitations related to this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your acknowledgment and interest in our work! We sincerely appreciate your thorough and insightful comments on our work, and we will address each of your main concerns below:\n\n---\n**Q1:** The 'Exploration of MM-ICL Prompt Construction' section can be explained more clearly.\n\n**R1:** Thanks for your constructive suggestion. We will follow your suggestion to explain more details about the 'Exploration of MM-ICL Prompt Construction' section, including providing more description in Figure 4 and adding more details in three instruction categories explanation.\n\n---\n**Q2:** Some parts of the appendix, such as the implementation details of the baseline, can be moved to the main text.\n\n**R2:** Thanks for your insightful feedback. We totally agree with your point. We will add the implementation details of the baseline to the main text in the next version.\n\n---\n**Q3:** The 'Limitations' section can include some specific limitations related to this work.\n\n**R3:** Thanks for your insightful suggestion. We will discuss more limitations in the next version. For example, one of limitations in our work lies in the mis-consideration of some image instructions, such as grounding and adding extra arrows. These may require more sophisticated human design and are not supported by most current models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JtGs7SSuww", "reviewer": "Reviewer_fN5J", "summary": "The paper studies what influences multimodal ICL and finds that using multimodal retrievers to find demonstrations, ordering of modalities in the context and introductory instructions help to improve the few-shot performance. The study spans 6 models and several multimodal datasets.", "questions": "Check weaknesses (e.g. 6. 7)", "limitations": "Few limitations are discussed in the paper.", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- The paper addresses an important problem. Multimodal ICL is little explored.\n- The paper conducts experiments spanning several models and datasets.", "weaknesses": "1. The paper claims to be the first to study multimodal ICL. However, previous works [1,2] conducted very similar studies and there is no mention for these previous works.\n2. The paper does not consider standard benchmarks. In addition it uses metrics not generally used to report scores in other papers. For example, the performance on VQAv2 is measured with Accuracy not BERTscore. I think the considered benchmarks and metrics are leading to false conclusions as detailed later, either because these metrics/benchmarks are flawed/limited or just saturated. To support the paper claims it is important to consider a typical evaluation setup. For example on benchmarks such as VQAv2, COCO captioning, NoCaps, TextCaps, OK-VQA, TextVQA ..  using typical scores such as Accuracy, CIDEr/BLEU…\n3. Related to previous points. Table 1 shows that IDEFICS 8B is comparable to GPT4V ! (e.g. VQA BertScore 85.42 GPT4V vs 86.11 IDEFICS few-shot random) IDEFICS 8B is significantly worse on almost all multimodal benchmarks. I am not sure at this point how much I can rely on the paper results and findings.\n4. I have some issues with some claims that I don’t find consistent with previous studies. These claims should be refined or supported with more evidence:\n- “The number of demonstrations does not significantly impact MM-ICL”: usually the more shots the better the performance [3,4,5] and the improvement is significant.\n- “increasing model parameters from 8 billion to over 100 billion does not primarily drive performance improvements, suggesting that multi-modal context understanding and alignment are more crucial for MM-ICL than model scaling”: Other works clearly show that increasing model size increases ICL performance [3,5]. To make such claim, only the scale of the model should change, but in Table 1 the authors seems to compare different models (e.g. GPT4-V vs Qwen-VL 10B)\n\n5. Wrong citations: IDEFICS 8B is not done by Awadallah et al (Table 1) !\n\n6.  The paper figures are not properly explained:\n- What the colors means in Fig 10?\n- Which scores are you showing in Fig 8 and 9?\nPlease write detailed captions to explain the paper figures.\n\n7. What is BELU score? Is it the same as BLEU?\n\n\n\nReferences:\n\n[1] Baldassini, Folco Bertini, et al. \"What Makes Multimodal In-Context Learning Work?.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Chen, Shuo, et al. \"Understanding and Improving In-Context Learning on Vision-language Models.\" ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models (2024).\n\n[3] Alayrac, Jean-Baptiste, et al. \"Flamingo: a visual language model for few-shot learning.\" Advances in neural information processing systems 35 (2022): 23716-23736.\n\n[4] Laurençon, Hugo, et al. \"Obelics: An open web-scale filtered dataset of interleaved image-text documents.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[5] Shukor, Mustafa, et al. \"Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning.\" The Twelfth International Conference on Learning Representations. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough and insightful comments on our work. In the following, **we will clarify your concerns and we would greatly appreciate it if you can reconsider the work in light of our clarification**.\n\n---\n**Q1:** The difference between our work with previous studies [1,2].\n\n**R1:** Thanks for your insightful feedback. We sincerely clarify the main differences from two aspects:\n1. **Unified Perspective.** Our work makes the first attempt to unify the current multi-modal In-context learning (MM-ICL) process, which takes a meaningful step to offer a unified view and guideline for researchers to build better MM-ICL. Previous works [1,2] can also be integrated in our perspective.\n2. **Comprehensive Investigation.** According to the unified perspective, we conduct a comprehensive exploration and provide insightful observations for each process within the unified perspective. However, previous studies [1,2] ***mainly focus on exploring the sample representation subpart of the retrieval process***, which is only a subset of our exploration.\n\nMore intuitive comparisons can be found in Figure 1 of the supplementary material. We will follow your suggestions to add a detailed discussion in the next version.\n\n[1] Baldassini et al. What Makes Multimodal In-Context Learning Work? CVPR 2024 Workshop.\n\n[2] Chen et al. Understanding and Improving In-Context Learning on Vision-language Models. ICLR 2024 Workshop.\n\n---\n**Q2:** The paper does not consider standard benchmarks and metrics.\n\n**R2:** Thanks for your constructive comments and we will answer your concerns point by point.\n- **Benchmark Concern:** We use the standard benchmark M3IT [1], which consists of a series of common datasets divided into four categories. Detailed information is shown in Table 1. We will add more data descriptions in the next version.\n- **Metric Concern:** In our experiment, we use the standard CIDEr metric for image caption task. However, since M3IT includes various VQA tasks with many free-form answers, the Acc. metric is not suitable. Therefore, inspired by the success of free-form and precise answer hybrid evaluation in machine reading comprehension, we use the Token F1 and BERTScore as metrics for evaluating semantics and exact keyword accuracy. We will add more discussion in the next version.\n\n|Dataset|Data Class|\n|:--:|:--:|\n|COCO Caption|IC|\n|TextCap|IC|\n|Paragraph Captioning|IC|\n|COCO Text|CLS|\n|ImageNet Image Classification|CLS|\n|IQA|CLS|\n|Image-Text Matching|CLS|\n|e-SNLI-VE|CLS|\n|Multi-modal Fact Checking|CLS|\n|VQA v2|VQA|\n|DocVQA|VQA|\n|OCR-VQA|VQA|\n|ST-VQA|VQA|\n|Text-VQA|VQA|\n|GQA|VQA|\n|OKVQA|VQA|\n|A-OKVQA|VQA|\n|ScienceQA|R|\n|M3CoT|R|\n||\n\nTable 1: Dataset in M3IT, where IC: Image Captioning, CLS: Classification, VQA: Visual Question Answering, R: Reasoning (with NL rationale).\n\n[1] Li et al. M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning. Arxiv 2024.\n\n---\n**Q3:** Table 1 shows that IDEFICS 8B is comparable to GPT4V! However, IDEFICS 8B is worse on almost all benchmarks. Why？\n\n**R3:** Thanks for your insightful comment. Sorry for confusing you. We totally agree that IDEFICS 8B is worse on almost all benchmarks. However, in our work, Table 1 shows that the results of IDEFICS2 8B [1], not IDEFICS 8B. In fact, IDEFICS2 is comparable with GPT4V, as shown in the following Table 2. We will add more discussion in the next version to make it clearer.\n\n||MathVista|MMBench|\n|:--:|:--:|:--:|\n|GPT4V|49.9[2]|74.3[3]|\n|IDEFICES2-8B|51.6[1]|76.8[1]|\n||\n\nTable 2: Performance comparison.\n\n[1] Laurençon et al. What matters when building vision-language models? Arxiv 2024\n\n[2] Lu et al. MathVista: Evaluating Math Reasoning in Visual Contexts. ICLR 2024\n\n[3] Liu et al. MMBench: Is Your Multi-modal Model an All-around Player? Arxiv 2023\n\n---\n**Q4:** Some claims should be refined or supported with more evidence:\n1. Other works usually show the more shots the better the performance.\n2. Other works show that increasing model size increases ICL performance. \n\n**R4:** Thanks for your valuable suggestions. We will answer your questions point by point.\n- **Response to (1):** Yes, your understanding is correct. In fact, previous works [1-3] concluded that 'more shots lead to better performance' mainly for the VQA task.\nWe also observed this trend in VQA task. But in other tasks like multi-modal chain-of-thought reasoning, it can even worsen results with more demonstrations [4]. In the future, we will follow your suggestion and provide more fine-grained conclusions, such as offering fine-grained observations for different tasks. \n- **Response to (2):** We totally agree with you. With the same model, a larger model generally performs ICL better. In our work, we were surprised to find that IDEFICS2 (8B) outperformed GPT-4V (>100B) on some tasks, indicating that beyond parameter size, multi-modal context understanding and alignment are also crucial for MM-ICL. We will refine our conclusion in the next version according to your suggestion.\n\n[1] Alayrac et al. Flamingo: a visual language model for few-shot learning. NeurIPS 2022.\n\n[2] Laurençon et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. NeurIPS 2024.\n\n[3] Shukor et al. Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning. ICLR 2023.\n\n[4] Chen et al. M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought. ACL 2024.\n\n---\n**Q5:** Other writing typos and tips (e.g., wrong citations; paper figures are not properly explained)\n\n**R5:** Thanks for your valuable feedback. We will follow your suggestions to thoroughly fix wrong citations and add detailed figure captions. Specifically, different colors denote different models in Fig 10. The scores in Fig 8 and 9 represent the Average score (AVG).\n\n---\n**Q6:** What is BELU score?\n\n**R6:** Sorry for the confusion. It is BLEU. We will polish our work in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for your insightful and thoughtful feedback.

1. We are greatly encouraged that all reviewers observe that our work addresses **an important research topic** by conducting a thorough exploration of the factors affecting MM-ICL (Reviewer #Pm2y, Reviewer #Wrbn, Reviewer #jbhA, Reviewer #fN5J).
2. We are pleased that reviewers found that our work provides some **insightful and interesting findings**, which can offer **a foundational guide for optimizing MM-ICL strategies in future research** (Reviewer #Pm2y, Reviewer #Wrbn, Reviewer #jbhA).
3. We are also glad that all reviewers found that our work conducts **comprehensive analysis** by exploring 20 strategies across 4 tasks with 6 representative models which is impressive and encouraging (Reviewer #Pm2y, Reviewer #Wrbn, Reviewer #jbhA, Reviewer #fN5J).

We will address all concerns to polish our work according to reviewers’ comments in the next version. Thanks once again for the valuable contributions of all the reviewers.

Meta-review: This submission presented a multimodal framework for In-Context Learning (MMICL). The authors conducted extensive experiments on the three stages of MM-ICL (demonstration retrieval, scheduling and prompting) to explore the factors that affect MMICL performance.  
The reviewers raised many questions. The authors were able to answer a number of them. And there was a rich discussion in particular on the type of evaluation carried out to validate the ICL hypotheses, and on the conclusions that are drawn from these “non-standard” Vision-Language evaluations.  
In the end, the reviewers were on the whole very convinced. The AC agreed that the strengths in this case outweighed the weaknesses. The final recommendation is to accept. Authors are strongly encouraged to take all comments into account in their final version.
Decision: Accept (poster)