What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration

Authors: libo  qin
qiguang  chen
hao  fei
zhi  chen
min  li
wanxiang  che
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 14:22:33

Abstract: Recently, rapid advancements in Multi-Modal In-Context Learning (MM-ICL) have achieved notable success, which is capable of achieving superior performance across various tasks without requiring additional parameter tuning. However, the underlying rules for the effectiveness of MM-ICL remain under-explored. To fill this gap, this work aims to investigate the research question: "_What factors affect the performance of MM-ICL?_" To this end, we investigate extensive experiments on the three core steps of MM-ICL including demonstration retrieval, demonstration ordering, and prompt construction using 6 vision large language models and 20 strategies. Our findings highlight (1) the necessity of a multi-modal retriever for demonstration retrieval, (2) the importance of intra-demonstration ordering over inter-demonstration ordering, and (3) the enhancement of task comprehension through introductory instructions in prompts. We hope this study can serve as a foundational guide for optimizing MM-ICL strategies in future research.

Main paper: 
 introduction: 
 Recently, Large Language Models (LLMs) have demonstrated remarkable advancements, showcasing proficiency in a wide range of tasks [Zhao et al., 2023a , Qin et al., 2023 , 2024 , Hu et al., 2023 , Pan et al., 2023] . Notably, advanced LLMs exhibit the emergence of novel capabilities such as In-Context Learning (ICL) [Wei et al., 2022a , Dong et al., 2022 , Zhuang et al., 2023] , which optimize task performance by incorporating demonstrations into input prompts [Giannou et al., 2023 , Li et al., 2023d , Wies et al., 2023 , Zhou et al., 2022] . In particular, multi-modal in-context-learning (MM-ICL) is capable of utilizing multi-modal demonstrations to quickly adapt to the downstream task without parameter tuning [Yin et al., 2023 , He et al., 2023 , Zhang et al., 2024 , Li and Lu, 2024 ].In the literature, a series of works emerge to enhance MM-ICL. Specifically, Gong et al. [2023] manually create a general template with multiple images and corresponding responses during instructiontuning (IT) stage to improve MM-ICL. Tsimpoukelli et al. [2021] , Li et al. [2023b] , Doveh et al. [2024] and Zhao et al. [2024] develop task-specific MM-ICL templates during the IT stage, further extending its capabilities across more domains. Li et al. [2023a] introduce OtterHD, adapting MM-ICL for high-definition image tasks. Furthermore, Sun et al. [2023] and Tian et al. [2024] explore the potential of MM-ICL in the image generation tasks. Jin et al. [2024] provide compelling evidence for the effectiveness of MM-ICL in comprehending game instructions. Zong et al. [2024] and Shukor et al. [2024] develop fine-grained benchmarks and evaluate the MM-ICL in classification tasks.While significant progress has been witnessed in MM-ICL, the existing work still mainly focuses on how to optimize MM-ICL, ignoring the underlying factors that influence its effectiveness and performance. Such gap impedes a comprehensive understanding of the mechanisms and performance determinants of MM-ICL, thereby limiting further exploration and research in this field. Motivated by this, this paper aims to systematically investigate the research question: What factors affect the performance of MM-ICL?, hoping to offer a unified view and guideline for researchers to build better MM-ICL. Specifically, as illustrated in Figure 1 , the MM-ICL process comprises three steps: demonstration retrieval, demonstration ordering, and prompt construction. Therefore, We systematically investigate the following sub-questions: (a) how to select multi-modal demonstrations (Sec. 3.1); (b) how to order multi-modal demonstrations (Sec. 3.2); and (c) how to construct MM-ICL prompts (Sec. 3.3) to this end. To achieve this, we conduct detailed experiments on MM-ICL using 20 strategies across 4 tasks with 6 representative vision large language models (VLLMs).Through extensive investigations, the main findings are as follows:• Multi-modal alignment is the bottleneck for MM-ICL. Our analysis confirms that, on average, multi-modal retrieval methods outperform single-modal ones. Furthermore, multimodal alignment in VLLMs has a greater impact on MM-ICL effectiveness than parameter size, identifying alignment as the key limitation in both backbone structure and demonstration quality. • Intra-demonstration ordering holds greater importance than inter-demonstration ordering.Our investigation first indicates that the intra-demonstration ordering, particularly the ordering of modalities, greatly influences model performance more than demonstration arrangement. • Introductory instruction guides better task understanding for MM-ICL. To construct a comprehensive MM-ICL prompt, it is essential to include introductory instructions preceding the demonstrations. This approach consistently enhances the performance of MM-ICL campared with summative instruction placed after demonstrations, and intra-demonstration instruction. 
background: 
 In this work, we formally present the prompt building process for MM-ICL. As depicted in Figure 1 , the process of prompt building for MM-ICL involves three sequential stages:(1) Demonstration Retrieval: The core MM-ICL requires retrieval to obtain demonstrations that can help MM-ICL. Formally, given a validation dataset V n = {x 1 , x 2 , . . . , x n }, each multi-modal sample x i includes textual input I txt i , visual input I vis i , and output O i . For a specific test query q, this step aims to identify a subset of relevant demonstrations C k = {x πj } k j=1 , where x πj ∈ V n .(2) Demonstration Ordering: Researches [Lu et al., 2022b , Wu et al., 2023 , Xiang et al., 2024] show that LLMs are highly sensitive to the order of demonstrations. Thus, arranging these demonstrations effectively is crucial for MM-ICL. After retrieving relevant demonstrations, we must rearrange the sequence L k = [x σj ] k j=1 , which will be used to construct the prompt.(3) Prompt Construction: Previous research indicates that using delimiters and instructions can significantly enhance textual ICL capabilities [Min et al., 2022 , Qin et al., 2023] . Therefore, the final core step is to transform the ordered demonstrations into a structured prompt P, incorporating delimiters and instructions to optimize MM-ICL.3 What Factors Affect Multi-modal In-Context Learning? 
exploration of mm-icl demonstration retrieval: 
 The efficacy of ICL heavily depends on the quality of the retrieved demonstrations C, which provide essential prior knowledge for MM-ICL. As illustrated in Figure 2 , the retrieval process encompasses Sample:𝑥 𝜋1 … Sample:𝑥 𝜋𝑘 Sample Representation. It involves defining an encoder (Encoder(•)) to map each input sample x j ∈ V and user query q into a shared representation space:EQUATIONSpecifically, we evaluate various encoder architectures across modalities, focusing on the impact of visual encoder (Encoder vis ), text encoder (Encoder txt ), and multi-modal encoder (Encoder multi ) on model performance.Sample Comparison. After deriving the representations, we employ a metric M to evaluate the quality Q j of the sample h j in comparison to the query representation h q and the dataset samples h j :EQUATIONSpecifically, we explore various comparison metrics, including cosine similarity M cos [Liu et al., 2022a] , L2 similarity M L2 [Liu et al., 2022a] , and semantic diversity M div [Li and Qiu, 2023a] , to assess sample quality and understand the correlation with model performance.Sample Selection. After quality assessments, we apply a selection criterion S to identify the k most advantageous samples x πj for inclusion in the demonstration set C:EQUATION)Sample selection is guided by factors such as domain information [He et al., 2023] , demonstration style [Agrawal et al., 2023] , and token distance [Liu et al., 2022a] . Specifically, we systematically examine samples from both in-domain and out-of-domain collections. And we also assess the impact of image style on the selected demonstrations. Further, we investigate the token distance between modalities to understand its effects on sample selection for MM-ICL.Following Lu et al. [2022b] and Wu et al. [2023] , the order of the demonstration set C significantly impacts MM-ICL performance. As shown in Figure 3 , this section explores two key aspects:Intra-demonstration Ordering. The sequence within a demonstration, especially modalities (e.g., text and image), is an important component that might affect the MM-ICL capabilities. Therefore, we introduce a intra-demonstration ordering permutation (IOP) to define this sequence:EQUATIONWe conduct a systematic exploration of various IOP configurations, including text-image-text (IOP tvt ), text-text-image (IOP ttv ), and image-text-text (IOP vtt ). These order analyses aim to evaluate the impact of different modal sequences on the model's performance.Inter-demonstration Ordering. The sequence in which demonstrations are organized within C also is the key component that might impact the performance of MM-ICL. Formally, we define a sample ordering permutation σ j to specify the arrangement:EQUATIONwhere x σj represents the j-th demonstration in the ordered demonstration list.VLLMs are highly sensitive to input instructions [Kojima et al., 2022 , Qin et al., 2023] . Inspired by this, to enhance task comprehension, we incorporate different instructions to explore the performance influence for MM-ICL. Formally, we construct instruction methods I(•) that describe the task and position them within the prompt. The prompt construction process is:EQUATIONSpecifically, as shown in Figure 4 , we explore three instruction categories to bolster MM-ICL process:• Introductory Instruction (I intro ) refers to the initial guidance that offers an overview of the task prior to any demonstrations. As shown in Figure 4 (a), this instruction, denoted as I intro , is positioned at the start of the ordered demonstration sequence, L. • Summative Instruction (I sum ) offers a summary after the examples, guiding the model to apply the learned concepts to real-world problems. As shown in Figure 4 (b), this instruction I is added at the end of the demonstration list L.• Intra-demonstration Instruction (I intra ) embeds task instructions within each demonstration, helping VLLMs understand the task requirements during the learning process. As shown in Figure 4 (c), this instruction I is included within each demonstration x i in the list L.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [No] Justification: The code for exploratory prompt work generally does not need to be released, and readers can easily use the prompts we report to directly reproduce the results.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setup: 
 Following the setting of Li et al. [2023c] , we systematically explore 4 tasks, including image-caption, visual question answering (VQA), image classification, and chain-of-thought reasoning, which come from M 3 IT [Li et al., 2023c] and M 3 CoT [Chen et al., 2024b] (as shown in Tables 2), providing a universal paradigm can help researchers conduct unified and fairer comparisons and studies within a unified framework. In order to evaluate the MM-ICL performance accurately, we use two indicators for each task. Following Zhang et al. [2019] , Li et al. [2023b] , and Zong et al. [2024] , we use CIDER [Vedantam et al., 2015] and BertScore [Zhang et al., 2019] as image-caption metrics. Since M 3 IT includes various VQA tasks with free-form answers, inspired by the success of free-form and precise answer hybrid evaluation in machine reading comprehension, following Rajpurkar et al.[2016], Zhang et al. [2019] , we adapt Token-F1 [Rajpurkar et al., 2016] and BertScore as visual question answering (VQA) metrics (The correlation analysis of the indicators and accuracy as shown in Table 3 ). Following Li et al. [2023c,b] , we use accuracy and F1 score as indicators of image classification. Following Lu et al. [2022a] , Golovneva et al. [2022] and Qin et al. [2023] , we use accuracy and reasoning alignment score [Golovneva et al., 2022] (RAS) as indicators of reasoning.To ensure rigorous experimental control, we established a baseline using a multi-modal encoder for data representation and cosine similarity for sample comparison, limiting retrieval to within the same task. This baseline ranks samples based on similarity, with a delimiter and a 3-shot setting (see Appendix A for details). In addition, all open source models complete inference on 2 A100 80G. For all experiments, we select top-p from {0.95, 1} and adjust the temperature parameter within [0, 1]. Among them, temperature is the main error variable in this work.5 Empirical Analysis of Factors Affecting MM-ICL 5.1 Empirical Analysis of MM-ICL Demonstration Retrieval 
sample representation: 
 Multi-modal alignment is the bottleneck for MM-ICL in both backbones and demonstrations.To evaluate the impact of semantic representation in different modalities for MM-ICL, we assessed three distinct encoders: RoBERTa [Liu et al., 2019] as a textual encoder for Textual Retriever, CLIP-Vision Encoder [Radford et al., 2021] for Visual Retriever, and BridgeTower [Xu et al., 2023] as multi-modal encoder for Multi-Modal Retriever. As illustrated in Table 1 , multi-modal retrieval consistently outperforms zero-shot, randomly selected, and single-modality methods, highlighting the advantages of multi-modal semantic learning for MM-ICL. What's more, as shown in Table 1 , our results reveal that increasing model parameters from 8 billion to over 100 billion does not significantly enhance performance, suggesting that beyond parameter size, multi-modal context understanding and alignment are more crucial for MM-ICL than model scaling. Our analysis demonstrates that multi-modal alignment is the critical factor in both the backbone and demonstrations.Current multi-modal encoders still lack modeling of multi-modal logic. Actually, multi-modal retrieval attains better performance in many scenarios like Image Caption and VQA. However, our experiments show that textual retrieval works well for classification and reasoning tasks. Based on the qualitative analysis, we observe that due to the semantic richness of the labels and rationales, textual retrieval can obtain more similar samples. However, the current multi-modal retrieval struggles with complex text semantics, often favoring image similarity. This aligns with recent work [Tong et al., 2023 , 2024 , Fei et al., 2024c] , which is valuable for future exploration. significant compared to text-only ICL scenarios. Specifically, retrieved demonstrations yield an average performance boost of 3.84%, compared to random demonstrations. In contrast, text-only scenarios show performance increases of over 10% with carefully selected samples [Shi et al., 2023] . Furthermore, the model remains unaffected by irrelevant samples, and the performance of almost all models is higher than zero-shot. This indicates that multi-modal context significantly reduces the need for careful demonstration selection, unlike in text-only scenarios.VLLMs learn semantic representations instead of token pattern representations for MM-ICL.As depicted by Agrawal et al. [2023] , textual ICL primarily learns token patterns (e.g., similar output formats, reasoning paths) among demonstration outputs. To investigate whether VLLMs rely on repetitive token patterns, we utilize the average BLEU score across demonstration outputs as a representation of token repetition. Figure 5 shows that only the image captioning task exhibits a positive correlation. In contrast, other tasks show a decline as BLEU scores exceed 30%. This underscores that MM-ICL primarily learns semantic rather than token pattern representations for effective performance.To further analyze the influencing factors of MM-ICL in sample retrieval, this study employs similarity and diversity metrics, which help assess how MM-ICL processes sample similarities and differences, enhancing our understanding of its mechanisms. See Appendix B for more details and results. Cosine similarity matters for sample comparison. Following Liu et al. [2022b] , we compare two representative similarity metrics, cosine similarity and L2 similarity. As shown in Figure 6 (a), cosine similarity, which measures the directional semantic alignment, emerges as the superior metric in MM-ICL than L2 similarity. Supported by Deza et al. [2009] and Steck et al. [2024] , it indicates that MM-ICL prioritizes semantic directional consistency over complete semantic alignment.Diversity does not show significant influence for sample comparison. He et al. [2023] , Li and Qiu [2023b] have shown that demonstrations with better diversity can effectively improve textual ICL.To explore whether it exists in MM-ICL, following Li and Qiu [2023b] , we ultilize the "diversity retriever", which selects the top-10 samples and further chooses the best 3 samples based on semantic diversity to obtain a more diverse MM-ICL. As demonstrated in Figure 6 (b), although diversity significantly enhances performance in text-based ICL, our experiments show limited improvement in MM-ICL tasks. This suggests that diversity may not directly correlate with better MM-ICL.Domain interval matters for sample selection. Prior research highlights the critical role of domain relevance in enhancing ICL performance. Inspired by this, we employ the multi-modal retriever to select samples from both in-domain and out-of-domain pools. Figure 7 (a) shows a nearly 4% performance drop when out-of-domain demonstrations are included, underscoring the necessity of in-domain demonstrations for optimal MM-ICL.Visual style is not a crucial factor in sample selection. Although stylistic similarity in text samples is known to bolster ICL, its effect on the visual modality remains ambiguous. Utilizing CLIP for image classification, we investigate the impact of stylistic coherence in multi-modal samples on MM-ICL performance. As depicted in Figure 7 (b), significant enhancements are observed solely in the VQA task, while captioning and classification show minimal effects and reasoning tasks decline. This indicates that diverse visual styles are not crucial in general MM-ICL.Token distances between modalities need to be considered for different tasks to improve sample selection. For textual ICL, excessive token distance between samples can impede performance [Liu et al., 2022a] . We extend this inquiry to MM-ICL, analyzing how token distance across modalities influences results. Specifically, during the sample selection process, we considered the impact of the average token distance between two images on the model within the entire prompt of MM-ICL. As illustrated in Figure 7 (c), the effect of token distance varies by task, typically showing an initial performance increase followed by a decline as distance grows, particularly in non-captioning tasks. This highlights the task-dependent nature of optimal token distance in MM-ICL.Intra-demonstration ordering significantly impacts performance. Within the demonstration, organizing the ordering, especially the relationship between modalities is a crucial topic. We investigate this by arranging inputs and outputs across modalities using three methods: text input→text output→image input (Text-Image), text input→image input→text output (Text-Image-Text), and image input→text input→text output (Image-Text). As shown in Figure 8 (a), positioning the image at the start significantly enhances model performance. This suggests that presenting visual information first improves multi-modal comprehension, thereby boosting its learning abilities.Inter-demonstration ordering demonstrates minimal impacts. Following Lu et al. [2022c] , we investigate how the order of demonstration presentation influences model efficacy. We explore various strategies: random rearrangement, a "similar-last" approach where samples similar to the query are shown last, and a "similar-first" approach where similar samples are presented first. Figure 8 (b) illustrates that inter-demonstration ordering has a negligible impact on MM-ICL performance. This suggests the order-robustness, with the presentation sequence having minimal effect. Introductory Instruction is consistently effective for better MM-ICL. To investigate the impact of inserting task-related instructions within prompts, we conduct the following experiment on three categories of instruction: Introductory Instruction, Summative Instruction, and Intra-demonstration Instruction. As depicted in Figure 9 , our analysis indicates that introductory instructions stably enhance model performance. In contrast, other instructions generally decrease performance. This finding suggests that introductory instructions facilitate targeted contextual learning and more effective semantic comprehension in demonstrations. We show more prompts and details in Appendix C.3.MM-ICL is affected by the number of demonstrations depending on the task. Contrary to traditional text-based ICL, where performance improves with more samples, our findings in Figure 10 (a) suggest that MM-ICL does not experience significant gains from more demonstrations. To further understand the reason behind, we analysis the performance on different tasks. As shown in Figure 10 (b), increasing the number of demonstrations enhances performance in caption and VQA tasks, a trend also reported in prior studies [Alayrac et al., 2022 , Laurençon et al., 2024a , Shukor et al., 2024] . However, performance declines when demonstrations exceed three across all tested VLLMs. In more complex reasoning tasks, such as multi-step multi-modal chain-of-thought reasoning, additional demonstrations do not yield effective improvements, aligning with the findings of Chen et al. [2024b] , and Fei et al. [2024a] .Moreover, we attribute it to the following reasons for this limitation: (1) Cognitive Overload: For complex tasks, understanding numerous demonstrations can overwhelm the model, impeding its ability to process and integrate information effectively [Chen et al., 2024a] .(1) Complexity of Reasoning Tasks: In reasoning tasks, the performance improvement from more demonstrations is often less pronounced than when using diverse retrievers. This suggests that reasoning tasks require sophisticated integration of information, where quality outweighs quantity. See Appendix C.1 for more detailed description.The importance of delimiter lessens by text-image interleaved demonstrations. Previous research suggests that specific delimiters for input and output data can demonstrably influence textual ICL capabilities [Min et al., 2022] . Therefore, we utilize ablation experiments to omit these delimiters to examine their necessity (see Appendix C.2 for details). As shown in Figure 11 , the resulting minor performance decline suggests that while these delimiters are less critical in MM-ICL, the modality switch inherent to MM-ICL may serve as an implicit delimiter, compensating for the absence of explicit delimiters. 
related work: 
 Recent advancements in vision large language models (VLLMs) have achieved great success in various vision-language tasks [Yin et al., 2023 , Wu et al., 2024a ,b, Wang et al., 2024 , Fei et al., 2024b] . Initially, VLLMs lack Multi-modal In-context Learning (MM-ICL) capabilities. To address this, researchers explore incorporating MM-ICL directly into the training phase. This involves constructing training samples with multi-modal interleaved data by manual and general templates, which unlock the MM-ICL capability [Alayrac et al., 2022 , Awadalla et al., 2023] . Building on this, Li et al. [2023b] , Doveh et al. [2024] and Zhao et al. [2024] extend the MM-ICL to construct a series of task-specific templates, which improves generalization for MM-ICL. Further, Li et al. [2023a] introduce OtterHD and adapt the former process for high-definition images. The potential of MM-ICL is further explored in scene text recognition, image generation, and game instructions [Zhao et al., 2023b , Sun et al., 2023 , Jin et al., 2024] .Recognizing the effectiveness of MM-ICL, researches shift towards prompt optimization. These methods focus on directly optimizing multi-modal prompts to understand the task and generate the expected output, without parameter adjustments [Gong et al., 2023 , Tsimpoukelli et al., 2021 , Li et al., 2023b] . This approach has significantly improved performance in visual reasoning tasks [Yang et al., 2022 , Zheng et al., 2023] . Another approach involves textualizing visual information to enable VLLMs to leverage their background knowledge through in-context learning, further enhancing visual reasoning [Yang et al., 2023 , Lu et al., 2024 , Gupta and Kembhavi, 2023 , Shen et al., 2024] . In addition, in order to better explore the MM-ICL, Zong et al. [2024] and Shukor et al. [2024] also provide a dataset to test the MM-ICL capabilities of the multi-modal classification. Furthermore, Shukor et al. [2024] take the first step to conduct an instruction modification exploration for MM-ICL.Meanwhile, Baldassini et al.[2024], Chen et al. [2024c] pioneer the first naive multi-modal retrieval exploration to enhance MM-ICL. Different from the existing work, our study mainly focuses on a systematic exploration of the effectiveness of key factors influencing the effectiveness of MM-ICL in a unified perspective. To this end, we conduct a detailed analysis and exploration on 6 VLLMs and 20 factors across 4 tasks, aiming to provide systematic and practical guidance for future research.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: As detailed in Section 4, we have thoroughly described the experimental setups for all our explorations.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
discussion: 
 Broader Impacts. Our work is the first to systematically explore the factors influencing MM-ICL. We aim to enhance the understanding of MM-ICL mechanisms and guide future developments in this field. Additionally, our findings could foster a more comprehensive comprehension of MM-ICL within the community. For social impact, this research may influence the creation of more effective multi-modal large language models and relevant applications.Limitations & Future Work. Due to time and cost constraints, this work is limited to the exploration of image and text modalities. In future research, we can extend our exploration to video modal ICL and multi-lingual MM-ICL scenarios. Another limitation of this work involves the insufficient consideration of certain image instructions, such as grounding or the inclusion of additional arrows. These aspects often require more complex human input and are not adequately supported by most current models.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Error bars are shown in Figure 5 and Figure 7 , with an explanation of the error variables provided in Section 4. However, we do not report error bars for all tasks due to the high costs associated with human annotation and computational resource consumption.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We have addressed the broader impacts of our work in Section 7. Additionally, as our research is primarily an empirical exploration and poses no additional social risks, we have not included a discussion on potential harmfulness. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion: 
 This study is the first to systematically explore MM-ICL by identifying key performance determinants.Our experiments with 6 models and 20 factors across 4 tasks show that multi-modal retrieval significantly outperforms single-modal approaches and the intra-demonstration ordering critically influences learning efficacy. Additionally, incorporating task-specific instructions into prompts enhances model performance. We hope these findings will refine our understanding of MM-ICL mechanisms and guide more effective developments and future research in this evolving field. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: The paper does not use existing assets. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: To ensure rigorous control of experimental variables, we establish a standard baseline for our study. This baseline utilizes a multi-modal encoder for data representation and cosine similarity for sample comparison, with retrieval restricted to the same task. The following sections provide detailed insights into the implementation of this baseline.Multi-modal Encoder for Sample Representation We employ BridgeTower [Xu et al., 2023] as a multi-modal encoder to represent the data in a unified embedding space. This encoder integrates both visual and textual information, prioritizing single modality to capture the rich semantic content present in images.To compare samples effectively, we use cosine similarity, a metric that measures the cosine of the angle between two non-zero vectors in a multidimensional space. This choice is motivated by its effectiveness in capturing the similarity between high-dimensional vectors, which are typical outputs of our multi-modal encoder. Specifically, we compute the cosine similarity between the query and each candidate sample, which is given by: cosine(h q , h i ) = h q • h i ∥h q ∥∥h i ∥ where h q and h i are the embedding vectors of the query q and candidate sample x i , respectively.To ensure the relevance and accuracy of the retrieval process, for sample selection, we first confine retrieval to the same task and domain. This means that comparisons and rankings are conducted exclusively among samples within the same task and domain category, ensuring the contextual appropriateness of the retrieved results.In addition, for sample selection, samples are ranked according to their cosine similarity scores. Higher similarity scores indicate a closer alignment with the query sample, enabling the efficient identification of the most relevant samples. This ranking process involves two main steps: (1) Sorting: Candidate samples are sorted in descending order based on their cosine similarity scores relative to Dataset Category COCO Caption [Chen et al., 2015 ] IC TextCaps [Sidorov et al., 2020] IC Paragraph Captioning [Krause et al., 2017] IC COCO Text [Veit et al., 2016 ] CLS ImageNet Image Classification [Russakovsky et al., 2015] CLS IQA [Duanmu et al., 2021] CLS COCO-ITM [Chen et al., 2015 ] CLS e-SNLI-VE [Kayser et al., 2021] CLS Mocheg [Yao et al., 2023] CLS VQA-v2 [Goyal et al., 2017] VQA DocVQA [Mathew et al., 2021] VQA OCR-VQA [Mishra et al., 2019] VQA ST-VQA [Biten et al., 2019] VQA Text-VQA [Singh et al., 2019] VQA GQA [Hudson and Manning, 2019] VQA OKVQA [Marino et al., 2019] VQA A-OKVQA [Schwenk et al., 2022] VQA ScienceQA [Lu et al., 2022a ] R M 3 CoT [Chen et al., 2024b] R Table 2 : Dataset in M 3 IT and M 3 CoT, where IC: Image Captioning, CLS: Classification, VQA: Visual Question Answering, R: Chain-of-Thought Reasoning (with NL rationale). Due to the cost, for each task, we evenly sampled 500 items according to the sub-dataset.the query. (2) Selection: Subsequently, the top-k ranked samples are selected based on their relevance as determined by the similarity scores.By default, we utilize the methodology for ordering demonstrations within our baseline model. By default, we adopt a text-after-image (Text-Image) approach for intra-demonstration sorting. This means that, within a single demonstration, textual information is positioned after the corresponding image. This ordering is chosen based on preliminary findings suggesting that such a sequence aids in better contextual understanding and retention of the demonstrated information.Furthermore, for the ordering of inter-demonstration sequences, we employ a similarity-based method. This method ranks demonstrations according to their similarity to the query, with more similar demonstrations placed higher in the order. The similarity is determined using a metric that assesses the alignment of key features between the query and the demonstrations. This approach ensures that the most relevant and contextually aligned demonstrations are prioritized, potentially enhancing the model's performance and the user's comprehension.To ensure consistency and comparability in our baseline, we introduce both a delimiter and a 3-shot setting (following Wei et al. [2022b] , Qin et al. [2023] ). The delimiter serves to clearly demarcate different segments of the input data, preventing any potential confusion or overlap between distinct portions of the input. This clear separation is crucial for the model to accurately process and understand the structure of the data it receives.The 3-shot setting, on the other hand, involves providing three examples for each task within the prompt. This approach is designed to stabilize the learning process by presenting the model with sufficient contextual information. By offering three examples, we strike a balance between providing enough context to guide the model's understanding and avoiding the cognitive overload that might occur with too many examples. This setting not only enhances the model's performance but also ensures a more robust and reliable learning process.In the context of using Vision-and-Language Large Models (VLLMs), it is essential to carefully structure the input prompts to ensure accurate processing. The prompt format typically used is illustrated below:[REQUEST] % Shot 1 <Visual Input I vis 1 > <Textual Input I txt 1 > [RESPONSE] <Textual Output I vis 1 > [REQUEST] % Shot 2 <Visual Input I vis 2 > <Textual Input I txt 2 > [RESPONSE] <Textual Output I vis 2 > [REQUEST] % Shot 3 <Visual Input I vis 3 > <Textual Input I txt 3 > [RESPONSE] <Textual Output I vis 3 > [REQUEST] % User Query <Visual Input I vis q > <Textual Input I txt q >where any gray text following the percent sign (%) is treated as a comment. These comments are not processed as part of the primary input but serve to provide additional context or instructions within the coding environment. This convention helps in maintaining the clarity and functionality of the given prompting.In conclusion, the standard baseline established here integrates a multi-modal encoder, cosine similarity, and task-specific retrieval with a focus on visual modalities. It ranks samples based on similarity and employs a delimiter with a 3-shot setting to ensure robust and consistent performance across different tasks.Cosine Similarity (M cos ) Compute the cosine similarity between h q and h j using the formula:M cos (h q , h j ) = h q • h j ∥h q ∥∥h j ∥ (7)L2 Similarity (M L2 ) Calculate the L2 similarity by computing the negative Euclidean distance between h q and h j :M L2 (h q , h j ) = -∥h q -h j ∥ 2 (8)Since Euclidean distance measures dissimilarity, we use the negative value to represent similarity, where a higher value indicates greater similarity.Semantic Diversity (M div ) Semantic diversity is assessed by evaluating the differences in the distributional properties of h q and h j . This assessment involves analyzing the variance in how these properties are distributed across different samples. To determine the presence of semantic diversity within Multi-Modal In-Context Learning (MM-ICL), we adopt the methodology proposed by Li and Qiu [2023b] . Specifically, we employ the "diversity retriever," designed to enhance the diversity of the selected samples. The diversity retriever operates by first selecting the top 10 samples based on a preliminary measure of relevance. From these top 10 samples, it then identifies the 3 samples that exhibit the highest semantic diversity. This two-step process ensures that the final selection of samples for MM-ICL is not only relevant but also diverse in terms of their semantic content.Comparing the results obtained using different metrics (M cos , M L2 , M div ) provides a comprehensive understanding of their effectiveness and suitability for specific applications. It is essential to analyze the trade-offs associated with each metric and interpret the results to draw meaningful conclusions about sample quality and relevance.As shown in Figure 12 , cosine similarity, which measures directional semantic alignment, emerges as the superior metric in MM-ICL compared to L2 similarity. This observation is supported by the findings of Deza et al. [2009] and Steck et al. [2024] , who highlight that MM-ICL prioritizes semantic directional consistency over complete semantic alignment. Cosine similarity's ability to capture the nuances of directional alignment allows for more precise interpretations of semantic relationships within the data, making it particularly effective for MM-ICL tasks.In contrast, Figure 13 illustrates that while diversity, as measured by M div , enhances performance in text-based in-context learning, our experiments reveal limited improvement in MM-ICL tasks. This finding suggests that diversity may not directly correlate with better performance in MM-ICL. The limited impact of diversity on MM-ICL performance could be attributed to the specific nature of multi-modal data, where the interplay between different modalities requires a more nuanced approach than simply maximizing diversity.Further analysis of these metrics reveals the inherent trade-offs between them. For instance, while cosine similarity offers advantages in maintaining semantic directional consistency, it may not capture the full extent of semantic similarity that L2 similarity can provide. On the other hand, L2 similarity, though comprehensive in measuring complete alignment, might lack the precision needed for tasks that rely heavily on directional semantic cues. Similarly, while diversity is beneficial in certain contexts, its role in MM-ICL needs to be reconsidered, potentially focusing on optimizing other aspects of sample quality.In summary, the evaluation of M cos , M L2 , and M div underscores the importance of selecting appropriate metrics based on the specific requirements of the task. Understanding the trade-offs and context-specific effectiveness of these metrics is crucial for optimizing performance in multi-modal in-context learning applications. Specifically, we have designed distinct instructions tailored to different types of tasks, ensuring clarity and appropriateness for each unique context. For image captioning tasks, the prompt is:Please provide a caption for the image following the structure of the provided example.In this context, the objective is to generate descriptive captions that accurately reflect the content and context of the image. For Visual Question Answering (VQA) tasks, our prompt is:Examine the image and answer the question by closely following the structure shown in the example provided.The VQA tasks require the model to analyze visual content and respond to specific queries. By following the example, users can produce answers that are precise and directly related to the visual stimuli. For image classification tasks, the prompt is:Carefully review the image and categorize it based on the options provided in [REQUEST],following the classification format illustrated in the example.Image classification involves categorizing images into predefined classes based on visual content. The provided example demonstrates the expected classification format. For chain-of-thought reasoning tasks, the prompt is:Carefully review the given image and the associated text. Utilize the reasoning format illustrated in the provided examples, breaking down your thought process. Ensure that each reasoning step is explicitly connected to observable details in the image or text, and articulate your conclusion in a clear and logical manner.Chain-of-thought reasoning tasks require a more complex interaction between visual and textual information. The prompt encourages users to break down their reasoning process into clear, logical steps, each supported by specific details from the image or text.Furthermore, we explore three categories of instructions to enhance the MM-ICL process:Introductory Instruction (I intro ) This instruction provides an overview of the task before presenting any demonstrations. As depicted in Figure 4 (a), the introductory instruction I intro is positioned at the beginning of the ordered demonstration list L. This setup aims to set the context for the subsequent examples. Specifically, the overall prompt template is as follows:<Instruction I intro > [DEMONSTRATIONS] [REQUEST] % Shot 1 <Visual Input I vis 1 > <Textual Input I txt 1 > [RESPONSE] <Textual Output I vis 1 > • • • [QUERY] [REQUEST] % User Query <Visual Input I vis q > <Textual Input I txt q >Summative Instruction (I sum ) This instruction offers a summary after the examples, guiding the model to apply the learned concepts to real-world problems. As shown in Figure 4 (b), the summative instruction I is added at the end of the demonstration list L. This helps in reinforcing the learning objectives and expected outcomes. Specifically, the overall prompt template is as follows:<Instruction I intro > [DEMONSTRATIONS] [REQUEST] % Shot 1 <Visual Input I vis 1 > <Textual Input I txt 1 > [RESPONSE] <Textual Output I vis 1 > • • • In summary, <Instruction I sum > [QUERY][REQUEST] % User Query <Visual Input I vis q > <Textual Input I txt q >Intra-demonstration Instruction (I intra ) This instruction embeds task instructions within each example, assisting the model in understanding the task requirements during the learning process. As illustrated in Figure 4 (c), the intra-demonstration instruction I is included within each demonstration x i in the list L. This method ensures that the task instructions are continuously reinforced throughout the learning process. Specifically, the overall prompt template is as follows:Model OKVQA [Marino et al., 2019] VQA-v2 [Goyal et al., 2017] Accuracy BERTScore Token F1 Accuracy BERTScore Token F1OpenFlamingo [Awadalla et al., 2023 [DEMONSTRATIONS][REQUEST] % Shot 1<Visual Input I vis 1 > <Textual Input I txt 1 > <Instruction I intra > [RESPONSE] <Textual Output I vis 1 > • • • [QUERY][REQUEST] % User Query <Visual Input I vis q > <Textual Input I txt q > <Instruction I intra >By systematically incorporating these instruction categories into the MM-ICL framework, we aim to investigate their impact on model performance and task comprehension.In our preliminary experiments, we observed that variations in prompts do not significantly alter the overall conclusions. Specifically, we employed multiple prompts-differing in instructions and delimiters-while maintaining equivalent semantic content but varying linguistic expression. As demonstrated in Table 4 , the influence of these different prompts on the results is minimal. This suggests that our findings are robust to changes in prompt formulation, thereby supporting the reliability of the experimental outcomes. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: This submission presented a multimodal framework for In-Context Learning (MMICL). The authors conducted extensive experiments on the three stages of MM-ICL (demonstration retrieval, scheduling and prompting) to explore the factors that affect MMICL performance.  
The reviewers raised many questions. The authors were able to answer a number of them. And there was a rich discussion in particular on the type of evaluation carried out to validate the ICL hypotheses, and on the conclusions that are drawn from these “non-standard” Vision-Language evaluations.  
In the end, the reviewers were on the whole very convinced. The AC agreed that the strengths in this case outweighed the weaknesses. The final recommendation is to accept. Authors are strongly encouraged to take all comments into account in their final version.

Reviews: review: paper_summary: paper_summary: The paper investigates the underlying factors that influence the effectiveness of Multi-Modal In-Context Learning (MM-ICL). The authors conducted extensive experiments on three core steps of MM-ICL: demonstration retrieval, demonstration ordering, and prompt construction using six vision large language models and 20 strategies. Their findings highlight the necessity of a multi-modal retriever for demonstration retrieval, the importance of intra-demonstration ordering, and the enhancement of task comprehension through introductory instructions in prompts. This study aims to provide a foundational guide for optimizing MM-ICL strategies in future research.
strength_weakness: strength_weakness: Some weaknesses are listed below: 

1. The results of the experiments seem apparent and intuitive. For example, the multi-modal retriever performs better than the single modality retriever because it incorporates more information. Additionally, intra-demonstration ordering is crucial as it affects the structure of the input sample and how the model processes the input. As a result, the findings do not seem particularly inspiring.

2. Some qualitative analysis is missing. It would be beneficial to compare how different in-context examples affect the answers.

3. Considering that the margin of improvement in some metrics is quite small, certain designs of in-context examples may not be as important. For instance, as shown in Table 1, in most scenarios, a single textual retriever is sufficient. Therefore, using a multi-modal retriever, which may consume more computational resources, is unnecessary.

4.	Is it appropriate to combine all tasks to provide a universal paradigm for constructing MM-ICL strategies? For example, in classification tasks, the multi-modal retriever does not seem to perform the best.

There are also some minor mistakes in the paper:

1. It is more common to use the term "Large Vision-Language Models (LVLMs)" instead of "Vision Large Language Models (VLLMs)," which you use throughout the paper (e.g., line 43).

2.The reference for the model Qwen-VL in Table 1 is incorrect; the model is not from OpenAI.

3.In Table 1, the Otter Model results are not highlighted in bold.
questions: questions: In addition to the list of weaknesses above, here are some questions for the authors to address:

1.For multi-modal demonstration retrieval, why do you use a multi-modal encoder to calculate the similarity instead of using the product of text and vision single modality similarity? Exploring this alternative approach might offer additional insights into multi-modal retrieval.

2.For the OpenAI GPT-4V model, how do you reconstruct the order of the text and the image? As far as I know, the API does not provide a way to structure the order of text and images.
limitations: limitations: Limitations are discussed in the paper. and the authors do not foresee any negative societal impact.
paper_summary: paper_summary: The work presents an in-depth analysis of the factors influencing the performance of Multi-modal In-Context Learning (MM-ICL). The authors systematically investigate the core steps of MM-ICL, including demonstration retrieval, ordering, and prompt construction. Utilizing six vision large language models and a variety of strategies, the study uncovers the significance of multi-modal alignment, the importance of intra-demonstration ordering, and the role of introductory instructions in enhancing task comprehension.
strength_weakness: strength_weakness: 1. The captions in the paper can be enriched to help readers gain a better understanding.
2. In Section 5.3, can you provide more analysis on why the number of demonstrations does not significantly impact MM-ICL?
3. The experimental section needs to be supplemented with a description of the LLM backbone.
questions: questions: 1. In Section 5.3, can you provide more analysis on why the number of demonstrations does not significantly impact MM-ICL?
2. In your experiments, how did you handle multiple image inputs?
3. Will different prompts affect the exploration experimental results?
limitations: limitations: This work presented some limitations in the submitted paper including: (1) extending the exploration to video modal ICL and (2) multi-lingual multi-modal ICL scenarios.
paper_summary: paper_summary: This work explores an interesting research question: “What factors affect the performance of MM-ICL?” To this end, they conduct comprehensive experiments on the three fundamental steps of MM-ICL: demonstration retrieval, demonstration ordering, and prompt construction. In addition, they explore 20 strategies across 4 tasks with 6 representative vision large language models (VLLMs).
strength_weakness: strength_weakness: Overall, I have no major issues. There are a few points that can be improved: 
(1) The 'Exploration of MM-ICL Prompt Construction' section can be explained more clearly. 

(2) Some parts of the appendix, such as the implementation details of the baseline, can be moved to the main text. 

(3) The 'Limitations' section can include some specific limitations related to this work.
questions: questions: See above comments.
limitations: limitations: See above comments.
paper_summary: paper_summary: The paper studies what influences multimodal ICL and finds that using multimodal retrievers to find demonstrations, ordering of modalities in the context and introductory instructions help to improve the few-shot performance. The study spans 6 models and several multimodal datasets.
strength_weakness: strength_weakness: 1. The paper claims to be the first to study multimodal ICL. However, previous works [1,2] conducted very similar studies and there is no mention for these previous works.
2. The paper does not consider standard benchmarks. In addition it uses metrics not generally used to report scores in other papers. For example, the performance on VQAv2 is measured with Accuracy not BERTscore. I think the considered benchmarks and metrics are leading to false conclusions as detailed later, either because these metrics/benchmarks are flawed/limited or just saturated. To support the paper claims it is important to consider a typical evaluation setup. For example on benchmarks such as VQAv2, COCO captioning, NoCaps, TextCaps, OK-VQA, TextVQA ..  using typical scores such as Accuracy, CIDEr/BLEU…
3. Related to previous points. Table 1 shows that IDEFICS 8B is comparable to GPT4V ! (e.g. VQA BertScore 85.42 GPT4V vs 86.11 IDEFICS few-shot random) IDEFICS 8B is significantly worse on almost all multimodal benchmarks. I am not sure at this point how much I can rely on the paper results and findings.
4. I have some issues with some claims that I don’t find consistent with previous studies. These claims should be refined or supported with more evidence:
- “The number of demonstrations does not significantly impact MM-ICL”: usually the more shots the better the performance [3,4,5] and the improvement is significant.
- “increasing model parameters from 8 billion to over 100 billion does not primarily drive performance improvements, suggesting that multi-modal context understanding and alignment are more crucial for MM-ICL than model scaling”: Other works clearly show that increasing model size increases ICL performance [3,5]. To make such claim, only the scale of the model should change, but in Table 1 the authors seems to compare different models (e.g. GPT4-V vs Qwen-VL 10B)

5. Wrong citations: IDEFICS 8B is not done by Awadallah et al (Table 1) !

6.  The paper figures are not properly explained:
- What the colors means in Fig 10?
- Which scores are you showing in Fig 8 and 9?
Please write detailed captions to explain the paper figures.

7. What is BELU score? Is it the same as BLEU?



References:

[1] Baldassini, Folco Bertini, et al. "What Makes Multimodal In-Context Learning Work?." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Chen, Shuo, et al. "Understanding and Improving In-Context Learning on Vision-language Models." ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models (2024).

[3] Alayrac, Jean-Baptiste, et al. "Flamingo: a visual language model for few-shot learning." Advances in neural information processing systems 35 (2022): 23716-23736.

[4] Laurençon, Hugo, et al. "Obelics: An open web-scale filtered dataset of interleaved image-text documents." Advances in Neural Information Processing Systems 36 (2024).

[5] Shukor, Mustafa, et al. "Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning." The Twelfth International Conference on Learning Representations. 2023.
questions: questions: Check weaknesses (e.g. 6. 7)
limitations: limitations: Few limitations are discussed in the paper.

Comments: title: Thanks for Your Time and Effort
Gratitude for Your Detailed Feedback
Thanks Again for Your Time and Effort
Gratitude for Your Detailed Feedback
Clarification for Your Feedback
Thanks for your time and effort
Gratitude for Constructive Feedback
Concerns Addressed
comment: We sincerely appreciate the time and effort you have dedicated to reviewing our response. We are very pleased that all your concerns have been addressed. We sincerely hope that you can consider raising the score after we have addressed all the concerns. We greatly appreciate your selfless contributions to the community.
We sincerely thank you for investing your time to review our work. We are encouraged by your acknowledgment and interest in our work. We will incorporate your suggestions to polish our work in the next version. Thank you once again for your valuable contributions.
We are writing to express our heartfelt gratitude for your valuable feedback. Your insightful and valuable suggestions can significantly contribute to enhancing the solidity of our work. We will follow your suggestions to polish our work in the next version.

Thank you once again for your valuable contributions.
Thanks for the authors' detailed response. My concerns have been addressed.
Thanks for your response. 

It seems that the accuracy is correlated with the metrics used in the paper (despite the fact that this needs more elaborated experiments), and better evaluation is less likely to change the main messages of the paper. 

However, I believe the paper should consider the benchmarks and metrics commonly used to compare recent  models to draw its conclusions, and can be improved on this side. Also, the authors shouldn't include any scores that raise suspicions about the work, I again ask the authors to reconsider the GPT4V scores (this model is significantly better than IDEFICS2). 

Given the authors clarifications and other reviewers responses, and to encourage the authors to include what is discussed here, as well as other reviewers responses, I will increase my score from 3 to 4.

Note: the VQAv2 score of GPT4V is 77.2 not 71.28 (Table 4 in [1])

[1] McKinzie, Brandon, et al. "Mm1: Methods, analysis & insights from multimodal llm pre-training." arXiv preprint arXiv:2403.09611 (2024).
Thanks for your response. I read this work again and went through the author's responses to other reviewers. I think the quality and contributions of this paper are solid and clear. The topic studied is important. Although some progress has been made in text-only In-context Learning, comprehensive investigation of multi-modal In-context Learning remains underexplored. The findings and practices in this paper, including the design of demonstration ordering and instruction prompts, can provide valuable insights for future research. I am inclined to raise my score to 8 and champion this paper.
We would like to extend our sincere gratitude for your thoughtful and detailed feedback. In addition, we are very grateful for the opportunity to further clarify your worry. We notice that your main concern is “Does F1/BertScore correlate with the accuracy?” and we sincerely agree with your concern. In the following, we will do our best to address your concern.

**Q:** Does F1/BertScore correlate with the accuracy?

**A:** Thanks for your insightful feedback. The answer is “**Yes**”. According to your suggestion, we have adopted accuracy for evaluating several VQA datasets including OK-VQA and VQAv2. The results are shown in Table 1 and Table 2 below:

|Model| Acc. | BERTScore | Token F1 |
|:--:|:--:|:--:|:--:|
|OpenFlamingo | 40.28 | 78.10 | 17.45 |
|GPT4V | 54.28 | 85.97 | 25.23 |
|IDEFICS2 | 55.32 | 87.61 | 27.81 |
Table 1: Performance on OKVQA.

|Model| Acc. | BERTScore | Token F1 |
|:--:|:--:|:--:|:--:|
|OpenFlamingo | 53.33 | 83.34 | 25.67 |
|IDEFICS2| 69.69 | 84.89 | 29.18 | 
|GPT4V| 71.28 | 87.98 | 35.46 |
Table 2: Performance on VQAv2.

From the results, we observe that accuracy is positively correlated with BERTScore and F1. We attribute it to the fact that higher semantic relevance and exact keyword performance can lead to higher accuracy. Such observation also demonstrates the main paper messages cannot be changed. We sincerely believe that the findings in our work can directly contribute to the MM-ICL community. **In the next version, we promise to add more experiments and discussions to enrich our work**. 

Your insights and suggestions can significantly contribute to enhancing the quality and clarity of the work. **We are very encouraged that the further clarifications can address your concern**. Thank you once again for your selfless contributions to the community.
Thanks again for the detailed response. 

To clarify, my concerns about the evaluation is related mainly to the metrics and in particular for VQA tasks. I think VQA tasks are very important as they cover the main use cases of such models. The accuracy is what mainly reported on VQA tasks as seen in tons of recent papers. 

- Does F1/BertScore correlate with the accuracy? 
- If the authors think that the accuracy is not suitable here they should justify why and it needs a separate study to shows this (which I don't think is within the scope of this paper). 

It is very likely that if the authors report the accuracy on VQA tasks they will see different observations (e.g. the significant gap between GPT4V and IDEFICS 2, for example GPT4V [1] (zero-shot) got 56.8 compared to 43.5 for IDEFICS 2 on MMMU benchmark). 

It is possible to find some benchmarks where weaker models have close scores to much stronger ones. This might be due to many reasons, some of them could be the benchmark/metrics are biased/flawed/saturated or do not reflect the real model capabilities, the weaker model is finetuned on the training set or similar datasets to the evaluated benchmarks ...

**Nonetheless for the purpose of this paper and similar papers that conduct analysis to understand these models, the benchmarks/metrics should be well chosen to properly draw conclusions.** 

My worries are if the paper is properly evaluated, different conclusions might be drawn. If the authors can show that for example the VQA accuracy on several VQA datasets is correlated with F1/BertScore or does not change the main paper messages, I might consider increasing my score.
Thanks for your timely feedback. We would like to further clarify your concerns.

**Q1:** Seeing the evaluation and conclusions, it seems that using the M3IT benchmark is not a good choice. 

**R1:** Thanks for your suggestion. In our detailed response R2 in previous rebuttal, M3IT includes four classic types of tasks that **contain almost all of your mentioned datasets** (e.g., VQAv2, COCO captioning, TextCaps, OK-VQA, TextVQA). We sincerely believe that these cover a broad range of categories for evaluation. We will add more details in the next version.

---
**Q2:** Showing 2 scores of IDEFICS2 that are worse but close to GPT4V is not enough to conclude that these 2 models are even comparable.

**R2:** Thanks for your comment. We sincerely believe that there are some misunderstandings. In fact, in some previous findings [1,2,3], **IDEFICS2 even outperforms GPT4V by 2 scores** (see the Table below). As shown in Table 1 in the submitted paper, the performance of these two models are also comparable. The core difference lies in the image captioning task. Nevertheless, the metrics you mentioned, such as CIDEr, are inappropriate for evaluating open-ended image caption. Human judgments of alignment often differ significantly because GPT-4V tends to provide more detailed responses, whereas IDEFICS2 prefers shorter ones. However, in most image captioning datasets, the golden captions are brief, which presents challenges for traditional metrics [4]. Therefore, such observations motivate us to explore semantic-level BERTScore metric in addition to CIDER for open-ended image caption task.

||MathVista|MMBench| 
|:--:|:--:|:--:|
|GPT4V |49.9[2]|74.3[3] |
|IDEFICES2-8B|51.6[1] |76.8[1] | 
Table: Performance comparison.

[1] Laurençon et al. What matters when building vision-language models? Arxiv 2024

[2] Lu et al. MathVista: Evaluating Math Reasoning in Visual Contexts. ICLR 2024

[3] Liu et al. MMBench: Is Your Multi-modal Model an All-around Player? Arxiv 2023

[4] What you see is what you read? improving text-image alignment evaluation. NeurIPS2023

---
**Q3:** I highly encourage the authors to work on the evaluation and avoid any claim that goes against previous serious works.

**R3:** Thanks for your constructive comment. Actually, **our findings are not in conflict with previous work**. For example, our findings in multi-modal chain-of-thought reasoning that simply adding more shots does not always lead to improved performance, which is consistent with the previous work [1]. We think that the following might be reasons why more demonstrations may not improve performance in these tasks, which is acknowledged by Reviewer #Wrbn.

1.	**Cognitive Overload**: For complex tasks, understanding complex demonstrations is difficult. More demonstrations can overwhelm the model, making it harder to process and integrate information effectively.
2.	**Complexity of Reasoning Tasks**: We observe that for reasoning tasks, the performance improvement brought by the number of demonstrations is not even as good as using different retrievers. It shows that reasoning tasks require sophisticated integration of information, where quality trumps quantity.

In addition, in our experiments, we also observe that more shots lead to better performance in VQA task, which also align with [2][3][4].  We will add more discussion in the next version.

[1] Chen et al. M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought. ACL 2024.

[2] Alayrac et al. Flamingo: a visual language model for few-shot learning. NeurIPS 2022.

[3] Laurençon et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. NeurIPS 2024.

[4] Shukor et al. Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning. ICLR 2023.

---
**Q4:** Regarding the originality

**R4:** Thanks for your kind mention. We sincerely think that our work is the first to unify the current multi-modal In-context learning (MM-ICL) process and conduct a comprehensive exploration for each process within the unified perspective. The provided findings in this work are meaningful and interesting, **which have been recognized by Reviewer #Pm2y, Reviewer #Wrbn, Reviewer #jbhA**. 

- Multi-modal alignment is the bottleneck for MM-ICL.
- Intra-demonstration ordering holds greater importance than inter-demonstration ordering.
- Introductory instruction guides better task understanding for MM-ICL.

In addition, when the mentioned work [1] is presented at CVPR Workshop, the NeurIPS deadline has already passed.

[1] Baldassini et al. What Makes Multimodal In-Context Learning Work? CVPR 2024 Workshop.

**We greatly appreciate the time you've invested in reviewing our response**. We hope the further clarification can address your concerns and **we sincerely hope that you can reconsider your score in light of our clarifications**. We **promise** to incorporate your all suggestions to improve our work.
Thanks for the detailed response. My main concerns about this submission is the evaluation. Without proper evaluation no serious conclusions can be drawn. I suggest to evaluate the model on a "broad" range on benchmarks "commonly" used by the multimodal community. Seeing the evaluation and conclusions, it seems that using the  M3IT benchmark is not a good choice. Showing 2 scores of IDEFICS2 that are worse but close to GPT4V is not enough to conclude that these 2 models are even comparable (the authors can check the evaluation of these models on broad range of tasks). I highly encourage the authors to work on the evaluation and avoid any claim that goes against previous serious works, unless a "rigorous, extensive and convincing" arguments/experiments support that.  Also regarding the originality, the work is "among" the first and not "the first". I think the paper needs additional work on the claims and experiments to be accepted at top venues. Given the importance of the studied topic, and the authors promises to refine and clarify some claims I will not decrease my score.
Thank you very much for your time and valuable feedback. Hope our clarifications can address your concerns, and we sincerely hope that you can reconsider our work in light of these clarifications. If you have any further comments, please do not hesitate to contact us. We greatly appreciate your selfless contributions to the community.
We sincerely thank you for investing your time to review our response. Your insightful and valuable suggestions have significantly contributed to enhancing the solidity of our work. We will follow your suggestions to enhance our work in the next version.

Thank you once again for your valuable contributions.
Thanks the authors for your clarifications, which I think possibly have addressed all my previous concerns. I like the insightful explanation about demonstrations for complex reasoning tasks, including Cognitive Overload and the Complexity of Reasoning Tasks. Actually, I’m working on MM-ICL for reasoning, and from my practice, for some complex reasoning tasks, providing just a few demonstrations may not be sufficient. So it is reasonable and necessary to explicitly construct some clues from multimodal reasoning examples when creating demonstrations. Overall, I think the findings authors give are meaningful and interesting, and believe they will show more impact in MM-ICL or relevant topics. One tip for authors: as future work, I think it would also be interesting to explore a unified theoretical framework for MM-ICL.