Many-Shot In-Context Learning in Multimodal Foundation Models

## Abstract

Abstract Large language models are well-known to be effective at few-shot in-context learning (ICL). Recent advancements in multimodal foundation models have enabled unprecedentedly long context windows, presenting an opportunity to explore their capability to perform ICL with many more demonstrating examples. In this work, we evaluate the performance of multimodal foundation models scaling from few-shot to many-shot ICL. We benchmark GPT-4o and Gemini 1.5 Pro across 14 datasets spanning multiple domains (natural imagery, medical imagery, remote sensing, and molecular imagery) and tasks (image classification, visual question answering, and object localization). We observe that many-shot ICL, including up to almost 2,000 multimodal demonstrating examples, leads to substantial improvements compared to few-shot ( < < < 100 examples) ICL across all of the datasets. Further, Gemini 1.5 Pro performance continues to improve log-linearly up to the maximum number of tested examples on many datasets.
We also find open-weights multimodal foundation models like Llama 3.2-Vision and InternLM-XComposer2.5 do not benefit from the demonstrating examples, highlighting an important gap between open and closed multimodal foundation models.
Given the high inference costs associated with the long prompts required for many-shot ICL, we also explore the impact of batching multiple queries in a single API call. We show that batching up to 50 queries can lead to performance improvements under zero-shot and many–shot ICL, with substantial gains in the zero-shot setting on multiple datasets, while drastically reducing per-query cost and latency. Finally, we measure ICL data efficiency of the models, or the rate at which the models learn from more demonstrating examples. We find that while GPT-4o and Gemini 1.5 Pro achieve similar zero-shot performance across the datasets, Gemini 1.5 Pro exhibits higher ICL data efficiency than GPT-4o on most datasets. Our results suggest that many-shot ICL could enable users to efficiently adapt multimodal foundation models to new applications and domains.

## 1 Introduction

Large language models (LLMs) have been shown to substantially benefit from the inclusion of a few demonstrating examples (shots) in the LLM context before the test query (Brown et al., ; Parnami & Lee, ; Wang et al., ). This phenomenon, commonly referred to as in-context learning (ICL), enables LLMs to learn from few shots without any updates to model parameters, and therefore improves specialization to new tasks without any further model training. More recently, large multimodal models (LMMs) have also demonstrated the capability of learning from in-context examples Achiam et al. (); Han et al. (); Zhang et al. (). Han et al. () and Zhang et al. () both show that few-shot multimodal ICL specifically helps to improve LMM performance on out-domain or out-of-distribution tasks.

While few-shot ICL has enabled promising performance improvements for both LLMs and LMMs, limited model context windows have constrained research on the impact of increasing the number of demonstrating examples on performance. This is especially true for LMMs as most use a large number of visual tokens to represent images. However, due to recent advancements enabling substantially longer context windows – for example, 128,000 tokens for GPT-4o and up to one million tokens for Gemini 1.5 Pro – it is now possible to explore the effect of drastically increasing the number of demonstrating examples.

To investigate the capability of state-of-the-art multimodal foundation models to perform many-shot ICL, we conduct a large suite of experiments benchmarking model performance on 14 datasets spanning several domains and multimodal tasks after scaling up the number of demonstrating examples by multiple orders of magnitude. Specifically, our contributions are as follows:

- 1.
We show that providing close-weights multimodal foundation models with many demonstrating examples leads to substantial performance improvements compared to providing only a few demonstrating examples. We observe that the performance of Gemini 1.5 Pro generally improves log-linearly as the number of demonstrating examples increases, whereas GPT-4o exhibits less stable improvements as the number of in-context examples increases.
- 2.
We find open-weights multimodal foundation models like Llama 3.2-Vision and InternLM-XComposer2.5 do not benefit from the demonstrating examples, highlighting a significant gap and an important direction for the open-weights community.
- 3.
We measure the data efficiency of the models under ICL as the number of demonstrating examples increases, and find that Gemini 1.5 Pro exhibits higher ICL data efficiency than GPT-4o on most datasets.
- 4.
We demonstrate that batching multiple queries into a single request can achieve similar or better performance than single query requests in a many-shot setting, while enabling substantially lower per-example latency and much cheaper per-example inference cost.
- 5.
We find that batching multiple questions can lead to substantial performance improvements in a zero-shot setting. We design experiments to explain this phenomenon, and find that the improvements are due to a combination of domain calibration, class calibration, and self-generated demonstrating examples due to autoregressive decoding.
- 6.
We release a new tool [(anonymized link)](https://drive.google.com/file/d/1kczLN_yic2CoBHvY2f3Cj5dNhaZ_iccg/view?usp=sharing) to allow users to easily test the many-shot ICL capabilities of multimodal foundation models, facilitating future work on studying them.

Figure: Figure 1: Many-shot multimodal in-context learning compared to zero-shot and few-shot multimodal ICL. In zero-shot and few-shot settings, respectively, no demonstrating examples or only a small number of demonstrating examples are provided in the context before the test query. In a many-shot ICL setting, we include a large number of demonstrating examples in the prompt, whereas in batched many-shot ICL, we perform multiple queries at once using query references.
Refer to caption: extracted/5902889/img/fig1_v6.png

## 2 Related Work

Scaling ICL.  The seminal work of Brown et al. () discovered performance improvements for LLMs from increasing the number of in-context examples, but the tested number of demonstrating examples was low (10 to 100), likely due to the restrictive context size (2048 tokens for GPT3). Increasing the number of in-context examples has only been explored recently by a few works Li et al. (); Agarwal et al. (); Bertsch et al. (). Both Li et al. () and Agarwal et al. () explore scaling in-context learning to more than 1,000 demonstrating examples and find performance improvements across multiple tasks. However, their experiments are limited to text-only benchmarks and do not compare performance across different models.

Multimodal ICL.  Due to the recent emergence of LMMs, research on multimodal ICL is still nascent. One prior work developed a new model to leverage complex prompts composed of multimodal inputs in order to allow models to compare images Zhao et al. (), while other recent works explored the generalizability of GPT-4V and Gemini to multimodal out-domain and out-of-distribution tasks, and found that ICL leads to performance benefits for both models across many tasks Zhang et al. (); Han et al. (). However, none of these works have leveraged the new largely expanded context windows to investigate the effects of increasing the number of demonstrating examples.

Batch Querying.  Multiple prior works have explored batching queries (also commonly referred to as batch prompting) for more efficient and cheaper inference. Batch prompting was first introduced in Cheng et al. (), leading to comparable or better performance than single prompting, while achieving substantially reduced inference token cost and latency. Lin et al. () observe performance degradation with batched prompts in longer contexts, and propose a variety of techniques to mitigate the performance loss. More recently, additional variations of batch prompting have been proposed, including grouping similar questions together Liu et al. (), batching prompts of different tasks Son et al. (), and concatenating multiple images into a single image collage Xu et al. (). We again note that batch prompting with high numbers of demonstrating examples and high numbers of queries has only become feasible due to larger context windows of recent models.

## 3 Methods

Figure: Figure 2: Gemini 1.5 Pro and GPT-4o performance from zero-shot to many-shot ICL. The x-axis is in log scale. For Gemini 1.5 Pro, we observe log-linear improvement on 9 out of the 10 datasets and. For GPT-4o, we observe improvement from more demonstrating examples on most datasets, while the improvement is substantially less stable than Gemini 1.5 Pro. Error bars are estimated standard deviations using bootstrapping with 1,000 bootstrap replicates.
Refer to caption: extracted/5902889/img/fig2_GPT4O.png

**Table 1: Summary of datasets. We use 14 datasets spanning multiple domains (natural imagery, medical imagery, remote sensing, and molecular imagery) and tasks (multi-class, multi-label, and fine-grained classification, visual question answering, object localization).**
| Dataset | Task and image type | Demo / test set size | Example image |
| --- | --- | --- | --- |
| HAM10000<br>(Tschandl et al., ) | 7-category skin disease classification on clinical photos | 805 / 210 |  |
| FIVES<br>(Jin et al., ) | 4-category eye disease classification on fundus images | 400 / 120 |  |
| CheXpert<br>(Irvin et al., ) | Multi-label 5-category lung disease detection on chest X-rays | 200 / 150 |  |
| Camelyon17<br>(Bandi et al., ) | Binary tumor detection on pathology images | 2000 / 100 |  |
| TerraIncognita<br>(Beery et al., ) | 9-category animal species recognition on camera images | 1035 / 270 |  |
| UCMerced<br>(Yang & Newsam, ) | 21-category land use classification on satellite images | 1470 / 420 |  |
| EuroSAT<br>(Helber et al., ) | 10-category land use / land cover classification on satellite images | 1000 / 300 |  |
| Oxford Pets<br>(Parkhi et al., ) | 35-category pet classification on camera images | 1750 / 700 |  |
| DTD<br>(Cimpoi et al., ) | 47-category texture classification on synthetic images | 2350 / 940 |  |
| DrugOOD Assay<br>(Ji et al., ) | Binary drug binding prediction on molecular images | 1600 / 200 |  |
| RSVQA<br>(Lobry et al., ) | Visual question answering on satellite images | 200/200 |  |
| VQA-RAD<br>(Lau et al., ) | Visual question answering on radiology images | 200/200 |  |
| DIOR<br>(Li et al., ) | Object localization on satellite images | 200/100 |  |
| DeepLesion<br>(Yan et al., ) | Lesion localization on CT images | 200/100 |  |

We conduct several experiments to test the effect of increasing the number of demonstrating examples on the performance of state-of-the-art multimodal foundation models: close-weights models like GPT-4o and Gemini 1.5 Pro, and open-weights models like Llama3.2-Vision (Section 3.1). We benchmark their performance using standard performance metrics as well as an ICL data efficiency metric (Section 3.3) on 14 datasets spanning several vision domains and multimodal tasks (Section 3.2). We conduct ablation studies to test the impact of batching queries on model performance and explain the substantial improvement in zero-shot settings (Section 4.2). We refer to the many-shot in-context learning framework as many-shot ICL. Figure 1 provides an illustrative summary of many-shot ICL and batched many-shot ICL compared to zero-shot and few-shot ICL.

### 3.1 Models

We use three state-of-the-art multimodal foundation models with public API access, namely GPT-4o, GPT4(V)-Turbo (Achiam et al., ), and Gemini 1.5 Pro (Reid et al., ). In addition, two open-weights multimodal foundation models (Llama3.2-11B-Vision () and InternLM-XComposer-2.5 (Zhang et al., )) are also tested. Because GPT-4o performs substantially better than GPT4(V)-Turbo, we focus on the results of GPT-4o and Gemini 1.5 Pro in the main text, and include GPT4(V)-Turbo results in the Appendix. We do not utilize Claude3-Opus in our experiments, as it only accepts up to 20 images in one request at the time of writing. The specific endpoint for for GPT-4o is “gpt-4o-2024-05-13”, for GPT-4(V)-Turbo is “gpt-4-turbo-2024-04-09”, and for Gemini 1.5 Pro is “gemini-1.5-pro-preview-0409”. We use the API service provided by OpenAI for GPT-4o and GPT-4(V)-Turbo, and the API service provided by Google Cloud on Vertex AI for Gemini 1.5 Pro. We set the temperature to zero for all models and a random seed for GPT-4(V)-Turbo and GPT-4o to obtain more deterministic responses. To prevent models from abstaining (which happens rarely), we rerun the query until an answer is provided. We call the APIs on a virtual machine instance of type “c2-standard-8” and run inference using open-weights models on a “a3-highgpu-8g” machine (with 8 H100 GPUs) hosted on Google Cloud Platform.

### 3.2 Datasets

We benchmark the model performance on 14 datasets spanning multiple domains (natural imagery, medical imagery, remote sensing, and molecular imagery) and tasks (image classification, visual question answering, and object localization). We acknowledge most LMMs are not yet capable of accurately producing localizations (Wu et al., ; Zang et al., ). Table 1 provides a summary of the datasets used in this study.

For all datasets, we construct a set of demonstrating (demo) examples from the original training and validation splits used for in-context learning and a test set from the original test split (if one exists) to evaluate the performance of the models. We randomly sample the demo and test sets from the original dataset without replacement. For the multi-class and fine-grained classification datasets, we perform a class-stratified sampling, ensuring an equal number of examples per class in both the demo and test sets. For the multi-label classification dataset (CheXpert), we sample an equal number of positive and negative samples per class in both the demo and test sets. We note that, since the task is multi-label, this sampling procedure does not result in an exactly equal number of examples per class. For the scaling experiments, we increase the number of demonstrating examples while ensuring class balance. Besides the 10 classification datasets shown in Table 1, we also include two visual question answering datasets (RSVQA (Lobry et al., ) and VQA-RAD (Lau et al., )) and two object localization datasets (DIOR for bridge localization (Li et al., ) and DeepLesion for lung lesion localization (Yan et al., )).

### 3.3 Evaluation Metrics

We use standard metrics to evaluate model performance on each dataset. Specifically, we measure performance using accuracy for all multi-class classification and visual question answering datasets as they are sampled to have a balanced class distribution. For multi-label classification on CheXpert, we use the macro-averaged F1 metric. In the rare case of parsing errors, we consider the response as incorrect. For object localization datasets, we use mean Intersection over Union(IoU). To estimate the variability around the evaluation metrics, we compute standard deviation using bootstrapping with 1,000 bootstrap replicates, and it captures the variability in data sampling.

In addition to standard performance metrics, we measure the data efficiency of each model. Specifically, we compute a linear regression between $\log_{10}(N+1)$ (with $N$ the number of examples) and model performance, enforcing that the line passes through the zero-shot performance point. This value approximates the amount of performance improvement from zero-shot expected from including an order of magnitude more demonstrating examples.

Figure: Figure 3: Llama3.2 and InternLM-XComposer2.5 performance compared with Gemini 1.5 Pro. While Gemini 1.5 Pro shows substantial improvement from zero-shot to many-shot ICL on all three categories of tasks, both open-weights models do not benefit from demonstrating examples.
Refer to caption: extracted/5902889/img/open-models.png

## 4 Results

We present many-shot ICL performance using batched queries in Section 4.1, investigate the impact of batching queries on performance in Section 4.2, and provide an analysis on cost and latency in Section 4.3. Results using GPT4(V)-Turbo are in Appendix C.

### 4.1 Increasing number of demonstrating examples

Main Results. Gemini 1.5 Pro exhibits consistent and substantial improvements as the number of demonstrating examples increases across all datasets except for DrugOOD Assay (Figure 2 and  3). Gemini 1.5 Pro shows particularly large improvements from many-shot ICL on HAM10000 (+23% accuracy compared to zero-shot, +16% compared to 7 examples), FIVES (+29% compared to zero-shot, +27% compared to 20 examples), and EuroSAT (+38% compared to zero-shot, +31% compared to 10 examples). Notably, for 8 out of the 14 datasets (FIVES, UCMerced, EuroSAT, Oxford Pets, DTD, VQA-RAD, DIOR and DeepLesion), Gemini 1.5 Pro performance continues to improve up to the highest number of demonstrating examples considered (~1,000 examples). On the other 6 datasets, the optimal performance occurs prior to the highest number of demo examples, with the maximum number of demo examples leading to similar or slightly worse performance than the optimal demo set size. On the other hand, Gemini 1.5 Pro performance on DrugOOD Assay does not substantially benefit from many-shot ICL, with high variance in performance across demo sizes and the peak performance at 40 demo examples.

Similarly, GPT-4o shows substantial performance improvements on all datasets except FIVES and DrugOOD Assay using many-shot ICL, but the improvement is not consistent. For many datasets, performance drops sharply at first and then improves significantly as the number of demonstrating examples increases further, resulting in V-shaped scaling curves (Figure 2). We also note that we were unable to increase the number of demo examples to the same level as considered for Gemini 1.5 Pro because GPT-4o has a shorter context window and is more prone to timeout errors with longer inputs. GPT-4o performance on DrugOOD Assay shows high variance, similar to Gemini 1.5 Pro, with the peak performance observed at 50 demo examples.

Open-weights Model Results. We also find open-weights multimodal foundation models like Llama 3.2-Vision and InternLM-XComposer2.5 do not benefit from the demonstration examples (Figure 3), highlighting a significant gap between open and closed multimodal foundation models.

Sensitivity to prompt selection. We also explore a different set of prompts to test the robustness of many-shot ICL to differences in prompt wording on two datasets. While there is a small deviation in performance between different prompts, the overall log-linear improvement trend is consistent across the prompts. Details can be found in Appendix B.

**Table 2: Many-shot ICL performance and efficiency comparison. We report the performance under a zero-shot regime and performance at the optimal demo set size as well as the many-shot ICL data efficiency of GPT-4o and Gemini 1.5 Pro. We measure performance using accuracy on all datasets except CheXpert, for which we use macro-averaged F1. The highest ICL data efficiency between the two models on each dataset is marked in bold.**
| Dataset | GPT-4o | Gemini 1.5 Pro |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Zero-shot | Best | Efficiency | Zero-shot | Best | Efficiency |  |
| HAM10000 | 34.93 | 53.59 (+18.66) | 5.91 | 33.33 | 56.46 (+23.13) | 6.94 |
| FIVES | 31.67 | 37.50 (+5.83) | 0.30 | 25.83 | 55.00 (+29.17) | 7.56 |
| CheXpert | 28.47 | 42.54 (+14.08) | 3.70 | 22.16 | 42.23 (+20.08) | 9.06 |
| Camelyon17 | 77.00 | 90.00 (+13.00) | 1.00 | 71.00 | 83.00 (+12.00) | 3.00 |
| TerraIncognita | 29.26 | 59.26 (+30.00) | 20.50 | 59.63 | 66.67 (+7.04) | 3.50 |
| UCMerced | 90.95 | 98.57 (+7.62) | 1.20 | 91.19 | 98.57 (+7.38) | 4.36 |
| EuroSAT | 55.37 | 84.23 (+28.86) | 19.40 | 36.24 | 74.16 (+37.92) | 20.61 |
| Oxford Pets | 83.14 | 94.14 (+11.00) | -3.72 | 85.29 | 97.43 (+12.14) | 4.26 |
| DTD | 39.26 | 74.47 (+35.21) | 4.48 | 69.89 | 83.19 (+13.30) | 3.89 |
| DrugOOD Assay | 50.00 | 55.00 (+5.00) | 2.02 | 48.00 | 55.50 (+7.50) | 2.03 |

ICL data efficiency.
We find Gemini 1.5 Pro demonstrates higher ICL data efficiency than GPT-4o across all datasets except TerraIncognita and DTD (Table 2). Gemini 1.5 Pro ICL efficiency is especially high on EuroSAT, with 20.61% improvement in accuracy for every 10x more demo examples, and lowest on DrugOOD Assay (2.03), Camelyon17 (3.00), and TerraIncognita (3.50). GPT-4o ICL data efficiency is especially high on TerraIncognita (20.50%) and EuroSat (19.40). Gemini 1.5 Pro has a positive efficiency on all datasets and GPT-4o has a positive data efficiency on 9 of the 10 datasets (excluding Oxford Pets). Importantly, both models benefit substantially from many-shot ICL at the optimal demo set size, with an average improvement of +17% for both Gemini 1.5 Pro and GPT-4o.

### 4.2 Impact of batching queries

As including a large set of demo examples in the prompt leads to much longer sequence lengths and therefore higher inference time and cost, we consider batching queries in a single prompt to reduce per-query cost, and examine the impact of different batch sizes on model performance. Due to its superior performance and free preview access, we use Gemini 1.5 Pro for these experiments.

Main Results. We find minimal performance degradations, and sometimes performance improvements, as we increase the number of queries included in each batch across under both zero-shot and many-shot (at the optimal demo set size) regimes (Figure 4). Notably, using a single query each time with many-shot ICL is suboptimal across many of the datasets. We find that the optimal batch size is among the three largest sizes on every dataset except CheXpert and EuroSAT, which both see optimal performance with a single query at a time.

We additionally observe that including a single query at a time is suboptimal on most datasets in the zero-shot regime. Surprisingly, performance with the highest batch size is substantially higher across three datasets under the zero-shot regime, with a consistent performance improvement as the batch size is increased on both UCMerced and Terraincognita.

Figure: Figure 4: Gemini 1.5 Pro performance under many-shot and zero-shot ICL with varying amount of queries included in every request. We show performance per batch size with the optimal number of demo examples (many-shot) and no demo examples (zero-shot). The $x$-axis is in log scale. Under the many-shot regime, batching queries leads to no substantial drop in performance compared to individual queries when we choose a suitable batch size. For zero-shot, including only one query is suboptimal for many datasets. Error bars are estimated standard deviations using bootstrapping with 1,000 bootstrap replicates.
Refer to caption: extracted/5902889/img/ManyICL_Fig3_new.png

Zero-shot performance improvements from batching queries.  We conduct several additional experiments to investigate why batch querying can lead to large performance improvements under the zero-shot regime on TerraIncognita and UCMerced. We hypothesize that this improvement may be due to three potential benefits from ICL: (1) domain calibration, where the model benefits from seeing more images in the domain in order to adapt to it, (2) class calibration, where seeing images of different classes enables the model to better calibrate its outputs(Min et al., ), and (3) self-ICL (shown to be effective in prior work Chen et al. ()), where the model can learn from self-generated demonstrations due to autoregressive decoding. We design experiments to isolate the potential benefits from each of these types of ICL between asking a single query to batching 50 queries together.

First, to measure potential improvement from domain calibration, we include 49 images from the same class in the prompt without including any label. We find a 3.0% improvement on TerraIncognita and 2.6% degradation on UCMerced, suggesting domain calibration is helpful for the former but not the latter. Second, to capture performance gains from class calibration, we include a random sample of 49 images in the prompt, again without including the label. We see a further 3.5% improvement on TerraIncognita (6.5% improvement from a single query) and a 4.5% improvement from a single query on UCMerced, suggesting including the context of class-balanced images is helpful even without labels. Third, to capture additional performance improvements from the self-generated labels, we obtain predicted labels from the zero-shot model using a single query for each of the 49 randomly sampled images and add them to the prompt. We observe further performance increase on both datasets, with 5.5% on TerraIncognita and 2.7% on UCMerced. The final total accuracy is similar to asking the 50 questions each round, which suggests these three components mostly explain the reason for improved zero-shot performance under a larger query batch size.

Figure: Figure 5: Ablation study to investigate why batching queries leads to performance improvements when using Gemini 1.5 Pro in a zero-shot setting. The first bar shows performance when including a single query, the second adds 49 unlabeled images from a single class, the third adds 49 unlabeled images in total from all classes, the fourth adds model responses to include self-generated demonstrations, and the last includes 50 queries in one request.
Refer to caption: extracted/5902889/img/ManyICL_batchexp.png

### 4.3 Cost and latency analysis

Many-shot ICL incurs zero additional training cost, but per-query inference can be costly and slow due to long input contexts. To quantitatively measure this, we compute the latency and cost associated with the zero-shot and many-shot requests with and without batching when using Gemini 1.5 Pro on HAM10000 and TerraIncognita. We calculate the costs using the Gemini 1.5 Pro preview pricing ($7 per 1 million input tokens and $21 per 1 million output tokens). For fair comparison and to minimize data transfer artifacts, all requests are sent to the same location where the VM instance is held (“us-central1”). We run the query three times under each setting and report the average.

In the zero-shot regime, we see substantial per-example latency reductions due to query batching, close to a 10x reduction on HAM10000 and 2x on TerraIncognita (Table 3). The per-example cost is similar between the two as there is no additional context needed for including demonstrating examples. In the many-shot regime, we observe substantial reductions in both per-example latency and cost on both datasets. Specifically, for HAM10000, we find a near 35x reduction in latency and 10x reduction in cost, and 20x reduction in latency and 45x reduction in cost for TerraIncognita.

## 5 Discussion

In this study, we evaluate many-shot ICL of state-of-the-art multimodal foundation models across 14 datasets and find consistent performance improvements across most of the datasets. Batching queries with many-shot ICL further exhibits substantially reduced per-example latency and inference costs without compromising performance. We also highlight one future direction for improving open-weights multimodal foundation models as they are unable to learn from demonstrating examples.

Our findings suggest that these multimodal foundation models have the capability of performing ICL with large numbers of demonstrating examples, which may have significant implications on their practical use. For example, it was previously impossible to adapt these large, private models to new tasks and domains, but many-shot ICL would enable users to leverage demonstrating examples to adapt the models. One significant advantage of many-shot ICL is its ability to get quick results even on the same day of model release, and that’s why we can finish our evaluation using GPT-4o and Llama3.2 within days.
Furthermore, fine-tuning open-source models is the standard practice when practitioners have access to moderately sized datasets, but many-shot ICL may remove the need for fine-tuning, making it much easier to develop customized approaches. We note that it remains to be seen how traditional fine-tuning of these models compares to many-shot ICL with foundation models in terms of absolute performance and data efficiency, so future work should explore this. In addition, it is important to study general issues which plague those foundation models, such as hallucinations and biases, under the context of many-shot ICL and batching queries. For example, it would be interesting to explore if carefully curated and large sets of demonstrating examples can reduce biases across different sub-groups. We leave this to future work.

Our study has limitations. First, we only explore performance under many-shot ICL on some common multimodal tasks which we believe are the most practically relevant and common multimodal settings, but it is worthwhile for future work to explore potential benefits from many-shot ICL on other tasks such as VL-ICL Bench Zong et al. (). Second, even after recent developments to increase context size, the size prohibits many-shot ICL  from being used on datasets with a large number (several hundred or more) of classes. We anticipate that context window sizes will continue to increase in size over time which will mitigate this issue. Third, we only run the ablation experiments on Gemini 1.5 Pro due to budget limit, but it will be interesting to whether these trends hold for GPT-4o. Fourth, the datasets which were used to train these private models have not been disclosed, so it is difficult to tell whether the models have been trained on the datasets we selected. We argue that the zero-shot performance across the datasets is far from perfect, which suggests that the datasets have not been used for training, but we cannot determine that with certainty.

**Table 3: Inference latency and cost using Gemini 1.5 Pro with and without query batching. We use 50 queries per batch. In the zero-shot setting, we can achieve lower per-example latency with batching, but the per-example cost remains identical. In the many-shot setting, the per-example cost and per-example latency both drop substantially with query batching.**
| Dataset | Without Query Batching | With Query Batching |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Per-batch Latency | Per-example Latency | Per-example Cost | Per-batch Latency | Per-example Latency | Per-example Cost |
| HAM10000 (zero-shot) | 2.2s | 2.2s | $0.0038 | 11.4s | 0.23s | $0.0038 |
| TerraIncognita (zero-shot) | 2.0s | 2.0s | $0.0037 | 51.6s | 1.0s | $0.0038 |
| HAM10000 (350-shot) | 17.3s | 17.3s | $0.8420 | 26.9s | 0.54s | $0.0877 |
| TerraIncognita (810-shot) | 34.9s | 34.9s | $1.8420 | 85.9s | 1.7s | $0.0406 |

## 6 Conclusion

In summary, we show that state-of-the-art multimodal foundation models, especially Gemini 1.5 Pro, are capable of many-shot ICLto achieve substantial performance improvements with cost efficiency across multiple domains and tasks. However, open-weights models like Llama 3.2-Vision do not exhibit the same benefits, highlighting a significant future direction. We believe that these results pave a promising path forward to improve the adaptability and accessibility of large multimodal foundation models.

## Appendix A Prompts used for ICL experiments

### A.1 Prompt used for image classification experiments

### A.2 Prompts used for image classification experiments with batching

### A.3 Prompts used for batching ablation experiments

#### A.3.1 Prefixing images

### A.4 Prompt used for visual question answering experiments

### A.5 Prompt used for object localization experiments

## Appendix B Prompt selection

We utilize a different set of prompts to test the robustness of ManyICL to differences in prompt wording. We randomly sample two datasets (HAM10000 and EuroSAT) for this experiment due to budget limit.

### B.1 Prompts used for prompt selection experiments

Note that only the question section is shown here, and prompt 1 is used for all other image classification experiments.

#### B.1.1 Prompt 1

#### B.1.2 Prompt 2

#### B.1.3 Prompt 3

### B.2 Prompt selection results

Figure 6 shows the sensitivity of performance to prompt selection on two datasets with three prompts. While there exists a small deviation in performance, but the overall log-linear improvement trend is consistent.

Figure: Figure 6: Sensitivity analysis of many-shot ICL. These plots show the change in task performance on two datasets as the number of demonstrating examples increases, using three different prompts. For all experiments on sensitivity analysis, the Gemini 1.5 Pro model is used. The $x$-axis is in the logarithmic scale, representing the number of demonstrating examples plus one. The log-linear improvement until the optimal performance is consistent across all prompts selected.
Refer to caption: extracted/5902889/img/ManyICL_Fig5.png

## Appendix C GPT4(V)-Turbo performance under many-shot ICL

GPT4(V)-Turbo shows mixed results for many-shot ICL, with substantial performance improvements on HAM1000, UCMerced, EuroSAT, and DTD, but minimal improvements or no improvement across the other six datasets (Figure 7). However, we note that we were unable to increase the number of demo examples to the same level as Gemini 1.5 Pro because GPT4(V)-Turbo has a shorter context window and is more prone to timeout errors when scaling. Additionally, GPT4(V)-Turbo seems to generally underperform Gemini 1.5 Pro across the datasets excluding FIVES and EuroSAT for which it seems to mostly match the Gemini 1.5 Pro performance. GPT4(V)-Turbo performance on DrugOOD Assay shows high variance, resembling that of Gemini 1.5 Pro with the peak performance at 40 demo examples.

Figure: Figure 7: GPT4(V)-Turbo and GPT-4o performance from zero-shot to many-shot ICL. We show the accuracy of GPT4(V)-Turbo and GPT-4o as we vary the number of demonstrating examples across the 10 classification datasets. The $x$-axis is in log scale. HAM10000 and TerraIncognita exhibit a relatively smooth log-linear improvement for both models. On the UCMerced, Oxford Pets, and DTD datasets, GPT-4o displays a large drop in performance before reaching the best performance with the largest number of demonstrating examples.
Refer to caption: extracted/5902889/img/ManyICL_gpt4vturbo.png

## Appendix D Performance of many-shot ICL on medical QA tasks

### D.1 Prompt used for medical QA experiments (MedQA, MedMCQA)

### D.2 Results

Figure 8 shows the results on medical QA tasks.

Figure: Figure 8: Many-shot ICL performance of medical QA tasks. The $x$-axis is in log scale. GPT4(V)-Turbo consistently shows better performance compared to Gemini 1.5 Pro. The accuracy tends to increase for GPT4(V)-Turbo, but Gemini 1.5 Pro performance is more variable.
Refer to caption: extracted/5902889/img/ManyICL_medqa.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer the Weaknesses.
soundness: 3
strengths: + The overall paper is well organized and easy to follow. 
+ The idea of many-shot ICL is promising and interesting. This paper first benchmarks the many-shot ICL capabilities of state-of-the-art multimodal foundation models.
+ The dataset construction procedure is well designed, such as the diverse domain, various task and the class-balanced sampling.
summary: This paper investigates the many-shot in-context learning (ICL) capabilities of state-of-the-art multimodal foundation models. The authors benchmark multiple multimodal foundation models (GPT-4o, Gemini 1.5 Pro, Llama 3.2-Vision and InternLM-XComposer2.5) across 14 datasets covering multiple domains (natural, medical, remote sensing and molecular images) and tasks (image classification, visional question answering and object localization). Benchmark results indicate that the close-source models especially Gemini 1.5 Pro achieves substantial performance from many-shot ICL, while open-source models do not exhibit the same benefits.
weaknesses: - This paper benchmarks a limited number of models. There are only two open-source and two close-source models are evaluated in the main text. I doubt that it is hard to draw a general and convincing conclusion from such limited assessed models.
- The ablation experiments are only conducted on Gemini 1.5 Pro. Such analysis is not convincing enough, since some phenomena could be distinct when different models are utilized.
- Although this paper observes some interesting phenomena, but their deeper causes are not analyzed and explained, e.g., the reason why Llama 3.2-Vision and InternLMXComposer2.5 do not benefit from the demonstrating examples. It would be better to add more insightful analysis highlighting the future direction.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to weakness.
soundness: 2
strengths: 1.The paper provides a comprehensive evaluation of many-shot ICL across multiple datasets, showcasing the model's performance in various domains and tasks, with a rigorous research methodology.

2.The authors suggest that batching multiple queries can result in significant performance improvements in zero-shot settings. They design experiments to elucidate this phenomenon from various perspectives, including domain calibration and class calibration. These findings could have important implications for practical applications.
summary: The authors assess the many-shot in-context learning (ICL) capabilities of state-of-the-art multimodal foundation models across 14 datasets, discovering consistent performance enhancements in the majority of them. Additionally, employing batching with many-shot ICL significantly lowers per-example latency and inference costs, all while maintaining performance levels.
weaknesses: 1.Balanced sampling seems very important, but what if we test with random sampling or other class-wise sampling?

2.The paper explores query batching using examples from the same dataset. What if we batch images from different datasets together to explore the impact of cross-dataset queries?

3.We can observe the scaling performance with the increasing amount of the ICL. In addition to quantity,  does the quality matter? It is great to see the presented results on close-weight and open-weights LLM. Yet, Since the community did research on the ICL for a long while, I was also expecting a more theoretical or more convincing discussion and analysis on your findings.

4.Presentation is straightforward and easy to understand, I also think it is good to see such explorations on LMM. But, on the other hand, I did concern whether such study is qualified for a ICLR paper or just being a technical report? BTW, I cannot see any reference to Figure 5 in the paper.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Figure 5: The figure is not referenced in the text. 

2. Around Line 368: Do the authors have any hypothesis as to why CheXpert and EuroSAT do not scale up with the number of queries? Providing more insight or potential explanations would strengthen the analysis.

3. Hyper-Parameters for Inference: It would be beneficial for the authors to include details on the hyper-parameters used during inference, such as temperature settings, to enhance the reproducibility of the experiments.

4. Image Classification in A.1: How are the negative choices constructed for the image classification task? Are they sampled from other class names in an offline manner or generated dynamically in an online manner?
soundness: 2
strengths: 1. The authors conducted tests on domains that are not commonly used for benchmarking multimodal LLMs, such as medical and remote sensing. While prior studies have explored these areas, there has been relatively less compared to the natural image domain.

2. The authors also address inference efficiency, a major concern when introducing many-shot in-context learning (ICL). 

3. Error bars are presented in most of the figures.
summary: This work examines the capabilities of multimodal foundation models in the context of many-shot in-context learning (ICL). The authors evaluated both proprietary and open-source models using 14 datasets spanning multiple domains. Given the high inference costs associated with many-shot ICL, the study also explores the potential of batching multiple queries within a single API call. The findings reveal that batch calling not only reduces latency but also enhances model performance.
weaknesses: 1. Lack of In-Depth Analysis: The authors point out that open-source mLLMs struggle to scale effectively with an increasing number of in-context examples, whereas proprietary models handle this scaling better. However, this discrepancy could simply be attributed to differences in model size. This explanation is not fully explored in the paper, and a deeper analysis would make the argument more compelling.

2. Concerns About Cross-Domain Adaptation: The paper uses many-shot ICL to evaluate LLM performance on cross-domain adaptation (stated in the opening). However, it is unclear whether proprietary models have already been pretrained on these datasets (as acknowledged in the limitations section). This raises the possibility of data leakage, which could explain the performance gap between open-source and proprietary models. Furthermore, it remains uncertain whether the paper's main claim—that cross-domain adaptation can be mitigated by many-shot ICL—holds true, as some supposedly out-of-domain datasets may actually be in-domain for these proprietary models. I suggest the authors only test on open source models that are more transparent with their visual instruction tuning datasets. 

3. Benchmark Datasets: While the benchmark includes two small Visual Question Answering (VQA), it primarily focuses on image classification tasks/objection detection that are relatively simple for classic computer vision models, many of which already achieve 95% or 100% accuracy (eurosat, UCmerced). Improving the diversity of the tasks will improve the solidity of the claim. 

4. Minor issue: Oxford pets is the dataset from conventional CV domain.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: For open-weighted multimodal models that do not benefit from many-shot ICL, can improvements be proposed to enhance their performance in many-shot ICL?
The work mainly focuses on GPT-4o and Gemini 1.5 Pro closed LMMs. How to choose the appropriate model for Many-Shot ICL?
soundness: 3
strengths: 1.The authors reveal the performance trends of LMMs as the number of instances increases, noting that many-shot ICL can significantly enhance open-weighted multimodal models.
2. The work conduct extensive experiments on 14 datasets spanning different domains and multimodal tasks to evaluate model performance from various perspectives. This includes comparing the inference speed after employing batch query techniques.
3. In order to mitigate the increase in inference speed with the number of instances, this study utilize batch query techniques to reduce the inference speed and unveiled the impact of number of queries on network model performance with batch query technology.
summary: The study assesses the performance of multimodal foundation models in context learning from few-shot to many-shot, comparing the performance of GPT-4o and Gemini 1.5 Pro across 14 datasets in different domains and tasks. It is found that many-shot ICL led to significant improvements, with Gemini 1.5 Pro demonstrating higher ICL data efficiency on most datasets. Additionally, multimodal foundation models with open weights did not benefit from demonstration examples, highlighting the gap between open and closed models. The research also explored the impact of batch processing multiple queries, showing performance improvements and cost reductions under zero-shot and many-shot ICL.
weaknesses: 1. The introduction of this paper is simple. The benefits of many-shot ICL for LLMs have been well-documented in some works, and this work merely briefly highlights the differences from these works in the related works section, failing to adequately showcase the unique advantages of this work.
2. Although the work reveals that open-weighted models can achieve better gains than closed LMMs as the number of instances changes, it does not explain the reasons behind this phenomenon.
3. The work primarily focuses on model performance but lacks an in-depth analysis. The authors do not provide explanations for why closed LLMs perform poorly on certain datasets while others perform well. They merely present the corresponding experimental results.

Meta-review: This paper obtains four negtive scores from all reviewers. Lacking of in-depth analysis is a common concern and the authors do not resolve this issue. Most of reviewers further point out that this paper is a technical report rather than a paper. Meanwhile, the link posted in the response to reviewer DvdU also violates the double blind strategy, which should be desk rejected.

Decision: Reject