Many-Shot In-Context Learning in Multimodal Foundation Models
Authors: name: Yixing Jiang, name: Jeremy Irvin, name: Ji Hun Wang, name: Muhammad Ahmed Chaudhry, name: Jonathan H. Chen, name: Andrew Y. Ng

Large language models are well-known to be effective at few-shot in-context
learning (ICL). Recent advancements in multimodal foundation models have
enabled unprecedentedly long context windows, presenting an opportunity to
explore their capability to perform ICL with many more demonstrating examples.
In this work, we evaluate the performance of multimodal foundation models
scaling from few-shot to many-shot ICL. We benchmark GPT-4o and Gemini 1.5 Pro
across 10 datasets spanning multiple domains (natural imagery, medical imagery,
remote sensing, and molecular imagery) and tasks (multi-class, multi-label, and
fine-grained classification). We observe that many-shot ICL, including up to
almost 2,000 multimodal demonstrating examples, leads to substantial
improvements compared to few-shot (<100 examples) ICL across all of the
datasets. Further, Gemini 1.5 Pro performance continues to improve log-linearly
up to the maximum number of tested examples on many datasets. Given the high
inference costs associated with the long prompts required for many-shot ICL, we
also explore the impact of batching multiple queries in a single API call. We
show that batching up to 50 queries can lead to performance improvements under
zero-shot and many-shot ICL, with substantial gains in the zero-shot setting on
multiple datasets, while drastically reducing per-query cost and latency.
Finally, we measure ICL data efficiency of the models, or the rate at which the
models learn from more demonstrating examples. We find that while GPT-4o and
Gemini 1.5 Pro achieve similar zero-shot performance across the datasets,
Gemini 1.5 Pro exhibits higher ICL data efficiency than GPT-4o on most
datasets. Our results suggest that many-shot ICL could enable users to
efficiently adapt multimodal foundation models to new applications and domains.
Our codebase is publicly available at
https://github.com/stanfordmlgroup/ManyICL .