VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning
Authors: Yongshuo Zong
Ondrej Bohdal
Timothy Hospedales
Venue: ICLR 2025 Poster (2025)
Keywords: Vision-Language Models
In-Context Learning
LLMs
Primary area: datasets and benchmarks
TL;DR: A comprehensive multimodal in-context learning benchmark for both text-output and image-output.

Abstract: Large language models (LLMs) famously exhibit emergent in-context learning (ICL) - the ability to rapidly adapt to new tasks using few-shot examples provided as a prompt, without updating the model's weights. Built on top of LLMs, vision large language models (VLLMs) have advanced significantly in areas such as recognition, reasoning, and grounding. However, investigations into multimodal ICL have predominantly focused on few-shot visual question answering (VQA), and image captioning, which we will show neither exploit the strengths of ICL, nor test its limitations. The broader capabilities and limitations of multimodal ICL remain under-explored. In this study, we introduce a comprehensive benchmark VL-ICL Bench for multimodal in-context learning, encompassing a broad spectrum of tasks that involve both images and text as inputs and outputs, and different types of challenges, from {perception to reasoning and long context length}. We evaluate the abilities of state-of-the-art VLLMs against this benchmark suite, revealing their diverse strengths and weaknesses, and showing that even the most advanced models, such as GPT-4, find the tasks challenging. By highlighting a range of new ICL tasks, and the associated strengths and limitations of existing models, we hope that our dataset will inspire future work on enhancing the in-context learning capabilities of VLLMs, as well as inspire new applications that leverage VLLM ICL. Project page is at https://ys-zong.github.io/VL-ICL/

Published as a conference paper at ICLR 2025
VL-ICL BENCH: THE DEVIL IN THE DETAILS OF MULTIMODAL IN-CONTEXT LEARNING
Yongshuo Zong*, Ondrej Bohdal*, Timothy Hospedales
University of Edinburgh Co-first authors
Lyongshuo.zong ondrej. bohdal,, t.hospedalest @ed uk
ABSTRACT
Large language models (LLMs) famously exhibit emergent in in-context learning (ICL) the ability to rapidly adapt to new tasks using few-shot examples pro- vided prompt, without updating the model's weights. Built on top of LLMs, vision large language models (VLLMs) have advanced significantly such recognition, visual question answering (VQA), reasoning, and ground- ing. However, investigations into multimodal ICL have predominantly focused few-shot VQA and image captioning, which we will show neither exploit the strengths of ICL, nor test its limitations. The broader capabilities and lim- itations of multimodal ICL remain under-explored. In this study, introduce a comprehensive benchmark for multimodal in-context learning. Our VL-ICL Bench encompasses broad spectrum of tasks that involve both images and text as inputs and outputs, and different types of challenges, from perception reasoning and long context length. We evaluate the abilities of state-of-the the- art VLLMs on this benchmark suite, revealing their diverse strengths and weak- nesses, and showing that even the most advanced models, such as GPT-4, find the tasks challenging. By highlighting a range of new ICL tasks, and the as- sociated strengths and limitations of existing models, we hope that our dataset will inspire future work on enhancing the in-context learning capabilities of VLLMs, as well as inspire new applications that leverage VLLM ICL. Project page: /ttt:ps:/ysszoooe github. 1o/VL-ICL/.
areas
INTRODUCTION
With the scaling of model size, large language models (LLMs) famously exhibit the emergent capa- bility of in-context learning (ICL) (Brown et al., 2020; Dong et al., 2024). This refers to the ability to learn from analogy within a single feed-forward pass thus, enabling the model to learn com- pletely new tasks using a few input-output examples, without requiring any updates to the model parameters. This training-fir free nature of ICL has led to its rapid and broad application across wide range of scenarios and applications, as illustrated by benchmarks such as Hendrycks et al. (2021); Zhong et al. (2024); Srivastava et al. (2023); Cobbe al. (2021).
Vision large language models (VLLMs) are typically built on base LLM,, by augmenting it with vision encoder and/or decoder connected by some stitching mechanism (Liu al., 2024a; 2023; Bai et al., 2023; Zong et al., 2024; Alayrac et al., 2022: Ge et al., 2024). These models have rapidly advanced alongside LLMs, and attracted significant attention for their remarkable multi-modal capa- bilities in zero-shot recognition, reasoning, grounding, and visual question answering (VQA) among other capabilities. These capabilities have been thoroughly tested by a range of recent benchmark suites (Liu et al., 2024d; Fu et al., 2023; Li et al., 2024b; Yu et al., 2024). Mcanwhile, VLLMS are also widely presumed to inherit in-context learning (ICL. capabilities from their base LLM, however, their abilities in this respect are poorly evaluated and poorly understood. Current VLLMS studies mainly report their zero-shot capabilities measured by the benchmarks above, while ICL is usually only evaluated qualitatively, or as a secondary consideration via few-shot visual question answering (VQA) or image captioning (Bai et al., 2023; Awadalla et al., 2023; Sun et al., 2024a; Laurençon et al., 2023), with notable deficiency in quantitative assessment across wider spectrum of ICL tasks. This is presumably due to the ready availability of VQA and captioning benchmarking infrastructure. However, we will show that captioning and VQA tasks not ideal for ICL. evalu-

---

Published as a conference paper at ICLR 2025
Fast Oent-Open-eeeee nded MinilmageNNet
CLEVR
Large
Perpo
Perpo
Operator Induction
TextOCR
5
GOOD
Interleaved Operator
Fast Matching MinilmageNee
2
3
15
No
Textto-Ilmgge MinilmageNee
CoBSAT
Statin
Station
Lion
Shously
Fast Counting
FastAttribute Matching
Blicktt
Zorp ball
Box dogs
Zorp box
Froop
Figure 1:
Illustration the different tasks in VL-ICL Bench. Image-to-text tasks are in the first three rows, while text-to-image tasks are in the bottom two rows. Image-to-text tasks in the third row do reasoning on interleaved image-text inputs.
They neither truly exploit the ability of ICL to improve performance from examples; nor do they test the limits of what ICL can do. Thus there is lack of motivation for future VLLM research to better exploit and expose the underpinning LLM''s ICL ability.
ation:
To enhance the understanding of multimodal ICL and assess the ICL capabilities of state-of-the-art VLLMs, introduce novel benchmark suite VL-ICL Bench (Figure 1), tailored for assessing VLLM in-context learning. Our benchmark suite incorporates both text-output and image-output tasks, and designed test various facets of VLLMs, including fine-grained perception, reason- ing, rule induction, and context-length. We conduct comprehensive evaluations of state-of-the-ard VLLMs that are capable of processing interleaved image-text as inputs on our benchmark. Results reveal that although certain models exhibit reasonable performance on specific tasks, none demon- strate uniform excellence across the entire spectrum of tasks, and some models perform near chance level on some tasks. We hope that this systematic study of different opportunities and challenges for multi-modal ICL will support practitioners to know what currently possible and impossible in terms of training-free learning of new multi-modal tasks, and spur VLLM model developers to study how to expose as much as possible of the LLM's ICL ability to the multi moltil world.
To summarise contributions: (1) We demonstrate the limitations inherent in the common practice of quantitatively evaluating VLLM ICL via VQA and captioning. (2) We introduce the first thor- ough and integrated benchmark suite of ICL tasks covering diverse challenges including perception, reasoning, nule-induction, long context-length and cetttt-immage/mmage-tee benchmark suite, and highlight their diverse
(3) We rigorously
evaluate range of state of the art VLLMs strengths and weaknesses, as well the varying maturity of solutions to different ICL challenges.
BACKGROUND AND MOTIVATION
2.1 THE ICL PROBLEM SETTING
Given pre-trained VLLM an optional text instruction context set S {(x,, vi) of query example v and labels y, and a test example x", ICL models estimate
Po Po ",,",,,,)))
with a foed-forward pass. For LLMs, x and y are typically text. For VLLMs, x can be text and/or images, and be text (image-to-He ICL) images (text-to-image ICL).
We context and support interchangeably this paper.
2

---

Published as a conference paper at ICLR 2025
TexLOCR
liduction
Long-Contest
Interieavod Operaper Induyfian
CDLVR Count
Inouuction
Perception
leaving
LLVV OneVision 728
GPTOV
Fast Matehirig Mint
MinliaageNet
upen Ended
Fast
Fast Binging
Reasoning
PN3 Vision
DEFICS2:40 88
Text Fast Kinimagghhi
Aitribute Marching
SEED-LLaMA-14B
Inoucconn
Text Image
GILL
CaBSAI Fau Counting
Figure 2: Performance of top-5 image-to-text models and top-3 text-to-image models on our bench-
mark. Left: By dataset separately. Right: By capability evaluated, averaging over datasets.
This ICL setting is in contrast to the simpler zero-shot scenario, where pre- pre-trained models estimate po(y" lx= (T) purely based the pre-learned knowledge in 0 with additional training data pro- vided in S. The zero-shot scenario has been rigorously evaluated by diverse benchmarks (Liu et al., 2024d; Fu al., 2023; Li et al., 2024b; Yu al., 2024), and in the following section we discuss the limitations of existing ICL evaluations that motivate our benchmark.
2.2 COMMON PRACTICE IN ICL EVALUATION
The benchmarks that have been most popular in prior attempts quantitative evaluation of multi- modal ICL are VQA and image captioning. We focus discussion in this section image-to-tens models (Alayrac et al., 2022; Bai et al., 2023; Li et al., 2023b; Laurençon et al., 2023; Awadalla et al., 2023), in-context pext-to-image models (Ge et al., 2024; Koh et al., 2023) are relatively less common and less mature, so there no common evaluation practice yet. In the case of captioning, the context set S contains examples of images and captions y; while for VQA the context S contains image-question pairs and answers y.
Figure 3(a) plots the ICL performance of six popular VLLMS on three widely used benchmarks Math VQA (Lu et al., 2024), VizWiz VQA (Gurari et al., 2018), and CÓCO Captioning (Lin all.2201 2014) for varying numbers of training examples (shots). While the performance of the different models varies, the key observation is that most of lines/models show only limited improvement with ICL (shots > 0) compared to the zero-shot case (shots 0). This is because, while the context set illustrates the notion of asking and answering a question captioning images, the baseline VLLMM is already quite good VQA and captioning. The limiting factors in VLLM captioning and VQA are things like detailed perception, common sense knowledge, etc. all of which are fundamental challenges to the VLLM, and not things that can reasonably be taught by a few-shot support set.
Given the discussion above, it is unclear why performance should improve with shots all? We conjectured that this is largely due to the VLLM I about each dataset's preferred answer style, rather than learning to better solve multi-modal inference tasks per-se. For example, in captioning zero-shot VLLMM tend to produce more verbose captions than COCO ground-truth, and they learn to be more concise through ICL. Meanwhile, for VQA, there is standard practice of evaluating based on string match between the ground-truth answer and the model-provided answer. For example, VizWiz has unanswerable questions, which some VLLMS answer with "I don't t know" which would not be string matched against a ground truth "Unanswerable about mnsser-formattig (c.g., preferred terminology; avoid using any preface or postface that may throw of a string match) from the context set. This is indeed a kind of ICL, but perhaps not what one expects to learning in VQA. To validate this conjecture, we repeat the previous evaluation, but using soft matching to eliminate the impact of answer format learning. For VQA, we use pretrained LLM to determine whether the prediction has semantically equivalent to the ground-truth while for captioning, we use GPT-3.5 to score the quality of the generated caption on scale of score 1-10 (details in appendix). Fig. 3(b) shows that the curves have almost fully flattened out, with zero-shot performance having improved. Fig. 3(c) quantifies this difference by showing the average rate of improvement with shots for exact match and LLM match. The change to LLM validation almost completely eliminates any benefit of ICI over zero-shot.
Some models thus learn
3

---

Published as a conference paper at ICLR 2025
MathVista
VizWiz
COCO
140
OpenFlamingo-98
Accuracy (%)
20
Accuracy (%)
120
Otter
CIDEr
IDEFICS-9B
10
100
IntemLM-X2
Qwen-VL-Cha
EMU2-Chat
20
Number of shots
Number of shots
Number of shots
(a) Exact match evaluation.
MathVista
VizWiz
COCO
30
Accuracy (%)
rate score
OpenFlamingo-991
Otter
Accuracy
IDEFICS-9B
IntemLM-X2
40
20
Qwen-VL-Chat
5
EMU2-Chat
8
0
2
4
Number shots
Number shots
Number shots
(b) LLM as a judge for evaluation.
MathVista
VieWiz
COCO
Accurary Incrament (8)
Accuracy Increment (%)
Score Increment
LILL
4+
Standard metric evailaatio
7.
041
2-44 Number of shots
1-42
4-8
6+1
1-42
2-44
6-4
0-11
1-42
2-44
4-48
Number shots
Number shots
(c) Accuracy increments when adding more support examples.
Figure 3: VQA and Captioning are poor benchmarks for image-to-text ICL, (a) Evaluating state-of- the-art VLLMs representative examples of popular image-to-text ICL benchmarks Math Vista, VizWiz, and COCO with standard exact match evaluation protocol. The amount of in-context learning is limited in many cases. (b) Re-evaluation of VLLMS with LLM-based evaluation almost eliminates improvement with context size. (c) The impact of ICL on performance goes from limited negligible when moving from traditional to LLM-based evaluation. ICL on these benchmarks primarily learns answer style/format.
In contrast the above, popular LLM ICL benchmarks in the language domain do usually exhibit non-trivial ICL learning (Brown et al., 2020; Dong et al., 2024). Figure 4 shows three state of the VLLMs along with their corresponding base LLMs. evaluated on three popular NLP tasks (AG- News (Zhang et al., 2015), MIT Movies (Ushio & Samacho-Collados 2021) and TREC ( (Voorhees & Tice, 2000)). We can see that contrast to the VQA/Captioning benchmarks, models' zero-shot performance is often substantially improved by few-shot ICL. This result confirms that the LLM components in VLLMs do inherit the ICL ability their base LLM. However, it raises the question of how we can meaningfully exploit and measure the ICL ability of VLLMs in the multi-modal context. In the next section, we introduce our benchmark VL-ICL Bench, which does exactly this.
VL-ICL BENCH
3.1
MAIN MULTIMODAL BENCHMARK
Our VL-ICL Bench covers a number of tasks, which includes diverse ICL capabilities spanning concept binding, reasoning fine-grat fine-grained perception. It covers both image-to-tex, and text-to- image generation. Our benchmark includes ten tasks detailed below, illustrated in Figure 1. Table summarises the diverse capabilities tested by each VL-ICI. Bench task, and demonstrates its com-
4

---

Published as a conference paper at ICLR 2025
AGNews
MIT Movies
TREC
Accuracy (80)
60
Next-7B
Vicuna-78
70-
QWen-VL-Chat
60-
QwanLM477
40
InternLM4XX2
40
InternLM2 Chat-78
2 4
Number of shats
Number of shats
Number of shats
Figure 4: Evaluating state-of-ther art VLLM/LLM pairs popular text-to-text ICL benchmarks. Few-shot ICL often substantially improves on zero-shot performance, indicating that meaningful in- context-learning is taking place, unlike for the popular image-to-text VLLM benchmarks in Fig. 3.
Table 1: VL-ICL Bench overview. It evaluates diverse capabilities and challenges of ICL with VLLMs, while being compact and easy to be use without prohibitive resource requirements.
Dataset
Capabiliies Tested
Train Set
Test Set
Size (GB)
Open MinilmageNeet
12T. Binding
CLEVR Count Induction
12T. Fine Grained Perceptiom, Induccion
8000
200
0.18
Operator Induction
12T. Induction, Reasoning
80)
60
0.011
Interleaved Operator Inclaction
12T. Induction, Reasoning, Interleaving, Long-Context
80
60
0.01
TeaOCR
12T. Fine Grained Perception, Indlaction
Matching MimilmapeNer
12T, Induction, Interleaving, Long-Context
1,6010)
Test-to-lmage MinilmageNet
T21. Fast Binding
5,0000
200
0.18
CoBSAT
T2L. Induction
800
200
0.07
Counting
T2L. Fist
Matching
T2L, Fast Binding
Total
T2L, 12T, Rinding, Perception, Long-Context, Interieaving, Induction, Reasoning
15,260
1,760
1.82
pactness. We follow the typical protocol of the ICL community (Dong et al., 2024; Tsimpoukelli et al., 2021; Min et al., 2022)² and split each dataset into train and test splits. Few-shot ICL is then performed/evalluated by sampling the support/context set from the training split, and the test/query examples from testing split. The final performance is the average of number of such ICL episodes.
Fast Open MinilmageNet We use the variant of MinilmageNet few-shot object recognition (Vinyals et al., 2016) repurposed for ICL in Tsimpoukelli et al. (2021). In open-ended classification, VLLMs must name objects based on a few examples, rather than classifying them into predefined categories, making the chance rate effectively zero. Fast-binding tasks test models' ability to asso- ciate novel names or symbols with concepts, without prior knowledge. Tsimpoukelli et al. (2021) assign synthetic names (e.g., dax perpo) to object categories, and the model must learn these associations to name test images. We use the two-way version of this task.
CLEVR Count Induction In this dataset, models must learn solve tasks like " "How many red objjctt are there in the scene?" from examples rather than explicit prompts. We input CLEVR scene images (Johnson et al., 2017) along with attribute: value pair identifying a specific object type based on four attributes: size, shape, color, or material. The output is the count of objects matching the attribute. To succeed, the model must ground the attribute in the image, differentiate object types, and induce that the required operation is counting."
Operator Induction In this image-to-text task, models must solve tasks of the type 2 given training examples like 1 ? 3 = 4. I.c., besides parsing an image I to extract the numbers and operator, models need to induce that the role of the unknown operator is addition, and then conduct arithmetic reasoning on parsed test examples. Available mathematical operations are plus, minus and times, and we only consider single digit numbers. We generate our own images for this task.
Interleaved Operator Induction This task tests the ability of models to reason over multiple images within x to produce single answer y. In this variation of operator induction we give the model two query images input containing each number in the expression, rather than single image containing the whole expression, as above. While separating the images simplifies perception difficulty of parsing expressions, it also increases difficulty by requiring reasoning between two
"This is different than the shot meta-learning community (Wang et al., 2020; Hospedales al., 2021), which samples support/query sets from the same pool.
"This task could he performed zero-shot with suitably detailed VQA prompt. But the goal is to test whether models learn the task from few examples by ICI.
5

---

Published as a conference paper at ICLR 2025
images, whereas VLLMs are typically trained on single-image tasks. Additionally, multiple images increase the token count, challenging the VLLMs' ability to handle larger context lengths.
TextOCR We repurpose the TextOCR dataset (Singh et al., 2021) to create a task where the model should learn to output the text shown in the red rectangle, as inspired by (Sun et al., 2024a). Images contain image with window of text highlighted, and outputs y are the OCR text. This task could be achieved by a suitably detailed zero-shot prompt, but unlike Sun et al. (2024a), we focus on evaluating whether the task can be induced by way of example through ICL. Thus this task tests both fine-grained perception and induction capabilities.
Fast Matching MinilmageNet This task is the simplest example of supervised learning of relation between two images. For relation learning, inputs contain an image pair x = {x1, }, and output y indicates whether specific relation holds between them. VLLMs are required to learn the relation from examples where it does (r(z1, 22) true) and does not (r(z1, 22) false) hold, using artificial keywords to name the relation. We re-use MinilmageNet (Tsimpoukelli et al., 2021; Vinyals et al., 2016) and the relation to learn is whether the two images come from the same class or not (Sung et al., 2018). This task tests induction, long-context, and multiple interleaved images.
Fast Text-to-Image MinilmageNet We introduce further variation of MinilmageNet (Tsim- poukelli et al., 2021; Vinyals et al., 2016), which inputs synthetic category names (for fast bind- ing), and outputs images y. The model should learn from the context set to associate synthetic names with distributions over images, and thus learn to generate a new image of the corresponding category when prompted with the artificial category name. This task tests image generation and fast binding.
CoBSAT We also utilize a recent text-to-image CoBSAT (Zeng et al., 2024) benchmark as part of our larger VL-ICL Bench suite (Figure 1). This is text-to-image task where the model must learn to synthesise images y of a specified text concept x (e.g., object category), but furthermore there is latent variable common to the context set examples that must be induced and correctly rendered in novel testing images (c.g., common color of objects). This task tests image generation and latent variable induction. We also use selected CoBSAT images for Fast Counting and Fast Attribute tasks.
Fast Counting In the fast counting task we associate artificial names with counts of objects in the image. The task is to generate an image that shows a given object in quantity associated with the keyword (e.g. perpo dogs where perpo means two). We use counts between one and four to make the task solvable, and ask the model to distinguish between two counts in a task.
Fast Attribute Matching Similarly to the other tasks, we ask the model to learn to associate artificial names to a concept, in this task to attributes (e.g. color red). We then ask the model to generate an image of an object with the given attribute, when given the name of the object and the keyword associated with the attribute. We show the model two attributes in the support set.
Capability Summary The VL-ICL Bench suite described above goes far beyond any individual existing ICL benchmark to test diverse capabilities of multi-modal ICL including (Table I): Both text-to-image and image-to-text generation: fast-binding the ability to rapidly ground new symbols to visual concepts and re-use those symbols in the context of new data; fine- grained perception - as required to count or read text; interleaving the ability reason over the content of multiple images when generating a single output; rule induction - inducing non-trivial concepts such as mathematical operators and latent variables from examples; simple reasoning such as arithmetic; and long-context the ability of a VLLM to usefully exploit a large number of context tokens.
Text Variation In order to compare the impact of multimodality we also include text-version alternatives for our tasks. For datasets such as open-ended MinilmageNet, instead of images we provide image captions and use those for reasoning. For example, in CLEVR we provide enumer- ation of the objects in the scene, including their attributes. Note that text versions are not practical for all of the tasks, particular Tex1OCR is difficult to translate into a suitable text alternative.
4 RESULTS
4.1 EXPERIMENT SETUP
Models We evaluate a diverse family of sfate-of-the-ar models with various sizes (ranging from 0.5B to 80B) and different LIM backbones our benchmark. Specifically, for image-to-text

---

Published as a conference paper at ICLR 2025
VLLMs, we select Open Flamingo (9B) (Awadalla al., 2023), IDEFICS (9/80B) (Laurençon et al., 2023), IDEFICS-v2 (8B) (Laurençon et al., 2024). Otter (9B) (Li et al., 2023b), InternLM- XComposer2 (7B) (Zhang et al., 2023a), LLaVA-Next 7B) (Liu al., 2024b), Qwen- Chat (9B) (Bai et al., 2023), Emu2-Chat (34B) (Sun et al., 2024a), VIIAA (7B) (Lin et al., 2024), Mantis (-Idefics2) (Jiang al., 2024a). Phi-3-Vision (4B) (Abdin et al., 2024), LongVA (Zhang et al.. 2024b), 1eemmLMXXCompser2.555 (7B) (Zhang et al., 2024a), and LLaVA-One (0.5/7772B) (Li al., 2024a). For Text-to-image VLLMs, we use GILL (7B) (Koh et al., 2023), SEED-LLaMA (8B. 14B) (Ge et al., 2024), Emul (14B) (Sun et al., 2024b), Emu2-Gen (34B) (Sun al., 2024a). We also evaluate GPT4V (Achiam et al., 2023) on our benchmark. We use officially released model weights or GPT4 API and adopt greedy decoding for reproducibility. All experi- ments are conducted using three different random seeds and we report the average performance.
Evaluation Metrics All our experiments evaluate test accuracy shots (ef: Fig. 3 and Fig. 5). To summarise these curves for easy comparison we use three main metrics: zero-shot accuracy, peak (max.) accuracy over all shots and ICL efficiency (the area under the accuracy vs shots curve above the zero-shot starting point, normalized over the whole area). For text-to-imat -mage models, we employ LLaVA-Next-778 as the judge model to determine whether the generated images are correct (e.g., whether contain the target attribute). Implementation details are detailed in Appendix A.
function of the number of
Prompt For consistency, we employ the following standard prompt format for in-context learning.
[Task Description]
Support Set: [Imagg][Question//Annvee]]] (n-shot)
Query: [Image][Question]
Prediction: [Answer]
4.2
MAIN RESULTS
The main results for VL-ICL Bench are illustrated for selective subsets in Fig. 5 and summarised quantitatively in Tab. and Tab. 3 for 12T and T2I benchmarks respectively. We make the follow- VL-ICL Bench tasks. Unlike the common VQA and captioning benchmarks (Fig. 3), our tasks have low zero-shot performance and in every task at least one model shows a clear improvement in performance with number of shots. Thus, ICL capability is now indeed being demonstrated and exploited. (2) VLLMS often struggle to make use of larger number of ICL examples. For several tasks and models performance increases with the first few shots; but the increase is not monotonic. Performance often decreases again as we move to a larger number of shots (e.g.. GPT4V CLEVR Count Induction; MitemLL-XCooposerr Operator induction in Fig. 5). More extremely, some models obtain neg- ative impact from more shots, leading to negative ICL efficiency in Tab. We attribute this to difficulty of dealing with the larger number of images and tokens confusing the model and over- whelming the value of additional training data. is exacerbated by the difficulty of extrapolation over context length and number of input images, which for higher-shot ICI. becomes greater than the context length and image number used for VLLMM training. This shows an important limit of the current Mate-of-the-ard in ICL: Future models must support longer contexts and more images to benefit from larger support sets. (3) LLaVA-OneVisile
ing observations: (1) VLLMs demonstrate non-trivial in-context learning
72B is the best overall image-to-text model, closely followed by GPT4V. (4) Zero-Shot Performance is not strongly indicative of ICL ability. For example on VL-ICL Bench, which surprising as is a strong model in mainstream zero-shot benchmarks. This is due to point (2): Its training protocol uses one image at time, and it uses large number of tokens per image - thus ICL requires to extrapolate substantially input image number and token number, which it fails to do. (5) There is No clear winner among text-to-image models. However, text-to-image models have more consistent shot scaling than image-to-tes sange-to-texx models. This is due to training with more diverse interleaved datasets that provide multiple input images per instance, and using fewer tokens per image for better scaling.
(Liu et al., 2024b) is perhaps one of the worst overall
We remark that our zero-shot and ICL efficiency summary metrics (Tabs 2, 3) can sum to one most. These metrics are visualised together in Fig. 14, along with the best possible Pareto-frrom The differing degree to which each model/dataset relies on 7S ICL performance is easily visible.
7

---

Published as a conference paper at ICLR 2025
Table 2: Average zero-shot, peak and efficiency scores (%. 1) of different models on VL-ICL Bench image-to-text datasets. LLaVA-OneVision- and GPT4V show the best ICL abilities.
Oyen Ended Minimageeee
Operator
Intectcared Operalur
Fust Minilmagy
Ang.
Za..
PL.
Ent.
Z.s.
Fl.
Edl
Z.s.
PL
ER
Zs.
PL
Edt.
Zs..
PL
ER.
Zs.
Ps.
Edr.
Z.s.
Ps.
EIC
Upenilumuugg- 96
0.01
58.2
46.2
18.8
16.6
5.0
7.8
0.0
0.0
0.0
0.0
8.9
0.0
29.1
20.2
9.1
13.3
8.7
IEEFCCS-B
0.0
10.3
IDEFICS2-65
0.0
37.3
31.5
35.3
-18.1
16.3
Uner
60
206
53
-100
0,0
0.8
-10
0.0
0.9
14.5
130
InsenLM X2
0.0
50.3
36.1
26.0
194
26.1
40,
10.0
K.7
160
3,3
25.3
25.3
-18.2
0.0
49.9
25.3
5.5
9.3
10,0
Quea VI CChe
8.0
16.0
17.
00
10.2
76
15.0
35.0
13
48
243
16.1
163
163
-83
0.0
0.3
0.1
33
10.8
8.1
LLaVs Neat
0.0
37.2
29.4
0.0
25.2
14.5
106
106
488
24.7
24.7
23.0
13.9
13.9
.78
0.0
0,0
08
13
13.7
13.2
0.0
31.6
13.1
0.0
10.3
11.0
VILA Maniislasiis2
15.7
103
0.0
55.2
ViesaV
0.0
50.0
33.3
0.0
34.5
204
0.0
54.4
46.8
K.0
41.5
25.0
6.0
27.8
18.9
0.0
90.3
42.4
9.2
6.3
53
LongNA 71I
0.0
12.7
19
0.0
79.
11.6
183
18.3
-1.8
76.0
76.0
-1.0
100
100
-33
0.0
50.0
45.0
6.3
10.5
93
LLAV LLLVW-OeVVion-775
0.0
98.7
58.4
0.5
42.3
127
33.3
75.6
298
48.5
51.7
244
36.3
45.8
28
0.0
70.3
505
2.5
1.7
38
Inoent M-X388
0.01
62.7
61.
161
0.3
13.0
43.5
43
38.
18.1
0.0
66
26.
17
0.0
418
42.0
74.0
322
0.0
35.
Table 3: Average zero-shot, peak and efficiency scores (%, 1) of different models on VL-ICL Bench text-to-image datasets.
Model
Text-to-Image Fast MinilmageNet
CoBSAT
Fast Counting
Fast Attribute Matching
Avg. Rank
Z.s.
Pk.
Eff.
Z.s.
Ph.
Eff.
Z.s.
Pk.
Eff.
Z.s.
Pk.
Eft.
Z.s.
Pk.
Eff.
GILL
16.0
13.6
34.2
34,2
-26.3
20.5
20.5
-13.2
3.5
SEED-LLaMA
0.0
16.5
13.3
0.5
33.7
24.5
34.2
56.7
12.7
21.0
34.3
10.5
3.5
2.5
2.2
LHEDD-LLMM-14B
21.2
16.3
5.5
43.8
30.7
41.3
51.6
1.9
22.4
35.6
8.4
2.0
Emul Gen
0.5
31.5
22.5
0.3
9.7
7.1
33.2
47.8
8.2
23.5
29.1
3.1
3.5
3.8
2.8
Fmm22-Ge Gen
0.0
37.0
28.6
8.7
28.7
15.6
44.2
59.2
5.2
26.5
34.0
-0.2
1.5
2.0
4.3 ADDITIONAL ANALYSIS
We next use VL-ICL Bench to analyse several challenges and factors influencing ICL performance.
Fast Concept Binding In our open minilmageNet task, we follow Tsimpoukelli al. (2021) to require fast-binding of synthetic concept names so as to purely test models' ICL ability, without confounding by VLLMs' zero-shot ability to associate visual concepts with names. Tab, compares the fast and real-world minilmageNNe recognition. Our fast-binding case is much more challenging.
Direct comparison of multimodal and text ICL We can disentangle the role of text versus im- age inputs for some image-to-text VL-ICL Bench tasks, where we can easily provide semantically equivalent text input describing the image, in place of image tokens. Fig. 6 shows comparison between image-input vs text-input for count induction, operator induction, and interleaved operator induction tasks. With text-input, performance grows much more sharply and consistently with num- ber of shots. This is attributable to both (i) reduction of perception difficulty, and (ii) reduction in the total number of tokens compared to image input.
Disentangling context length and in-context learning One direct factor limiting VL ICL is the model's context length, as images translate to large number of tokens (c.g., in LLaVA, one image translates to 576 tokens, causing 8-shot setting exceed 4k context window). This makes difficult to disentangle whether the limitation on VL-ICL performance arises from context length constraints or inherent challenges in the models' ability perform ICL. To further understand this issue, we apply SelfExtend (Jin et al., 2024), a training-free position encoding extrapolation strategy that has been shown to effectively extend the context length of LLMs. We use SelfExtend to extend the context length of LLaVA and VILA from 4k to 16k. As shown in Tab. 5 (Fig. 15 in the Appendix), though it can help in some cases, performance often remains similar, suggesting
Table 4: Average zero-shot, peak and efficiency scores (%, 1) on fast open-ended and real-world MinilmageNet dataset. Real-world version is much easier, with larger shoo and peak accuracies.
Dataset
IEFICS--0B
Otler
Mo-e
Oven-YL-Chut
LLaVA-Nest-7B
GPT4Y
Z,s
Pk.
E3ft
Z,5
Pk.
E3f.
Z,5.
Pk.
E2f.
Z.s.
Pk.
EHf.
Z.5.
Ph.
Eft.
2.6.
Eff.
Open-Eidee
0.00
28.50
2062
0.00
33.67
World
30.50
94.77
38.75
35.12
32.17
64.50
8

---

Published as a conference paper at ICLR 2025
CLEVR Count Induction
Operator Induction
Text-to-lmage Fast MinilmageNet
Accuracy (%)
Accuracy (%6
Accuracy (96)
30
16
26
20-
10
2 Number of shots
Number of shots
Number of shots
IDEFICS-808
Otter
LLavA-Next-78
PhiB-Vision
InternLM-225
IDEFICS2-88
InterLM-X2
VILA-B
LLaWW-Oneiisoon725
GPT4V
GILL
SEED-LLaMN-8B
SEED-LLaMA-14B
Emul-Gen
Emu2-Gen
Figure 5: Illustrative VL-ICL Bench results. Our tasks better exploit and evaluate ICL compared to the mainstream in Fig. 3. They have lower zero-shot accuracy and more substantial gains with shots.
CLEVR Count Induction
Oper stor Induction
Interleaved Operator induction
Accuracy (88)
40-
8
60
30
Accuraicy
Accuracy 20
26 15
40
20-
Number shots
Number of shots
Number of shots
InternLM:X2
Qwen-VLLChat
LLaVA-Next-78
Figure 6: Comparison of multimodal (dashed line) and text (solid line). Performance increases more sharply and consistently with text inputs, highlighting the greater difficulty of multimodal ICL.
that long-context capabilities are necessary but not sufficient for improving ICL. Furthermore, even recent models with 1 128k context windows, like LLaVA-One Vision, do not always show improved performance with more shots as seen in Tab. 2. This highlights that VL-ICL is a more complex challenge, requiring understanding of interleaved few-shot examples, which cannot be solved solely by increasing context length.
Emergent threshold of multimodal ICL We further investigate whether there is threshold in model size at which emergent ICL abilities appear, as described by Wei et (2022a) for LLMs. To test this, we use the LLaVA-One Vision series (Li et al., 2024a), which share the same architecture and training data but differ in LLM size (0.5B, 7B, and 72B parameters). Notably, we find that the 72B model demonstrates emergent ICL abilities. As shown in Figure 7, the 72B model improves with more shots across all tasks, particularly in Interleaved Operator Induction and Fast Matching MinilmageNet, where the 0.5B and 7B models perform worse than chance as shots increase. This indicates that the 72B model understands the tasks, while the smaller models fail, highlighting the impact of model size on ICL and the presence of emergent threshold.
Fast Open-Ended MinilmageNet
Interleaved Operator Induction
Fast Matching MinilmageNet
Accuracy 1%)
100
Accuracy (96)
Accuracy (%)
40
20
2
Number shots
Number shots
Number shots
LLaA--Oneiisi 0.5B
LLaVA- DOVV--Oneiision-225
Figure 7: Comparison different model sizes. Dashed line indicates random chance performance assuming available options (the models do open-ended generation can be worse than random).
Additionally, we provide thorough analyses in the Appendix B and C on various factors, including scaling to many shots (up to 64), ICL efficiency versus zero-shot performance, chain-of-thouge prompting, repeating the support set, varying levels of task descriptions, and qualitative results.
9

---

Published as a conference paper at ICLR 2025
Table 5: Comparison of models with and without context extension strategy (SelfExtend), While is helpful in some cases, context extension does not necessarily improve the performance of ICL.
Fast Open-Ended MinilmageNet
CLEVR
Dataset
Operator Induction
TextOR
Count Induction
Z.s.
Pk.
Eff.
Z.s.
Pk.
Eff.
7.s.
Pk.
Eff.
Z.s.
Pk.
Eff.
-aaV--Net-7BB (w/ SelfExxendd)
0.0
37.2
29.4
0.0
25.2
19.3
10.6
10.6
24.7
24.7
-23.0
LLaVA Next-BB (w/ SelfExtend)
0.0
51.0
38,9
0.0
29.0
25.4
11.7
11.7
-5.8
26.0
26.0
-23.7
VILA-7B (wlo SelfExtend)
0.0
38.2
32.3
3.5
34.3
27.5
28.3
28.3
-18.9
28.0
30.2
-3.7
VILA-7B (w/ SelfExtend)
0.0
54.0
40.0
4.0
34.8
27.5
28.3
28.3
-20.3
28.0
29.7
-.5
5 RELATED WORK
VLLM Evaluation With the rapid development of VLLMs, researchers are creating evaluation benchmarks to thoroughly assess their capabilities from diverse perspectives. These evaluations range from zero-shot aggregated benchmarks like MME (Fu et al., 2023), MMbench (Liu et al., 2024d), and MM-VET (Yu et al., 2024) to datasets designed for fine-tuning on specific aspects, such as visual reasoning (Hudson & Manning, 2019) and knowledge knowledge-grounded QA (Lu et al., 2022). They predominantly focus on single-image, leaving in-context learning evaluation undenexplored.
In-Context Learning Evaluation The term "in-context" has been used in a few ways, including to describe scenarios with interleaved inputs, such multiple video frames or multi-turn conversa- tions (Li et al., 2023a;b; Zhao et al., 2024; Ge et al., 2024). Although the study of interleaved inputs presents an intriguing subject, it does not align with the core definition of in-context learning that we consider following (Brown et al., 2020; Dong et al., 2024; Min et al., 2022), which involves the emergent ability to learn function from x + y from few-shot support input-output pairs. Prior evaluation of ICL (Awadalla et al.. 2023; Laurençon et al., 2023: Sun et al., 2024a; Baldassini et al., 2024; Chen al., 2024) in this sense limited, and comes with serious drawbacks as discussed in Sec. 2.2. Concurrent to our work, CobSAT (Zeng et al., 2024) introduces a benchmark designed to evaluate in-context learning in text-to-image models, focusing particularly on latent variable induc- tion capabilities. Our work spans much wider range of tasks and capabilities, encompasses both image-to-ted and text-to-image generation and subsumes CobSAT as one of our ten total bench- marks (Tab. 1). The concurrent work of Jiang al. (2024b) studies ICL shot-scaling, but only for 12T recognition tasks in contrast to our wide range of task types (Tab. 1), and without elucidating the limitations of mainstream VQA/captioning paradigm we have in Sec. 2.2.
Visual In-Context Learning The term "in-context" has also been used in pure vision models, which aim to perform diverse image-to-image tasks without task-specific prediction heads (Bar et al., 2022: Wang et al.. 2023a;b), such as semantic segmentation, depth estimation, object detection, etc. However, these models are explicitly trained on paired in-context input-output data to be able to perform visual ICL during inference. In this paper, we focus on multimodal vision-language ICL. which is based on the emergent ability of LLMs.
DISCUSSION
We have introduced the first comprehensive benchmark suite VL-ICL Bench for multi-modal vision- and-language in-context learning. This benchmark suite avoids the issue with the existing main- stream but limited approach to evaluating image-to-text ICL-that that ICL provides limited demonstra- ble benefit over zero-shot inference, and VLLMs learn answer formatting at best rather than any true multi-modal capability. In contrast, VL-ICL Bench tests a wide variety of multi-modal capabilities including both text-to-image and image-to-text generation, fine-grained perception, rule-induction, reasoning, image interleaving, fast concept binding, long context, and shot scaling. We hope this benchmark will inspire model developers to consider all these capabilities in VLLM development, and inform practitioners about the evolution of what VLLM ICL can and cannot do as the field de- velops. One limitation of this work is that we evaluate only a small number of text-to-image models, in contrast to the more comprehensive set of image-to textttt models. This is due to the limited avail- ability of VLLMs capable of handling interleaved inputs and generating images. Developing such models presents promising direction for future research.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
Yongshuo Zong is supported by the United Kingdom Research and Innovation (grant EP/S02431X/I), UKRI Centre for Doctoral Training in Biomedical Al at the University of Edin- burgh, School of Informatics. For the purpose of open access, the author has applied creative commons attribution (CC BY) licence to any author accepted manuscript version arising.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint 1iti:24044.42111 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint (oXix:2333.08777 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 2022.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.00 01390, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 2023.
Folco Bertini Baldassini, Mustafa Shukor, Matthieu Cord, Laure Soulier, and Benjamin Piwowarski. What makes multimodal in-context learning work? CVPR Workshop, 2024.
Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. NeurlPS, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurlPS, 2020.
Shuo Chen, Zhen Han, Bailan He, Mark Buckley, Philip Torr, Volker Tresp. and Jindong Gu. Under- standing and improving in-context learning on vision-language models. ICLR Workshop, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint
2021.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. EMNLP, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer. ICLR, 2024.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. CVPR, 2018.
11

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ICLR, 2021.
Timothy M Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey. Meta-Learning in Neural Networks: A Survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. CVPR, 2019.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. TMLR, 2024a.
Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H Chen, and Andrew Y Ng. Many-shot in-context learning in multimodal foundation models. arXiv preprint 209.09900.09798. 2024b.
Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia- Yuan Chang, Huiyuan Chen, and Xia Hu. Lim maybe longlm: Self-extend Ilm context window without tuning. ICML, 2024.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. CVPR, 2017.
Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. NeurIPS, 2023.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. NeurIPS, 2023.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision- laiguale models? NeurlPS, 2024.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Fanyi Pu, Jingkang Yang. Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023a.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726. 2023b.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.00 03326,, 2024a.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal IIms with generative comprehension. CVPR, 2024b.
Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, and Xu Yang. How to configure good in-context sequence for visual question answering. CVPR, 2024c.
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. CVPR, 2024.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. CVPR, 2024a.
12

---

Published as a conference paper at ICLR 2025
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, oct, and world knowledge, January 2024b. URL https://
1lava-vl- vw.github..... 2ttpbllogg/24--1122220 11000--1aaaaexx///
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models long contexts. TACI., 2024c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, al. Mmbench: your multi-modal model an all-around player? ECCV, 2024d.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurlPS, 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hanmaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. ICLR, 2024.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetalCL: Learning to learn in context. NAACL, 2022.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. ICLR, 2022.
Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojeiech Galuba, and Tal Hassner. Tex- (OCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. CVPR, 2021.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. TMLR, 2023.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al. Generative multimodal models are in-context learners. CVPR, 2024a.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang. Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. ICLR, 2024b.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales. Learning compare: Relation network for few-shot learning. CVPR, 2018
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Mul- timodal few-shot learning with frozen language models. NeurIPS, 2021.
Asahi Ushio and Jose Camacho-Colladed Collados. T-NER: An all-round python library for transformer- based named entity recognition. EACL System Demonstrations, 2021.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. NeurlPS, 2016.
Ellen M Voorhees and Dawn M Tice. Building question answering test collection. SIGIR, 2000.
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images:
A generalist painter for in-context visual learning. CVPR, 2023a.
Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Towards segmenting everything in context. ICCV, 2023b.
Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. Generalizing from a few exam- ples: A survey few-shot learning. ACM Comput. Surv., 2020.
13

---

Published as a conference paper at ICLR 2025
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. TMLR, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurlPS, 2022b.
Xu Yang. Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng. Exploring diverse in- context configurations for image captioning. NeurIPS, 2023.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. JCML, 2024.
Yuchen Zeng. Wonjun Kang, Yicong Chen, Hyung Koo, and Kangwook Lec. Can milms perform text-to-image in-context learning? COLM, 2024.
Pan Zhang. Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuan- grui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. Intemonm-cooppoer: A vision- language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.1511 2023a.
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, al. Intemln-xcoomoosr-2555. A versatile large vision language model supporting long-contextual input and output. arXiv preprint mxXiv2440.033320 2024a
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language vision. arXiv preprint arXiv:2406. 0064006666852 /6852, 2024b.
Xiang Zhang. Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. NeurIPS, 2015.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? NeurIPS, 2023b.
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering vision-language model with multi-modal in-context learning. ICLR, 2024.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A human-centrie benchmark for evaluating foundation models. NAACL Findings, 2024.
Yongshuo Zong, Oisin Mac Aodha, and Timothy Hospedales, Self-supervised multimodal learning: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
Table of Content
Section Implementation and Evaluation Details
Section B: Further Experimental Analysis
Subsection B.1: Full Main Results (Figure 8).
Subsection B.2: Results of Scaling to Many Shots (Figure 9).
Subsection B.3: Results of Shain-of-Thougie Prompting (Figure 10).
Subsection B.4: Results of Repeating the Support Set (Figure 11).
Subsection B.5: Influence of Instruction Fine-tuning ICL (Figure 12).
Subsection B.6: Different Levels of Task Descriptions (Figure 13).
Subsection B.7: ICL Efficiency vs Zero-Shot Performance (Figure 14).
Subsection B.8: Analysis on Context Extension (Figure 15).
Subsection B.9: Average Performance over Time (Figure 16).
Subsection B.10: Impact of In-Context Demonstration Selection (Table to Table 10).
Subsection B.11: Impact of Different Model Architectures (Figure 17).
Section C: Qualitative Results
Section D: Complete Results
Raw results of all figures in main paper and supplementary materials from Table 11 70.
15

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION AND EVALUATION DETAILS
VL-ICL Bench Evaluation Metrics We use accuracy as the metric across all subsets our benchmark. For text-to-image generation tasks, we utilize the state-of-the-art VLLM LLaVA- Next (Liu et al., 2024b) as the judge model to decide whether the generated images contain the required object or attribute.
Models Configurations We additionally provide a summary of the configurations for the models benchmarked our paper in Table 6, with a particular focus on the number of tokens per image and the context length. This information helps elucidate why some models exhibit poor scalability with increasing shots, as the total lengths exceed the maximum context window.
Table 6: Detailed configurations of the models used our benchmark.
Model
Connection Module
Image Tokens
Context Length (Train)
Context Length (Test)
OpenFlamingo-99
Perceiver
64
2048
2048
IDEFICS-9B
Perceiver
64
2048
2048
Otter
Perceiver
2048
2048
InternLM-- M-MCooppsosr2222
Perceiver
2048
4096
Qwen-VL- VL-Chat
Cross-Ateention
256
2048
8192
LLaVA-Next
MLP
576
2048
4096
Emul
C-Forner
512
2048
2048
Phi3-Vision
MLP
AnyRes
128K
VA
MLP
UniRes
200K
IDEFICS2
MLP
32K
LLaVA-One
MLP
AnyRes
128K
IntemLM-X2255
Partial-LoRA
AnyRes
24K
96K
Emu2
Linear layers
64
2048
2048
GILL
Linear layers
2048
2048
SEED-LLaMA
Q-Former
32
2048
4096
Prompts
We list specific prompts below that we use for specific experiments.
Prompt judge image generation for Fast MinilmageNet and CobSAT dataset
User: Decide whether the image contains the following concept: {GT). Answer with 'yes' or 'no'.
Prompt to judge the answer for Vizwiz VQA.
User: Based the image and question, decide whether the predicted answer has the same meaning as the ground truth. Answer with 'yes' or 'no'. Question: {Question} Predicted answer: (Prediction) Ground Truth: (GT)
16

---

Published as a conference paper at ICLR 2025
Prompt to rate the quality of COCO captioning (Main text, section 2.2)
User:
Given the following image, you to evaluate the provided generated caption based on its relevance, accuracy, completeness, and creativity in describing the image. Rate the caption scale from to 10, where 10 represents an exceptional description that accurately and completely reflects the image's content, and represents poor description that does not accurately describe the image.
Generated Caption: (Prediction)
Ground Truth Caption: {GT}
Consider the following criteria for your rating:
(Very Poor): The caption does not correspond to the image's content, providing incorrect information or irrelevant descriptions. It misses essential elements and may introduce non- existent aspects.
(Poor): The caption only slightly relates to the image, missing significant details or containing inaccuracies. acknowledges some elements the image but overlooks key aspects.
5 (Fair): The caption provides a basic description of the image but lacks depth and detail. It captures main elements but misses subtleties and may lack creativity or precision.
(Good): The caption accurately describes the main elements of the image, with some atten- tion to detail and creativity. Minor inaccuracies or omissions may be present, but the overall description is sound.
The caption provides a detailed and accurate description of the image, with good creativity and insight. It captures both essential and minor elements, offering well- rounded depiction.
(Very Good):
9 (Excellent): The caption delivers accurate, detailed, and insightful description, demon- strating high creativity and a deep understanding of the image. It covers all relevant details, enhancing the viewer's perception.
(Exceptional): The caption offers flawless description, providing comprehensive, accurate, and highly creative insights. It perfectly aligns with the image's 's content, capturing nuances and offering an enhanced perspective.
Please provide your rating. You should ONLY output the score number.
Details on Dataset Sampling and Filtering We include additional details on how we performed dataset sampling and filtering when creating our benchmark:
Fast Open-Ended MinilmageNet: we take the first 200 query examples provided in the original open-ended MinilmageNet, together with their associated support examples.
we randomly sample 800 examples from the original CLEVR dataset training scenes to use as support examples, and we select 200 examples randomly from the valida- tion split as query examples.
CLEVR:
we generate this dataset ourselves and we consider all single digit combinations. We use randomly selected 80 combinations of digits the support set and 20 combinations digits as the query set. Each combination is used with different operators.
Operator Induction:
Tex1OCR: we use 800 randomly selected examples from the original Tex1OC training set as support examples and 200 randomly selected examples from the validation set as query examples. Before selecting these we did filtering to ensure we use only valid texts that are not marked as rotated. To make the task manageable, we select the largest text in the image for OCR.
Interleaved Operator Induction: same as for the operator induction task.
Fast Matching MinilmageNet: this is derived from our Fast Open-Ended MinilmageNet, using the same underlying data.
17

---

Published as a conference paper at ICLR 2025
Text-to-Image Fast MinilmageNet: repurpose Fast Open- Ended MinilmageNet for this task, so we use the same underlying data.
CoBSAT:
take the first 80% of scenarios from the original CoBSAT dataset for support set and the remaining 20% for the query set.
Fast Counting: take action and colour categories from CoBSAT, using half of animals and objects for the support examples and another half for the query examples. Create composed images by repeating the image one, two, three or four times.
Fast Attribute Matching: take all attribute-object pairs from CoBSAT and use 30 of them support and 20 query sets, with overlaps across objects between the sets.
B FURTHER ANALYSIS
B.1 FULL VL-ICL BENCH RESULTS
We show the main results across all datasets in Fig. 8.
B.2 SCALING TO MORE SHOTS
To examine the maximum number of shots the models can handle and whether the model can still benefit from more shots, further increase the support set size to 16, 32, and 64 shots. We choose three models for this experiment: OpenFlamingo 9B (Awadalla et al., 2023), IDEFICS-9B- Instruct (Laurençon
al., 2023), InternLM--Composer (Zhang al., 2023a), LA (Zhang et al., 2024b), and Mantis-Idefics2 (Jiang et al., 2024a). These models were selected because each image they process translates fewer tokens (Table 6) they have longer context win- dow, ensuring that they do not exceed the maximum context length when evaluated with 64 shots. CBGFIC--B-IIntrrenn demonstrates better scaling capability compared other models most of the datasets. Besides, while InternLM-XX M-moLMXXs-Cmpsrr/ has strong performance in a low-shot regime, the performance quickly decreases with many shots. This may be due the mismatch between training (4096) and testing (2048) context length (Table 6) where the extrapolation of context length has been a well-known challenging task (Press et al., 2022; Liu et al., 2024c).
B.3 CHAIN-OF-THOUGHT PROMPTING
To investigate whether there is any strategy that enhance in-context learning, straightforward method is Chain-of Thought (CoT) prompting (Wei et al., 2022b). CoT prompts the model to artic- ulate its reasoning process concerning latent variables from the support set, potentially improving its learning and inference capabilities. We experiment with Qwen- VL-Chat (Bai et al., 2023) and DiernL--XCooposse (Zhang et al., 2023a) that have tf-teoof-hheaar LLMs with strong reasoning ability. Below is the specific prompt we use.
18

---

Published as a conference paper at ICLR 2025
Open-Ended MinilmageNee:
CLEVR Count Induction
Operator Induction
100
60
Accuracy (%)
80
60-
Accuracy
Accuracy
20
20
Interleaved Operator Induction
TextOCR
1
Matching MinilmageNett
50-
Accuracy (%6
60-
40
60
Accuracy
30
Accuracy
40
20
20-
20
5
Number of shots
Number of shots
Number shots
OpenFlamingo-98
Otter
Emu2-Chat
Mantis-ldefics2
IDEFICS-98
InternLM-22
VILA-7B
IDEFICS-80B
Owen Chat
Phi3 Phi3-Vision
InternLM X2d5
IDEFICS2-8B 8B
LextA-Next-78
LongVA-78
GPT4V
Text-to-lmage Fast MinilmageNet
CoBSAT
Accuracy (%)
Accuracy (%6)
20
20
GILL
SEED-LLAMA-SB
2
5
8
Number of shots
Number of shots
SEED-LLAMA-14B
Emul-Gen
Emu2-Gen
Fast Counting
Fast Attribute Matching
Accuracy (%)
Accuracy (%))
30
40
20
20
10
Number of shots
Number of shots
Figure 8: VL-ICL Bench results. Top: Image-to-Text. Bottom: Text-to- Text-to-Image Image tasks.
19

---

Published as a conference paper at ICLR 2025
CLEVR Induction
Operator Induction
Accuracy (%))
30
30
Accuracy
10
024
024
32 TextOCR
Interleaved Operator Induction
30
30
Accuracy (%6)
cuuuacy (%)
20
15
5-
Number of shots
Number shots
OpenFlamingo-98
IDEFICS-98
InternLM-X2
LongVA-78
Mantis-ldefics2
Figure 9: Results of scaling to many shots (Max 64). IDEFICS-9B Instrutttt exhibits strong scal-
ing capabilities across most datasets compared other models. Additionally, while InternLM- XComposer2 shows strong performance in low-shot scenarios, its performance diminishes rapidly as the number of shots increases.
20

---

Published as a conference paper at ICLR 2025
[Co'I' Prompt]: Let's first think step by step and analyze the relationship between the given few-shot question-answa pairs. Give reasoning rationales.
User:
[Task Despription|[Support Set][Query][CoT Prompt]
VLLMs:
[Generated rationals]
User: [Task [nneriptiin][Supppaaa [e]][Queyy]] rationals]
VLLMs: Prediction
We do not observe a consistent improvement with chain-of-thought prompting: benefits perfor- mance some datasets while detracting from it others. These findings underscore the complex- ity of in-context learning tasks, suggesting that fundamental advancements in model development necessary. Such tasks cannot be readily addressed with simple prompting techniques like CoT.
CLEVR Count Induction
Operator Induction
Interleaved Operator Induction
30
40
Accuracy (%)
Accuracy (9%)
20
20
Accuracy
15.
20
15
10
5-
0
1
2
TextOCR
Matching Mini-ImageNNet dataset
Number shots
10-
3
Accuracy 1%))
Accuracy (%)
20-
4 88
15-
Qwen-VL-Chat InternLM-X2
5
Number shots
Number of shots
Figure 10: Comparison of Chain-of- Thoughtt prompting (dashed line, diamond markers) with base- line results (solid line, circle markers) across selection of datasets and models. Chain-of-thought prompting does not consistently improve performance across datasets, highlighting the complex- ity of in-context learning tasks and the need for fundamental model development beyond simple prompting techniques.
B.4 REPRATING SUPPORT SET
In this subsection, we experiment with interesting setting: we duplicate the same support example multiple times to assess whether repetition enhances performance. We employ the Qwen- Qwen-VI VI,-Chat model for these experiments, with the results presented in Figure 11. We found that duplicating shots is particularly beneficial in the 1-shot scenario for Fast Open-Ended MinilmageNet, although this is not consistently observed across other datasets. The likely reason is that Fast Open-Ended MinilmageNet gains from the reinforcement of binding the concept through repeated examples, whereas for tasks like operator induction, diverse examples are necessary to facilitate the learning process.
B.5 INFLUENCE OF INSTRUCTION FINE-TUNING
We investigate how enstruction-folloiinn fine-tuning affects in-context learning capabilities. We compare two model families, each with a pre-trained version and an instruction-followin fine-tuned version: Qwen-VI. versus -L-Chat (Bai al., 2023) and IDEFICS-9B versus IDEFICS-
21

---

Published as a conference paper at ICLR 2025
Fast Open-Ended MinilmageNet
Operator Induction
CLEVR Count Induction
32.5
Accuracy (%6
Accuracy (%6
Accuraci (6
30.0
27.5
15
25.0
10-
22.5
5
2
2
4
2
Number shots
Number shots
Number
No repeat
Repeat x2
Repeat x3
Repeat x4
Figure II: Investigation of the impact of repeating the in-context examples across a selection of datasets, using Qwen-VL-Chat model. The X-axis represents the number of unique shots, the total number of shots. For example, 1-shot Repeat x2 means there is one unique shot and it is repeated twice.
Fast Open-Ended MinilmaaeNNet
CLEVR Count Induction
Operator Induction
80
50
Accuracy (%)
Accuracy (96)
20
40
Accuracy
40
15
30
20
20
5
10
0
1
2
5
Interleaved Operator Induction
TextOCR
Number of shots
15
Accuracy (%)
IDEFICS-9B
10
Accuracy
IDEFICS-98 w/o Tuning
Qwen-VL-Chat
5
Qwen-VL w/o Tuning
0-
2
Number shots
Number of shots
Figure 12: Comparison of using (solid line) and not using instruction tuning (dashed line). Although the outcomes vary, models not fine-tuned with instructions exhibit marginally better scalability with respect the number shots, as evidenced in datasets like Tex1OCR.
9B- Instruct (Laurençon et al., 2023). Their performance differences are illustrated in Figure 12. Although the outcomes vary, models fine-tuned with instructions exhibit marginally better scal- ability concerning the number of shots, as seen with the Tex1OCR dataset. Further studies are needed understand whether cnstucction-oollovinn fine-tuning harm the in-context ability.
B.6 DIFFERENT LEVELS TASK DESCRIPTION DETAILS
We show the impact of different levels of details the prompt description in Figure 13. The results show that generally the best results are obtained with the most detailed descriptions. but this is not necessarily the case all settings and in some cases, even no descriptions can better. The per- formance is often similar across different levels of details, but in some cases, it can be significantly worse, e.g. for TextOCR. We also provide tables with the full results. our main experiments, we adopt detailed task descriptions for all datasets.
The task descriptions that we use for different datasets are as follows:
22

---

Published as a conference paper at ICLR 2025
Fast Open-Ended MinilmageNet
Detailed: Induce the concept from the in-context examples. Answer the question with single word or phase.
Concise: Answer the question with single word phase.
CLEVR Count Induction
The image contains objects of different shapes, colors, sizes and materials. The question describes the attribute and its value. You need to find all objects within the image that satisfy the condition. You should induce what operation to use according to the results of the in-context examples and then calculate the result.
Detailed:
Concise: Find objects of the given type, induce what operation to use and calculate the result.
Operator Induction
Detailed:
The image contains two digit numbers and ? representing the mathematical opera- tor. Induce the mathematical operator (addition, multiplication, minus) according to the results of the in-context examples and calculate the result.
Concise: Induce the mathematical operator and calculate the result.
TextOCR
An image will be provided where red box is drawn around the text of interest. Answer with the inside the red box. Ensure that the transcription is precise, reflecting the exact characters, including letters, numbers, symbols.
Detailed:
Concise: Answer with the text inside the red box.
Fast Open-Ended MinilmageNet
CLEVR Count Induction
30
50
25-
Accuracy (%)
Accuracy
20
30
20
0
0
0
2
Operator Induction
TextOCR
25
Accuracy (%6
30
Accuracy 199)
20
15
tn.
10
Number of shots
Number shots
Owen-VL-Chat
Lext
OpenFlamingo-ppe
InternLM-X2
IDEFICS-9B
Comparison of detailed task description (solid line, circle markers), concise task de- scription (dashed line, x markers) and no task description (dotted line, diamond markers) across a selection of datasets and models.
Figure 13:
23

---

Published as a conference paper at ICLR 2025
B.7 ICL EFFICIENCY VS SERO-SHOT PERFORMANCE
We analyse the in-context learning efficiency and compare it against the zero-shot performance. We define ICL efficiency the ability improve the performance after sceing few examples. We calculate its value as a proportion of the area between the zero-shot curve and the few-shot curve, number of shots). The results are shown
the total available area (0- (0-100% accuracy, in Fig. 14, where we only show models with non-negative efficiency. We also show the pareto front line that deseribes the constraint
the best possible models: zero-shot performance ICL efficiency = 100%. The results indicate that there a good variability between different models in how efficiently they can learn from the provided support examples.
Fast Open Ended MinilmageNet
CLEVR Count Induction
Operator Induction
100
100
100
ICL efftciency
80
ICL efficiency
80
efficiency
60
60-
60
40
20
20
20
Intereaveed Operator Induction
40
20
TextOCK
80
Fast Matching MinilmageNeet
60
100
100
100
100
ICL efficiency (66)
efficiency (%6
efficiency
60-
60-
60
40
20
20
20
40
00
100
20
60
80
60
100
Zero-shot accuracy (%)
Zero-shot accuracy
(%)
Zero-shat accuracy (%)
OpenFlamingo 98
Otter
Emu2: Chat
Mantis Idefics2
IDEFICS-98
InternLM-22
VILA-7B
LLavA-OneVisiin Dnviiin-iiis
IDEFICS-80B
Qwen-VL-Chor
Phi3-Vision
InternLM-X2o0
IDEFIC52-8B
LLaVA-Next-7B
LongVA-7B
GPT4V
texttto-Image Fast MinilmageNet
CoBSAT
ICL efficiency (%6
100
ICL efficlency (%
100
75-
75
25
25
GILL
25
75
100
25
75
100
SEED-LLaMA-8B
Zero-shot accuracy (%)
Zero-shat accuracy (%)
SEED-LLaMA.144 14B
Emul-Gen
Emu2-Gen
Fast Counting
Fast Attribute Matching
ICL efficiency (%6)
ICL efficiency (%6
75-
75
50
25
25
25
75
100
25
75
100
Zero-shot accuracy (%)
Zero-shat accuracy (%)
Figure 14: ICI. efficiency vs zero-shot performance results. Top: Image-to-Text, Bottom: Text-to-
Image tasks.
B.8 SELF-EXTEND ANALYSIS
We analyse the impact of using self-extend mechanism to improve the ability to handle longer con- text. The results are shown in Figure 15 and were descrihed earlier in the main text.
24

---

Published as a conference paper at ICLR 2025
Fast Open-Ended MinilmageNet
CLEVR Count Induction
50
Accuracy (%)
Accuracy
20-
10
0-
0-
0
2
2
Operator Induction
TextOCR
25
Accuracy (%)
20
20
Accuracy
15-
10
5-
0
Number of shots
Number of shots
LLavA-Next-78
VILA-7B
Comparison of models without self-extend (solid line, circle markers) and with self- Figure 15:
extend (dashed line, x markers).
B.9 AVERAGE PERFORMANCE OVER TIME
We analyse in Figure 16 how the average accuracy and ICL. efficiency of image-to-text and text- to-image models evolved over time on our tasks. We can see that the performance as well as ICL efficiency have been continuously improving even though there are some outliers (e.g. GPT4V) that were released earlier and obtained some of the stronger performances. This shows that our benchmark can effectively assess the ICL capability the VLMs.
Average Accuracy over
Average ever
Average Accuracy (%)
b0. 20
30
Average ICL
Im mage-to-Text Text-to-Image
Im Image-to-Text
Text-to-lmage
2033-05
2323-09
2324-11 Date
2326-33
20405 2004-07
2033-35
2003-11 2026-31
2024-23
200665 202487
Date
Figure 16: Average accuracy and eff. in terms of when the model was released.
B.10 IMPACT OF IN- IN-CONTEXT DEMONSTRATION SELECTION
As shown in previous work (Zhang et al., 2023b; Li et al., 2024c; Chen et al., 2024; Yang et al., 2023), different in-context demonstration selection strategies can have large impact on the final performance. To further understand this, we employ a similarity-basee selection strategy (Zhang al., 2023b; Li al., 2024e; Chen et al., 2024; Yang et al., 2023). Specifically, for image-to-text tasks (CLEVR and TextOCR), we use CLIP image embeddings to retrieve the top-k most similar images and their corresponding text. Conversely, for text-to-image tasks (CoBSAT), we use CLIP text embeddings to retrieve the top-k most similar texts and their associated images. Additionally, we experiment with three different arrangements the selected top-k demonstrations: (1) from the most similar to the least similar one, followed by the query; (2) from the least similar one the most similar one, followed by the query; and (3) a random order of the top-k demonstrations. These compared to the original setup which used random selection.
25

---

Published as a conference paper at ICLR 2025
We select several pe-performing models for experiments, with the results presented in Table to 9. In each cell, the first three numbers correspond to the three arrangements the selected top-k demonstrations (1), (2), and (3), while the final value represents the baseline performance with ran- dom selection. For both text-to-image and image-to-text tasks, similarity- baaaed retrieval consistently enhances performance. Specifically, similarity-basee selection methods have ~3% improvement in peak accuracy and a ~33% boost in ICL efficiency compared to the baseline. These findings demonstrate that selection strategies can effectively enhance ICL tasks, highlighting promising direction for future work aimed at improving performance across various tasks. Detailed analysis of the changes peak accuracy and ICL efficiency in Table 10.
Table 7: (Similarit--hased Selection on CLEVR (Accuracy %).
Model
1-shot
2-shot
4-shot
8-shot
LLaVA-OneVision -7B
48.0 / 48.0 / 48.0 38.2
38.2
42.5/37.0/
39.77
33.5/34.5 /33/34.//350/33
33.0/ 33.5 34.0/ 28,3
Phi3 Vision
4 34,5
27.5/
28.5/28.5 32.5/177
36.5
Mantis Mantis-Idefies2
36.0/36.0
/ 36.0/ 40.2
36.5
/2.0
28.5
/28.5/24
22./21.5/263/11 52.0555//577/52
IntemLMMX225
62.0/ 62.0/ 62.0 63.2
62.0/
57.5/55./5558//55
Table 8: Similarity-based Selection on Tex1OCR (Accuracy %).
Model
1-shot
2-shot
4-shot
8-shot
LLaVA-OneVision
-7B
40.0 40.0
35.7
42.0 /2.0/42.0/4
42.2
4.0/444.//33//422
45.0/46.0 46.0 44.8 44.7 45.0/43.0
Phi3- Phi3-Vision
32.5/225/333 /
40.0 /9.0/39.5/
38.0
42.5/42./422./339 22..2///22..//2
44.5
Mantis-Ideficss.
12.5 12.5 12.5/
13.8
24.5
25.2
27.5/280//20 22.7
29.0 / 27.8
IntemL_MXX2D5
38.0/ 38.0 38.0 36.5
38.0 42.0/ 39.8/ 37.0
42,07 42.5 42.5
27.5/ 26.5
275
Table 9: Similarity-based Selection on CoBSAT (Accuracy %).
Model
1-shot
2-shot
4-shot
8-shot
Soed-LLama-88
24.0/24.0/ 23.7 24.0/24../23.7/55...
35.0/34.0/ 33.54..333.//// 33.5/21.8
36.0/35.5/ 37.2/7788
34.5/34.0/33.77 /33.7/3
Seed-I.1 II-II.Lamm Lam.mmaaaaa 148
27.0 27.0 27.0/ 77.0222.//77.//22
37.0/ 34.0/ 36.5
36.5 /6.5//00 /3.3 33.3
43.5 43.5/41.5/423 4.2/41/5//2///40
42.0/ 43.0 47.8 /338
Emu2-Gen
27.5/275 27.5//27.////
39.0 34.0/ 36.0 / 28.7
28.5 /8.5/ 27.5
23.0/ 38.5 20.8
B.11 IMPACT OF DIFFERENT MODEL ARCHITECTRRES
We analyse the impact of different model architectures Figure 17. More specifically we split the models into three categories: cross cross-attention based, VAaVi-like and GPT4V for which the architec- ture is unknown. To give more context, we include time in our analysis. The results indicate that the best accuracies and ICL efficiencies obtained by LLa VA-liee models (LLaVA- Vision- 72B, InternI.M Laan__MXX200 M-X2d5, Phizonision). However, more broadly there is no clear trend between cross- attention and VLalk-l models, and both categories obtain comparable performance. The first models were typically cross-attention based, but researchers have continued developing them also more recently.
C QUALITATIVE ANALYSIS
We also include a qualitative analysis on selected cases, where we analyse the impact of using more support examples on the quality of the output. We analyse text-to-image tasks in Fig. 18, using Emu2-Gen model. For text-to-image MinilmageNet the model should learn from the support exam- ples that the artificial names slation and shously correspond a lion and school bus, respectively. Emu2-Gen is able to do it to certain extent, but may get confused by additional support examples as more support examples are not necessarily helpful. In CoBSAT, the support set induces that the animal should have glacier and desert background. With no support examples the model only dis- plays the animal, but with more support examples it learns that it should use glacier background. In the second example, the model is able to capture that it should desert background, but is less successful in showing the required animal zebra. The quality of the generated images is not necessarily better with more support examples.
26

---

Published as a conference paper at ICLR 2025
Table 10:
Zero-shot accuracy, peak accuracy and ICL efficiency scores (%, 1) of different models for the three arrangements of similarity-based selection as well as the original random selection.
Z.s.
Eff.
CLEVR Count Induction
VVA-Onneission78
5.5
48.0 48.0/ 48.0
38.2
29.6 29.1 29.9
24.8
Viii-nisionn
0.0
42.0/ 42.0 42.0
34.5
31.3 / 30.6/
31.0/
20.4
Mantis-Ideficss
31.5
36.5 / 36.0
36,0
40.2
-1.9/ -1.9
-12/
-3.9
Mn-2m5
6.0
62.0/ 62.0 62.0
63.2
48.6/ 47.0/
47.5/
47.5
TextOCR
LLaVA-Onee VieVononn
39.0
45.0 46.0//4.0//
/4.///4..//////////////// 44.7
3.7
4.3 /
3,9/
2.8
Phi3-Vision
8.0
45.0 43.0/ 44.7
41.5
31.2/
30.5
31.1
29.0
Mantis-Idefices
21.0
29.0
29.7
27.8
4.0/
3.8/
4.6
4.1
IntemLMMX22d5
32.0
42.0/ 42.5
42.5
42.5
4.5/
5.2
5.0/
4.2
CoBSAT
SEED-LLaMA-SB
0.5
36.0 35.5
33.2 /33.7
31,2/
30.7
31.1/
24.4
SEED-LLaMA-
5.5
43.5/43.00
47.8 /33.8
32.0/
30.9
32.8/
30.2
Emu2-Gen
8.7
39.0/ / 38.5
36.0 28.7
19.0/
25.7
18.5/ 15.5
Average Accuracy per Type over Time
Average ICL per Type over Time
Average Accuracy (96)
20-
Average
2725-55 202507
2325/23
2005-11
2724-11 Date
212645
200400 2004-07
2025-35 200500
202547
2025-11
2004-11 212655 Date
20040s 272447
Crass-attn.
LLaVA like
GPT4V
Cross attn.
like
GPT4V
Figure 17: Average accuracy and ICL eff. per model architecture, with information on when the model was released. The best models are LLaVA-like, but otherwise the performance is comparable.
For qualitative analysis of image-to-text tasks, we discuss some of the common mistakes that the models make for each task.
is relatively common for the models to predict the real-world class, even if it is asked to the artificial names from the support set. With more support examples such mistakes are less likely to occur as the model learns to use the artificial names.
Open-Ended MinilmageNet
In many cases the model rephrases the question, while in others it talks that e.g. the described object is present. Such behaviour is more common with fewer or no support examples. With more support examples the model learns predict count but gets incorrect answer. It can be because some objects are more difficult to recognize, e.g. if one partially covers another.
CLEVR
Operator Induction A very common mistake is use different operator than what would be induced from the support examples. For example, the model may guess it should add two numbers instead of multiplying them and vice versa.
Interleaved Operator Induction The model sometimes predicts the first displayed number or direct combination of them, e.g. if the two numbers are and 2, it returns 12. It is also relatively common to use an incorrect operator between the numbers.
TexoCC In many cases the model returns more words than are highlighted the red box, but includes the highlighted word as one of them. It is also common that the model misses letter in the text or returns a word that similar but different from the correct answer. some cases though the answer may be very different from what is highlighted, possibly returning different word in the image.
27

---

Published as a conference paper at ICLR 2025
Text-to-lmage MinilmageNet
CoBSAT
0-shot
1-shot
2-shot
4-shot
0-shot
1-shot
2-shot
4-shot
Slation: Lion
Tiger: Tiger with
Glacier Background
0-shot
1-shot
2-shot
4-shot
0-shot
1-shot
2-shot 4-shot
Shously: School Bus
Zebra: Zebra with Desert Background
Qualitative analysis: Images generated by Emu2-Gen show the ability to learn the con- cept induced by the support examples.
Figure 18:
28

---

Published as a conference paper at ICLR 2025
D COMPLETE RESULTS
In this section, we show the raw results of the figures in the main text in Section D.1 to D.3, and results of supplementary materials in Section D.4.
D.1 INITIAL ANALYSIS OF VQA AND IMAGE CAPTIONING
Table 11: Results of Math Vista using string match.
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
OpenFlamingo-99
9.00
17.00
20.00
21.80
22.00
Otter
14.00
16.50
18.00
18.50
19.50
IDEFICS-
0.50
5.50
16.60
20.50
21.00
IntemnMM-XComposczZ
22.50
21.00
22.50
24.00
24.00
Qwen-VL-Chen
20.00
21.50
22.00
22.00
22.00
LLaVA-Next-V Vett--icna
23.00
22.00
15.00
10.00
9.50
Emu2-Chat
18.50
22.50
23.50
23.50
20.00
Table 12: Results of Math Vista using LLM for answer extraction.
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
OpenFlamingo-98
21.00
22.00
22.00
22.00
23.50
Otter
20.00
20.00
17.50
19.50
19.50
IDEFICS-9B
22.50
22.00
22.00
23.50
23.50
InternLM-XCCom
XXCopoosrrr
28.00
28.50
28.50
29.50
26.00
19.00
21.00
19.50
18.50
19.00
LaaVA-Next-Vicunaa
23.00
25.00
18.00
15.00
10.50
Emu2-Chat
24.50
22.50
23.50
23.50
22.00
Table 13: Results VizWiz using exact match.
Model
O-Shot
1-Shot
2-Shot
4-Shot
8-Shot
OpenFlamingo-99
24.57
29.31
30.62
32.14
35.11
Otter
20.13
24.21
25.78
26.33
25.71
IDEFICS-99
33.20
31.67
45.33
42.80
41.87
InternLM-X2
71.93
66.33
66.73
66.27
71.73
Owen-VL-Chie
32.40
34.80
35.20
40.80
40.90
LLaVA-Next
Sext-Vicuna-BB
54.12
28.13
10.20
6.60
0.40
Emu2-Chat
31.06
34.20
36.13
40.12
42.66
29

---

Published as a conference paper at ICLR 2025
Table 14: Results of VizWiz using LLM as the judge.
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
OpenFlamingo-9B
31.54
35.85
36.10
36.4
37.28
Otter
23.40
24.12
25.88
26.19
24.93
IDEFICS- 9BB
35.10
36.90
40.66
42.30
41.35
72.90
68.20
69.10
66.30
68.42
Qwen-VL.-Chan
37.40
37.53
39.22
40.38
41.20
LLaVA--NettVicuna--
55.26
26.08
11.38
8.72
2.50
Emu2-Chat
35.68
35.83
37.67
40.51
41.99
Table 15: Results of COCO captions (CIDEr).
Model
0-Shot 1-Shot
2-Shot
4-Shot
8-Shot
OpenFlamingo-999
78.2
85.6
88.1
89.0
93.3
Otter
86.2
91.0
97.0
98.6
100.5
IDEFICS
71.6
84.20
90.8
93.1
98.5
125.74
125.26
138.82
135.22
129.8
Owen-VL-Cher
121.10
132.6
135.3
135.4
136.9
BLaVA-Next-Vieunse
131.24
81.75
40.49
34.47
26.26
Emu2-Chat
126.3
127.0
132.06
131.10
133.5
Table 16: Results of COCO captions. Scores are rated by LLaVA-Next from 1-10 (higher the better).
Model
O-Shot
1-Shot
2-Shot
4-Shot
8-Shot
OpenFlamingo-98
2.71
2.42
2.75
2.88
2.90
Otter
5.60
5.78
5.68
5.72
5.56
IDEFICS- -BB
6.15
6.22
6.15
6.21
6.22
MarrnL.-XCCoposeeer
6.67
4.17
4.67
5.11
4.58
Qwen-VL-Chat
6.17
6.17
6.12
6.11
6.12
MaaVV--Next-iicnaaa
6.34
3.54
3.75
3.43
3.57
Emu2-Chat
6.15
5.89
6.05
6.11
6.15
Table 17: Comparisons of VLLMMs and LLMSs for text ICL on AGNews dataset (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
LLaVA-Ner --ext--iiuun--P
74.66
81.16
81.61
84.05
85.38
Vicuna-7B
65.83
79.22
80.20
82.24
82.41
Qwen-VL--Chal
75.49
72.74
73.78
78.62
80.91
Qw
54.80
59.51
67.53
72.80
74.64
Mieeml.MXXCoopssrr
83.99
82.87
82.80
83.28
83.97
InternLM2-Chat-77
81.89
84.11
84.25
84.32
84.33
Table 18: Comparisons of VL.MM and LLMS for text ICL on MIT Movies dataset (Accuracy %).
Model
O-Shot
1-Shot
2-Shot
4-Shot
8-Shot
47.47
29.88
59.04
85.06
89.16 16
Vicuna-7B
50.36
63.61
77.83
86.99
89.88
Qwen- VL--Cha
50.36
26.99
46.02
74.70
85.54
QwenLM-7B
66.27
46.99
64.10
85.30
92.53
69.64
69.64
78.31
88.19
90.12
MammLMM2-hht--2
61.45
63.61
73.98
87.71
94.46
30

---

Published as a conference paper at ICLR 2025
Table 19: Comparisons of VLLMs and LLMs for text ICL on TREC dataset (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
49.80
51.80
55.60
65.40
71.00
Vicuna-7B
46.20
55.00
63.60
64.20
70.40
Qwen-VL-Chal
43.60
51.00
55.40
61.20
72.60
QwenLM-7B
40.80
54.00
61.00
63.00
73.90
InternL MeemmlMMXCooposrr2
59.00
71.20
75.00
81.80
82.80
MamnLMM2-hhtt77
62.00
80.20
85.00
87.00
87.20
31

---

Published as a conference paper at ICLR 2025
D.2
MAIN RESULTS
Table 20: Results of different models on Fast Open-Ended Mini-ImageNet (Aceuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
5-Shot
OpenFlamingo-99
0.00 = 0.00
39.50 = 1.22
58.17 + 3.57
51.17 + 0.85
54.50 5.66
IDEFICS-9B
0.00 = 0.00
22.00 0.41
52.00 2.94
53.83
0.94
59.17 6.20
IDEFICS-80B
0.00 = 0.00
2.25
49.50
1.28
52.47 + 3.25
62.50 2.00
IDEFICS2-8B
0.00 = 0.00
0.00
43.50 + 1.47
61.50 1.41
58,00
Otter
0.00 = 0.00
10.00
25.00 = 1.22
25.67 2.25
InternL.-X
0.00 0.00
1.08
38.00
49.00
50.33
Qwen-VL--Cin -L-Catt
0.00
0.40
47.33 2.49
55.17 2.83
LLaVA-Next-7B
0.00
2..17
33.67 + 2.25
36.95 0.24 1.02
Emu2-Chat
0.00 0.00
8.00
29.33 + 1.84
28.18
27.54
VILA-2.7B
0.00
5.27
50.33 1.18
45.00
42,00 1.78
VILA-7B
0.00
1
38.17 2.72
37.67
37.33 1.18
Mantis-Idefics2
0.00
1.08
72.83 + 2.39
84.33
76,00 = 0.41
Phi3-
0.00
0.85
50.00 + 1.08
39.00
40.17 1.25
Von
0.00
22.33
52.67 3.09
48.83
49.50 1.08
LLaVA-One Vision-72B
0.00 - 0.00
12.67 0.94
88.00 0.82
98.67 0.47
98.33 1 0.62
InterLL.-X2d55
0.00 - 0.00
12.00 1 0.00
82.50 1 1.22
88.17 1 1.70
91.00 1 1.47
GPT4V
0.00
14.00
48,00
56.00
78.00
Table 21: Results of different models on Real-nan Mini- Mini-ImageNet (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
5-Shot
OpenFlamingo
0.00 0.00
26.00 + 2.86
53.33 3.27
52.83 + 0.94
49.50 1.22
IDEFICS-9B
26.50 = 0.00
41.83 = 2.25
74.50 + 2.27
89.00 + 0.41
91.17 1.89
IDEFICS-80B
30.50 = 0.00
41.83 + 1.18
82.00 2.68
94,67 : 0.62
91.33 1.43
Otter
13.00 = 0.00
51.00
57.33 3.09
56.50
61.00 1.87
InternLM-XX
20.00 = 0.00
31.30
6.24
66.67 1.89
Owen-VL-Chie
32.17 : 0.24
40.67 = 1.03
58.00 0.71
8.06
2.005
LLaVA-Next-7B
20.50 0.00
6..50
52.83 1: 1.25
7.83
Emu2-Chat
29.89 0.00
1.41
51.43 4.45 1.52
57.25 1 3.06
Mantis-Idefics2
18.00 0.00
27.00 1.08
49.33
1.25
16.67 1 1.70
17.83 1 2.05
IDEFICS--BB
25.50 0.00
24.17 1 1.25
57.83 1 1.31
79.00 2.55
79.83 1 1.25
In2er
18.50 0.00
41.67 1.31
69.83
1 0.24
83,50 0.82
84.50 1 0.71
LLaVA-One Vision-7B
24.50 1 0.00
54.17 0.85
68.17
1.31
67.00 1.08
67.17 3.42
GPT4V
48.00
56.00
78.00
90.00
86.00
D.3 ADDITIONAL ANALYSIS
32

---

Published as a conference paper at ICLR 2025
Table 22: Results of different models on Operator Induction dataset (Aceuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
OpenPlamingo-911
5.00 + 0.00
2.22 + 3.14
1.67 +: 1.36
2.78 + 0.79
7.78 = 2.08
IDEFICS-99 9B
11.67 0,00
14.44 +:44:00 0.79
10.56 + 2.08
7.78 1 2.08
11.11
IDEFICS 80B
13.33=000
15.00
14.67 2.36
21.67 + 1.36
16.11
IDEFICS--BB
36.67 0.00
#1.67:4.00
39.45
38.89 3.14
4
Otter
21.67 1
11.67
1.33
12.22 1.57
=
InternI .M-X2
26.11+3.114
40.00
40.00
+
28.89
19.83
Qwen-VLa VL-Chat
15.00 = 0.00
1.36
17.22
18.89
25.00
LLaVA-Next-78
10.56 + 1.57
1.51
5,56 + 2.08
3.77
0.00
Emu2-Chat
28.56 1.57
21.67
1.57
21.67
2.50
VILA-2.7B
16.67
12.78
#311
11.67
11.67
VILA-7B
2
1.31
6.67
7.78
8.33
Phi3-Vision
0.00
1.57
54,44 1.36 0.79
Long VA-7B
18.33 0.00
8.33 + 3.60
6,11
5.56
3.89
Mantis-ldefie22
16.67 + 0.00
12.78 +.08
8.89 + 2.08
14.44 + 4.16
16.11 0.79
LLaVA-One 72B
33.33 = 0.00
3.44
60.56 + 6.71
68.89 + 1.57
57.78 + 2.08
InternlM-X2255
36.67 + 0.00
40.00 40.00+3.66 3.60
41.67 + 2.72
35.00 + 2.36
35.00 + 4.08
GPT4V
24.00
66.00
84.00
92.00
92.00
Table 23: Results of different models on TextOCR dataset (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
OpenFlamingo-99
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
0.00 0.00
IDEFICS-9P 9B
16.50 = 0.00
22.50 + 1.08
19.83 + 0.62
22.83 + 1.31
28.00 + 1.63
IDEFICS-800 80B
20.00 = 0.00
2.110
2.738
IDEFICS2-8B
23.00 0.00
1..00
2.85
1.25
30.33 0.85
Otter
0.00 0.00
0.00
0.17
0.47
0.67 0.24
Internl .M-2
8.67 4.01
3.62
10.50+0..
2
11.83 + 2.95
Qwen-VI.-Chal VI.-Chat
4.83+6.8
17.17+1..3
+
22.33
24.17 + 0.24
0.24
LLaVA-Next-7B
24.67 = 2.25
0.24
0.33 +
0.00
0.00 0.00
Emu2-Chat
25.83 + 0.24
1.47
31,50 1.87
36.50
29.50 1 1.78
VILA-2.7B
0.50 0.00
14.83 1 2.39
16.67 0.62
18.50 1 1.22
VILA-7B
28.00 1 0,00
6.17 1.03
22.17 1 0.24
26.83 0.47
30.17 1 1.03
Phi3-Vision
8.00 100+0000
2.25
38.17 0.85
39.67
41.50 + 1.78
LongVA-7B
26.00 0.00
0.82
11.17 0.85
17.17
17.50 + 2.12
Mantis-ldefics2
21.00 = 0.00
13.83
25.17 + 0.62
27.67 + 0.47
LLaVA-OneVisionn-Pn
48.50 + 0.00
50.67 + 1.70
51.17 1 0.24
50.67 1 2.09
51.67 1 0.62
Intem1
32.00 + 0.00
36.50 + 1.41
37.00 + 0.41
42.50 + 0.71
27.02 + 0.24
GPT4V
39.29
32.14
48.00
50.00
49.00
Table 24: Results of different models on CLEVR dataset (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
OpenFlamingo-99
0.00 0.00
17.83 + 2.25
17.00 + 2.27
18.83 + 1.03
16.33 + 1.43
IDEFICS 9B
0.00 + 0.00
30.33
2.25
29.50 + 1.47
27.67 1: 2.05
27.17 1: 2.87
IDEFICS-80B
0.00 0.00
31.1622.10
30.82
1.59
31.50 1 1.00
32.43 3,62
IDEFICS2-8B
2,00
51.00
48.83 4.09
43.83 1 0.62
51.50 1: 1.87
Otter
0.00 0.00
1.06
2.22
8.17
0.17 0.17
InternL-XX
1.83 +
26.00
24.67 5.25
20.00
22.83 + 0.85
Qwen-VE VL-Chat
0.00 100
44.55
25.33
26.83
30.17
LLaVA-Next-7B
0.00 0.00
6.61
24.83 4.90
0.17 17.83
Emu2-Chat
5,33
2.73
14.00 + 3.49
14.83 1.89
17.67 + 1.03
VILA-227B
0.00 0.00
30.67 2.09
29.17 0.85
29.17 1.03
29.83 + 0.24
VILA--B
0.00
24.00
32.67 1.43
31.50 + 2.16
34.33 + 2.39
Phi3-Vision
0.00 0.00
4
25.50 2.45
17.67 2.09
18.67 + 0.85
LongVA-78
0.00
2
22.83 2.39
1.08
23.83 1 3.47
Mantis- Idefics2
31.45 2.18
40.17 3.12
32.00 1 0.71
24.50 1.58 1.08
21.67 1 2.05
LOVVA-OOeissone
0.50 0.00
37.17 2.01
32.00 0.71
31.83
1.84
42.33
1 0.85
IntemL_MXX255
6.00 + 0.00
63.17 + 1.25
58.50 + 2.86
56.00 2
53.17 2
GPT4V
6.00
30.00
38.00
42.00
32.00
33

---

Published as a conference paper at ICLR 2025
Table 25: Results of different models on Fast Matching MinilmageNer dataset (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
5-Shot
OpenFlamingo-99
0.00 + 0.00
18.25 + 1.32
19.28 5.17
26.33 3.02
29.11 1.30
IDEFICS-9B
0.00 + 0.00
0.08 + 0.12
0.00 + 0.00
0.00 0.00
0.00 + 0.00
IDEFICS-80B
0.00 = 0.00
28.30 1.94 3.70
2 2
Otter
0.00 = 0.00
0.00
0.00 0.00 0.00
0.00
0.00
IDEFICS2-8B
0.00 = 0.00
0.18
8.33 + 8
2 1.93
Mantis 2
0.00 0.00
49.25 1.59
47.08
47.17
InternLM-XX.
0.00
49.92 0.12
0.32
0.24
Owen-VL-Chan
1.00
0.24
0.00 0.00
0.08
0.08 0.12
Phi3-Vision
0.00 0.00
0.85
46.08 + 1.48
1.23
50.75 1.43
LLaVA-Next-7B
0.00
0.00
0.00 0.00
0.00
0.00 0.00
LongVA-7BB
0.00 0.00
50.00
50.00 0.000
1.000
VILA-7B
0.00 1 0.00
49.67 0.311
49.75 0.35
49.08
49.92 1
Emu2-Chat
0.00 1 0.00
32.59 0.58
36.80 1 1.80
37.62
40.25 2.43
Inter1L
0.00 0.00
9.25 0.00
46.75 1 0.00
31.50 1
8.00 T 0.00
LLaVA-OneVisionn/77
0.00
28.50 T 2.07
58.08 0.92
68.83 0.31
70.33 0.66
GPT4V
0.00
2.00
48.75
58.25
58.00
Table Results of different models on Interleaved Operator induction (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
OpenFlamingo-99
0.00 + 0.00
5.56 + 1.57
3.89 2.83
2.78 + 0.79
8.89 + 3.42
IDEFICS-P9 9B
15,00 0.00
1.08
6.11 + 0.79
6.11 + 1.57
5.00
IDEFICS-80E
25.00 0.00
36.67 1.21
31.67 +: 2.46
3.13
20.00 2.77
IDEFICS2-8B
38.33 0.00
16.11
17.78 3.14
20.00 1.36
20.00 4.71
Otter
8,33 +
1.57
9.44
7.22
5.56
InternLM-X2
28.33 0.00
2.83
2.84
1..9
4.44 2.83
Qwen-VL-CHi VL-Chat
16.67 0.00
9.44
8.33 1.36
8.89 + 2.83
5.56 0.79
LLaVA-Next-77
13.89 1.57
2.22
3.11
5.00 +
5.00
Emu2-Chat
26.67 0.00
18.33
20.56 3.42
10.00
VILA-2.7B
0.00
10.56
0.79
11.11
11.67 1.36
VILA-7B
28.33 0.00
6.01
11.67 + 3.60
12.2
13.33 2.36
Pision
0.00 + 0.00
16.67
2.72
21.67 + 1,36
15.56 166++15
27.78 + 3.14
LongVA-7B
10.00 0.00
1.36
0.79
4.45
3.89 4.37
Mantis-Idefics2
30.00 0.00
5.56
4.44
6.67 1.36
7.22 1 0.78
MWVVA-OneVision7222
38.33 0.00
42.78 2.08
41.11
36.67 2.72
47.78 10.03
IntemLM-X266
38.33 0.00
0.22233.2
31.67 + 3.60
33.89 = 2.08
21.11 2.08
GPT4V
36.00
58.00
72.00
74.00
70.00
Table Results of different models on CobSAT: Total accuracies (Aceuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
GILL
2.67 + 0.24
12.33 + 1.31
9.33 +: 0.24
11.50 1.47
8.00 + 1.63
SEED-LLaMA-SB
0.50 + 0.41
15.83 + 1.65
21.83 - 1.65
27.83 2.36
33.67 - 2.32
SEED-LLaMA-14B
5.50 + 0.71
26.83 + 1.65
33.33 +: 3.32
40.83 - 1.65
43.83 +: 2.87
Emul-Gen
0.33 + 0.47
4.83
0.47
4.72
8.67 + 1.18
9.67 = 0.24
Emu2-Gen
8.67 + 0.62
23.00 + 3.24
28.67 2.01
27.33 2.72
20.83 0.85
Table 28: Results different models on CobSAT: Latent accuracies (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
GILL
7.67 0.24
47.67 + 1.43
53.17 1.65
67.33 + 1.03
72.33 + 0.85
SEED-LLaMA-SB
8.00 0.41
38.50 1.47
44.33 0.62
49.50 + 0.82
56.00 1.41
15.17 0.62
41.50 + 1.41
52.17 = 2.09
53,67 + 1.93
57.17 + 1.84
Emul-Gen
8.00 0.41
55.50 1:5000 2.55
71.00 - 0.71
77.00
+ 0.82
82.00 - 0.00
Emu2-Gen
18.00 +: 1.63
43.83 4.33
1.25 72.33
81.50 + 0.41
78.33 1.25
34

---

Published as a conference paper at ICLR 2025
Table 29: Results of different models on CobSAT: Non-latent accuracies (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
GILL
19.33 + 0.24
21.33 1.65
16.17 1.65
19.83 + 2.01
15.83 2.78
BLED--LLMM-8B
12.33 +: 1.03
47.33 + 4.03
52.00 = 1.87
58.83 +: 1.93
63.33 :: 2.01
SEED-LLaMA-14B
82.67 + 0.24
76.33 0.94
75.83 1.70
78.33 + 0.85
80.83 0.85
Emul-Gen
26.50 0.41
11.17
13.33
2.25
16.00 + 0.71
17.00 0.71
Emu2-Gen
62.00 + 0.41
49.17 + 4.29
42.33 = 2.62
35.67 + 2.05
29.33 + 1.43
Table 30: Results of different models on Text-to-Image Fast Mini-ImageNee (Accuracy %)
Model
0-Shot
I-Shot
2-Shot
4-Shot
5-Shot
GILL
0.00 + 0.00
16.00 + 2.27
15.17 + 2.72
14.83 0.24
14.33 2.25
SEED-LLaMA-SB
0.00 = 0.00
15.00 3.27
12.67 + 1.18
16.00 + 2.12
16.50 + 1.87
SEED-LLaMA-I4B
0.75 + 0.25
17.25 2.75
16.75
1.75
21.25 -: 1.75
21.00 3.00
Emul-Gen
0.50 + 0.41
31.50 1.87
22.83
+ 2.72
25.00 - 0.71
23.17 1.03
Emu2-Gen
0.00 + 0.00
24.33 + 3.30
30.67
1.31
37.00 = 1.22
Anole-7B
0.50 = 0.41
11.00 = 2.86
7.00 + 0.71
0.17 = 0.24
0.17 = 0.24
Table 31: Results of different models on Fast Counting (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
5-Shot
GILL
34.17 + 2.36
8.33 + 4,25
3.33 + 2.36
3.33
1.18
7.50 + 2.04
BGED-LLaMA-88
34.17 = 2.36
50.00 + 6.12
47.50 -:00::0 4.08
45.00
+ 2.04
56.67 6.24
SGED-LLaMA-IE
41.26 + 1.25
42.72 + 5.14
37.50 = 2.36
51.57
3.25
37.55 + 5.54
Emul-Gen
33.19 + 4.27
39.55 + 4.51
37.56 + 3.17
47.81
3.94
45.52 +
Emu2-Gen
44.17 = 4.32
50.80 2.12
47.13 = 3.98
49.25
+ 4.04
59.25 5.23
Table 32: Results different models on Fast Attribute Matching (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
5-Shot
GILL
20.50 +: 1.08
6.50 +: 0.41
4.67 = 1.43
6.17 1.31
7.00 1: 1.47
SEED-LLaMA-SB
21.00 = 0.41
31.17 = 1.43
34.33 = 0.62
32.17 + 1.03
32.17 1.03
SEED-LLaMAN
22.36 1 0.28
26.35 1 1.93
33.57 =: 2.01
32.00 1: 1.94
35.59 1: 3,09
Emul-Gen
23.47 3.98
24.11 6.26
27.58 + 2.47
29.14
1 5.98
24.27
1.86
cmu2-gen
26.50 = 1.78
18.67 + 1.03
23.17 = 1.43
34.00 + 2.27
28.00 1.08
Table 33: Results of different models on the Text version of Operator Induction (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
InternLM--XCn M-XXoooosss
15.00
50.00
73.33
75.00
83.33
Qwen-VL-CCi
0.00
45.00
56.67
63.33
71.67
ALaVA-Next-Vicun-77
10.00
40.00
53.33
60.00
68.33
Table 34: Results of different models the text version of Interleaved Operator Induction (Accu-
racy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
InternLM MaanLLM--XCCpperr
8.33
35.00
36.67
46.67
78.33
Qwen-VL-Char VL-Chat
0.00
50.00
55.00
61.67
66.67
16.67
41.67
45.00
53.33
70.00
35

---

Published as a conference paper at ICLR 2025
Table 35; Results of different models the text version of CLEVR dataset (Aceuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
0.00
45.00
43.00
42.00
41.00
Qwen-VL-Char VL-Chat
0.00
49.50
47.50
54.00
53.50
LLLaVA-Nex--iieuna-7
0.00
43.00
38.50
37.50
36.50
Table 36: Results of different models on the text version of toaetto-mmmge Fast Mini -mageee (Aceuracy %.)
Model
0-Shot
1-Shot
2-Shot
4-Shot
5-Shot
GILL SEED-LLaMA-SB
0.00
18.50
20.00
20.50
18.50
0.00
16.30
15.20
16.50
14,20
SEED-LLaMA-I4B
1.50
23.00
20.00
22.50
15.50
Emul-Gen
0.50
28.60
29.10
24.20
20.00
Emu2-Gen
0.20
32.40
38.80
40.50
42.10
Table 37: Results of different models on the text version of CobSAT: Total accuracies (%).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
GILL SEED-LLaMA-SE
6.00
13.00
20.50
22.50
23.50
0.50
14.50
15.50
30.50
32.00
SEED-LLaMA-14B
6.00
13.50
28.00
34.00
40.50
Emul-Gen
2.50
11.00
19.50
23.50
20.00
Emu2-Gen
7.50
19.50
32.50
46.50
45.00
Table 38: Results of different models on the text version of CobSAT: Latent accuracies (%).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
GILL
6.50
33.00
39.00
37.50
38.00
SEED-LLaMA-SB
4.00
17.50
18.50
35.00
46.00
SEED-LLaMA-145
6.50
60.00
55.50
60.00
66.00
Emul-Gen
6.00
24.00
31.50
43.50
42.00
Emu2-Gen
12.00
74.00
86.00
92.50
88.50
Table 39: Results of different models on the text version of CobSAT: Non-latent accuracies (%).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
GILL
86.00
44.50
62.50
67.00
71.50
SEED-LLaMA-SB
21.00
80.00
83.50
80.50
74.50
SEED-LLaMM--44
90.00
21.50
56.50
63.00
67.50
Emul-Gen
30.00
33.50
52.00
48.50
45.50
Emu2-Gen 68.50
22.50
37.50
50.50
49.00
Table 40: Comparisons in-context learning ability with and without instructioon-oollovingg fine- tuning on Fast Open-Ended MinilmageNet Dataset.
Model
O-Shot
1-Shot
2-Shot
4-Shot
5-Shot
IDEFICS-9B
0.00
16.00
47.50
58,00
56.00
0.00
22.00
52.00
53.83
59.17
Qwen-Vi VL
0.00
35.50
79.50
68.00
67.00
Qwen-VL-Chat VL-Chat
0.00
0.50
47.33
58.00
55.17
36

---

Published as a conference paper at ICLR 2025
Table 41: Comparisons of in-context learning ability with and without instruction-follonn following fine-
tuning on TextOCR Dataset (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
IDEFICS-9B 3.50
16.50
22.50
25.00
26.00
IDEFICS-9B-Instrue
16.50
22.50
19.83
22.83
28.00
Qwen-VL
0.00
27.00
28.50
30.50
37.00
Qwen-VL-Chie
4.83
17.17
21.50
22.33
24.17
Table 42: Comparisons of in-context learning ability with and without instructionnfollovinng fine-
tuning on CLEVR Dataset (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
IDEFICS-9P
0.00
19.50
26.00
25.00
29.00
PHEFICS-9B-Instuud
0,00
30.33
29.50
27.67
27.17
Qwen- VL
2.50
18.50
17.50
24.00
26.00
Qwen- VL-Chat
0.00
29.83
25.33
26.83
30.17
Table 43: Comparisons of in-context learning ability with and without instruction-followwe fine-
tuning on Operator Induction Dataset (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
IDEFICS-9B
5.00
16.67
8.33
10.00
3.33
IDEFICS-9B. IB-Insruttt
11.67
14.44
10.56
7.78
11.11
Qwen-Vi VL
15.00
26.67
36.67
46.67
56.67
Qwen-VL-Chne
15.00
10.00
17.22
18.89
25.00
Table 44: Comparisons of in-context learning ability with and without instruction-follone following fine-
tuning on Interleaved Operator Induction Dataset (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
IDEFICS-9B
13.33
8.33
5,00
10.00
3.33
IDEFICS-9B-Instreed
15.00
5.56
6.11
6.11
5.00
Qwen-Ve VL
0.00
13.33
13.33
8.33
11.67
16.67
9.44
8.33
8.89
5.56
37

---

Published as a conference paper at ICLR 2025
D.4
SUPPLEMENTARY RESULTS
D.4.1 SCALING TO MANY SHOTS
Table 45 to 48
D.4.2
CHAIN-OF-THOUGHT PROMPTING
Table 49 to 53
D.4.3 REPEATING SUPPORT SET
Table 54 to 56.
D.4.4 DIFFERENT LEVELS TASK DESCRIPTIONS
Table 57 60.
D.4.5
EMERGENT BEHAVIOR ANALYSIS LLAVA-ONEVISHI WITH DIFFERENT MODEL SIZES
Table 61 to 66.
D.4.6 ANALYSIS ON CONTEXT EXTENSION
Table 67 to 70.
Table 45: Results of many shots CLEVR dataset.
Model
16-Shot
32-Shot
64-Shot
OpenFlamingo-99 9B
22.00 = 1.47
25.33 + 1.65
25.67 = 2.39
IDEFICS- -----------tttttt
28.17 = 2.66
29.00 + 1.08
30.50 = 1.78
IntemLMM-X2
14.67 = 1.70
15.50 + 1.08
16.33 = 1.03
LongVA-7B
23.33 + 1.03
23.67 + 1.65
29.50 + 3.63
Mantis- Idefics
19.50 + 0.82
22.33
+ 1.31
10.17 + 2.25
Table 46: Results many shots on Operator Induction dataset.
Model
16-Shot
32-Shot
64-Shot
OpenFlamingo-99
13.33 3.60
8.89 1.57
11.67 + 1.36
In-trruuct
5.00 + 3.60
7.78 1.57
5.00 + 1.36
InternLM-X2
0.00 = 0.00
0.00
0.00
0.00 0.00
LongVA-7B
2.22 - 0.78
3.33 0.00
3,89 0.79
Mantis-Idefics2
20.00 4.91
16.67 + 1.36
17.78 + 0.78
Table 47: Results of many shots on Interleaved Operator Induction dataset.
Model
16-Shot
32-Shot
64-Shot
OpenFlamingo-98
8.89 + 3,42
8.33 = 3.60
11.67 + 3,60
IDEFICS 9B-Instruct
8.89 2.08
7.78 2.08
7.78 + 2.83
InternLM-X2
3.89 + 0.79
5.00 = 1.36
5.00 + 1.36
Long VA-7B
6.11 1.57
4.44 0.79
5.00 - 0.00
Mantis-ldefies2
7.22 + 2.08
3.89 2.08
3.89 + 2.08
38

---

Published as a conference paper at ICLR 2025
Table 48: Results of many shots TextOCR dataset.
Model
16-Shot
32-Shot
64-Shot
OpenFlammino-99
0.00 0.00
0.00 0.00
0.00 0.00
IDEFICS-9B-Innsee I-Instucct
29,00 + 1.22
$3.17
33.50 1.47
InterLM-X22
3.11
0.00 = 0.00
0.00
LongVA-77
15.33 1.55
10.83 + 0.47
5.50
Mantis-Ideficss
26.33 = 1.43
29.83 + 1.65
22.83 = 1.70
Table 49: Results with Chain-of- Chain-of-Thought prompting Operator Induction dataset.
Model
0-shot
1-shot
2-Shot
4-Shot
8-Shot
Qwen-VL-Che VL-Chat
13.67
21.00
15.33
9.33
16.67
IntemLM-X22
28.33
32.00
35.67
30.00
5.00
Table 50: Results with Shain-of-Thoughe prompting on Interleaved Operator Induction dataset.
Model
0-shot
1-shot
2-Shot
4-Shot
8-Shot
Qwen- Owen-VL-Chal 3.33
10.00
10.00
8.33
8.33
InternLM-X2
18.33
5.00
5.00
10.00
8.33
Table 51: Results with Chain-of-Thought prompting on TexOOR dataset.
Model
0-shot
1 1-shot
2-Shot
4-Shot
8-Shot
Qwen-VIL VL-Chat
7.00
28.50
27.50
30.50
22.50
InternLM-X2
12.00
1.50
0.50
2.00
0.50
Table 52: Results with Chain-of-Thought prompting on CLEVR dataset.
Model
0-shot
1-shot
2-Shot
4-Shot
8-Shot
Qwen-Vi VL-Chat
0.50
10.50
21.50
18.00
26.00
InternLM-X2
4.00
21.50
20.00
26.00
27.00
Table 53: Results with Chain-of-Thoughh prompting on Matching Mini- Mini-ImageNNt dataset.
Model
0-shot
1 1-shot
2-Shot
4-Shot
5-Shot
56.00
56.75
51.25 53.00
56.75
53,00
InternL--X2
58.25
51.50
50.00
48.50
Table 54: Result of Qwen-VL-Chat on Fast Open-Ended MinilmageNet dataset with repeated in- context examples.
Model
1-shot
2-Shot
4-Shot
5-Shot
No Repeat
0.50
47.33
58,00
55.17
Repeat 41.00
62.50
54.50
56.50
Repeat x3
62.50
55.50
61.00
62.00
Repeat
60.00
56.50
60.00
58.50
39

---

Published as a conference paper at ICLR 2025
Table 55: Result of Cheht on Operator Induction dataset with repeated in-context exam- ples.
Model
1-shot
2-Shot
4-Shot
8-Shot
No repeat
10.00
17.22
18.89
25.00
Repeat x2
5.00
15.00
23.33
25.00
Repeat x3
11.67
15.00
20,00
26.67
Repeat x4
13.33
18.33
21.67
18.33
Table 56: Result Qwen- Veeeh- on CLEVR dataset with repeated in-context examples.
Model
1-shot
2-Shot
4-Shot
8-Shot
No Repeat
29.83
25.33
26.83
30.17
Repeat 25.50
30.50
27.50
28.50
Repeat x3
22.50
32.50
26.50
32.00
Repeat 19.50
31.50
23.00
27.50
Table 57: Results of different models on Fast Open-Ended MinilmageNet dataset (Accuracy %).
Model
O-Shot
1-Shot
2-Shot
4-Shot
5-Shot
Qwen- VL-Chat Detailed
0.00 0.00
0.50 0.41
47.33 2.49
58.00 + 2.83
55.17
Qwen- VL-Chat Concise
0.00 0.00
0.83 + 0.62
48.00
2.45
59.00 + 0.41
52.50
Qwen- VL-Chat None
0.00 0.00
6.33 + 0.47
56.17 + 1.65
57.67 + 0.85
53.83 2.78
LLaVA-Next-7B Detailed
0.00 + 0.00
22.17 + 4.03
33.67 + 2.25
0.00 0.00
0.33 + 0.24
LLaVA-Next-77 Concise
0.00 + 0.00
24.00 0.71
34.50 + 2.68
0.00 0.00
0.33 + 0.24
LLaVA-Ncxt-7B None
0.00 + 0.00
2.01 16.67
32.00 + 2.55
0.33 0.24
0.17 + 0.24
OpenHlamingo-99 Detailed
0.00 +.00
39.50 1.22
58.17 + 3.57
51.17 + 0.85
54.50 + 5.66
OpenFlamingo-
Concise
0.00 + 0.00
36.50 + 0.41
51.67 + 2.78
52.17 + 0.62
49.33 + 1.25
OpenFlamingo-99 9B None
1.00
38.17 1.03
52.17 + 2.46
49.17 + 0.85
49.33 = 1.25
InternLMXX2 Detailed
0.00
14.83 1.03
38.00 1.78
49.00 + 1.78
50.33
Intern_M-X2 Concise
0.00
19.50 1.47
40.33 + 1.89
48.83 + 0.85
49.17 1.93
InterLM-X22 None
0.00 0.00
22.00 2
43.00 + 2.16
46.33 + 3.06
48.17 = 0.62
IDEFICS-9B Detailed
22.00 0.00 0.00
52.00 + 2.94
$3.83 0.94
59.17 6.20
IDEFICS-9B Concise
0.00 0.00
28.50 1.78
53.83 1 4.09
53.83 0.94
55.67 2.09
IDEFICS-99 None
0.00 0.00
37.17 4.29
52.17 4.48
53.17 = 1.25
55,50 1.47
Table 58: Results of different models CLEVR Count Induction dataset using different levels of task description (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
Qwen- VL-Chat Detailed
0.00 0.00
29.83
4.55
25.33 + 3.47
26.83 + 3.06
30.17 = 2.95
Qwen-
Owen-VL-Char VL-Chat Concise
0.00 0.00
24.67 + 2.32
25.67 + 0.85
25.33 - 1.65
24.83 = 2,32
Qwen-VL- VL-Chat None
1.00 + 0.00
25.17 + 2.72
24.33 + 1.31
24.83 + 1.31
24.67 = 2.36
LLaVA-Next-T7P Detailed
0.00
25.17 6.64
24.83 4,90
17.83 4.59
0.17 + 0.24
LLaVA-Next-778 Concise
0.00 0.00
25,00
27.00 3.89
20.00 + 2.48
0.00 0.00
LLaVA-Next-71B None
0.00
15.50 2.12
23.83 + 2.87
12.83 + 1.70
0.17 + 0.24
OpenHlamingo-99 9B
Detailed
0.00 +
17.83 2.25
17.00 + 2.27
18.83 + 1.03
16.33 + 1.43
OpenFlamingo- 9B Concise
0.00 0.00
15.33 2.39
19.00 + 2.27
20.00 + 0.71
18.33 3.09
OpenFlamingo-991 None
15.33 0.94
18.17 + 1.03
21.33 + 1.89
19.33 2.78
Intern M-X2 Detailed
183+00224
26.00 + 1.63
24.67 + 5.25
20.00 + 2.94
22.83 - 0.85
Intern M--2 Concise
1.00+0.000
19.33 + 2.25
20.17 + 1.31 1.31
9,50
12.33 2.32
InteraL--X2 None
1.00
26.67 +
24.67 2.01
25.17 1.18
23.17 - 2.25
IDEFICS-9B Detailed
0.00
30.33
29.50 1.47
27.67 2.05
27.17
IDEFICS-9B Concise
30.67
31.00
26.17 - 1.55
26.83
IDEFICS. -B None
0.00 0.00
30.83 1.43
31.33 2.95
28.50 + 1.78
28.00 - 0.41
40

---

Published as a conference paper at ICLR 2025
Table 59: Results of different models Operator Induction dataset using different levels of task description (Accuracy %).
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
Owen-Vll-Chie Detailed
15.00 0.00
10.00 + 1.36
17.22 + 3.14
18.89 1.57
25.00 2.72
Owen-VL-CCie Concise
15.00 0.00
2.08
15.56 3.42
17.78 2.08
27.22 0.79
Owen-VI.s Vh-Chat- None
15.00 0.00
8.33
14.44 + 2.83
18.33
27.22 +: 0.79
LLaVA-NexI- 7B Detailed
10.56 1.57
6.11 1.57
5.56 + 2.08
3.33 + 2.72
0.00 + 0.00
LLaVA-Next- 7B Concise
5,00 = 0.00
7.22 = 0.79
5.56 + 2.08
4.44 + 2.08
1.11 + 0.79
LLaVA-Next- 7B None
+.00
6.11 =
5.56 + 1.57
4.44 + 1.57
0.56 = 0.79
OpenFlamingo-99 Detailed
5.00 = 0.00
2.22 3.14
1.67 + 1.36
2,78 + 0.79
7.78 + 2.08
OpenFlamingo-99 Concise
6.67 + 0.00
5.00 1.0++3.0 3.60
4.44
4.44 + 1.57
9.44 + 1.57
OpenFlamingo-999 None
6.67 + 0.00
5.00 + 3.60
3.33 2.36
4.44 + 2.08
11.67 + 3.60
IntemLM-XX Detailed
26.11 3.14
40.00 + 10.80
40.00 4,91
39.44 7.49
19.83
IntemLM-X2 Concise
18.33
29,44 3.42
22.78 + 2.83
18.33 1.36
= 2.36
InterL
None
18.33 -
13.33 +: 2.36
12.78 + 2.83
12.22
- 2.72
IDEFICS-9P S9B Detuiled
11.67 = 0.00
14.44 + 0.79
10.56 + 2.08
7.78 + 2.08
11.11 = 1.57
IDEFICS-9E
Concise
15.00 = 0.00
13.89 + 2.83
12.22 = 0.79
8.89 + 0.79
8.33 + 3.60
IDEFICS-9E None
15.00 = 0.00
17.22 + 2.83
10.56 + 0.79
10.56 + 2.08
7.78 # 3.93
Table 60: Results of different models on Tex1OCR dataset using different levels of task description (Accuracy %).
Model
0-Shot
I-Shot
2-Shot
4-Shot
8-Shot
Qwen- VL-Chat Detailed
4.83 + 6.84
17.17 + 1.43
21.50 + 1.08
22.33 + 1.31
24.17 + 0.24
Qwen- Wven-VL-GGre VL-Chat
Concise
0.00 + 0.00
8.00 + 0.82
9.50 + 0.41
9.83 + 0.62
9.17 0.24
Qwen- Lan--L--haaa None
0.00 1 0.00
9.67 + 0.62
10.33 + 0.47
10.67 1 0.47
9.33 0.47
LLaVA-Next-7B Detailed
24.67 = 2,25
0.83 + 0.24
0.33 + 0.24
0.00 - 0.00
0.00 = 0.00
LLaVA-Next-7B Concise
8.50 + 0.00
0.00 + 0.00
0.00 + 0.00
0.00 0.00
0.00 = 0.00
LLaVA-Next-7B None
10.00 = 0.00
0.00 + 0.00
0.00 + 0.00
0.00 = 0.00
0.00 = 0.00
OpenFlamingo-99
Detailed
0.00 + 0.00
0.00 + 0.00
0.00 + 0.00
0.00 = 0.00
0.00 0.00
OpenFlamingo-99
Concise
0.00 + 0.00
0.00 + 0.00
0.00 + 0.00
0.00 + 0.00
0.00 0.00
OpenFlamingo-99
None
0.00 + 0.00
0.00 + 0.00
0.00 0.00
0.00 + 0.00
0.00 0.00
InternLM-XX X2
Detailed
8.67 + 4.01
3.83 + 0.62
10.50 + 0.71
16.00 + 2.48
11.83 + 2.95
InternL X2 Concise
0.50 0.00
0.50 + 0.41
0.83 + 0.47
2.33 + 1.03
0.00 = 0.00
InternL
None
0.50
0.50 + 0.41
1.33 + 0.47
3.67 + 2.09
0.00 0.00
IDEFICS-99 Detailed
16.50
22.50
22.83
28.00
IDEFICS-9B
Concise
+ 3.00
2.50
5.50 0.41
5.83 - 0.24
0.24
6.17 = 0.47
IDEFICS-9B None
4.00 + 0.00
2.67 + 0.62
5.33 + 0.47
6.00 = 0.41
6.33 = 0.62
Table 61: Performance LLa Vision of different model sizes on CLEVR dataset.
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
LIa V.VA--Oneiisiin-....B
0.00 0.00
3.83 + 0.24
11.50 + 2.68
12.83 + 0.62
24.33 + 2.05
LLaVA-OneVision-778
5.50 0.00
38.17 2.25
33.83 + 0.85
31.50 + 4.64
28.33 + 0.24
LiaVA-OneVisionn728
0.50 0.00
37.17 2.01
32.00 0.71
31.83 + 1.84
42.33 + 0.85
Table 62: Performance of LLaVA-One Vision of different model sizes TextOCR dataset.
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
VA Oisionni
16.50 0.00
2.33 + 0.62
3.00 - 0.41
13,33 + 0.94
25.50 + 0.71
LLa Va-AVAOOOeiisinn Vision-7B
39.00 0.00
35.67 4: 1.18
42.17 + 1.03
42.17 1 0.24
44.67 +:2 1.84
LLa VisV-n-nniisiin-221
48.50 + 0.00
50.67 4: 1.70
51.17 + 0.24
50.67 4 2.09
51.67 + 0.62
41

---

Published as a conference paper at ICLR 2025
Table 63; Performance of LLaVA-One Vision of different model sizes on Fast Open-Ended Matching MinilmageNet dataset.
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
LLaVA-One Vision-O.5B
0.00 0.00
20.33 2.25
36.83 1.43
36.83
2.01
41.33 1 1.55
LLaVA-One Viniooonn
0.00 0.00
13.50 1 1.08
48. 17 1 3.57
49.50 1 1.87
49.50 1 1.08
LLaVA-OneVision-2225
0.00 0.00
12.67 1 0.94
88.00 1 0.82
98.67 1 0.47
98.33 1 0.62
Table 64: Performance of
of different model sizes Operator Induction dataset.
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
LLa Vision
21.67 0.00
6.67 + 1.36
6.11 2.08
6.67 2.36
6.67 3.60
VaaaVA-Oee 7B
35.00 0.00
45.00 2.72
41.11 + 3,42
45.56 + 0.79
28.33 + 0.00
LLaVA-OneVision-222
33.33 0.00
75.56 3.42
60.56 + 6.71
68.89 + 1.57
57.78 + 2.08
Table 65: Performance of LLaVA-One of different model sizes on Interleaved Operator In- duction dataset.
Model
O-Shot
1-Shot
2-Shot
4-Shot
8-Shot
LLaVA-One VaeioOOOeViiiioOOo Vision-0.5B
13.33 + 0.00
5.56 +: 2,83
4.44 +: 2.83
5.56 2.83
1.67 1: 0.00
LLaVA-One VAVV-OOOiiiionn Vision-7B
38.33 + 0.00
15.00 + 1.36
15,56 + 0.79
22.22 + 3.43
20.55 + 2.08
VAA Visionnn
38.33 + 0.00
42.78 + 2.08
41.11 4: 2.08
36.67 + 2.72
47.78 + 10.03
Table 66: Performance of LLaVA-One Vision of different model sizes Fast Matching Minilma-
geNet dataset.
Model
0-Shot
1-Shot
2-Shot
4-Shot
5-Shot
LLaVA-OneVision-O..
0.00 + 0.00
8.92 +: 1.04
34.77 + 2.61
49.92 + 0.72
48.15 15 +:: 0.90
LLaVA-OneVisiion-T
0.00 0.00
15.67 1.12
49.58 + 0.12
50.00 + 0.00
50.00 + 0.00
LL.VA-OneVision-P7
0.00 0.00
28.50 2.07
58.08 0.92
68.83 0.31
70.33 + 0.66
Table 67: Performance comparisons before and after context extension (SelfExtend) on CLEVR
dataset.
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
ALaVA-Next--P (w/o SelfExtend)
0.00 : 0.00
25.17 t 6.64
24.83 i: 4,90
17.83 + 4.59
19.17 + 0.24
LLaVA-Next-77 (w/ SelfExtend)
0.00 1 0.00
28.83 i 1.31
29.00 1 2.27
27.17 1: 1.89
24.50 1 1.87
VILA-7B (wlo SelfExtend)
3.50 0.00
34.00 + 2.86
32.67 1.43
31.50 +: 2.16
34.33 2.39 32.00 3.08
VILA-7B (w/ SelfExtend)
4.00 + 0.00
34.83 + 1.89
32.00 0.82
34.50 + 3.08
Table 68: Performance comparisons of before and after context extension (SelfExtend) TextOC dataset.
Model
0-Shot
1-Shot
2-Shot
4-Shot
8-Shot
LLaVA-Next-77B (wfo SelfExtend)
24.67 1: 2.25
0.83 0.24
0.33
0.00 + 0.00
0.00 0.00
LLLVV-NNex--BB (w/ SelfExtend)
26.00 0.00
0.83 0.24
0.17 0.24
1.00 0.41
0.50 0.71
VILA-7B (wlo SelfExtend) VILA-7B (w SelfExtend)
28.00 1 0.00
6.17 : 1.03
22.17 1 0.24
26.83 1 0.47
30.17 1 1.03
28.00 1 0.00
6.83 + 0.85
20.67 1 0.24
25.50 + 0.82
29.67 +:: 1.55
Table 69: Performance comparisons of before and after context extension (SelfExtend) on Operator
Induction dataset.
Model
0-Shot
I-Shot
2-Shot
4-Shot
8-Shot
LLaVA-Next-7B (wlo SelfExtend)
10.56 + 1.57
6.11 + 1.57
5.56 + 2.08
3.33 = 2.72
0.00 + 0.00
LLaVA-Next-7B (w/ SelfExtend)
11.67 + 0.00
7.22 + 0.78
5.00 + 2.72
6.11 = 0.79
3.89 + 0.79
VILA-7B (w/o SellExtend)
28.33 - 0.00
11.11 + 4.37
6.67 3.60
7.78 = 0.78
8.33 2.72
VILA-7B (w SelfExtend)
28.33 - 0.00
9.44 4.38
5.55
6.67 = 1.36
6.11 1.57
42

---

Published as a conference paper at ICLR 2025
Table 70: Performance comparisons of before and after context extension (SelfFxtend) Fast
Open-Ended MinilmageNeet dataset.
Model
0-Shot
1-Shot
2-Shot
4-Shot
5-Shot
V.L.VA-Next-7B (w/o SelfFxtend)
0.00 0.00
22.17 + 4.03
33.67 2.25
37.24 + 1.02
36.95 0.24
I.LVV--Nex--7B (w/ SelfExtend)
0.00 0.00
29.33
1.43
42.33
2.46
51.00 + 2.68
50.17 1.18
VILA-7B (w/o SelfExtend)
0.00 1 0.00
28.83 + 1,43
38.17 1: 2.72
37.67 1: 3.68
37.33 1 1.18
VILA-7B (w/ SelfExtend)
0.00 + 0.00
27.00 + 2.04
54.00 + 2.86
45.67 + 3.12
47.00 + 1.47
43


Reviews:
[{"review_id": "jHz0xAxhVd", "reviewer": "Reviewer_vZo4", "summary": "This paper investigates an important property for vision large language models (VLLMs): evaluting the in-context (ICL) ability in multi-modal scenes. The authors first point out current VQA and captioning benchmarks are not ideal for evaluting multi-modal ICL via quantitative results. To bridge the research gap, the authors proposes the first ICL benchmark, VL-ICL bench, which encompasses 10 tasks and is used evaluate both the image-to-text and text-to-image models. The authors provide the comprehensive evaluation for a range of VLLMs on the proposed VL-ICL bench.\n\nOverall, I appreciate the motivation to evaluate the ICL capabilities of multi-modal models, as this is a significant yet under-explored area. The experiments are extensive and yield useful conclusions.", "questions": "1. Some tables do not bolden the best performance number, such as Table 5.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper starts from a good motivation for evaluating the ICL ability of current multimodal models.\n2. This paper proposes the VL-ICL bench convering 10 tasks to evaluate the diverse capacibilities such as perception, reasoning, rule-induction.\n3. The authors conduct extensive and thorough experiments with the current multimodal models on the proposed benchmark", "weaknesses": "1. Some details about the construction of VL-ICL should be clarified. For the datasets used in Table 1, do you use all the samples from the original sources? Do you perform some filtering strategies?\n2. Could the authors give more explanations about the metric ICL efficiency? Why the ICL efficiency has negative numbers in Table 2?\n3. Based on the curve figures (e.g., Figure 5 and Figure 6), it appears that the multimodal models do not significantly benefit from additional ICL examples, with performance gains primarily saturating at the 1-shot level. What do you think might be the reason for this? Additionally, why do you believe that the proposed VL-ICL offers a better evaluation of ICL capability compared to traditional VQA benchmarks, as the Figure 3(a) also reveals the same trend?\n\nI would like to hear the response from authors, and then make my final score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6IB3uq8ad3", "reviewer": "Reviewer_vd4e", "summary": "This paper reveals the limitations inherent in the common practice of quantitatively evaluating VLLM in-context learning (ICL) via VQA and captioning, and then introduces a comprehensive benchmark (i.e., VL-ICL Bench) for multimodal in-context learning. The introduced VL-ICL Bench incorporates both image-to-text (captioning and VQA) and text-to-image tasks, and evaluated various facets of VLLMs including fine-grained perception, rule-induction, reasoning, image interleaving, fast concept binding, long context, and shot scaling. The authors benchmarks over 20 VLLMs and highlight their strengths and limitations.", "questions": "- The few-shot multimodal ICL typically helps to improve the performance out-domain or out-of-distribution tasks. I am wondering why this paper only focuses on VQA and captioning?\n- Why the peak accuracy is used rather than average accuracy over all shots？\n- In Lines 361-363, the authors claim that ‘we attribute this to difficulty of dealing with the larger number of images and tokens confusing the model and overwhelming the value of additional training data’. It there any evidence to support such statement?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The overall paper is well organized and easy to follow. \n+ The research problem (i.e., benchmarking the multimodal ICL capabilities of VLLMs) is quite valuable in VLLM communities. Comprehensive analysis experiments are provided to show the limitations of existing benchmarks.\n+ The introduced benchmark covers multiple practical ICL tasks and assesses numerous VLLMs. Multiple discussions are provided to show the promising direction for future research.", "weaknesses": "- Some texts are not consistent with the figures. For instance, in Line 159-160 and 224-225, the authors claim that the ICL exhibit more significant improvement on text-to-text benchmarks compared to image-to-text benchmarks. However, the differences between Figures 3a and 4 are marginal. These line charts show similar trends. \n- Some design choices are not clear. The authors want to show different trends of VLLMs on multimodal and LLM benchmarks in Figure 3 and 4. I am wondering why different models are evaluated?\n- The data contribution of the proposed benchmark is somewhat weak, as all the data and annotations are collected from existing datasets. In addition, there is no dataset statics of the introduced VL-ICL Bench. It would be better to show the distribution of the data as well as the tested capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OBjcAFYIh0", "reviewer": "Reviewer_vjqe", "summary": "This article introduces a comprehensive benchmark named VL-ICL Bench for assessing the capabilities of multimodal in-context learning (ICL). The suite includes a range of tasks involving both image and text inputs and outputs. The authors evaluate the performance of state-of-the-art vision large language models (VLLMs) on this benchmark, revealing their strengths and weaknesses across various tasks, and note that even the most advanced models, such as GPT-4, find these tasks challenging. The article hopes that the dataset will inspire future work on enhancing the in-context learning capabilities of VLLMs and inspire new applications that leverage VLLM ICL.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Point out the limitations of the common practice of quantitatively evaluating VLLM ICL through VQA and image captioning.\n2. Propose a comprehensive benchmark suite of ICL tasks covering diverse challenges, including perception, reasoning, and so on.\n3. It rigorously evaluates a range of state-of-the-art VLLMs on the benchmark suite and highlights their diverse strengths and weaknesses.", "weaknesses": "1. The evaluation seems too weak. For example, for the ICL tasks of image generation, the community might focus more on generating images based on complex instructions. For example, researchers in such fields prefer using VLMs to evaluate the generated images given complex instructions as described in [1].\n2. The possible usage of this model is still unclear. There are two situations when we want to evaluate multi-modal tasks, but I think the proposed benchmark is not suitable for any situation. \n\nIn the first situation, we evaluate the generation and understanding abilities separately, which is more reasonable nowadays because existing VLMs are usually good at one type of task even for the recent ones like [2], thus they prefer to be evaluated on benchmarks for specific targets.\n\nIn the second situation, for the models that could generate and understand in the same framework, it is better to evaluate them in more complex in-context-learning settings. For example, the input and output both contain images and texts. Only in the way the upper bound of the abilities of such models could be measured. However, this setting is not contained in this paper.\n\n[1] Lin, Zhiqiu, et al. \"Evaluating text-to-visual generation with image-to-text generation.\" European Conference on Computer Vision. Springer, Cham, 2025.\n[2] Xiao, Shitao, et al. \"Omnigen: Unified image generation.\" arXiv preprint arXiv:2409.11340 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b25bTkDzHT", "reviewer": "Reviewer_EuE8", "summary": "This paper introduces a comprehensive dataset, VL-ICL, for evaluating VLLM. Unlike previous datasets that primarily focus on tasks such as question answering, OCR, or image caption generation, this dataset takes into account the contextual learning capabilities of both image-to-text (I2T) and text-to-image (T2I) modalities. It assesses the in-context learning (ICL) abilities of a wide range of multimodal models and reflects on potential issues these models may face. Additionally, it summarizes some phenomena that might arise during multimodal contextual learning, offering valuable insights for future experiments and research in the multimodal field.", "questions": "(1) I noticed that the number of dataset samples/test samples is relatively small. Does this dataset suffer from a significant issue of imbalanced class sample distribution?\n\n(2) In section 4.3, you mentioned an approach where language text is used to replace images for ICL inference. How exactly was this implemented? Did you directly remove the images from the ICL setup and only keep the questions and answers, or did you generate captions for the images to perform contextual learning inference? Or was another method used?\n\n(3) Can you explore the impact of different model architectures on ICL abilities? In your experiments, there are models with cross-attention architectures similar to Flamingo, as well as models like LLAVA that embed visual features into the text embedding space. Does this have a significant impact on the ICL performance of the models?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) Compared to previous image captioning or visual question answering tasks, this dataset introduces new tasks designed to assess multimodal contextual learning capabilities. This provides significant benefits for future model design, enhancing models' visual understanding abilities, and conducting comprehensive testing of multimodal models. Additionally, it includes detailed evaluations of few-shot language-to-image generation tasks.\n\n(2) The paper thoroughly tests the contextual learning abilities of open-source VLLM with different architectures, backed by extensive experimentation.\n\n(3) The paper documents and provides detailed explanations of a series of phenomena observed in vision ICL, offering insights for future research exploration.", "weaknesses": "(1) I believe this paper should expand on the impact of different ICD selections on ICL. There are already some papers that have demonstrated the role of ICD in ICL within VLLM.  Furthermore, in section 2.2 of the article, some phenomena related to multimodal ICL (mentioned in Section 2.2 :**For example, in captioning zero-shot VLLMs tend to produce more verbose captions than COCO ground-truth, and they learn to be more concise through ICL. Meanwhile, for VQA, there is a standard practice of evaluating based on string match between the ground-truth answer and the model-provided answer. For example, VizWiz has unanswerable questions, which some VLLMs answer with \"I don’t know\" which would not be string matched against a ground truth \"Unanswerable\". Some models thus learn about answer-formatting (e.g., preferred terminology; avoid using any preface or postface that may throw of a string match) from the context set. This is indeed a kind of ICL, but perhaps not what one expects to be learning in VQA. To validate this conjecture, we repeat the previous evaluation, but using soft matching to eliminate the impact of answer format learning.** ) have been previously discussed in several papers and thus the authors should refer to these studies.\n- What Makes Good Examples for Visual In-Context Learning?\n-How to Configure Good In-Context Sequence for Visual Question Answering\n- Understanding and Improving In-Context Learning on Vision-language Models\n- Exploring diverse in-context configurations for image captioning \nIf you could test the impact of these sample selections within VL-ICL and include appropriate citations, I would consider raising the score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)