Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
Authors: Brandon Huang
Chancharik Mitra
Leonid Karlinsky
Assaf Arbelle
Trevor Darrell
Roei Herzig
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Multimodal Models
Vision-and-Language
In-Context Learning
Primary area: machine_vision
TL;DR: We demonstrate the existence of multimodal task vectors and leverage them for many-shot ICL in LMMs.

Abstract: The recent success of interleaved Large Multimodal Models (LMMs) in few-shot learning suggests that in-context learning (ICL) with many examples can be promising for learning new tasks. However, this many-shot multimodal ICL setting has one crucial problem: it is fundamentally limited by the model's context length set at pretraining. The problem is especially prominent in the multimodal domain, which processes both text and images, requiring additional tokens. This motivates the need for a multimodal method to compress many shots into fewer tokens without finetuning. In this work, we enable LMMs to perform multimodal, many-shot in-context learning by leveraging Multimodal Task Vectors (MTV)---compact implicit representations of in-context examples compressed in the model's attention heads. Specifically, we first demonstrate the existence of such MTV in LMMs and then leverage these extracted MTV to enable many-shot in-context learning for various vision-and-language tasks. Our experiments suggest that MTV can scale in performance with the number of compressed shots and generalize to similar out-of-domain tasks without additional context length for inference. Code: https://github.com/Brandon3964/MultiModal-Task-Vector

Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
Brandon Huang!"
Chancharik Mitra
Assaf Arbelle²
Leonid Karlinsky
Trevor Darrell
Roei Herzig1.2
University of California, Berkeley
2 IBM Research
MIT-IBM Watson AI Lab
Abstract
The recent success of interleaved Large Multimodal Models (LMMs) in few- shot learning suggests that in-context learning (ICL) with many examples be promising for learning new tasks. However, this many-shot multimodal ICL setting has crucial problem: it is fundamentally limited by the model's context length set at pretraining. The problem especially prominent in the multimodal domain, which processes both text and images, requiring additional tokens. This motivates the need for a multimodal method to compress many shots into fewer tokens without finetuning. In this work, enable LMMs perform multimodal, many-shot in-context learning by leveraging Multimodal Task Vectors (MTV)- compact implicit representations of in-context examples compressed in the model's attention heads. Specifically, we first demonstrate the existence of such MTV in LMMs and then leverage these extracted MTV to enable many-shot in-context learning for various iision-and-annuuae can scale in performance with the number of compressed shots and generalize similar -ft-oi- tasks without additional context length for inference. Code: https: //github. thub. cwwmcromaaaaee hom///rannoo33666/aatttaaaaaa Task Vector
tasks. Our experiments suggest that MTV
Introduction
Large Multimodal Models (LMMs) such as GPT-4V 60), LLaVA 49 50]. and the BLIP 13
family of models demonstrate state-of-the-an performance on variety of vision and language (VL)
tasks due to their strong reasoning capabilities over both text and images. Recent works show that LMMs pre-trained particular, few-shot, in-context learning (ICL) in text-only LLMs has been scaled with an increasing number of examples in long-conter language models setting called many-shot learning [1]] A natural question arises on how to perform many-shot learning in the multimodal domain.
interleaved text-image data do multimodal in-context learning [6] 39]. In
The first issue with directly applying many-shot learning regimen to LMMs is the intrinsic limitation of context length. This especially true in the multimodal domain, LMMS must encode both text and images, whose embeddings LMMs leverage for reasoning, struggle to use their entire context length effectively for ICL [45] 51]. Secondly, perhaps due to the misalignment of pretraining tasks with ICL, many instruction-tunee LMMs underperform on tasks in the ICL. setting
loken-expensive Moreover, long-context language models, which
[16), suggesting the importance of interleaved LMMss. Finally, there is also the challenge of the increasing memory and run-time required for processing long contexts for every inference call. These challenges motivate a method for compressing multimodal in-context examples into compact, implicit representations. Therefore, in this paper, propose
Denotes Equal Contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

N= 300 shots
Task Vector from LMM
Activation Space
"Tulip"
""Dandolion"
LMM
Replace Activations for Downstream Inference
LMM
"Darsdelion"
LMM
"Tulip"
Figure 1: Multimodal Task Vectors (MTV) Overview. We overcome an LMM's context length limitation by encoding many shots of multimodal examples activations in the LMM's latent space. We then directly replace this encoding into the LMM's activation space during downstream inference.
Multimodal Task Vectors
representations of multimodal in-context tasks- the attention heads of LMMs to enable many-shot ICL. In particular, we show the existence of MTV in interleaved LMMs, and we use them to compress large numbers of multimodal ICL examples.
Recent research in explainability has demonstrated the existence of task vectors both the lan- guage 25 [ and vision (27] domains. These task vectors implicit representations of in-context tasks represented by sets of activations in the model. These activations compactly summarize the information in ICI examples. In our work, we go beyond proving the existence of these task vectors in the multimodal domain by demonstrating their ability to compress examples for many-shot ICL in LMMs without the need for finetuning.
Our method can be described in three steps. First, given a set of many-shot multimodal ICL examples, calculate the mean activations corresponding to the last token across multiple inference iterations. Second, to avoid the context length constraint, we select a set of attention heads the model to store the mean activations of the ICL examples. However, since the downstream task may be zero-shot use a different number of ICL. examples, we select a set of examples aligned with its form. We then these examples find an optimal set of LMM head locations where the many-shot examples will be encoded. We refer to these mean activations and locations as MTV, which implicitly encodes the many-shot multimodal examples for use in the downstream task. Finally, for downstream inference, we replace the mean activations from Step I with the attention head locations found in Step 2. Since we input examples to the LMM across different iterations in Step 1, Multimodal Task Vectors can implicitly encode more examples than are allowable by the context limit. We find that utilizing many examples for extracting MTV surpasses performance on zero-shot and most standard few-shot ICL settings, suggesting the effectiveness of our method. Another key benefit of our method is that it frees up tokens for the model during downstream inference compared standard few-shot ICL methods. An overview of our method is shown in Figure]] [
We summarize our main contributions as follows: (i) We show the existence of Multimodal Task Vectors, compact implicit representations of in-context functions in LMMs. (ii) MTV can encode more examples than allowed by an LMM's context length, enabling both runtime and memory- efficient multimodal many-shot in-context learning. (iii) MTV surpasses zero-shot and few-shot ICL settings on various VL benchmarks without finetuning. (iv) MTV can scale larger numbers of examples and can generalize to similar ofo-ain tasks.
2 Related Works
Many-Shot In-Context Learning. Few-shot in-context learning (ICL) is a significant area of study text-only LLMs
89 A natural question arises about the possibility of using larger number of shots (e.g., hundreds) to further improve performance or learn more complex tasks. Indeed, some early work in text-only many-shot, in-context learning suggests performance on different tasks can scale with a larger number of examples
45

---

However, scaling ICL in text-only LLMs is a challenge due to the intrinsic context length. One method to increase context length in these models is to apply positional interpolation methods [T0 63
However, research on these longer-context models finds that they struggle to use the entire context for ICL 45 [1]. Moreover, as inference on long contexts of inputs is also time and memory-expensivv, it is unclear whether simply scaling the context of models is practical for enabling multimodal many-shot ICL in open-source models. There is some early evidence of multimodal many-shot ICL being effective in closed-source models [5], so the question arises to how to achieve something similar for open-source models. This has led to work that looks to compress explicit input 34 58 77). But crucially, many of these methods require finetuning and only try to preserve performance. Our work is different in that it is the first to enable multimodal models with many- ICL capabilities, while also improving on complex VL tasks without finetuning.
tokens [1 20
Task Vectors. Our work builds off of research text-only and vision-only domains showing that internal representations of these models called task vectors 22 27 81 (or function vectors) can encapsulate tasks outlined by ICL examples. Our is the first demonstration of Multimodal Task Vectors (MTV) in LMMs. Going beyond previous work, however, we show that MTV enable LMMs not only to use many-shot, multimodal ICL examples but also scale with more samples, be used alongside explicit ICL shots, and even generalize to unseen classes or similar tasks.
Model Domain Adaptation Methods. As LLM and LMM model architectures have advanced, so have methods to allow these models to generalize beyond their pretraining distributions. Methods like 88 have shown strong zero-shot generalization to some out-of-domain tasks, but forgetting remains an issue. One popular solution to this issue involves Parameter Efficient
instruction tuning [ 50
68
Fine-tuning (PEFT) (28] finetuning either a set of soft prompt imput tokens [4 46). low-rank model
weights [1 29 99 or a separate adapter from the main model [8] 30, 100]
Prompting methods are a well-explored area for adapting models without finetuning. LLM prompt-
ing includes zero-shot methods [36 83 885, few-shot and ICL methods ( 13 53 56] expert
prompting [93] and Chain-of-Thougee (CoT) (90 [O1], with extensions like self-consistency (86), Tree-of- Thought (ToT) [94], and Graph-of of Th-uhht (Go'T) [8] 40 95] for more complex structures.
Similar multimodal prompting methods exist for LMMs as well [57 84 87 102 104).
Large Multimodal Models (LMMs). The state-of-the-arr performance of LMMs 2
13
17
21
43
49
50
96
97.
105] on multimodal tasks stems from combining LLMs' reasoning capabilities
1212
26
66
71
78 with the perception abilities of vision models. L.Ms' generative reasoning also Such
makes them more applicable complex tasks than previous contrastive methods 42
43
65
tasks include visual question-answering [4] 23 24 31
32 54 667 as well as object identification and 48 59 82). Visual Programmatic Models (VPMs) are another class of multimodal
localization 37
methods that makes use of in-context APIs code generation 19 22 52
69
72
74
76
92).
However, context length limits both LMMs' and VPMs' ability to use multimodal prompting methods such as ICL [9]. Another key challenge is that many LMMs are pre-trained on single text-imaae pair data. Recently, many LMM models now pretrain on interleaved text-image data 2 6
16
33
39
75
103), making effective multimodal ICL possible. In our work, MTV goes beyond simple few-shot multimodal ICL and scales to many-shot multimodal ICL.
3 Multimodal Task Vectors
To address the challenge of performing many-shot multimodal in-context learning, we demonstrate the existence of MTV in LMMs and then leverage them for many-shot multimodal ICL. We begin by describing some background on multimodal ICL and task vectors (Section (.T). We then introduce our three-step approach: (i) We calculate the mean activations of the attention heads from the many- shot multimodal ICL examples (Section 3.2); (ii) We then extract the set of LMM attention heads locations that best align to the downstream task using an adapted version of the REINFORCE [ algorithm (Section 3,3; and (iii) We replace the calculated mean activation values into the LMM for a downstream task (Section 3.4). The detailed method visual is shown in F
3.1 Preliminaries
In the multimodal in-context learning setting, an LMM learns new task outlined by a set of multimodal examples. The input to the LMM would be outlined as follows:
Lie
[[x1 31), (x2 v2).... ...., (In Bx), Q]
(I)
3

---

Step 1: Calculate Mean Activations:
LMM Last- fast-Token
N shots
Activation Space
"Daisy"
"Rose"
Mean Activations MTV
T = 100 iterations
-T 17--000 300 shots
N- 3 shots
"Tulip"
""Dandelion"
Step 2: Find Attention Head Locations:
Candidate Locations
Attention-Heal Locations X
"Cattleya"
N 100 shots
MTV EXTRACT
"Cape Flower"
Step 3: A:plyy MTV for Inference:
Replace HTTTV directly locations XTTV in the model.
"Dandelion"
LMM
"Tulip"
Figure 2: Multimodal Task Vectors (MTV). In the standard multimodal in-context learning (ICL) paradigm, the number of shots is limited by an LMM's context length. We solve this issue by first finding the mean activations corresponding to the last token of the examples' input (Step 1), and then caleulating of attention head locations (Step 2) that best align with the downstream task. These mean activations are then replaced directly in these attention head locations (Step 3), enabling many-shot multimodal ICL.
where the model is prompted to answer query Q given set of input-output examples (each ar being a multimodal input and each y a text output).
We note that in-context examples are commonly passed sequentially to the LMM, necessarily restricting multimodal ICI. to being small numbers of shots due limited context length. Furthermore, the images require more tokens to embed, which means enabling many-shot ICL is even more challenging in the multimodal domain. To solve this, we utilize our method CTV-whiic are implicit representations in the model's attention heads that encode a many-shot multimodal ICL task.
We start with a background task vectors for some task j. Given model F, denote the of attention-head locations as À = {1 VI E F} where each location is indexed as = (h, m) for the th layer and mtth attention head. Now, task vectors utilize the intermediate outputs of an LMM, called activations. For given input sequence of written in terms of its tokens {x1, 22, Tr); each attention head (h, m) produces an activation z & for each token ws, where is the model's embedding dimension and H is the number of heads. These activations are simply the output vectors of each attention head before any linear projection. While each head's activation typically concatenated with others and projected to form the layer's output, task vectors specifically utilize the

---

prejpreoection activations of the final token Fr from each attention head. We thus define the task vectors as follows: (I) the task vector values Hj are a subset of mean activations produced by the attention heads of F given examples of a task, and (2) the task vector locations Aj. which denotes subset of the attention head indices per task. Thus, the task vector (1);Ay)) For inference, Pj replaces the activation values of the heads in the locations given by Aj-
In prior work [25] 27 81] the calculation of the mean activations μj and the extraction of the attention-head locations X are used together to extract the task vector. Interestingly, we find that these two steps should be decoupled in order better align with the downstream task. In our work, we calculate the mean activations M4 corresponding to the last token specifically to encode a dataset of many-shor multimodal ICL examples by averaging them across multiple inference calls. However, the downstream task may not always be in the same ICL format as the many-shot examples (e.g., the downstream task uses different number of shots or is zero-shot). To solve this, we use separate set of examples that are of the exact format of the downstream task to align the extracted attention-head locations X with the inference task. This separation of responsibilities, wherein H4, captures the essential information from the many-shot examples and Xj identifies the specific attention head locations for the downstream task, optimizes the utilization of the encoded information at relevant locations within the model.
Our approach to finding Multimodal Task Vectors (MTV) (I XMTV) allows LMMs to actually leverage many-shot multimodal ICL examples for complex vision- lasionlanguuag tasks without being limited by context length. We proceed by first describing how to calculate the mean activations.
3.2 Step 1: Calculate MTV Mean Activations
The ultimate objective of many-shot multimodal ICL is to use a large number of input-output examples when solving task j. However, it not trivial to get the LMM to more examples during inference time than its context length allows.
To address this issue, we pass few-shot input J, for each inference call / for a total of T > 1 inference calls. Each It consists of N shots (where N 1) of multimodal in-context examples the form of randomly-selecter input-output response pairs (xz vt), and Qt, which is the query to be answered by the LMM in that iteration.
J = [<1] y1), (I2 y2), (æN UN), Q.)
(2)
Thus, over T' LMM inference calls, we have a many-shot multimodal dataset (of N x T examples):
Imany
[4, 12....,.777 !T!
However, this dataset is still just disconnected set of few-shot examples. Next, we would like to connect the separate examples into one unified many-shot multimodal ICL representation.
For each inference call, the LMM is given N-shot ICL examples. We calculate the mean of the activations corresponding the last token of the input z1, for each attention head index vI E (Section 3.1 across T inference calls, yielding:
T
1 T
VI E X:
MLJ = T
Σ E E(z
E [2L (x1 11). (x2 Ve)...., (IN UN), Qi] (4)
In this step, we have found the mean activations μi,j, which encode an internal LMM representation of many shots of multimodal ICL examples. the next subsection, we describe our methodology for selecting the set of attention heads where these mean activations will be used.
3.3 Step 2: Extract MTV Attention Head Locations
After Step we have mean activations for the attention heads of the last token in given many-shot multimodal task. Yet, we still need to find which set of attention heads X should be chosen encode our task.
To choose the set of attention heads, we first prepare a separate set of examples specifically aligned to the format of the downstream task. For instance, if the downstream setting is a 2-way, one-shot
5

---

classification task, then the S examples should conform to this paradigm. For our explanation, let's consider a downstream task that zero-shot such that there is single query Q₂ and corresponding response R, for all E [S].
From these examples, we utilize adapted version of the REINFORCE 91 algorithm-an iterative policy optimization method that can be used to find task vector locations 27 Given an LMM F, we first select proposed set of attention head locations by sampling Bernoulli distribution over the locations multiple times. Next, we directly replace the values of the selected attention heads with the corresponding mean activations μl.j. Then, after prompting the model with the query Qs. we use the negative cross cross-entropy loss between the LMM's 's output logits and the logits of the ground-trute response Rg to optimize the Bernoulli distribution. By optimizing the Bernoulli distribution across S iterations, we are finding the best attention head locations X for patching in our mean activations. Finally, we can extract 2 XMTV, the optimized indices of attention heads, by sampling optimized Bernoulli distribution.
MTV = MTV WWW.EETRAATTFCCCM
[R,, Ro... Rsl)
(5)
is important to note that MTV EXTRACT does not require finetuning the LMM parame- ters, but rather only inference calls. We describe further the underlying details of our adapted MTV_ EXTRACT algorithm in Section Af the Supplementary. Having found XMTV and pr.j we describe in what follows, the final procedure to use MTV for inference.
3.4 Step 3: Multimodal Task Vector Application
After we have identified a set of attention heads X XMTV it is straightforward to apply MTV for inference. We denote the set of mean activations μu MiV as follows μ1) 2 {uge/V E XATVV.
To run downstream inference on a new query Qnew with our model F, we directly replace the values of attention heads A XMTV with pi MTV and produce the following response Rnew:
Rnew FeQ .15 (TV
MTV
Rnew is thus response generated using many shots of multimodal examples as implicit context via MTV. The key insight our method is the importance of N (the number multimodal examples) and many T (the number of iterations) during the caleulation of MTV. This enables an LMM to go beyond its context length to learn more nuanced properties of the task from seeing many examples. Additionally, insertion of MT' directly into the LMM also obviates the need for any context length during downstream inference, actually freeing additional context for other use (e.g., an additional prompt, more ICL examples, etc.). Finally, because align the attention-heat locations with the downstream task, MTV can be effectively applied to zero-shot and different ICL settings.
4 Evaluation
In order for LMMs to perform multimodal ICL, it is important for interleaved data to be included in pretraining. We apply our MTV approach to Qwen- VL (6]. Idefics2-8B (38), and ViLA-1.5-8B [47] three popular interleaved LMMs. For each model, we compare our method to using few-shot ICL across different vision-and- lang-ang tasks like VQA and object identification.
4.1 Implementation Details
We implemented MTV using PyTorch [61. We used each model's respective official implementation. While the compute and memory requirements differ slightly between models, all our experiments can be run single NVIDIA A6000 GPU. For additional information, refer Supplementary Section B Our model and weights will be released upon acceptance, and our code is in Supplementary.
4.2 Models
In this work, we apply MTV to the following interleaved LMMs as they are better-suited for multimodal ICL as shown by [16] (1) QwenVL (6) is LLaMA-based model that has the ability to

---

questions, where process high-reat resolutio images, and its tto-stage pre- trainining methodology, which includes multi- task finetuning and interleaved text-image data. (2) Idefics2-8B [39] Mistral- model that benefits from its pre-training on the expansive OBELICS dataset, which comprises web-scale collection of interleaved image-text documents. We utilize the base version of the model. This demonstrates multimodal in-context learning abilities. (3) -LaMA3-ViLL--15588 (abbreviated as VILA-I.5-8B). ViLA-1.5-8B [47] is an architecture that leverages LLaMA-3 as the LLM backbone. As in others, significant portion of the model's pretraining data is interleaved text-image data. (4) MANTIS-LLAMAAA encoder and LLaMA3 [55] language model finetuned using the MANTIS dataset, specially curated multi-image dataset that emphasizes co-reference, reasoning, comparing, temporal understanding.
MANTIS-LLaMA3-1
[33 is a combination of a SigLIP [9]] visual
We show the number of tokens per image embedding for each model in Table to illustrate the especial importance of MTVs in the image-text domain:
Table 1: Per Image Embedding Token Length and Total Context Length for Models
Model Name
Per Image Token Length
Total Context Length
VILA-1.5-8B
144
8192
Idefics2-8B
64
8192
QwenVL
256
8192
64
8192
4.3 Datasets
We briefly describe the tasks and datasets evaluate our method on. More details about the datasets and their setup can be found in Section
VQA Datasets. We use the following commonly-evaluater datasets which emphasize different aspects of multimodal reasoning, including visual features (VizWiz) and outside knowledge (OK-VQA) (1) VizWiz (3 consists of images taken by visually impaired individuals paired with questions they pose about these images, making it crucial for developing Al systems that assist real-world, sacessibility-focusse visual understanding tasks. (2) OK-VQA dataset (54) is designed to push the boundaries of Visual Question Answering (VQA) by focusing on knowledge-bart
answers require external knowledge beyond the image content. (3)
Object Classification. We use the following datasets, which are commonly used for object clas- sification in multimodal ICL: (1) The Flowers dataset [59]. commonly known as the Oxford 102 Flowers dataset, is a collection specifically designed for image-based flower species recognition for fine-graines classification of 102 different categories of flowers. (2) Caltech's 's CUB Dataset on Birds [82] is a well-known resource for evaluating algorithms on the task of object identification, specifically focused on bird species. It features 200 bird species with roughly 30 images each, anno- tated with key attributes and bounding boxes. Both Flowers and Birds are formatted as 2-way, -hott classification episodes, with model inputs being positive and negative image for the class to be identified in the query image. The response format a short text response.
5 Results
Our main results are shown in Table For VQA, we show the results of MTV with shots per 100 iterations to calculate the mean activations and 100 examples for task vector locations (500 examples total). The task vector is extracted using examples from the train set of the dataset and evaluated on the validation set. For object classification, we extract MTV based on a 2-way, one-shot regimen per 100 iterations for both mean activations and task vector locations (200 examples total). The task vector is extracted using a train set of 30% of the object classes and evaluated on the remaining 70% of unseen classes. We demonstrate how Multimodal Task Vectors outperforms zero-shot and few-shot ICL settings on three different models on VI. tasks, highlighting the effectiveness of our method. Next, we describe the unique capabilities of our method, such as scaling to more samples and showing some generalizations to other tasks. More results can be found in Section A. of Supplementary.
7

---

Table 2: Results. (Left) MTV evaluated on VQA datasets. (Right) MTV evaluated on object
classification datasets. The baselines are shown in gray.
(a) MTV on VQA Benchmarks
(h) MTV on Object Classification
Model
VizWiz
OK- OK-VQA
Model
Flowers
CUB
Flamingo 9B
28.8
44.7
LLaVA-1.5--1B
+4-shot ICL
34.9
49.3
1-shot ICL
58.60
58.24
+8-shot ICL
39.4
50.0
LLaVA-1.6-131
Blip3
21.2
26.5
1-shot ICL
65.58
67.90
+4-shot ICL
38.4
49.2
Flamingo 9B
+8-shot ICL
44.3
49.1
1-shot 9B
48.78
51.2
Qwen-VL-7B
35.2
58.6
IDEFICS 9B
+4-shot ICL
42.0
62.0
1-shot ICL
55.29
62.0
+8-shot ICL
44.3
61.5
Emu 37B
+MTV
45.6
62.0
+ 1-shot ICL
52.76
53.56
Idefics2
31.3
52.4
Qwen-VL-7B
+4-shot ICL
40.8
51.5
+ 1-shot ICL
55.0
56.5
+8-shot ICL
43.8
52.3
+ +MV++I-soot ICL
78.1
80.0
+MTV
52.5
53.0
Idefics2
VILA--15-8B
28.0
32.8
+I-shot ICL
82.8
88.7
+4-shot ICL
39.3
35.6
+ ICL
83.8
89.8
+8-shot ICI.
44.2
36.5
VILA-L.5-8B
+MTV
55.2
40.6
1-shot ICL
87.4
88.4
MANTIS-LLaMA3-AA LLaMA3-8B
36.3
51.7
+MTV+1-shot ICL
89.3
89.7
+4-shot ICL.
26.4
52.5
MANTIS-LLaMAS,
+8-shot ICL
27.5
52.0
1-shot ICL
87.4
84.0
+MTV
51.0
52.8
+ MTV+I-shot ICL
89.8
89.7
5.1 MTV scales with more examples
We are interested in evaluating (i) the effect of different numbers of shots used per iteration to extract MTV and (ii) the effect of different numbers of iterations used. We the impact on accuracy when increasing both of these parameters for QwenVL on the VizWiz validation set. In Figure 3 we show on the left that the optimal number of multimodal ICL shots 16 shots per iteration. Further, we show on the right side of the figure that 1000 examples yield the best performance. These results illustrate that MTV can effectively scale by utilizing larger numbers of ICL. examples per iteration and also in aggregate.
5.2
MTV works with explicit few-shot examples
One of the benefits of MTV over few-shot ICL is the context length that is saved during inference. This is because the many-shot examples are encoded directly in the activation space rather than the input token space. Thus, ask whether the LMM use the freed context for additional few-shot examples. For object classification, we formulate both Flowers and CUB as a 1-shot comparison between positive and negative sample to identify the correct class (i.e., 2-way, 1-shot ICL by construction). We report results on I-shot ICL and MTV with I-shot classification during inference. MTV+1-shot ICL surpasses 1-shot ICL accuracy on these tasks, showing that MTV can be utilized alongside few-shot examples. Furthermore, it is vital to note that the evaluation classes are completely unseen by MTV, Thus, with just 1-shot ICL example, MTV is able generalize unseen classes.
5.3
MTV heads generalize to other tasks
In this experiment, we further ask whether the MTV heads MTV extracted on one task j can generalize to separate, but similar task k. To test this, use the attention heads extracted from

---

57.3
61.5
Method
Method Owen-MTV
55.0
Qwen-MTV
61.0
Vizwiz Accuracy
VizWiz Accuracy
60.5
50.0
59.5
40.0
59.0
5.10
100
No. of Shot Per Iteration
No. of Iterations Used
Figure 3: Scaling of Qwen-MTV on VizWiz: (Left) We show the effect of varying the number of shots per iteration for fixed 100 iterations. (Right) We also show the effect of varying numbers of iterations fixing 4 shots per iteration.
ViLA-1.5-8B on VizWiz for use on OK-VQA. Our results on the left of Table demonsttate that the extracted heads from one task can improve accuracy on another similar task. This generalizability of the heads is significant because it suggests that the heads from MTV may only have to be extracted once to applied to many other similar tasks. The only calculation necessary then would be the mean activations of the many-shot examples used for the target dataset, making the application of many-shot multimodal ICL even more efficient for similar tasks.
Table 3: Generalization & Method Comparison (Left) MTV-VizWiZ evaluated on OK-VQA
(Right) MTV compared to VizWiz finetuning, function vectors 3 and task vectors 2
(a) Attention Head Generalization
(b) Comparison to Other Methods
Model
VizWiz
OK- OK-VQA
Model
VizWiz
OK-VQA
ViLA-115-8B
28.0
32.8
Qwen-Vit VL-7B
35.2
58.6
+ VizWiz E.T
62.0
25.1
4-shot-ICL
39.3
35.6
+ FV
36.4
59.0
8 8-shot-ICL
44.2
36.5
VTV
37.0
58.9
MTV-Vizwiz
55.2
38.3
+ MTV
45.6
62.0
5.4 Finetuning as an upper bound
In Table 3b, we compare our method to finetuning. To do this, we use finetune on the same number of examples as MTV uses from the train set and evaluate not only on the validation set but also on the validation set of another similar dataset. In particular, for a ViLA-115-88 model finetuned on VizWiz, we report accuracy on both VizWiz and OK- VQA validation sets. can be seen that finetuning indeed an upper bound on the dataset the model was finetuned on. However, we show that finetuning leads to overfitting on the finetuned dataset and even forgetting the zero-shot capabilities. In contrast, we also show that MTV not only improves zero-shot capabilities but can generalize to similar tasks with only a few inference examples Table 2b and Table Bal
5.5 Comparison other methods
We compare our method to two different methods that can find task vectors: Visual Task Vectors (VTV) [27] and Function Vectors (FV) [8]] Originally, these works could be applied as-is to support multimodal ICL, but here, we have implemented version that follows the original exactly with only minor modifications to allow performing our evaluated multimodal tasks. More details about the methods can be found in Section A.2 2 in the Supplementary. In our experiments Table 3b we find that MT' surpasses both methods on VizWiz and OK-VQA VTV are image-only task vectors that use only one-shot image examples for fixed small T iterations, and they calculate the

---

mean activations and the locations together without aligning to the downstream task. FV are text-only task vectors that use Causal Mediation Analysis [62] to extract task vector locations from only the output activations of the last token. The results suggest the importance of finding the task vectors by decoupling the calculation of the mean activations and locations two separate steps to perform many-shot multimodal ICL more effectively for complex multimodal tasks.
5.6
Compute and runtime efficiency
Metric
O-shot
4-shot
8-shot
16-shot
MTV (400-shot)
Max GPU Memory (GB)
17.4
18.3
19.0
20.6
19.8
Runtime per 100 iterations (min)
1.1
2.7
3.1
3.3
1.9
Table 4: Efficiency: We show that even though MTV encodes 400 multimodal ICL. examples in the mean activations, still requires less runtime and memory than 8-shot and 16-shot multimodal ICL..
An important feature of our work is that multimodal ICL examples do not require explicit tokens during inference. Because of this, we are interested in the efficiency gains of our method. Intuitively, the longer MTV extraction time is amortized during downstream inference, where the runtime would be equivalent to the zero-shot case. Similarly, the memory requirements are maximal during the MTV extraction process but require the same memory as the zero-shot case afterward. In contrast, the ICL tasks have a slower runtime and larger memory requirement throughout due to running inference on N examples for every iteration. To demonstrate this, we calculate the maximum memory requirement in gigabytes (GB) for ViLA-1.5-8B on VizWiz using different ICL-shot counts and MTV with 400 examples. As shown in Table 4 MTV requires less runtime than 16-shot, 8-shot, and 4-shot ICL methods and also requires less memory than 16-shot ICL. These results demonstrate that MTV can encode many multimodal ICI. examples with greater efficiency than few-shot methods.
6 Conclusion
In this work, we present Multimodal Task Vectors a compact, implicit representation that can efficiently encode many-shot multimodal ICL examples for use in complex vision-language tasks. We demonstrate this implicit model representation not only encodes a multimodal ICL task but can also enable many-shot multimodal ICL to surpass zero-shot and few-shot performance VL tasks. Our method stands out from previous work in its ability to scale, use additional explicit multimodal ICL examples, and generalize to other similar VL tasks. Our work is a viable way to surpass the limit of context length of an LMM for multimodal ICL and demonstrates clearly that these additional examples aid in multimodal reasoning. Finally, we do not anticipate specific negative impact, but, as with any Machine Learning method, we recommend exercising caution.
variety of
Limitations
While Multimodal Task Vectors offers substantial benefits for handling complex vision-language tasks compared to finetuning or few-shot ICL, it is important to recognize certain limitations that accompany our approach. MTV requires access to the internal architecture of an LMM, so while it is an effective solution for all open-source models, its application restricted from proprietary models, such GPT-4 (60) and Gemini [79 80). Furthermore, while many-shot ICL is incredibly attractive for many applications. it may not practical for low-data scenarios where synthetic data [ or the transfer of MTV extracted from another dataset may be required. We feel these challenges represent great opportunities for future work in the many-shot multimodal in-context learning domain.
Acknowledgements
We would like to thank Deva Ramanan, Grace Luo, and Suzanne Petryk for their insightful feedback and discussions. This project has received funding from Prof. Darrell's group part by DoD, including PTG and/or LwLL programs, well BAIR's industrial alliance programs.
10

---

References
[1]
Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Stephanic Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal M. P. Behbahani, Alcksandra Faust, and Hugo Larochelle. Many-shot in-context learning. 2024.
[2]
lean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Scrkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei. Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. ArXiv, abs/2204.1411 14198, 2022.
[3] Amit Alfassy, Assaf Arbelle, Oshri Halimi, Sivan Harary, Roei Herzig, Eli Schwartz, Rameswar Panda, Michele Dolfi, Christoph Auer. Peter W. J. Staar, Kate Saenko, Rogerio Feris, and Leonid Karlinsky. FETA: Towards specializing foundational models for expert task applications. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[4] Stanislaw Antol. Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick. and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2433, 2015.
[5] Elad Ben Avraham, Roei Herzig, Karttikeya Mangalam, Amir Bar, Anna Rohrbach, Leonid Karlinsky, Trevor Darrell, and Amir Globerson. Bringing image scene structure video via frame-clip consistency object tokens. In Thirty- Sixth Conference on Neural Information Processing Systems, 2022.
[6] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, Qwen-vl: A frontier large vision-language model with versatile abilities. ArXiv, abs/2308.12960 2023.
[7] Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R. Gormley, and Graham Neubig. In-context learning with longecontext models: An in-depth exploration. 2024.
[8] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski. Hubert Niewiadomski. Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. ArXiv, abs/2308 09687, 2023.
[9] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan. Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry. Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020.
[10] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. ArXiv, hal/2306.15595 2023.
[11] Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. ArXiv, abs/2305. 14788, 2023.
[12] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prahhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacoh Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Gareía, Vedant Misra, Kevin Rohinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi. David Dohan, Shivani Agrawal, Mark Omernick, Audrew M. Dai. Thanumalayan Sankaranarayana ana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee. Zongwei Zhou, Xuczhi Wang, Brennan Sacta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Mellsem,l,stem,, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Ficdel. Palm: Scaling language modcling with pathways. Mach. Learn. Res., 1-:4::::224:1111 2022.
[13] Wenliang Dai, Junnan Li, Dongxu Li. Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi, Instructhlip: Towards general-purpose vision-language models with instruction tuning, 2023.
[14] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1ms.. ArXiv, abs/2305, 14314, 2023.

---

[15] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng. Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey in-context learning. 2022.
[16] Sivan Doveh, Shaked Perck, Muhammad Jehanzeb Mirza, Amit Alfassy, Assaf Arbelle, Shimon Ullman, and Leonid Karlinsky. Towards multimodal in-context learning for vision & language models, ArXiv, abs/2403. 2024.
[17] Danny Driess, F. Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian lehter, Ayzaan Wahid, Jonathan Tompson. Quan Ho Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar. Pierre Sermanet, Daniel Duckworth, Sergey Levine. Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Peter R. Florence. Palm-e: An embodied multimodal language model. In International Conference on Machine Learning, 2023.
[18] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, W. Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Jiao Qiao. Llama-adapter adapter v2: Parameter-efficiee visual instruction model. ArXiv, abs/2304.15010, 2023.
[19] Jiaxin Ge, Sanjay Subramanian, Baifeng Shi, Roei Herzig, and Trevor Darrel. Recursive visual program- ming. In European Conference on Computer Vision, pages 1-18. Springer, 2025.
[20] Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. ArXiv, abs/2307.06945, 2023.
[21] Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang. Miao Zheng, Qianmengke Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-ppt: A vision and language model for dialogue with humans. ArXiv, abs/2305. 04790, 2023.
[22] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. 2023 IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 111.311962..22220
[23] Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jicbo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. 2018 IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3608-3617, 2018.
[24] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726-9735, 2019.
[25] Roce Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors, ArXiv, 1b5/2310.159116, 2023.
[26] Roei Herzig, Alon Mendelson, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Darrell, and Amir Globerson. Incorporating structured representations into pretrained vision V language models using scene graphs. In The 2023 Conference on Empirical Methods Natural Language Processing, 2023.
[27] Alberto Hojel, Yutong Bai, Trevor Darrell, Amir Globerson, and Amir Bar. Finding visual task vectors. In European Conference on Computer Vision, pages 257-273. Springer, 2025.
[28] Neil Houlsby, Andrei Giurgiu. Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NIP. In Proceedings of the 36th International Conference on Machine Learning, pages 2790-2799, 2019.
[29] J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen, Lora: Low-rank adaptation large language models. ArXiv, abs/2106.0990 09685, 2021.
[30] Zhiqiang Hu, Yihuai Lan, Lei Wang. Wanyu Xu, Ee-Peng Lim, Roy KeiWei Lee, Lidong Bing, and Soujanya Poria. LIm- adapters: An adapter family for parameter-efficient fine-tuning of large language models. ArXiv, ahs/2304.01933, 2023.
[31] Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6693-6702, 2019.
[32] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yun- Hun-H Sung. Zhen Li, and Tom Duerig, Scaling up visual and vision-langueg representation learning with noisy text supervision. In International Conference on Machine Learning, 2021.
12

---

[33] Dongfu Jiang, Xuan He. Huaye Zeng, Cong Wei, Max W.F. Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv2405.01483, 2024.
[34] Huiqiang Jiang, Qianhui Wu, Chin- Yew Lin, Yuqing Yang, and Lili Qiu. LImlingua: Compressing prompls for accelerated inference of large language models. In Conference on Empirical Methods in Natural Language Processing, 2023.
[35] Yixing Jiang, Jeremy Andrew Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H Chen, and Andrew Y Ng. Many-shot in-context learning in multimodal foundation models. In ICML 2024 Workshop on In- In-Context Learning.
[36] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. ArXiv, abs/2205. 1111220011196,,,...11 2022.
[37] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis. Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations, International Journal Computer Vision. 123(1):32-73. 2017.
[38] Hugo Laurenccon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang. Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelisc: An open web-scale filtered dataset of interleaved image-text documents. ArXiv, (en//236..1527,, 2023.
[39] Hugo Laurenccon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision- language models? 2024.
[40] Bin Lei, Pei-Hung Lin, Chunhua Liao, and Caiwen Ding. Boosting logical reasoning in large language models through a new framework: The graph of thought. ArXiv, 0bh/2238..86144 2023.
[41] Brian Lester. Rami Al-Rfou. and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Conference on Empirical Methods in Natural Language Processing, 2021.
[42] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision- bainuala understanding and generation. arXiv preprint arXiv:2201.12086 2022.
[43] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2:
bootstrapping language-image pre-
training with frozen image encoders and large language models. In ICMI, 2023.
[44] Mukai Li, Shansan Gong, Jiangtao Feng, Yiheng Xu, Jinchao Zhang, Zhiyong Wu, and Lingpeng Kong, In-context learning with many demonstration examples. ArXiv, abs/2302.00431,, 2023.
[45] Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context Ilms struggle with long contett learning. 2024.
[46] Xiang Lisa and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volune 1: Long Papers), abs/2101.00190, 2021.
[47] Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023.
[48] Tsung-Yi Yi Lin, M. Maire, Serge J. Belongie, James Hays, P. Perona, Ramanan, Piotr Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[49] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023.
[50] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. In NewrlPS, 2023.
[51] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-172 2023.
[52] Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. ArXiv, abs/2304.09842, 2023.
13

---

[53] Huan Ma. Changying Zhang. Yatao Bian, Lemao Liu, Zhirui Zhang, Peilin Zhao, Shu Zhang, H. Fu, Qinghua Hu, and Bing Wu. Fairness-guiin guided few-shot prompting for large language models. ArXiv, abs/2303.13217 2023.
[54] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: visual question answering benchmark requiring external knowledge. 2019 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 3190-3199, 2019.
[55] AT Meta, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2, 2024.
[56] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? ArXiv, abs/2202.12832 2022.
[57] Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain of thought prompting for large multimodal models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patter Recognition (CVPR), 2024.
[58) Jesse Mu, Xiang Lisa Li, and Noah D. Goodman. Learning to compress prompts with gist tokens. ArXiv, abs/2304.08467, 2023.
[59] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision. Gruphics & Image Processing, pages 722-729, 2008.
[60] OpenAI. Gpt-4 technical report. ArXiv, ahs/2303.08774. 2023.
[61] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
[62] Judea Pearl. Direct and indirect effects. Probabilistic and Causal Inference, 2001.
[63] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. ArXiv, abs/2309.0000 2023.
[64] Yujia Qin, Shi Liang, Yining Ye, Kunlun Zhu, Lan Yan, Ya-Ting Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao. Runchu Tian, Ruobing Xie, Jie Zhou, Mare H. Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models master 16000+ real-world apis. ArXiv, Abs/2307.16788 2023.
[65] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-878 8763. PMLR, 2021.
[66] Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 11:14::1-14::66 2019.
[67] Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbba,, and Pushpak Bhattacharyya. Sciencequ: novel resource for question answering on scholarly articles. International Journal on Digital Libraries. 23:289 301, 2022.
[68] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaf- fin, Arnaud Stiegler, Arun Raja, Manan Dey, Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica. Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden. Thomas Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan. Teven Le Scao, Stella Biderman. Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
[69] Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. ArXiv, abs/2302.04461, 2023.
14

---

[70] Jacob Schreiber. Jeffrey Bilmes, and William Stafford Noble. apricot: Submodular selection for data summarization in python. Journal of Machine Learning Research, 21(161):1-6, 2020.
[71] Chuyi Shang, Amos You, Sanjay Subramanian, Trevor Darrell, and Roci Herzig. Traveler: A multi-Imm agent framework for video question-answering answering. ArXiv, abs/2404.01476, 2024.
[72] Yongliang Shen, Kaitao Song, Xu Tan, Dong Sheng Li, Weiming Lu, and Yue Ting Zhuang. Hugginggpt: Solving ai tasks with chaigpt and its friends in hugging face. ArXiv, 1tt/2333.1158000 2023.
[73] Charles Burton Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context. ArXiv, abs/2209.15189 2022.
[74] Sanjay Subramanian, Medhini G. Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, and Dan Klein. Modular visual question answering via code generation. ArXir, abs/2306.05392, 2023.
[75] Quan Sun, Yufeng Cui. Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. ArXiv, (bs//2312.132855 2023.
[76] D' D'idac Sur'is, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python exccution for reasoning. ArXiv, abs/2303.08 1822 2023.
[77] Sijun Tan, Xiuyu Li, Shishir G. Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph E. Gonzalez, and Raluca A. Popa. Lloco: Learning long contexts offline. 2024.
[78] Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier García, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. U12: Unifying language learning paradigms. In International Conference on Learning Representations, 2022.
[79] Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv, abs/2403.05530. 2024.
[80] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Nean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11800 2023.
[81] Eric Todd, Millicent Li, Arna Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. ArXiv, abs/2310.152133, 2023.
[82] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge J. Belongie. The caltech-uesd 2irds-200-2011 dataset. 2011.
[83] Xingehen Wan, Ruoxi Sun, Hanjun Dai, Serean Ö. Arik, and Tomas Pfister. Better zero-shot reasoning with self-adaptive prompting. In Annual Meeting of the Association for Computational Linguistics, 2023.
[84] Lei Wang, Yilang Hu, Jiabang He, Xingdong Xu, Ning Liu, Hui juan Liu, and Hengtao Shen. T-sciq: Teaching multimodal chain-of-thought reasoning via large language model signals for science question answering, ArXiv, fass/2355034555 2023.
[85] Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka- Wei Lee, and Ec-Peng Lim. Plan-and-solva prompting: Improving zcro-shot chain-of thought reasoning by large language models. In Anmual Meeting of the Association for Computational Linguistics, 2023.
[86] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Fd Huai hsin Chi, and Denny Zhou. Self- consistency improves chain of thought reasoning in language models. ArXiv, abs/2203. 1 11171, 2022.
[87] Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, and Heng Ji. Language models with image descriptors are strong few-shot video-language learners. ArXiv, abs/2205. 10042205510747, 2022.
[88] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Gmu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. ArXiv, abs/2109.01652, 2021.
[89] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Huai hsin Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang. Jeff Dean, and William Fedus. Emergent abilities of large language models. Trans. Mach. Learn Res., 2022, 2022.
15

---

[90] Jason Wei, Xuezhi Wang, Dale Schuurmans. Maarten Bosma, Ed Huai hsin Chi, F. Xia. Quoe Le. and Denny Zhou, Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201. 19903, 2022.
[91] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 9:229-256, 2004.
[92] Chenfei Wu, Sheng-Kai Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. ArXiv, abs/2303.0400 0,667,, 2023.
[93] Benfeng Xu, An Yang, Junyang Lin, Quang Wang. Chang Zhou, Yongdong Zhang, and Zhendong Mao. Expertprompting: Instructing large language models distinguished experts. ArXiv, abs/2305. 14688. 2023.
[94] Shunyu Yao. Dian Yu. Jeffrey Zhao, Izhak Shafran, Thomas Griffiths. Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. ArXiv, abs/2305.10601, 2023.
[95] Yao Yao, Z. Li, and Hai Zhao. Beyond chain-of-thoughh effective graph-of-thought reasoning in large language models. ArXiv, abs/2305. 16582, 2023.
[96] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, ZiZhou, Junyan Wang. Anwen Hu, Pengcheng Shi, Yaya Shi. Chenliang Li. Yuanhong Xu, Hehong Chen, Junfeng Tian, Qiang Qi. Ji Zhang, and Feiyan Huang. mplug-owl: Modularization empowers large language models with multimodality. ArXiv, abs/2304.14178, 2023.
[97] Qinghao Ye, Haiyang Xu, Jiabo Ye, Mingshi Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow12: Revolutionizing multi- modal large language model with modality collahoration. ArXiv, abs/231 1.04257, 2023.
[98] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages (1975-11986, 2023.
[99] Qingru Zhang, Minshuo Chen, Alexander W. Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-effiient fine-tuning, ArXiv, abs/2303. 10512, 2023.
[100] Renrui Zhang. Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Jiao Qiao. alama-adapter: Efficient fine-tuning of language models with zero-init attention. ArXiv, abs/2303. 16199, 2023.
[101] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alexander J. Smola. Automatic chain of thought prompting in large language models. ArXiv, abs/2210.0349g 2022.
[102] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alexander J. Smola. Multimodal thain-of-thought reasoning in language models. ArXiv, abs/2302.00923,, 2023.
[103] Haozhe Zhao, Zefan Cai. Shuzheng Si. Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang. Wenjuan Han, and Baobao Chang. Mmicl: Empowering vision-language model with multi-modal in-context learning. ArXiv, ahs/2309.07915, 2023.
[104] Ge Zheng, Bin Yang, Jiajin Tang, Hong- Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of tf-uhht prompting for multimodal reasoning language models. ArXiv, 01123300114366 2023.
[105] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, arXiv preprint arXiv:2304. 10592, 2023.
16

---

Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
Supplementary Material
Here, we provide additional information about our experimental results, qualitative examples, imple- mentation details, and datasets. Specifically, Section Alprovides more experiment results, Section provides additional method details, Section (B provides
providdes additional implementation details, and Sec- tion C provides qualitative visualizations to illustrate our approach.
A Additional Experiment Results
We present several additional experiments that further demonstrate the benefits of our MTV approach.
A.1 Additional Experiments
Here we provide additional experiments and ablations that further illustrate different characteristics of MTV.
Motivation for encoding shots the activation space.. We highlight our paper's motivation in addressing the context length token limitation of LMMs by encoding ICL shots in the activation space. An additional limiting factor in the token space is the physical constraints runtime, which ablated in Section of of the paper. For example. 25-shot ICL is actually the maximum number of vanilla ICL shots that can be run on a single 48GB A6000 GPU for Qwen- VL. We demonstrate the degradation of increasing numbers of multimodal token-space ICL shots (VizWiz-weeVVi) in Table 5a
memory and
Effect of shot quality MTV: We assess the connection between textual and activation-space shot quality by comparing MTV using random selection with MTV using high-quality shots selected with the Facility Location algorithm [70). We apply MTV to Qwen and use the Qwen GTE embedding model to obtain embeddings for the Facility Location algorithm and present the results in Table 5b Excitingly, we find that high-quality shots do indeed lead to significant improvements in MTV performance.
MTV with noisy exemplars.. We compare the robustness of MTV compared to that of vanilla ICL. For Qwen'V. on VizWiz and OKVQA, we replace of the 4 examples each iteration of 4-shot ICL. and 1aohot-000-iteration MTV with example from the opposite dataset. We report both accuracy and degradation in Table
Table 5: ICL Degradation, Shot-Quality Impact, and Stability (Left) Degradation of ICL with increasing number shots. (Right) Impact of shot quality on MTV and stability of ICL vs MTV with noisy examples.
(a) ICL Degradation with Increasing Shots
(b) MTV with High-Quality Shots
Model
VizWiz
ICL Shots
Acc.
% Acc. Increase
Qwen VL-7B
35.2
0
35.2
MTV
45.6
4
42,0
6.8
+ MTV F.L. Shots
58.1
8
44.3
2.3
(c) Stability of ICL MTV using VWenVL.
16
46.9
2.6
VizWiz
OK-VQ-
20
49.0
2.1
4-shot ICL
41.0 (-1.0)
61.5 (-0.5)
25
49.8
0.8
MTV
43.4 (-2.2)
61.9 (-0.1)

---

Attention head generalization on object classification tasks Table 6a We also test generalization for object classification tasks identical the formulation described in Section [3.3] For clarity, MTV shows another kind of generalization when it is leveraged alongside additional explicit ICL samples. This capability described in Section 5.2 To summarize our experiment, we calculate MTV using the Flowers dataset using 1-shot ICL example for 100 iterations for both the mean activations H2) u and the attention head locations X Then, we apply MTV the CUB task using the same of attention head locations from Flowers. We just calculate the mean activations for the CUB dataset using a 1-shot for 100 iterations (halving our data requirement for this specific scenario). Once again. we find that the heads of MTV can indeed generalize between similar classes.
Table 6: Generalization & Direct ICL Comparison (Left) MTV-Flowers evaluated on OK-VQA (Right) Direct comparison of MTV extracted from 4-shots, 1-iteration (MTV_4shot lit) compared to 4-shot ICL
(a) Attention Head Generalization
(b) Comparison to Other Methods
Model
Flowers
CUB
Model
VizWiz
OK-VQA
ViLA-1.5-8B
ViLA-1.5-8B
28.0
32.8
+ I-shot-ICL
87.4
88.4
I-stttLI
39.3
35.6
+ ..TT--Flowers+-ssho-II..
89.3
89.9
+ WMMV__4hho Bee___4hot____ lit
57.4
40.0
MTV one-to-one comparison with ICI. Table Although not directly comparable, we consider an extreme case of MTV where we encode only 4-shots of ICL examples for 1 iteration. This matches the exact setting used in standard 4-shot ICL. Interestingly, MTV applied to both VizWiz and OK-VQA exceeds performance on the 4-shot-ICL case and even MTV formulated on 4-shots per 100 iterations for calculating the mean activations. This result suggests that there may be scope for MTV to be effective both high and low-data regimens. More research needs to be done to explore this idea.
Method
Qwen+MTV
Flowers Accuracy
100
Total No. of Examples Used
Figure 4: Efficiency. We show that for Flowers, MTV does scale to but only up to 100 examples in our experiments.
Scaling on Flowers Dataset. We provide additional results on the scaling property of MTV on the Flowers dataset. We again note that the examples are 2-way, one-shot examples with 2 examples (one positive and one negative) for each sample. As in the main paper, we fix shot per iteration to calculate the mean activations, scaling up to 500 total examples used. Our results show that there is a saturation of MTV at 100 examples (i.e., example per 100 iterations). While this still indicates some scaling as the result is improvement over 20 examples, the results show that the task vector can reach its best accuracy with fewer shots depending on the complexity of the task. Future work to probe more deeply into the scaling nature of MTV across different tasks would be valuable.

---

MTV at extreme shot counts. We delve further into the scaling capabilities by evaluating the performance MTV the maximum number of VizWiz shots per iteration allowable by the memory constraints of single NVIDIA RTX A6000. The experiment shown in Table 7a indiccats that while MTV does continue scale, there also certainly saturation point for VizWiz. The exact saturation point likely depends on the specific task.
Effeet of permutation. We consider applying five random seeds different configurations of MTV comparing its variability under permutation of example order to standard few-shot ICL. We present the mean and standard deviation for these experiments s-shot and s-ohot settings, MTV shows less variabilityy to example permutation. This intuitively makes sense as more examples are averaged over multiple iterations, leading more stable performance across different seeds.
Table 7b Although not significant, both
Table 7: Qwen Generalization & Variability to Permutation (Left) Evaluation of MTV on extreme shot-iteration counts. (Right) Variability to permutation across seeds.
(a) QwenVLL-7B: Extreme shot-count performance
(b) Permutation Variability
Model
Accuracy (%)
Model
Accuracy (%)
Qwen VL-7B
QwenVL-7B
+ MTV_ MTV__MV2VoOO ____shot__000r
54.9
MTV +MTV_4-soot 00_-shot__1000 100it
45.2 (0.7)
+ MTV 20shot_200ii
55.1
+ MTV 1_-shott_200h
48.3 (0.4)
+ MTV MTV_25shot_L __5shot_100ir
56.4
+ +MTV_8shoot 8-shottshOmme
50.4 (0.9)
+ MTV UTV__2shot_200it
51.4
+ MTV 8-shot__0000
51.8 (0.6)
+ 4-shot ICL
41.3 (0.8)
8-shot ICL
42.9 (1.5)
MTV for language-only tasks. While we show the importance of MTV especially for vision- language tasks, the methodology can be powerful way to learn tasks in the language-only domain as well. We demonstrate in Table Sa the effectiveness MTV on two common LLM tasks using LLaMA-3-8B [55]:
Table 8: Performance on Language and Document Tasks (Left) Evaluation on English-Spanish and Antonym Generation tasks. (Right) MTV performance across different shot settings on document tasks.
(a) Performance on Language-Only Tasks
(b) MTV Document Task Performance
Snglish-Spanish
Antonym Generation
O-shot
4-shot
8-shot
MTV
10 shot
65.2
56.0
CharlQA
19.1
25.0
26.4
34.9
400 shol
68.5
57.6
TexIVQA
42.4
45.4
51.0
MTV 4-100
76.7
61.7
MTV on additional document datasets. Multimodal documents are form of data with complex compositions visual and textual modalities, with interleaved language, photograph, and chart information. As such, we provide some preliminary results on the effectiveness of MTV on these types of datasets in Table These encouraging results prompt future research into the domain of leveraging task vectors for learning challenging document tasks.
Here provide some additional method details about MTV, Visual Task Vectors (VTV) [2 and Function Vectors [1 (FV).
A.2 MTV-EXTRACT
We describe the particulars of our MTV-EXTRACT algorithm for finding the set of attention head locations that best align with the downstream task as follows (Q, and R2 are formatted identically to the downstream task):

---

Algorithm 1 MTV-EXTRACT for finding task vector locations
Require: (LMM), S (examples), Hi (mean activations), Qa, R. (queries and responses)
Ensure: XMTV (optimized attention head locations)
1: Initialize 0 randomly
2: for - 1 to 5 do
3: for - 1 to 32 do
Sampling heads 32 times
4: Sample A ~ Bernuuli(((())
5: Replace activations for X; in F with ml,
6: Compute output logits O₂ < F(Q2)
> Pass Qo to LMM F
7:
Ti++Negative Cross Dooss--Ettooy(O,,, R.) 8: end for
9:
Adam(0, Vo Σ 2i=1 T.)) end for
Update rule
10:
Sample final XMTV Bernoulli(a
b Final set of head locations
12:
return XMTV
We point out few important factors. It is important to note that none of the parameters of F are being finetuned through any gradient update. We take the negative cross-entropy (negative MTV EXTRACT draws inspiration from REINFORCE (9I) which is policy optimization algorithm) between the output logits O, and the first token of the target response Rt for a simple update scheme. This along with the choice of 32 samples of the Bernoulli distribution are encourage more experimentation with in future work.
A.3 Visual Vectors (VTV) Adaptation for Multimodal ICL
Visual Task Vectors (VTV) 27 were originally designed be applied large vision-transformer- based models. We make as few changes as possible to apply this method for multimodal tasks. We preserve VTVs distinct factors like the usage of 1-shot examples for both calculation of the mean activations and attention head locations regardless of the format of the downstream task. Furthermore, we fix the number of iterations for both mean activation and attention head calculation at 10. Finally, we replace the proposed MSE loss with a cross-entropy loss that is more suited for an LMM task.
A.4 Function Vectors (FV)
Because Function Vectors describe text-only task vectors, we follow the implementation of Function Vectors [8I] almost exactly LLMs and LMMs are similar. The only major change made is the use of many-shot multimodal ICL examples for mean activation calculation. We preserve the lack of an optimization method for the layer used to replace the mean activations. Rather than performing standard grid search over the set of layers, we set the layer number to as recommended for I.LaMA and LLaMA-based models by the paper. The only other difference is the encoding of multimodal ICL examples. Again due the the similarity between LMMs and text-only LLMs, these tests can be used needed as long as the multimodal inputs are properly processed by the LMM.
B Additional Implementation Details
To run all of our experiments, we use 1 NVIDIA RTX 6000 GPU. Importantly, this includes the runtime and efficiency ablations, which were evaluated on the same GPU for consistency. Please refer to the respective model's paper for their specific implementation details of the architecture. Besides the output token generation length, which varies depending on the standard setting for each task, we use the default generation parameters (e.g. temperature and no. of beams in beam search) recommended for each model. In the following sections, we describe some of the finer nuances of our MTV-EXTRACT process as well as our implementations of the Visual Task Vectors (VTV) and Function Vectors (FV) implementations.

---

B.1 VizWiz
Dataset. The VizWiz dataset is designed to challenge and evaluate the capabilities of Large Mul- timodal Models (LMMs) in understanding and responding to real-world visual questions. This dataset is comprised of images accompanied by spoken questions, which have been transcribed and paired with answers. Each image in this dataset is sourced from visually impaired individuals secking assistance, thereby incorporating a wide array of everyday challenges they face. This setup is inherently diverse and often requires high-level visual understanding combined with contextual reasoning, making them a robust benchmark for assessing the practical utility of LMMs assistive technologies. The format of the dataset samples is image paired with text question. The LMM is required to provide a short response limited to 10 tokens or respond with "unanswerable" if the question is not answerable give the image.
For this research paper, we specifically utilize the VizWiz dataset to benchmark the performance of our proposed task vectors in multimodal in-context learning (MM-ICL) on a dataset that challenges visual scene understanding of LMMs. We extract MTV on the training set and evaluate on the evaluation set containing 4,319 validation imagc/question pairs.
Inference details. We use the standard VQA question-answer response format that is outlined in the QwenVL repository https: g/ com com/QquenL//ween-- VL Put simply, the LMM is presented with an image and a corresponding text question. The response is then expected a short text format of no more than 10 tokens (set the "max_token" parameter in the LMM). One nuance is the special answer "unanswerable". We handle this by providing MTV and all baselines with the following prompt for every question: "First carefully understand the given examples. Then use the given image and answer the question in the same way as the examples. If the question can not be answered, respond unanswerable. The official dataset can be downloaded at phtttss:/// / org/ tasks-ak--saaaa and datasets/vqa/
B.2 OK-VQA
Dataset. The OK-VQA dataset, differs from traditional VQA datasets in its focus on necessitating knowledge beyond what is presented in the given images. This dataset encompasses over 14,000 questions that are not merely reliant on visual cues but require associative reasoning with external data sources, making it unique tool for evaluating Al's capability in handling complex, knowledge-driver queries. Thus, we evaluate on this dataset to test whether MTV can be beneficial for this type of reasoning.
We once again extract MTV on the train set and evaluate on the validation set. OK-V -KAV is formatted as image with corresponding text question. However, it is important to note that the text question heavily relies on external knowledge to answer. Examples of questions can be found in Section C
Inference details. We use the standard VQA question-answer response format that is outlined in the VW repository https: //gits com QwenLM//wen. VL. Put simply, the LMM is presented with an image and a corresponding text question. The response is then expected in a short text format of no more than 10 tokens (set as the "max tokens" parameter in the LMM). We do not add any additional prompts or special tokens apart from prompt format or image tokens required by the model being evaluated. The official dataset can downloaded at httttt:t//////e allenai..rr...
B.3 Flowers
Dataset. Flowers 59 an object classification dataset that requires fine-grained classification of 102 different flower species. The Flowers dataset is formulated 2-way, 1-shot task where one example is the positive sample and the other is the negative sample. In this way, the data poses a unique challenge for MTV having store examples with two associated images. Thus, given the 2-way examples and the query image, the LMM is tasked with selecting the correct class from the given two options. Examples can be found in Section C
Implementation Details. We use the official data released by the authors which is available at https://wwzz http://wwwwwbots.o.ttooo robots.t.......... ox. ac uk/ "vgg///g//a/ httg//daaaflooes////// We provide a Python code snippet below showing the Flowers data format:
def fiimat_floverccuuu (cur_data):
5

---

pos = cur cur_data["pos"
neg "adda""aa"""
pos_label cur cur_data[" "r__data""ps__lablll
neg_label cur curadata_ cur_datt_"neg__aael""
query - cur "ar_data("query]] rand_num candom.randint(0..1)
rand num 0:
pos_example #P<áng><foo></inn>>iha the type of flower the image? A.f
poa_label) B. (. Laaaaaeelnnnneer with the option's letter from the given choice directly. Answer: A\n"
neg_ reg_example (n poa_ boa_la B. fneg_ labell with the option's letter from the given choice directly. Answer: B\n"
the type flower the image? A.(
cur_query (uinagnn is the type flower in the image? A. pos_labell B. L.imeg_labellnnnnse with the option's letter from the given choice directly. Answer:"
query_label "A"
return pos_example + example cur_query, query_laber -1
else:
exa_pee fiddng>>ppaa/</nnn>aaaa is the type of flover in the image? A. neg_label> B. (.ppos_ Laoos_llabllnnnnsse with the option's letter from the given choice directly. Answer: B\n"
neg example f" neg. bagela B. (pos__laaeel
is the type of flower in the image? A.f label) AnAnswer with the option's letter from the given choice directly. Answer: Aln"
cur_ cur_query = #"kinn>(quer>)<<mm>>aat meg_label> B. l.-pos_label)\nnee with the option's letter from the given choice directly. Answer:"
the type flower the image? A.f
query_label "B" return neamexaele pos_example cur_query, query_label,
B.4 CUB
Dataset. CUB 82 or CUB-200-2011 is an object classification dataset that tests the fine-grained classification 200 classes birds. Similar the Flowers dataset, CUB formulated 2-way, 1-shot task where one example is the positive sample and the other is the negative sample. In this way, the data poses unique challenge for MTV having store examples with two associated images. Thus, given the 2-way examples and the query image, the LMM is tasked with selecting the correct class from the given two options.
Implementation Details. We use the official data released by the authors which is available at https://www. ssttp:://ww.vvsiion vision. caltech. snippet below showing the Flowers data format:
cectaasstscuu___eaea c/t_c/ub___00/00 200_2011/ We provide a Python code
def format_ cubiaat_uubdee -aturddaaaa:
pos cur cur_datte data["pos""
neg cur cur_data pos_label cur
label = cur [an__atta"nne__labe
query = "rr_daaaa"querr""
rand_num random. (0,1d
if vand_num 0:
jos_example #"King>(pps></innWWaat is the type of bird in the image? A.
pos_label} B. Lateg_la with the option's letter from the given
choice directly. Anaver: ANn"
neg_ example f" (ngg) is the type of bird in the image? A. label) AnAnswer with the option's letter from the given choice directly. Answer: B\n"
pos_ labela B. (nog_label))

---

cur_query f is the type bird the image? A. pos_ yos_label> B. LateeglabellAnAnsswe with the option's letter from the given choice directly. Answer:"
query_label "A" return pos_example + examexellle cur_query, query_label, -1
else:
pos_ example f" finng>>ppaa<</inn>iittt is the type of bird in the image? A. neg neg_label) B.
with the option's letter from the given choice directly. Answer: B\n"
neg. exampe #Paing>fnnegk//mmgnaaaat the type of bird the image? A.f \nAnswer with the option's letter from the given
neg_labell B. choice directly. Answer: A\n"
cur_query fiaiappa the type bird the image? A.f neg. neg_label> B. (apps_labellnnaeeee with the option's letter from the given choice directly. Answer:"
query_label "B" return neg_example pos_example cur_query, query_label,
C Qualitative Visualizations
We present further qualitative success and failure cases of QwenVL-MTV in Figure 5 on OK-VQA and Flowers.
D Licenses and Privacy
The license, PII, and consent details of each dataset are in the respective papers. In addition, we wish to emphasize that the datasets we use do not contain any harmful or offensive content, as many other papers in the field also use them. Thus, we do not anticipate specific negative impact, but, with any machine learning method, we recommend exercising caution.

---

Flowers Examples:
Positive Example
Negative
Posttive Example
What the
What the type of flower in the image' A.cape
What is the type flowor image?
the image?
A Autbyy lppaci cattleya Bisnapding ragon
Aruby loped calt eya B Banaadraaenn Answer with the option's, lutter from the givan cholca directly.
Answer with the option's latter from the givann choics direcily.
Answer given direcity.
directlyy
Ansower
Answer:
Query
Query
What
Nloer
What the type of flower mage?
Aruby ilaped cottleya B. B.snaperagon
Answer with the optorts.
letter from the glvan chokka directly.
Zero-shot:
Zero Paro-Shott:
MTV:B
Positive Example
Negative Example
Positive
Negathve
What is the type flower the image? A capa Answer directly.
What the type of flower the image? capa option's
What is the type flower in the image? A japanese anemone mexican aster Answerr given directly.
What the typo of flower in the image? japanese Answer option's
B.meslcan
directly.
directly.
Answer:
Answer:
Answer:
Quary
type
tyse
beardee
Alapanese
E.mestean
Answer with the aptionns letter from the given charce directly.
Answer with the option's latter from the choice directy..
Zoro- Shot: MTV
Zero-Shat:
MTV:
OK-VQA Examples:
CAUTION
THESH STIPP EDIES
whrt speed dows wnimal
What piece of apparel holds siy version of the item on the pole?
What does the color this sign represent America?
sporting event
Zero-shot: necklace
Zero:Vellow MTVCCaution
Zerossoccer
MTV: 30mph
Figure 5: Efficiency. show that for Flowers, MTV does scale to but only up to 100 examples experiments.

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims are supported both by the main results and additional ablations and experiments Sections
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discussed about limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
This point is not relevant; it is not theory paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Everything is reproducible. We include any necessary details in Section B| as well the Supplemental sections.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
10

---

Answer:
[Yes]
Justification:
Yes, code is provided in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
cariiinnooiyyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctpppp.ccppulli//guiii wwwiiccgguie///eeeeeeeei CoodeSubeiiieeee ssi wswinnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, all is included in Section 4 as well as the provided code in the abstract.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Our paper does not require eтror bars or statistical significance, only accuracy.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, describe required compute for our method in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics atttps://nat /tttur::///errps.... httpp//ccccpppp...ttttttttteeeea
Answer:
[Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts disccused in Section 6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
12

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We don't 't have any safeguards to discuss here.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: All data and code are credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswipteeeee wwworeeswthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
13

---

Answer: [NA]
Justification:
We present a method and no additional assets are introduced in the paper.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: No human subjects experimental data involving humans used in this research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification:
This research does not have any experiments with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
14


Reviews:
[{"review_id": "CAOI9VfuJ1", "reviewer": "Reviewer_dwXj", "summary": "In-context learning with many examples can be effective for learning new tasks. However, there are challenges with many-shot multimodal in-context learning (ICL), such as the limitation caused by the model’s context length. This issue is more challenging in the multimodal setting because it processes both images and text, requiring additional tokens. Therefore, a method to compress this information into fewer tokens without finetuning is necessary. The paper proposes using large language models (LLMs) to perform many-shot in-context learning with multimodal task vectors (MTVs). MTVs are compact, implicit representations of in-context samples, compressed by the model’s attention head. The paper provides experiments showing that outperform existing methods.", "questions": "please refer to the weaknesses.", "limitations": "The paper includes limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is generally well-written. \n\nThe motivation is clear, and the research question, and challenging and up-to-date. \n\nThe paper provides experiments showing that the proposed method outperforms existing methods. \n\nThe paper also provides ablation studies to better analyze and understand the model.", "weaknesses": "I would recommend the authors compare their work with “Many-Shot In-Context Learning\nin Multimodal Foundation Models”. I think this can be a concurrent work, however, a general comparison of the methods and adding the reference to the paper helps the paper. \n\nI am a bit confused with the number of data used in each case in Tables 1 and 2. Can the authors elaborate on that? Also, can the authors explain more about the MTV+ 1-shot ICL setting?\nIs the setting for MTV in Table 1 2-way 1-shot always?\n\nWhile the FV and VTV are other existing methods, the paper does not clearly explain those in the main text and related work. I would recommend adding an extra link to the explanation on the supplementary material. Also, would the authors elaborate on the main differences here as well?\n\nBased on Figure 2, the best shot is 16 and iterations 100. However, the paper does not provide any data on shots more than 16 and iterations more than 100, which makes the conclusion suboptimal. Similarly, the paper discusses the effect of permutation only with one experiment (running for different seeds) which is not enough. \n\n\n\nWriting:\n\nIn line 227, (3) is extra\n\nReferring to steps 1 and 2 in lines 42-55 while these steps are not clearly defined makes it hard to follow the text. \n\nI would recommend the authors explain more about the notion of interleaved data in the paper. The current writing can be difficult to read.\n\nIn section 5.2, the reference to the table is missing\n\nReference missing on “scaling on flowers dataset” section\n\nReferences to sections are missing in A1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and respond to all points in the following, incorporating all suggested changes into the paper:\n\n**Comparison with “Many-Shot In-Context Learning in Multimodal Foundation Models”.** We thank the reviewer for sharing this concurrent work with us and will add the reference to our final version. While the topic of many-shot in-context learning is a shared feature with our work, one should note that the suggested paper focuses exclusively on closed, proprietary models such as GPT-4o and Gemini-1.5 for image classification. Our work focuses on enabling many-shot, in-context learning capabilities for open-source, interleaved models with limited context lengths. We also evaluate on VQA benchmarks, not just image classification tasks.\n\n**Tables 1 and 2 format clarification.** We clarify the exact format and number of shots for each experiment. In Table 1, the prompt for zero-shot is formatted as simply ``<Query>`` (the question to be answered). For 4-shot, the prompt consists of four randomly-selected concatenated examples of multimodal question-answer pairs followed by the query: ``<Q1> <A1> <Q2> <A2> <Q3> <A3> <Q4> <A4> <Query>``. The 8-shot case is identical except with 8 exemplars instead of 4. \n\nFinally, we use a setting of MTV with 400 examples (4 shots per 100 iterations) for calculating mean activations and 100 examples (zero-shot per 100 iterations) for extracting the attention-head locations.\n\nFor Table 2, the 2-way-1-shot formulation is used for all experiments. The prompt structure includes one positive example, one negative example, and the query image: ``<image_1> <class_label_1> <image_2> <class_label_2> <query_image>``, where the task is to classify the ``<query_image>`` correctly as one of the given classes. The baseline consists of just a single one of these prompts, while the MTV case calculates mean activations from 100 iterations of one 2-way-1-shot example. The exact prompt we use is provided in Section C.3 of the supplementary material. \n\n\n**Function Vectors (FV) and Visual Task Vectors (VTV) methods explained.** We include a more detailed description of Function Vectors (FV) [1] and Visual Task Vectors (VTV) [2]:\n\n\nFV are a type of text-only task vector that has the following key differences from our method:\nFunction Vectors uses a fixed number of shots without exploring the variation of both the number of shots or iterations for many-shot ICL.\nFV utilizes Causal Mediation Analysis [3] to extract the attention head locations. At a high level, this method selects attention heads by calculating the causal impact of each attention head (i.e. how much the LLM’s probability of outputting a correct answer increases given a set of corrupted, incorrect, ICL examples). The layer to replace activations is chosen via a simple linear search. This is in contrast to MTV which selects attention heads by learning a distribution over the attention heads across all layers. \n\nVTV is a type of image-only task vector that has the following key difference from our method:\nVTV is a visual prompting method for vision-transformer models [2]\nVTV also uses a fixed number of single-shot examples without exploring the variation of shots and iterations to enable many-shot ICL.\nIn contrast to VTV, MTV uses a token-level loss to align with the usage of an LMM for a multimodal task.\nVTV uses a single set of one-shot examples for both mean activation calculation and attention head extraction. In contrast, MTV decouples this process by first calculating the mean activations and then using a separate set of examples that are specifically aligned with the format of the downstream task. \n\nWe find that our method’s distinct properties—increase in shots and iterations, token-level loss, and the decoupling of the activation aggregation and attention-head extraction—are key factors that enable MTV to improve over FV and VTV. \n\n**MTV with more shots and iterations.** Following the reviewer’s suggestion, we present results on MTV with higher shots (20, 25) and iteration counts (200, 400) for Qwen on Vizwiz. 25 shots is the highest number allowable by the memory constraints (48 GB) of a single A6000 GPU. We present the results here:\n\n| Shots | Iterations | Accuracy |\n|-------|------------|----------|\n| 20    | 100        | 54.9     |\n| 20    | 200        | 55.1     |\n| 25    | 100        | 56.4     |\n| 25    | 200        | 51.4     |\n\n\nThe purpose of our experiment shown in Figure 2 is to show the *ability of MTV to scale*, not necessarily to find the optimal shot-iteration setting. This exact value will differ based on the dataset and model. Nevertheless, the above results demonstrate the ability of MTV to scale even beyond 16 shots per iteration. \n\n**Measuring the impact of permutation.** As the reviewer suggested, we explore the effect of permutation in more depth by evaluating Qwen-VL on VizWiz using the following (shot, iteration) counts for calculating mean activations: (4,100); (4,200); (8,100); (8,200). We evaluate on 5 seeds for each setting as well as 4-shot and 8-shot ICL as a baseline. In the following table, we present the mean and standard deviation for each run:\n\n*MTV:*\n\n| Shots | Iterations | Accuracy |\n|-------|------------|----------|\n| 4    | 100        | 45.2 (.7)  |\n| 4    | 200        | 48.3 (.4)     |\n| 8    | 100        | 50.4 (.9)    |\n| 8    | 200        | 51.8 (.6)    |\n\n*Few-shot*\n\n| Shots | Accuracy |\n|-------|------------|\n| 4    | 41.3 (0.8)   |\n| 8    | 42.9 (1.5)   |\n\n\nMTV for 4 and 8 shots for different iterations is always more stable with respect to example permutation compared to vanilla ICL in our experiments. \n\nThank you. We hope these points address all of your concerns, and we are happy to clarify anything further. We also commit to fixing all paper writing errors pointed out in the final paper draft.\n\n\nReferences:\n\n[1] Todd et. al. FV\n\n[2] Hojel et. al. VTV\n\n[3] Imai et. al. \"A General Approach to Causal Mediation Analysis.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sbu60MYZpS", "reviewer": "Reviewer_3gQC", "summary": "In the context of multimodal understanding, this work studies the area of many shot, long-context ICL (in context-learning) in natively multimodal models (large multimodal models i.e. LMMs, where images and text are interleaved). The premise proposed is that the pretrain time context of existing LMMs is prohibitive w.r.t generalization to longer contexts for adding many-shot multimodal examples. To leverage many examples under this context budget, this work proposes a technique to compress i.e. encode these multimodal exemplars into the weight space (multimodal task vectors i.e. MTV) by encoding them via mean attention head activations and replacing these mean activations into heads which are most aligned for a downstream task (which is potentially out of domain / zero shot). This selection is done at inference time via a REINFORCE based selection mechanism. Empirically, consistent improvements are demonstrated compared to the ICL baseline across open models (where weights can be accessed) on 4 tasks covering visual question answering and object classification.", "questions": "Besides the above, here's a suggestion: there are many abbreviations (e.g. VL benchmarks) which could be expanded clearly.", "limitations": "Limitations are mentioned.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper addresses and important and relevant area of multimodal Q/A or understanding in natively multimodal models, where not many benchmarks exist to understand the long context multimodal capabilities of LMMs. I like the coherence in the methodology and the stepwise presentation in the paper. Figure 1 gives a concise overview of the method being proposed.", "weaknesses": "While the empirical improvements are encouraging, following are some weaknesses where I have concerns regarding the motivation of the method and it would be great to have a discussion on these:\n\n1. How is the proposed method uniquely situated in the context of \"multimodal\" QnA? The context length limitation, encoding exemplars efficiently and tuning head selection for a downstream task seem to be applicable in the text-only domain too. Can this same technique be applied to the text-only space as well? What is specific about the methodology proposed which would make sense for the multimodal domain but probably not work for text?\n\n2. There are repeated claims on images being \"more expensive\" in the token space e.g. “Image embeddings are expensive”, “the images require more tokens to embed”. I believe these claims could be concretized better by doing a simple analysis of difference in tokens, because I believe this difference is usually not prohibitive. It would also be nice to have numbers on the context lengths in the benchmarks mentioned. \n\n3. There is no evidence of degradation of model performance with increasing context length in the number of multimodal shots. Without this, it is hard to understand the motivation for choosing to encode shots in the weight space. If one can see the degradation being more rapid for vanilla ICL compared to this method, the motivation would be much more convincing.\n\n3. Why does one need REINFORCE based selection? Concretely, how would the current method compare with the baseline of simply replacing all heads with the mean activations? It is unclear why the authors chose to use REINFORCE. \n\n4. Having many shots in the context (v/s in the weight space) provides the benefit of interpretability - what is the intuition on losing on this benefit? Does shot quality in the textual space correspond to shot quality in the weight space? As a follow up, how does the \"cost of prompt iteration\" compare between vanilla ICL v/s MTV?\n\n5. How robust is this technique to shot quality e.g. noisy exemplars? Is it more or less robust than vanilla ICL?\n\n6. The empirical results focus on comparisons with vanilla ICL. While encouraging, it would be convincing if the method shows performance improvements on long context tasks where vanilla ICL fails but this method works. Further, the tasks considered are still limited and a wider range of longer-context multimodal benchmarks could be considered. (to name some e.g. MMMU, TextVQA, DocVQA).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and respond to all points in the following, incorporating all clarifications and additional results into the final paper:\n\n**MTVs for Text-only domain.** Using LLaMA3, we evaluate MTV on text-only tasks. The two tasks are English-Spanish translation and Antonym generation, as done in [1]:\n\n|             | English-Spanish | Antonym Generation |\n|-------------|-----------------|---------|\n| 10 shot     | 65.2            | 56.0    |\n| 400 shot    | 68.5            | 57.6    |\n| MTV 4-100   | 76.7            | 61.7    |\n\nWe find that MTV is also effective in the text-only domain.\n\n**MTV’s unique fit for multimodal tasks.** As suggested by the reviewer, we provide the token count per image for each model to highlight the cost of image inputs:\n\n| Model Name | Per Image Token Length | Total Context Length |\n|------------|-------------------|----------------------|\n| VILA       | 144               | 8192                 |\n| Idefics2   | 64                | 8192                 |\n| QwenVL     | 256               | 8192                 |\n\nA maximum of 8192/64 = 128 images can be encoded assuming no language inputs, already lower than the number in base MTV (400 shots). \n\nConsidering language tokens, we present the percentage of the input that is text and image on VILA 1.5 for datasets we evaluated on: \n\n| Dataset | % Input Text | % Input Image |\n|-------|----------|------------|\n| VizWiz     |  6.6   |    93.4  |\n| OK-VQA     |      8.4 |    91.6   |\n| Flowers     |   12.7       |     87.3    |\n| CUB  |  14        |    86      |\n\nThis shows that images quickly consume many of the input tokens for multimodal tasks. The data presented thus demonstrates the value of MTV, especially in the multimodal domain. \n\n**Motivation for encoding shots in the activation space.** We highlight our paper’s motivation in addressing the context length token limitation of LMMs by encoding ICL shots in the activation space. An additional limiting factor in the token space is the physical constraints of memory and runtime, which we ablated in Section 5.6 of the paper. For example, 25-shot ICL is actually the maximum number of vanilla ICL shots that can be run on a single 48GB A6000 GPU for Qwen-VL.\nRegardless, following the reviewer’s suggestion, we demonstrate the degradation of increasing numbers of multimodal token-space ICL shots (VizWiz-QwenVL):\n\n| ICL Shots | Accuracy | % Accuracy Increase |\n|-------|----------|------------|\n| 0     |  35.2    |    -     |\n| 4     |   42.0     |    6.8     |\n| 8     |    44.3      |    2.3     |\n| 16    |   46.9       |      2.6      |\n| 20    |      49.0    |    2.1        |\n| 25    |      49.8    |    0.8       |\n\nCompared to the results shown in Figure 2 of our paper, vanilla ICL improvement degrades at lower accuracy than MTV.\n\n**REINFORCE-based attention-head extraction.** We perform an experiment replacing mean activations of ALL heads for VizWiz. Interestingly, QwenVL achieves 0% accuracy on Vizwiz in this setting. We assume this is due to the query contexts being completely overwritten by the mean activations.\n\nPast work has shown that some subset of the heads include important ICL information and showed several ways to extract this information [1,2,3]. In our work, we select this subset of attention heads using a REINFORCE-based method to optimize a distribution over a *set* of attention heads as opposed to one-at-a-time like CMA [1], leading to superior performance (Section 5.5).\n\n**MTV Shot Quality.**  We assess the connection between textual and activation-space shot quality by comparing MTV using random selection with MTV using high-quality shots selected with the Facility Location algorithm [5]. We apply MTV to QwenVL and use the Qwen GTE embedding model to obtain embeddings for the Facility Location algorithm:\n\n| Model            | VizWiz | \n|--------------------------|---------|\n| QwenVL-7B         | 35.2   | \n| + MTV                    | 45.6 | \n| + MTV + F.L. Shots| 58.1   |\n\nWe find that choosing higher-quality examples drastically improves the performance of MTV, indicating that shot quality in the textual and activation spaces is closely connected.\n\n**MTV interpretability.** We appreciate the reviewer's note on interpretability. We posit an alternative perspective that MTV shots are no less interpretable but are simply the same set of examples represented as a set of activations. Furthermore, task vector methods like MTV give additional insight into which attention heads are relevant for an ICL task [1,4].\n\n**Cost of prompt iteration.** Please note that we provide an analysis of the cost of prompt iteration in Section 5.6 of the paper. We show that MTV is more runtime efficient than 4-shot ICL and more memory efficient than 16-shot ICL.\n\n**MTV with noisy exemplars.** We compare the robustness of MTV with vanilla ICL as suggested. For QwenVL on VizWiz and OKVQA, we replace 1 of the 4 examples in each iteration of 4-shot ICL and 4-shot-100-iteration MTV with an example from the opposite dataset. We report both accuracy and degradation.\n\n|        | Qwen VizWiz | Qwen OK-VQA |\n|--------|-------------|-------------|\n| 4-shot | 41.0 (-1.0) | 61.5 (-0.5) |\n| MTV    | 43.4 (-2.2) | 61.9 (-0.1) |\n\nThe results suggest that MTV and vanilla ICL have similar robustness to noisy examples.\n\n**Document datasets.** We evaluated MTV on document datasets ChartQA and TextVQA with VILA-1.5:\n\n|          | 0-shot  | 4-shot  | 8-shot  | MTV  |\n|----------|-----|-----|-----|------|\n| ChartQA  | 19.1| 25  | 26.4| 34.9 |\n| TextVQA  | 42.4| 45.4| 47.1| 51   |\n\nThese results indicate MTV’s capabilities on document-based tasks. \n\nThank you. We hope this addresses all concerns, and we are happy to clarify anything further.\n\nReferences:\n\n[1] Todd et. al. FV\n\n[2] Hendel et. al. \"In-Context Learning Creates Task Vectors.\" \n\n[3] Olsson et. al. \"In-context Learning and Induction Heads.\" \n\n[4] Hojel. et. al. VTV.\n\n[5] Schreiber et. al. Apricot.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZqNvB0ZPtA", "reviewer": "Reviewer_p88K", "summary": "This paper should be desk rejected as the single PDF submission does not have the paper checklist. \nIt would be unfair to make an exception as this requirement has been clearly stated at https://neurips.cc/Conferences/2024/CallForPapers and the latex template file.", "questions": "NA", "limitations": "NA", "rating": 1, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "NA", "weaknesses": "NA", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please note that our checklist is provided in our Supplementary pdf, and thus, the PCs have decided not to desk-reject this paper. As this has been universally applied to all submissions, our submission is thus in accordance with guidelines, and fairness should not be adversely affected.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AsuENpIepz", "reviewer": "Reviewer_Xn5t", "summary": "This work presents MTV to enhance the in-context learning (ICL) capabilities of LMMs, which typically have limited context length, especially when dealing with multimodal data that includes both text and images. MTV addresses this by compressing many-shot examples into compact implicit representations within the model's attention heads. The approach involves calculating mean activations of attention heads across multiple inference iterations and selecting optimal locations for these activations using an adapted REINFORCE algorithm. This method allows LMMs to effectively use more examples than their context length permits, improving performance on various vision-and-language tasks without finetuning. Experiments demonstrate that MTV not only scales with more examples but also generalizes to similar out-of-domain tasks and works alongside explicit few-shot examples, offering a practical solution for efficient and effective many-shot multimodal ICL.", "questions": "1. Can this method work on LMMs that can only take in one image? Because most of models currently being evaluated are models that can take multiple images. Since your work is like to use the in-context examples as the prior, then I think it's possible.", "limitations": "No significant limitations", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Interesting idea and finding, MTV is an efficient and lightweight method to use training data and off-the-shelf LMMs.\n2. The writing is well-written and easy-to-follow overall.\n3. The results seem good, models are evaluated on two tasks, four benchmarks and three models. In addition, the efficiency is also evaluated. I like the efficiency evaluation.", "weaknesses": "1. Minor: the dataset can always be included more: Try MMMU[1], OpenFlamingo and Otter did worth with 1/3/5-shot examples.\n2. Minor: the models can also be included more: Try Mantis[2]\n\n[1] Yue et al., MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. 2023 ArXiv.\n\n[2] Jiang et al., Mantis: Interleaved Multi-Image Instruction Tuning, 2024 ArXiv.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments. In the following, we provide a response to the questions raised in the review:\n\n**Additional MANTIS results.** As suggested by the reviewer, we present some additional results of our method using the MANTIS-LLama3 model on OKVQA, Vizwiz, Flower, and CUB:\n\n|       | OK-VQA | VizWiz | Flower | CUB  |\n|-------|-------|--------|--------|------|\n| Zero-Shot  | 51.7  | 36.3   | -      | -    |\n| 4-shot ICL   | 52.5  | 26.4   | 88.7   | 84.0 |\n| MTV-100   | 52.8  | 51.0   | 89.8   | 89.7 |\n\nWe will include these results in the final draft of the paper.\n\n**Additional Dataset results.** As suggested by the reviewer, we present additional results of our method on the following benchmarks using VILA-1.5:\n\n|          | 0-shot   | 4-shot   | 8-shot   | MTV  |\n|----------|-----|-----|-----|------|\n| ChartQA  | 19.1| 25  | 26.4| 34.9 |\n| TextVQA  | 42.4| 45.4| 47.1| 51   |\n\n**Leveraging MTV for Single-Image LMMs.** We thank the author for the suggestion to leverage MTV for single-image LMMs. This question is especially interesting as single-image pretraining causes the underlying model to struggle with handling multi-image inputs [1]. \n\nWe apply MTV to LLaVA-1.5-13B on VizWiz. The model has a zero-shot accuracy of 54.5% and an accuracy of *62.1% with MTV*. This result is encouraging as it indicates MTV’s ability to enable multi-image, few-shot, and many-shot multimodal reasoning for models pretrained on exclusively single-image data.\n\nWe hope the above points have clarified and addressed your concerns. We would be happy to provide any further clarifications as requested. \n\nReferences:\n\n[1] Doveh, S., Perek, S., Mirza, M.J., Alfassy, A., Arbelle, A., Ullman, S., & Karlinsky, L. (2024). Towards Multimodal In-Context Learning for Vision & Language Models. ArXiv, abs/2403.12736.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received 4 reviews, but the review that gave a score of 1 due to desk reject reasons has been ignored since the case was already handled by the PCs. The paper addresses an open and highly challenging problem in the MLLM space, namely to maximize its potential for many-shot in-context learning. The reviewers find that the paper is well written, with a good motivation, and good results. One reviewer had concerns about how the paper is unique in the large landscape of MLLMs, as well as open questions about design choices. These questions were positively answered in the rebuttal, resulting in an accept consensus amongst the reviewers. The AC agrees with the general applicability and broad use of the method and recommends acceptance.
Decision: Accept (poster)