LIVE: Learnable In-Context Vector for Visual Question Answering
Authors: Yingzhe Peng
chenduo hao
Xinting Hu
Jiawei Peng
Xin Geng
Xu Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: In-Context Learning
MultiModal
Primary area: natural_language_processing

Abstract: As language models continue to scale, Large Language Models (LLMs) have exhibited emerging capabilities in In-Context Learning (ICL), enabling them to solve language tasks by prefixing a few in-context demonstrations (ICDs) as context. Inspired by these advancements, researchers have extended these techniques to develop Large Multimodal Models (LMMs) with ICL capabilities. However, applying ICL usually faces two major challenges: 1) using more ICDs will largely increase the inference time and 2) the performance is sensitive to the selection of ICDs. These challenges are further exacerbated in LMMs due to the integration of multiple data types and the combinational complexity of multimodal ICDs. Recently, to address these challenges, some NLP studies introduce non-learnable In-Context Vectors (ICVs) which extract useful task information from ICDs into a single vector and then insert it into the LLM to help solve the corresponding task. However, although useful in simple NLP tasks, these non-learnable methods fail to handle complex multimodal tasks like Visual Question Answering (VQA). In this study, we propose \underline{\textbf{L}}earnable \underline{\textbf{I}}n-Context \underline{\textbf{Ve}}ctor (LIVE) to distill essential task information from demonstrations, improving ICL performance in LMMs. Experiments show that LIVE can significantly reduce computational costs while enhancing accuracy in VQA tasks compared to traditional ICL and other non-learnable ICV methods.

LIVE: Learnable In-Context Vector for Visual Question Answering
Yingzhe Peng 2, Chenduo Hao Xinting Hu Jiawei Peng' ,2, Xin Gengl 1.2, Xu Yang 1.2+
Southeast University
2 Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China fyingzhe peng, 213201447, bengjiawei, xgeng, xuyang_palmj Oseu. edu. cn Nanyang Technological University xinting001@e Ce ntu. edu.
Abstract
As language models continue to scale, Large Language Models (LLMs) have exhibited emerging capabilities in In-Context Learning (ICL), enabling them to solve language tasks by prefixing few in-context demonstrations (ICDs) as context. Inspired by these advancements, researchers have extended these tech- niques develop Large Multimodal Models (LMMs) with ICL capabilities. How- ever, applying ICL usually faces two major challenges: 1) using more ICDs will largely increase the inference time and 2) the performance is sensitive the se- lection of ICDs. These challenges are further exacerbated in LMMs due to the integration of multiple data types and the combinational complexity of multi- modal ICDs. Recently, to address these challenges, some NLP studies introduce non-leamable In-Context Vectors (ICVs) which extract useful task information from ICDs into single vector and then insert into the LLM help solve the corresponding task. However, although useful in simple NLP tasks, these non-learnable methods fail to handle complex multimodal tasks like Visual Ques- tion Answering (VQA). In this study, we propose Learnable In-Context Vector (LIVE) to distill essential task information from demonstrations, improving ICL performance in LMMs. Experiments show that LIVE can significantly reduce computational costs while enhancing accuracy in VQA tasks compared to tradi- tional ICI. and other non-learmable ICV methods. The code is available at phttps: //github.comt hra.shhh..cco/ooooddddeeeeeee//e -oee
Context- Vector
Introduction
As language models continue to scale up. Large Language Models (LLMs) 1+3 have demonstrated emerging capabilities in In-Context Learning (ICL) 4: these models solve language tasks when provided with a few similar examples, termed in-context demonstrations (ICDs), as context. Unlike traditional fask-specific fine-tuning, ICL, efficient method to adapt LLM downstream task [5] 6), achieves comparable performance without necessitating updates to millions or trillions of model parameters 77. By prefixing just a handful of data samples to the query input, ICL configures model's ss behavior to produce the corresponding output, thus facilitating rapid adaptation across a wide range of downstream tasks. Inspired by these advancements in the language domain, researchers have extended these techniques to develop Large Multimodal Models (LMMs) with ICL capabilities 8-10
Correspondnng author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Trainable
Interence Time 8.25X
Interence Time
LMM
1X
ICD
ICD Faz
Quarry 2
Number FLOPs
LIVE
+
Querr 2
FLOPs
Embeddling
24.97X
Embedeling
1X
Zaro Learning
Queetoor: How mare
Quention: Aee engii
Queetoon What rumbor txtily
Guentee: Whot numbor soily tean trowe?
Murnber connciaas
(6) Conventional In-Contect Learning (IC1)
(b) Learnable In-context Vector (UVE)
Figure 1: (a) Conventional ICL is more sensitive to the ICD selection and requires more inference time. (b) LIVE is more robust and reduces inference time by inputting shift vector.
Employing ICL in LLMs meets two challenges: Firstly, although increasing the number of ICDs typically enhances performance 77. this practice conflicts with computational efficiency constraints. As ICDs are prefixed to the query, the increase in input tokens severely impacts the Transformer's inference speed, causing marked slowdown in computational performance. Secondly, the effec- tiveness of ICL is vulnerable to the selection of demonstrations a limited number are used. It makes the process of choosing demonstrations critical for optimal performance. However, developing selection strategies and measuring the effectiveness of ICDs remain open questions [14] For LMMs, the challenges above are further exacerbated: 1) The computational complexity is significantly increased due to the integration of multiple data types as ICDs (as shown in Figure a)). 2) The task of selecting effective multi-modal ICDs becomes more complex and nuanced, as each modality contributes uniquely understanding the context [I8 19] further complicating the assessment of their combined effect in demonstration selection.
[3333, particularly when only
To alleviate these two challenges, recent research on LLMs has introduced In-Context Vector (ICV) to extract the most useful task information from ICDs, and then use it to directly influence the processing in LLMs 22). For example, (20) proposes that by using multiple demonstrations and a dummy query as inputs, the representation of the last token from middle layer of the model can be extracted the vector. This vector is then used replace the representation of the corresponding token in the same layer during inference, which can achieve performance comparable to ICL. Such in-context vector alleviates the requirement of multiple ICDs during inference, as well as effectively bypasses the complexity of the individual selection of demonstrations by representing the most effective components across many demonstrations.
However, these studies apply non-learnable strategies to extract ICVs, although useful in some simple NLP tasks, lose the efficacy in complex modal tasks like Visual Question Answering (VQA). Our preliminary experiments have demonstrated that directly applying these non-learnable ICVs yields unsatisfactory results. The principal reason is the intrinsic complexity of VQA compared to the language tasks addressed by these non-learnable s For example, the previous methods focus on simple NLP task, such as Antonym 23 and Country-
[21), whose distribution patterns can be easily identified by LLMs. In contrast, as unified vision-language task, VQA encompasses a diverse array of question types, where each one corresponds to a different vision-understame understanding task. For instance, questions like "What is this?" or "How many are there?" require classification and counting abilities, respectively. These varied requirements imply that the task information, which non-learnable methods attempt to abstract, cannot be effectively captured by single ICV.
In this study, to make ICVs abstract more useful VQA task information, we try distill the task information implied in demonstrations into single Learnable In-Context Vector (LIVE). Our method is motivated by the observation 200 that ICL can be treated as a process of "shifting" the direction of the latent states of query towards the target, i.e., adding this latent state with shift vector. Then we hope to learn suitable ICVs to replace the ICDs during inference to shift the direction. To achieve this, we train LIVE by minimizing the output distributions of a LMM got by only using LIVE and by inputting a few demonstrations. During training, we use different 32-shot randomly sampled demonstrations for different queries to distill task knowledge. Then LIVE is encouraged to capture the most essential task information from these different combinations by removing the individual characteristics of demonstrations, Moreover, 24 finds that during ICL, different layers of LLM

---

have diverse roles in addressing demonstrations. Then in our method, each layer is assigned with unique ICV to capture more fine-grained task information.
Our LIVE inherits the efficiency of previous non-learnable ICVs, i.e., during inference, under the same performance conditions, LIVE only needs 1/24.97 FLOPs number of 32-shot ICI. Additionally, in VQAV2/OKVQA, LIVE improves accuracy by 2.36/ 1.6 compared to 32-shot ICL. We also compare LIVE to LoRA [25] that when comparable trainable parameters are used, LIVE requires much fewer training samples than LoRA (500 vs. 8000) to achieve satisfactory performance. Besides, we design lots of analytical experiments validate whether LIVE can better shift the hidden states of queries to the target direction and analyze why previous non-learnable methods fail to solve VQA.
Related Work
In-Context Vector:
Recently, more and more researchers in NLP have begun focus on using an In-Context Vector (ICV) to modify the activation values during the forward propagation of LLM to simulate the effect of ICDs in ICL. [20) propose the "Task Vector", which extracts the representation
of the middle layer from the LLM during ICL inference as the ICV, and replaces the representation
of the same layer during zero-shot inference. Meanwhile, 2T introduced the "Function Vector", which uses attention weight analysis to take the mean of the activation values of the attention heads that most significantly affect the final result in ICL inference as the final ICV. This vector is then directly added to the representation of the middle layer during zero-shot inference to form new representation. On the other hand, (22 propose "PCA In-Context Vector". They believe that the ICV should be closer to the LLM's representation of the task output and farther from the task input representation. Thus, they extract the input and output representations of several demonstrations and using PCA to find the overall principal direction as the ICV. These efforts mainly focus using non-learnable methods to find the specific ICV for NLP tasks, achieving effects similar to ICL in various tasks. However, these methods only are tested on some simple tasks in NLP. When LMMs face with more complex tasks, the performance of these methods remains uncertain.
ICL in LLM: Prompt engineering allows LLMs to tackle specific tasks without requiring fine- tuning 26 A specific form of this approach, ICL, further improves these capabilities by creating prompts that include several demonstrations. ICL has already demonstrated superior performance and good generalization on many tasks [34
36), and can be easily adapted to downstream tasks. However, the use of ICL faces several issues: first, ICL is very sensitive to the selection and arrangement order of demonstrations 4 [3 39); poor demonstrations can severely impact ICL performance. Second, too many demonstrations can significantly slow down the inference speed of LLMs (4). While ICVs can effectively address these two issues, as it can use only queries as input to the model while preserving ICL performance, without the need for demonstrations as input.
ICL in LMM: As the performance of LLMs continues to improve, an increasing number of re- searchers begin to adapt LLMs to the multimodal domain 46. Relying on the powerful inference capabilities of L.Mss, some LMMs have started to exhibit ICL capabilities, such as Flamingo [8] and IDEFICS (9. Moreover, these models have significantly enhanced their ICL capabiliiiess by concate- nating multiple samples as contextual information during the training process. Currently, researchers mainly focus on how to configure demonstrations to address the sensitivity of ICL performance in LMMs. [8 19 have respectively adopted heuristic retrieval methods for selecting demonstrations in Image Captioning and VQA. However, no researchers have yet extracted ICV from LMMs and evalutaed it. Therefore, the effectiveness of ICV in LMMs still needs further exploration. Considering that the IDEFICS model shares the same model structure as Flamingo and possesses stronger ICL capabilitics, we primarily focus on valid our method on the IDEFICS model.
3 LIVE: Learnable In-Context Vector
Here we show how to derive the formulation of the shift vector from Self-Attention (SA) mechanism and then introduce how to design LIVE based on this formulation. Generally, to implement In-Context Learning (ICL) using language or multimodal model (LLM/LMMM) M, the input has the following form: X = (Xp, #). where Xn = (.., represents the concatenation of l In-Context Demonstrations (ICDs), and z denotes the query input, shown in Figure 2 Given X as Key and
3

---

Frozen
Trainable
Ground Truth
Output Pik aC)
KL Loss 4
Output gl
Factor
+
4
UMM M
Zero Shot Learning
Shot Contert
*
LMM M
Learning
Query
ICD X1
ICD Xx
Query 2
Cocotten Demo Detratttaooon
62
12
Query
ICD x
x%
Query
Lawr
81
F1
Query
Loger
Ouery
UVE
ICD X
ICD X
Query 2
Lwer
(e) Learnable
Learning
Figure 2: The LIVE training pipeline: (a) The distribution P , M) of LMMs output when using LIVE. (b) Adding LIVE into the representations of the query to simulate the shift effect brought by demonstrations. (c) The distribution (( M) of LMMs output when using demonstrations.
Value, for each token i; of at, applying Self-Attention (SA) once yields:
X
X
X
SA X) SA
softmax I 2,0 T])
(1)
where vector X
f. E 1x and denotes the sequence length of the entire input [X at.
D
Expanding the softmax function, we obtain:
exp (xp(ú;Xp)) X p)1
exp(;;XD))
exp(2;;0
exp(#;;0
softmax( ( 2æ
Z1
Z2
21 + Zz
21 Zz
Zi + Zz
expansion of 2,X
expansion of di
exp(2;XD))
xxp(2,
Z1 + Zz
Z1 + Zz
(2)
where le and lg represent the lengths of the XD and å, respectively. Zi and Zg are the sum of exponential scores between the query token à;; with each token in X and ø: Zı E,, exp(2,X]] and Zo = Ei, exp($tā This leads to the following formulation of SA:
SA X, X, exp(f; 1)DD
exp($ )
71 7/2
71 + 72
21 exp(f;
1)Xn
22
expl exp(
+
Zi Zz
Z1
21 + Z2
Zz
(3)
Z1
Zg
softmax((;; xI X
softmax($;0)
Z1 Zz2
Z1 Zz
JISA(;;; Xp.Xp) (1-1 SA (fis, अ., x),,
where μ Zz/(Z1 + Zz). Let h(2) SA(E,, 2,2). The output of SA(a:) can then be expressed as: (1-))(a))
SA X, X,
(4)
As noted in Equation 4 we observe that h(t)) ) is the representation obtained with self-attention over the query & without appending any ICD; h(Xp) functions similarly to a "shift" vector, altering the attention representation h())) by incorporating contextual information from the ICDs Xp. The coefficient y quantifies the degree of influence Xp has over the original query representation. For visual demonstration of how ICDs shift the representation space, see F Consequently, once learning general shift direction replace the effect of h (XX),, to simulate the ICL process of LMMs without actual demonstrations.
employ this shift direction
We propose novel method that involves Learnable In-Context Vector (LIVE) to simulate the ICL process without actual demonstrations. This approach aims to abstract general task information from demonstrations, enabling it to shift the model's representation toward the direction influenced by the ICDs. Figure 2 shows the training pipeline LIVE. The LIVE training dataset, denoted as D (d, dN ). is subset of the VQA dataset training split, created by randomly selecting N question-answe pairs from it. We use each training sample d, to simulate the query sample å in ICI. and randomly select & demonstrations from D }d,} for it. Additionally, [24] shows that during

---

ICL, each layer of an LLM performs distinct role. Motivated by this, we assume that for LMM, each layer also requires a specific shift direction. We assign a learnable vector e, and a weight factor or for each layer / to learn the unique shift effect. Our final LIVE comprises of the vector set V and the corresponding weight factor set a as:
V
{1) 22 vr.)
E IR 1xd
(5)
{ai a2..,, aL), as E RRxi
where L is the number of layers. To train V and a, we align the distribution of the model's outputs for the query when shifted by demonstrations, [[; with that shifted by our LIVE, P(o a: M). This alignment is achieved by minimizing the Kullback-Leibler (KL) divergence:
C = (L
|| V ax;
(6)
To obtain the distribution P(á M)) for each query & we randomly select As demonstrations to form X These are concatenated with the query to form the inputs for the model. The model's output for the query is then considered as the shifted distribution PM)
To obtain the output of & by using LIVE, we follow 120 22), we the vector vs to shift the each layer's s output representation Mı(â;) and get: M = Mā; + ayv, which is shown in Figure 2 a). After applying LIVE shift the representations at each layer, we obtain the output distribution P( ac; M). Notably, during training, XD for each query & includes randomly sampled 32-shot demonstrations. This strategy encourages LIVE extract the most useful common information from various demonstration combinations and prevents it from being influenced by the individual characteristics of certain demonstrations.
In addition, to facilitate the LIVE acquiring more task-specific information, we also optimize the P( V, a; M) with the ground truth by Lgt Thus, the overall loss L is defined as:
L = Agr + La
where
Lgt - Σ log P (6: V M)
where À is the hyper-paramettr to control the importance of ground truth loss.
4
Experiments
4.1 Setting and implementation details
Model and Dataset: We evaluate approach using the IDEFICS-9B model 9 across two datasets: VQAv2 (47) and OKVQA [48). VQAv2 emphasizes open-ended VQA tasks, encompassing 4, 570 question-answer pairs in its training split, supplemented by additional 2, 143, 540 pairs in the validation split. OKVQA is a large-scale dataset designed for models that require external knowledge to answer questions. It consists of 14, 055 question-answer pairs, with 9, 009 allocated for training and 5, 046 for validation. For both VQAv2 and OKVQA datasets, We train our LIVE on 8, 000 pairs from each training set. Due computational resource limitations, we randomly sample 10, 000 question-answer pairs from the VQAv2 validation split for evaluation [18], For OKVQA, we utilize the entire validation split.
During training, we assign 32-shot demonstrations for each query, enabling LIVE acquire better directions of shifting vectors for VQA tasks. The v, is initialized using normal distribution with a mean of and a standard deviation of 0.01, and all or are initialized to 0.1. More detailed training parameters be found in Appendix.
LIVE Setting:
4.2 Results
4.2.1 Compared Methods
We primarily compare the following methods:
Zero-Shot:
The model uses only the query as input.
k-Shot ICL:The model uses &o demonstrations, randomly selected from the VQA dataset training split, along with the query input.
5

---

Table I: Accuracy (%) with Different ICVs Methods and Finetuning Methods, where numbers in parentheses indicate multiples of LIVE trainable parameters.
Zero-Shot
32-shot ICL
TV
FV
PCA-ICV
LoRA
LIVE (Ours)
VQAv2
29.25
56.18 48.48
43.68
30.21
34.75
49.02
58.54
OKVQA
30.54
32.68
31.02
30.59
34.21
50.08
Total Trainable Parameters
155, 136(133()( x8.8)
131, 104(x1.0)
Non-Learnable ICV Methods: We extend three established non-learnable ICV methods from language models to our multimodal settings: (1) Task Vector (TV) [20] uses & demonstrations and a dummy query to extract the representation of the last token from a middle layer of the model as the ICV. During inference, this vector replaces the representation of the last token in the same layer. We conduct evaluations on TV by implementing across various layers and select the layer where it achieves the highest performance improvement. (2) Function Vector (FV) [21] employs small subset of the validation data to derive the mean output from critical attention heads, forming the ICV. During inference, this vector is added to the representations of the last token within specific layer. We conduct evaluations on FV by implementing it across various layers and select the layer where it achieves the highest performance improvement. (3) PCA In-Context Vector (PCA-ICV) 22 computes the ICV by applying PCA to the difference between the question and question-answer representations from & demonstrations. During inference, these vectors are added to the representations of all tokens at each layer,
LoRA [25]: This method finetunes the LMMs with the same number of samples of training LIVE. We add the LoRA module the token classification head of the last layer. this way, the number of trainable parameters is comparable to that of LIVE.
4.2.2 Performance and Inference Efficiency on VQA.
We present performance comparisons with various methods in Table Certain existing methods show only marginal improvements over Zero-Shot baselines, e.g., FV improves by 0.96/0.45 on VQAv2/OK and PCA-ICV improves by 0.04 on OKVQA. Besides, we observe that all the previous non-learnable ICV methods do not reach the performance of the standard 32-shot ICL, e,g., the best non-learnable method, TV, is still 12.5/15.8 lower than 32-shot ICL VQAv2/OKVQA. In contrast, our LIVE achieves an accuracy improvement of 2.36 on VQAv2 and 1.6 on OKVQA over 32-shot ICL. These results highlight the inefficacy of non-learnable methods in capturing essential task task-specific information for VQA, whereas LIVE, by leveraging diverse 32-shot ICL demonstrations for each query during training, manages to abstract useful task information effectively. We further show that our LIVE outperforms LoRA with less trainable parameters, suggesting LIVE can abstract task information more efficiently.
Figure 3 displays the efficiency of LIVE during inference compared to other methods. We average the FLOPs and actual inference time consumption per forward pass over 1000 randomly sampled P queries We observe that LIVE only needs 1/24.97 FLOPs and 1/8.25 inference time of 32-shot ICL per forward pass. Additionally, LIVE maintains almost the same inference speed as Zero-Sho Shot. These comparisons validate the efficiency of LIVE during inference.
4.3 Ablation Studies
We use ablation studies to explore the effects of diverse settings, including different training losses, the shot number of demonstrations & used during training, and the number of training data N.
Training Loss: Table compares the results of using different losses: only Lgg in Eq. 7 or La in Eq. (7). We find that only using Zgt (same as standard fine-tuning) significantly dam- ages the performance, e.g., Egt achieves 16.9/6. 12 lower accuracy on VQAv2/OKVQQ com- pared to using the combined loss L; yet using only La results in a smaller performance drop 3.78/3.14 lower VQAv2/OKVOL VQA than when using L. This suggests that with a limited number of trainable parameters, LIVE trained with La is more robust and capable of capturing essential
For detailed hardware information, refer to Appendix.

---

FLOPs Comparisons
Runtime Comparisons
25
24-
8.25X
11.00%
6.31X
LAXX
6.99X
18
0,90X
Zam-SSue
LIVE
4-hhet ICI,
R-shee ICI. I6-boot Ie- IC1.22hh ICI.
Zare-Shit
LIVE
4-bbt [CL. R-hhel ICI 16-hol I.L. ICL
Figure 3: The total number of FLOPs and real inference time consumption of ICL, Zero-Shot, LIVE for 1000 query samples.
task information than Cgt. It underscores that the LIVE cannot solely rely fine-tuning with LMMs on specific datasets, but should effectively leverage abstracted insights from demonstrations.
Number of Demonstrations k: We compare the performance of LIVE trained with & demonstra- tions per query and the corresponding k-shot ICL in Table 3 The result shows that increase the number of demonstrations enhances performance of ICI. and LIVE, indicating that more demonstrations can provide each query with a richer context to help train LIVE. Additionally, LIVE consistently surpasses the performance of k-shot ICL across different training sizes, showcasing the robustness of our LIVE in utilizing demonstrations. Notably, when the number of demonstrations limited, the performance gap between LIVE and ICL becomes more pronounced. This is because ICL is highly sensitive to the choice of demonstrations; with insufficient demonstrations, the model may shift the query representations in an incorrect direction. In contrast, LIVE continuously by learning the main shift direction of the query representations from the demonstrations, reduces the negative impact of poor demonstrations on the query during training and is more robust that can extract essential task information.
Table 2:
Accuracy (%) of LIVE with
Different Training Loss on VQA.
Ld
Egt
VQAv2 OKVQA
54.76
41.64
58.54
46.94
43.96
50.08
Size of Training Set: Figure illustrates how varying the number of training samples impacts the performance of LIVE and LoRA. On the VQAv2 dataset, both methods show improved performance with increasing data sizes. Notably, LIVE performs exceptionally well across both low and high training sizes. It achieves perfor- mance close to that of 1-shot ICL with just 700 training samples and surpasses 32-shot ICI with 4,000 training samples. In contrast, LoRA does exceed the performance of 1-shot ICL, even when expanded to 8,000 samples. For OKVQA, the performance of LoRA with small data sizes is even worse than Zero-Shot. This is because OK OKVQA requires external knowledge to answer questions, while learning external knowledge from small amount of data can disrupt the inherent knowledge of the pre-trained model, leading to significant drop in performance. Conversely, LIVE excels by focusing on learning shift direction, thus preserving the model's inherent reasoning abilities. With just 500 samples, LIVE outperforms 1-shot ICL, and with 4,000 samples, it nearly matches the performance of 32-shot ICL. These observations underscore LIVE's superior efficiency over LoRA capturing and utilizing complex reasoning capabilities with much fewer training samples.
Table 3: Accuracy (%) of Different Number of Demonstrations on VQA.
Number of Demonstrations
Task
Method
4
8
16
32
LIVE
56.84
57.60
58.25
58.27
58.54
VQAv2
ICL
51.39
53.72
54.24
55.70
56.18
LIVE
47.51
47.68
49.40
49.71
50.08
OKVQA
ICL
40.75
46.11
46.79
47.70
48.48
7

---

VQAV Accuracy
Accurasy)
2 RCT
Zan-lel
POANO
Calts
Samples
Samples
Figure Accuracy (%) of LIVE and LoRA with different size of training set.
VQAv2
OKVQA
TV
LAVE
Figure 5: T-SNE visualization of first answer token representations over 200 queries.
4.4
Analysis
4.4.1
The Shifting Effect in Latent Space
To better demonstrate the shifting effect of LIVE on query samples, randomly select 200 query samples and conduct different methods inference LMMs. We extract represen- tation vector of the first answer token for T-SNE dimensionality reduction, shown in Figure 5 Additionally, to quantitatively evaluate the effect shift directions of different ICV methods, calculate the following metrics. Given query 2, resentation of the first answer token obtained by 32-shot ICL, Zero-Shot, specific ICV meth- ods, respectively. Then caleulate the standard shift direction Syt Tiel T=s and the shift direction of specific ICV as s" = r Tis- Finally, we define the shift direction simi- larity the cosine similarity between s* and 8yt, indicating how closely the shift direction of the ICV method aligns with the standard shift direction. The results are presented in Table From Figure 5] we find that 32-shot ICI. exhibits a significant shift compared to Zero- Shot, visualizing the shift effects given in Eq. 4 Considering both Table 4 and we find that the shift direction similarity has positive correlation to the accuracy: if method has large direction similarity, also has better performance. For example, among non-learnable methods, TV has higher shift direction similarity than other ones.
Tial T=s. r' to denote the rep-
Table 4: The shift direction similarities different ICV methods.
TV
FV
PCA-ICV
LIVE
VQAv2 OKVQA
0.486
-0.106
0.027
0.742
0.326
0.218
-0.190
0.829
then it has better accuracy in Table T Furthermore, for LIVE which has the best accuracy in Table its shift direction similarity is also the highest, which is 0.742/0.82. on VQAv2/OKVQA, validating
that LIVE can produce shifts in query samples similar to 32-shot ICL, visualized in Figure

---

Table 5: Direct decoding of the different ICV methods.
Methods
Decoding Top-10 Tokens of different methods in order of decreasing probability
TV
'No'. "Yes', 'no". 'It', 'F. "No', "No'. 'A'. in., 'G, "for', "and", "..." "to", 'on', "none". "there", 'no", "the", 'not", 'None", "dep'. '_ye', ''unknown', 'yes' "Question', SShort, "?" "nn", "QUKST", "questions', 'S7, "answer', 'Short, questions"
"yes". "Not"
FV
PCA-ICV
LIVE
Table 6: The frequency of yes/no hallucinations and meaningless responses.
Zero-Shot
TV
FV
PCA-ICV
LIVE
yes/no Hallucination
111
7
4
Mcaningless Answer in yes/no Meaningless Answer in number
0
0
0
2
0
2
522
Mcaningless Answer in other
257
0
247
2072
(1) Question: Which side the trees have snow them?
(2)
Question: stove gas or electric?
TV:
PCA-ICY: innnnsswer
PCA-ICV:
LIVE:
Yes
X
X
right
yes
x
'n'nAnwwerii
X
gas
(3) Question: How many men are standing next ench other?
(4) Question: What color is the umbrella?
TV:
PCA-ICV:
PCA-ICV:
X
2
2
Red
'n'nAnswer:
Red
Figure 6: Visualizations of the cases where non- non-laarnable methods appear yes/no hallucinations and meaningless responses.
Such positive correlation validates the effectiveness of our motivation that single LIVE can indeed simulate the ICL capability of LMMs by shifting the direction of the query representation.
4.4.2 Why Non-Learnable Methods are Poor on VQA?
Decoding ICV To Tokens:
We follow previous studies 41 to analyze the parameters of
Transformers by directly decoding them into vocabulary tokens. Specifically, given vector a E Rixd it can be projected using the unembedding matrix E E dx LMMs obtain the corresponding token probability distribution p. where N is the vocabulary size:
exp(v E)
p softmax(v v E)
(8)
Ej exp(v E);
We calculate the p of the vectors got from different methods in VQAv2 and select the top-10 tokens with the highest probabilities in p shown in Table 5 We can see that the tokens got from FV are not highly relevant to the VQA task, which proves that FV does not capture the task information of VQAv2. On the other hand, the frequency of "yes" and "no" tokens is relatively high in the decoding results of PCA-ICV and TV, suggesting that they prefer to capture the simple patterns from demonstrations. yeg./o,/no, but struggle to grasp the overall task information of complex VQA. In contrast, the tokens of LIVE decoding are not biased to specific answers like yes/no, suggesting abstracts more summary task knowledge of VQA.
Hallucinations and Invalid Responses.
VQA contains various answer types and for convenience, VQAv2 divides them into three categories: "yes/no", "number". and "other". After delving deeper into the answer details, interestingly, we find that TV frequently answers "yes or no" to number/other questions as shown in (1)(2)33, We term this phenomenon the yes/no hallucination and count the frequency of the yes/no hallucination over all test data samples for different methods in Table 6 We can find that TV appears 111 times of yes/no hallucination, being consistent with the observations in Table suggesting TV is biased to yes/no type question. We also observe that non-learnable methods tend to respond meaningless text (e.g." "\n"") when responding to "number/other" questions as shown Figure (1)(2(4)) Table 6 shows the number of meaningless answers. We find that PCA-ICV and TV have more chance to return

---

meaningless answers for "other" questions, suggesting these methods do not capture the overall task information of VQA and are not able answer some less frequently appeared questions. However, for LIVE, it has less yes/no hallucination and meaningless responses, validating that LIVE captures more robust task information of VQA.
Conclusion
To address the two major drawbacks of ICL in LMM-long computation time and sensitivity to demonstration selection-we we try to apply non-lcarnable ICV methods from NLP to solve VQA. However, due the complexity of VQA and the significant biases often inherent in non-leamnble methods, the performance is unsatisfactory. Then we propose the Learnable ICV (LIVE) to overcome this drawback. By learning the general shift direction from large amount of ICL data, LIVE successfully replaces the role of demonstrations in ICL. Experiments validate that LIVE outperforms traditional ICL methods and other non-learnable ICV methods two VQA datasets. Experiments also show that LIVE, compared to LoRA, maintains excellent performance with minimal data, suggesting LIVE is new research direction for LMMs solve multimodal tasks. the future, will explore the application of LIVE on more multimodal tasks by various LMMs.
Acknowledgement
This work supported by National Science Foundation of China (62206048), Natural Science Foundation of Jiangsu Province (BK20220819) 19), 19), Young Elite Scientists Sponsorship Program of Jiangsu Association for Science and Technology Tj-2022-027, Fundamental Research Funds for the Central Universitdex2222244333333 and Big Data Computing Center of Southeast University
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad. Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ArXiv preprint, abs/2303.08874,, 2023.
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302.13971, 2023.
[3] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307 (00000337000288 2023.
[4] Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop Knowledge Extraction and Integration for Deep Learning Architectures, pages 100-114, Dublin, Ireland and Online, 2022. Association for Computational Linguistics.
[5] Shuxia Lin, Miaosen Zhang, Ruiming Chen, Xu Yang, Qiufeng Wang, and Xin Geng, Linearly decomposing and recomposing vision transformers for diverse- scale models. The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024.
[6] Qiufeng Wang, Xu Yang, Fu Feng, Jing Wang, and Xin Geng. Cluster-learngene: Inheriting adaptive clusters for vision transformers. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[7] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. ArXiv preprint, abs/2301.00234, 2023.
[8] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 15:23716-23733 2022.
[9] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.
[10] Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Perey Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 1021, NewrIPS 2021, December 6- 14, 2021, virtual, pages 200-212, 2021.
[11) Feng Nie, Meixi Chen, Zhirui Zhang, and Xu Cheng. Improving few-shot performance of language models via nearest neighbor calibration. ArXiv preprint, abs/2212.022 2022.
[12] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1086-8098 Dublin, Ireland, 2022. Association for Computational Linguistics.
[13] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference Natural Languuge Processing (Volume Long Papers), pages 3816-3830, Online, 2021. Association for Computational Linguistics.

---

[14] Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. Self-adaptive in- in-context learning: An information compression perspective for in-context example selection and ordering. ArXiv preprint, Abs/2212.10375, 2022.
[15] Tai Nguyen and Eric Wong. In-context example selection with influences. ArXiv preprint. abs/2302. 1042, 2023.
[16] Xiaonan Li and Xipeng Qiu. Finding supporting examples for in-context learning. arXiv e-prints, pages arXiv-2302, 2023.
[17] Yiming Zhang, Shi Feng, and Chenhao Tan. Active example selection for in-context learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9134-9148, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics.
[18) Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, and Xu Yang. How configure good in-context sequence for visual question answering. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 26710-26720, 2024.
[19] Xu Yang, Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng. Exploring diverse in-context configurations for image captioning. Advances in Neural Information Processing Systems, 36, 2024.
[20] Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. ArXiv preprint, abs/2310. 15916, 2023.
[21] Eric Todd, Millicent L Li, Arna Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. ArXiv preprint, abs/2310.15215. 2023.
[22] Sheng Liu, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. ArXiv preprint, 0tbs/2311.066666 2023.
[23] Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. Distinguishing antonyms and synonyms pattern-based neural network. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 75, Valeniia, Spain, 2017. Association for Computational Linguistics.
[24] Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng. Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. ArXiv preprint, abs/2305. 14160, 2023.
[25] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[26] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan. Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Mersert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse. Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tice Lin, editors, Advances in Neural Information Processing Systems 33; Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[27] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 188)99
[28] Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, et al. Ernie 3.0: Large-seale knowledge enhanced pre-training for language understanding and generation. ArXiv preprint, abs/2107.02 0221110770.17 2021.
12

---

[29] Jincen Jiang. Qianyu Zhou, Yuhang Li, Xuequan Lu, Meili Wang, Lizhuang Ma, Jian Chang, and Jian Jun Zhang. Di-pic: Domain generalized point-in-context learning for point cloud understanding. In European Conference on Computer Vision (ECCV). Springer, 2024.
[30] Tingbing Yan, Wenzheng Zeng, Yang Xiao, Xingyu Tong, Bo Tan, Zhiwen Fang, Zhiguo Cao, and Joey Tianyi Zhou. Crossglg: Lim guides one-shot skeleton-based 3d action recognition in cross-level manner. arXiv preprint 2rXiv:2403.1006 2024.
[31] Congzhi Zhang, Linhai Zhang, and Deyu Zhou. Causal walk: Debiasing multi-hop fact verification with front-door adjustment. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 19541, 2024.
[32] Congzhi Zhang, Linhai Zhang, Jialong Wu, Deyu Zhou, and Yulan He. Causal prompting: Debiasing large language model prompting based on front-door adjustment, 2024.
[33] Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. ArXiv preprint. abs/2404.11912, 2024.
[34] Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 12284 12314, Toronto, Canada, July 2023. Association for Computational Linguisties.
[35] Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. Structured prompting: Scaling in-context learning 1,000 examples. arXiv preprint arXiv:22 2022.
[36] Yixiao Yuan, Yawen Huang, and Yi Zhou. Rethinking unified generative adversarial model for mri modality completion. In Anirban Mukhopadhyay, Ilkay Oksuz, Sandy Engelhardt, Dajiang Zhu, and Yixuan Yuan, editors, Deep Generative Models, pages 143-153, Cham, 2024. Springer Nature Switzerland.
[37] Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. Advances in Neurul Information Processing Systems, 36, 2024.
[38] Chengwei Qin, Aston Zhang, Anirudh Dagar, and Wenming Ye. In-context learning with iterative demonstration selection. arXiv preprint
2023.
[39] Chengwei Qin, Wenhan Xia, Fangkai Jiao, and Shafiq Joty. Improving in-context learning via bidirectional alignment. arXiv preprint arXiv:2312. /7055, 2023.
[40] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 17453, 2023.
[41] Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Llava- yh: Efficient multi-modal assistant with small language model. ArXiv preprint, has/2401.023300 2024.
[42] Haotian Liu, Chumyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuming, ArXiv preprint, abs/2310.03744, 2023.
[43] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. ArXiv preprint, abs/2308.013 013900, 2023.
[44] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision- language understanding with advanced large language models. ArXiv preprint, abs/2304.10592. 2023.
[45] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. ArXiv preprint, abs/2306.05425, 2023.
13

---

[46] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. ArXiv preprint, abs/2306. 13549, 2023.
[47] Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI. USA, July 21-26, 2017, pages 6325 6334. IEEE Computer Society, 2017.
[48] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 3195 1994-3204. Computer Vision Foundation IEEE, 2019.
[49] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer beed-forward layers are key-value memories. arXiv preprint arXiv:2012.149111 2020.
[50] Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint vrXiv:2303.08122 2023.
[51] Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. arXiv preprint arXiv:2209.0m 02535, 2022.
[52] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[53] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1955) 2014.
[54] Xinlei Chen, Hao Fang, Tsung-Y Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. ArXiv preprint, abs/1554.00025, 2015.
[55] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr- Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947-952. IEEE, 2019.
[56] Yongshuo Zong, Ondrej Bohdal, and Timothy Hospedales. V1-icl bench: The devil in the details of benchmarking multimodal in-context learning. arXiv preprint arXiv:2403.13164 2024.
[57] Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? ArXiv preprint, abs/2405.022 0.224. 2024.
[58] Shuo Chen, Zhen Han, Bailan He, Mark Buckley, Philip Torr, Volker Tresp, and Jindong Gu. Understanding and improving in-context learning on vision- languaee models. ArXiv preprint, abs/2311.18021, 2023.
14

---

A Implementation Details.
A.1 LIVE Hyperparameters
Table presents the hyper-parameters utilized for training the LIVE. The optimizer denotes the optimization algorithm employed during model training. For V and at we use different learning rate to optimize. The À represents the weight assigned to Lgt in Eq during training. Precision refers the float precision type used for model weights and gradient descent throughout the ICV training process. Weight Decay signifies the rate of weight decay applied during training, the warm up value set to 0.01. while accumulate batches denotes the batch size for gradient accumulation during the training phase.
Table VQAv2 and OKVQA LIVE Training Parameters
Hyperparameter
VQAv2
OKVQA
optimizer
AdamW
[22
AdamW
learning rate of
le-2
le-2
learning rate of V À
le-3
5e-3
0.5
0.5
weight decay precision
le-3
le-3
FP16
FP16
batch size
2
2
warm up accumulate batches
0.1
0.1
8
8
number of epochs
10
10
A.2 LoRA Hyperparameters
Table details the hyper-parameters for the LoRA model trained during our experiment. Both the OKVQA and VQAv2 datasets use the same hyper-parameters.
Table 8: LoRA Training Parameters
Hyperparameter
Value
optimizer
AdamW
learning rate
le-3
LoRA matrix rank
32
LoRA dropout rate 53
0.05
batch size
warm up
0.1
number of epochs
10
A.3 Inference and hardware details
our ICV inference process, we employ the following hyperparameters: For ICV model inference, the maximum number of new tokens is set to 5, the number of beam searches is set to 3, the length penalty is set to O, and the minimum number of generated tokens is set 0. During the inference process, we utilize two Xeon Silver 3414 CPUs, one RTX 3090 GPU, and 384 GB of memory.
B Detailed Results
B.1
The Detailed Inference Speed Experiments
This subsection provides a detailed presentation of some experimental data, primarily including forward propagation FLOPs and runtime cost, as well as a comparison specific data between LIVE and LoRA. Table presents the detailed results of the runtime and the FLOPs shown in Figure 3
15

---

For forward operation LIVE compared to k-shot ICL. The average token length during the forward inference is 38 tokens for Zero-Shot and LIVE, 107 tokens for s-shot ICL, 187 tokens for 8-shot ICL, 330 tokens for 16-shot ICI., and 633 tokens for 32-shot ICI.
Table 9: Comparison of FL.OPs and Runtime for LIVE and k-shot ICL.
Mertic
Zero-Shot
LIVE
4-shot ICL
8-shot ICL
16-shot ICL
32-shot ICL
FLOPs (THLOPs)
0.935
0.936
3.568 92.44
6.375 158.21
12.341
23.364
Runtime (ms)
56.69
56.81
266.13
468.55
B.2
The Detailed Accuracy of Different Training Dataset
Table T0 presents a comparative analysis of the results between LoRA and LIVE results on different training dataset size in Section
B.3 Non-Learnable Methods Results
Function Vector:
This section examines the test results of the function vector employed in the experiment across different layers of the IDEFICS -BB model. Table [ presents the test results for VQAv2, and Table [2 shows the test results for OK VK. The Best result of VQAv2 is 10th layer's result, which reaches 30.21, and the best result of OKVQA is 31.02 from the first layer.
Table 10: Accuracy (%) of LoRA and LIVE on different training set sizes
Training set size
Dataset
Method
100
300
500
700
900
1000
2000
4000
6000
8000
LIVE
29.43
49.66
49.93
51.03
53.42
53.71
56.64
57.76
58.33
58.54
VQAv2
LoRA
29.52
39.02
40.01
40.99
41.99
42.81
45.26
47.71
48.51
49.02
LIVE
30.64
38.41
41.87
45.82
46.09
46.05
47.22
48.39
49.68
50.08
OKVQA
LORA
30.37
13.38
20.01
23.42
25.01
25.66
29.52
33.08
34.17
34.21
Table 11: The Function Vector Accuracy (%) acorss differnt layers on VQAv2.
VQAv2
layer:
layer:2
layer:3
layer:4
layer:5
layer:6
layer:7
29.28
29.0
28.5
27.43
27.94
28.7
29.17
layer:9
layer: 10
layer: 11
layer: 12
layer: 13
layer: 14
layer: 15
29.34
30.21
29.94
29.48
29.52
29.38
29.62
layer:17
layer:
layer:
layer:20
layer:21
layer:22
layer:23
29.51
29.59
29.5
29.28
29.24
29.29
29.11
layer:25
layer:26
layer:27
layer:28
layer:29
layer:30
layer:31
29.34
29.19
29.26
29.18
29.19
29.45
29.32
Task Vector: This section examines the test results the task vector utilized in the experiment across various layers of the IDEFICS model. Table 13 displays the test results for VQAv2, while Table 1hosoww the test results for OKVQA. The task vector is derived using 32 question-answer pairs. The best result of the task vector VQAv2 is at the 10th layer, reaching 43.68, while the best result on OKVQA is at the 12th layer, reaching 32.68.
PCA-ICV: PCA-ICV employs weighting factor to regulate the degree of interference ICV has on the model. We test various values of to assess performance. Table [5] shows the results of PCA-ICV on VQAv2 and OKVQA datasets with different value and extract samples. It is evident
16

---

Table 12:
The Function Vector Accuracy (%) acorss differnt layers on OKVQA
OKVQA
layer:
layer:2
layer:3
layer:4
layer:5
layer:6
layer:7
layer:8
31.02
30.57
30.23
30.27
30.04
30.36
30.61
30.23
layer:9
layer:10
layer:11
layer: 12
layer:13
layer:14
layer:15
layer:16
30.56
30.62
30.6
30.27
30.22
30.19
30.2
30.3
layer:17 17
layer: 18
1ayer:19
layer:20
layer:21
layer:22
layer:23
layer:24
30.44
30.25
30.27
30.17
30.22
30.16
30.31
30.23
layer:25
layer:26
layer:27
layer:28
layer:29
layer:30
layer:31
layer:32
30.42
30.32
30.34
30.25
30.3
30.34
30.4
30.28
Table 13: The Task Vector Accuracy (%) acorss differnt layers VQAv2
VQAv2
layer:
layer:2
layer:3
layer:4
layer:5
layer:6
layer:7
28.18
28.82
30.47
30.53
36.41
35.35
36.17
layer:9
layer: 10
layer: 11
layer: 12
layer: 13
layer: 14
layer: 15
41.72
43.68
40.33
34.32
16.91
15.42
14.53
layer:
layer:
layer:
layer:20
layer:21
layer:22
layer:23
12.94
12.44
12.28
11.89
11.98
11.78
11.44
layer:25
layer:26
layer:27
layer:28
layer:29
layer:30
layer:31
12.8
12.95
12.97
12.86
12.87
13.29
14.7
Table 14: The Task Vector Accuracy (%) acorss differnt layers OKVQA
OKVQA
layer:0
layer:1
layer:2
layer:3
layer:4
layer:5
layer:6
layer:7
13.58
14.4
15.3
15.0
15.79
18.06
19.18
22.37
layer:8
layer:9
layer: 10
layer: 11
layer: 12
layer: 13
layer: 14
layer: 15
21.71
22.93
32.5
31.99
32.68
29.38
21.4
17.27
layer:16
layer:17
layer:18
layer:19
layer:20
layer:21
layer:22
layer:23
6.49
0.94
0.5
0.53
0.46
0.32
0.34
0.29
layer:24
layer:25
layer:26
layer:27
layer:28
layer:29
layer:30
layer:31
0.28
0.3
0.3
0.29
0.33
0.33
0.33
1.11
that the optimal alpha for the VQAv2 dataset significantly differs from that for OKVQA. The optimal result of PCA-ICV on VQAv2 34.75 when alpha set to le-2, while the best result for PCA-ICV is 30.59 when alpha is set to le-5.
Table 15: The PCA-ICV Accuracy (%) acorss differnt alpha
Alpha
Dataset
Samples
le-2
le-3
le-4
le-5
VQAv2
32
34.75
30.95
30.06
30.0
OKVQA
32
22.46
30.06
30.23
30.59

---

C
the vector to General LIVE
LIVE is essentially shift vector, allowing control the shift directions of query representations, which means we can control the shift direction of LMMs by adding or subtracting different shift vectors. Therefore,
several LIVE trained different VQA dataset to get the General LIVE. Specifically, given the diverse VQA task set T {L_...., J. where I, is a VQA task and is the number of tasks, train the task-specific LIVE V {0),-.. ..p.... v and the weight factor a' = faj....,,c a²,} each VQA dataset. Then, have the LIVE set Vset [VV ..... V") and its weight factors set ge = {a''.... x²)) The General LIVE has the same shape of task specifc LIVE. For the [-th layer, is defined 2ı >, a/ot. During inference,
shift the original representation. resulting in (1 ([,[=)"==(((]) + 0t We average the LIVE trained in OKVQA and VQAv2 to get the general ICV and evaluate the performance of the general ICV
Vg = (1), Vr on OKVQA and VQAv2, with the results presented on Table 16
Table 16: Accuracy (%) for 32-shot ICL, task-specific LIVE, General LIVE.
Methods
32-shot ICL
LIVE
General LIVE
VQAv2
56.18
58.54 50.08
56.17
OKVQA
48.48
49.52
The results indicate that while the performance of the general ICV is somewhat reduced compared to task-specific LIVE; it decreased by 2.37 on VQAv2 and by 0.56 on OKVQA. However, its performance is very close to that of 32-shot ICL. Most importantly, it offers significant advantages in real-world scenarios where the distribution of test data unknown. This makes the general ICV more suitable for practical applications, providing robust solution that can be effectively utilized across varying environments. Furthermore, the findings highlight the scalability of LIVE: whenever a new VQA dataset is used train LIVE, we only need to simply recalculate the mean shift vectors of all previously trained task-specific LIVES and the new LIVE to generate a new general ICV, thus creating more general VQA LIVE. This approach not only simplifies the adaptation process for diverse tasks but also ensures that the model maintains a high level of performance across different applications. The results, therefore, underscore the potential of general ICVs enhance the flexibility and applicability of VQA systems in real-world settings.
D More exploratory experiments.
D.1
Generalization Ability
We conducted supplementary experiments on other vision- lasion-l tasks and LMMs to demonstrate the general applicability of our LIVE method.
D.1.1 Task Generalization
In Table utilized IDEFICSv1-9E the baseline model to evaluate the performance of LIVE against ICL and LoRA on the COCO caption (5taset[54], OcrVQA dataset (55), and VI-ICL [56) Experimental results demonstrate that our method consistently maintains advantages over both ICL and LoRA across various tasks.
Table 17: Accuracy (%) for 32-shot ICL, LoRA, and LIVE in various dataset.
Task (Metrie)
32-shot ICL
LoRA
LIVE
COCO (CIDEr)
106.31
109.18
117.38
VL-ICL Textocr (ACC)
21.5
22.5
24.0
VL-ICL Clevr (ACC)
33.5
35.5
37.0
OcrVQA (ACC)
16.3
15.9
17.5
18

---

D.1.2 Model Generalization
To test the generalizability of our method across different architectural models, we extended LIVE to IDEFICSv2 m5del[57]. It noteworthy that IDEFICSv2 (8B-base) differs from IDEFICSv1 in its handling of image-text interactions; while IDEFICSv uses a cross-attention mechanism for the fusion of image and text tokens, IDEFICSv2 utilizes an additional projection layer to map image information into the text space for LMM processing. These two architectures represent the primary approaches within LMM. Conducting experiments with this model can validate the generalizability of LIVE.
Due to the longer input sequence of IDEFICSv2 compared to IDEFICSvI, we reached the limits of our GPU memory during 8-shot ICL inference. Therefore, we compared the LIVE trained on 8-shot data with 1-8 shot ICL. As shown in Table [8] LIVE improves upon 8-shot ICL by 6. and 0.84% on VQAv2 and OKVQA, respectively. In terms of inference speed, it surpasses 8-shot ICL by factor of 8.95. These results demonstrate that the LIVE method is both generalizable and effective within LMM, excelling in both inference accuracy and speed compared to ICL.
Table 18: The performance comparison between LIVE and ICL on IDEFICS-v2-8B.
Method
VQAr2 ACC
OKVQA ACC
Inference Time
FLOPs (TFLOPs)
Zero-shot
55,39
43.08
0.087 (x 0.97)
3.110 (x 0.99)
1-shot ICL
60.33
45.65
0.168 (x 1,90)
6.497 (x 2.07)
2-shot ICL
63.49
50.40
0.237 (x 2.68)
9.610 (X 3,06)
4-shot ICL
64.88
53.18
0.395 (x 4,46)
17.139 (X 5,46)
8-shot ICL
66,20
57.68
0.792 (x 8,95)
32.130 (x 10.23)
8-shot LIVE
70.30
58.52
0.089 (x
3.141 I)
D.2 Performance with more shot
To investigate the impact of higher number of shots on LIVE, we conducted a comparison with ICL at 48 and 64 shots. Detail results are shown in T9blc|19].Ti results indicate that while increasing the number of shots improves the performance of both methods, they have reached performance plateau. Notably, the gains achieved by LIVE remain superior to those of ICL. For example, with 64 shots LIVE achieves 2.99% 1.85% improvement over ICL on VQAv2/OKVO. Thus, although using more ICDs improves ICL performance, LIVE has a large improvement
Table 19: The performance comparison of ICL and LIVE with more shots.
Benchmark
Method
Shot
Shot
Shot
Shot 64
ICL
55.7
56.18
56.67
56.71
VQAv2
LIVE
58.27 (+2.57)
58.54 (+2.36)
59.07 (+2.4)
59.70 (+2.99)
ICL LIVE
47.7
48.48
48. 68
48.60
OKVQA
49.71 (+2.01)
50.08 (+1.6)
50.55 (+1.87)
50.45 (+1.85)
D.3 Shared layer LIVE
We conducted experiments to investigate the impact of using shared bias vector across all layers (as shown in Table 20). and we observed a significant performance drop compared to LIVE: 20.6% on VQAv2 and 9.01% OKVQA. This performance drop aligns with the findings in 24 that different Transformer layers play diverse roles in ICL. Also, Eq 5 show that LIVE vectors are layer-specific. Then using single shared vector fails to capture the distinet information processed each layer, leading to the observed decline in accuracy.
D.4 Detail comparisopn LoRA
To provide a more detailed comparison with LoRA, we made the following modifications to LoRA. First, based on the loss function formula in Eq. 7 we applied the same loss function to LoRA. As shown in Table 21 even after modifying the loss function, LoRA's performance remains significantly lower
19

---

Table 20: The performance comparison of using shared LIVE across all layer (Share LIVE) with LIVE and ICL
Method
VQAv2
OKVQA
32 shot ICL
56.18
48.48
shot LIVE
58.54
50.08
32-shot Share LIVE
37.94 (-20.6)
41.07 (-9.01)
than LIVE both VQAv2 (49.48% vs. 58.54%) and OKVQA (37.02% vs. 50.08%). Additionally, we fine-tuned LoRA with more parameters, but similarly, increasing the parameter count did not close the performance gap between LoRA and our LIVE, further demonstrating the effectiveness of LIVE's unique design.
The performance comparison of the different implementations LoRA with LIVE. "LoRA" integrates LoRA into the token classification head of the last layer to minimize the trainable parameters, "LoRA with KI." uses 32-shot ICDs and applies the same KI. loss as our LIVE, and "LoRA O O_proj all layers" integrates LoRA into the O_proj of all layers.
Table 21:
Method
VQAv2
OKYQA
Parameters Number
LoRA
49.02
34.21
1.15M (x times)
LoRA with KL
49.48
37.02
1.15M (x 8 times)
LoRA O_proj all layers
57.57
45.03
10.5M (x 80 times)
LIVE
58.54
50.08
0.13M (x time)
D.5 Comparison between untrained LIVE
We evaluated the impact of untrained LIVE by initializing them randomly with normal distribution and tested VQAv2 (Table 22). revealing that untrained LIVE has similar results as zero zero-shot, which are largely worse than the trained LIVE. We also visualized the embeddings using t-SNE to see the shift effect in Fig (A), showing that the shift effect of untrained LIVE is similar to zero-shot ICL, while learned LIVES align more closely with 32-shot ICL, proving that the training process important.
Table 22: The performance comparison of untrained LIVE, trained LIVE, ICL, and Zero-shot.
Task
Zero Shot
Untrained LIVE
32-shot ICL
Trained LIVE (Paper)
VQAv2
29.25
30.3
56.
58.54
OKVQA
30.54
30.2
48.48
50.08
Table 23: The performance comparison between LIVE and diverse ICL methods.
Task
Method
Shot 1
Shot 4
Shot 8
Shot 16
Shot 32
Average
LIVE
56.84
57.60
58.25
58.27
58.54
57.90
ICL
51.39
53.72
54.24
55.70
56.18 18
54.25
VQAv2
RICE
47.83
53.54
55.04
56.89
58.07
54.27
MMICES
48.22
54.99
56.16
57.02
57.98
54.87
LIVE
47.51
47.68
49.4
49.71
50.08
48.88
ICL
40.75
46.11
46.79
47.70
48.48
45.97
OKVQA
RICE
40.67
47.07
48.92
50.73
51.11
47.70
MMICES
40.39
47.13
50.19
50.46
50.61
47.76
20

---

VQAv2
OKVQA
Figure 7: T-SNE visualization of first answer token over 200 queries. Left/right corresponds to VQAv2/OKVQA, respectively.
D.6 Comparison between LIVE and diverse ICL methods.
In previous works 8 19 18], the importance of selecting ICDs for ICL in VQA and other vision- language tasks has been demonstrated. reasonably compare the performance differences between LIVE and ICL, we adopted two methods for selecting ICL samples: RICE[43), which retrieves samples based on image similarity to the query, and MMICES(58), which combines both language and image similarity for ICD retrieval. The results comparing LIVE and ICL with different ICD selection strategies are presented
Table 23 showing that even with optimized ICD selection, ICL still lags behind LIVE in terms of accuracy.
We also tried to use RICE samples lear LIVE. However, the results were unsatisfactory, with only 57.68% accuracy VQAv2, while using random samples to train LIVE got 58.54% accuracy. This may be because using RICE samples will make each training individual characteristics, increasing the difficulty for LIVE to learn common task knowledge, and even causing LIVE to learn spurious correlations between the similar ICDs and the query.
sequence contain abundant
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have elaborated on the contributions of our approach in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [No]
Justification: Our method does have limitations, but due space limitations and focus on presenting the novelty of the method and the main results, we were unable to discuss these limitations in detail within the paper. However, recognize that comprehensive discussion of limitations is crucial for fully understanding the applicability and potential directions for improvement of the method. Currently, there are few open-source models with significant multimodal ICL capabilities. We hope to the release of more models with excellent multimodal ICL capabilities in the future, which will help to validate and improve our method. Additionally, we plan to explore the performance of the method across different application scenarios to provide more comprehensive and practical solution. Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
22

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: we provide complete proof in our Section 3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We show all hyperparameters and detailed experimental procedures for all experiments in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
23

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We submit all code for reproducing all experiments of our paper in supplemen- tary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.
http:///uuuidee//eeeuuuuiiiii /et////ooeSSssseeeeeeeeeee ississsinn ssionPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details in paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification: We conduct a large number of experiments, and the cost of repeating them is relatively high.
Guidelines:
24

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide all computer resources information in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://pww....a......o/ooo//o//e
Answer: [Yes]
Justification:
Our research conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction)
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
25

---

Justification: Our approach is primarily used to enhance the reasoning capabilities of the model without modifying the parameters of the large model. Therefore, does not involve any negative societal impacts. Additionally, the method is focused on improving technical aspects and does not directly interact with sensitive data or applications that could lead to ethical concerns misuse. As such, the potential for both positive and negative societal impacts is minimal, making the discussion of broader impacts not applicable in this context.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our approach is not related this.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We respect all licenses of the model and datasets.
Guidelines:
26

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do create new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our research is not related to the crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
27

---

Justification:
Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "Ap4ebmGGft", "reviewer": "Reviewer_VKDo", "summary": "This paper introduces learnable in-context vector (L-ICV) that extends the standard ICV method by training it on task samples. The method achieves performance benefits similar to ICV and the training methodology enables the L-ICV to outperform while still preserving the benefits of few-shot in-context learning. This is achieved through the use of a KL-loss term, that preserves the behavior rather close to ICV, but allowing adaptation as necessary. \n\nThe evaluation of the method is done on OKVQA and VQAv2 datasets. For both of datasets the performance of L-ICV outperforms that of LoRA fine-tuning. Authors provide an explanation and namely that due to need to access external knowledge, preserving the in-context learning behavior is advantageous compared to performing LoRA fine-tuning that disrupts the behavior, particularly when using few samples.", "questions": "Questions directly map to weaknesses discussed at previous point.\n\n1) Could authors ground the work in this line of interpretation of in-context learning and explain how L-ICV improves upon task recognition or task learning angles?\n\n2) Could the authors extend their analysis to such a task (e.g. OcrVQA, DocVQA, ChartQA, or InfoVQA)? Alternatively, why is comparing naively LoRA with L-ICV tuned to the output behavior of few-shot in-context demonstrations a fair comparison at all?\n\nI'm willing to raise my score if the authors help me better understand these angles.", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I believe the authors present a well-grounded method extending the strengths of in-context learning through few-shot learning. I find the learning approach balances well the specific model behavior with the task needs.  This is achieved through\n\n1) Strong theoretical grounding and explanation of the method, contrasting well the formulation and empirical results with other standard methods, such as ICV and LoRA.\n2) In-depth visualizations, qualitative assessment of the differences between the methods, such as the t-SNE visualization, decoding of the top-10 tokens and performance graphs compared effect of varying numbers of samples on L-ICV and LoRA", "weaknesses": "There are a few weaknesses, when contrasting it in-depth with methods focused on ICL for visual-language models.\n\n1) Ignoring some of the recent work on interpretation of in-context learning abilities (whether few-shot or zero-shot). It has been discussed as to whether the learning process performs task recognition (superficial format recognition, for example) or task learning (learning to map inputs to outputs) [[1]](https://arxiv.org/abs/2305.09731). The concept was extended to VLMs, by extending it to visual task recognition, studied particularly on VQA [[2]](https://arxiv.org/abs/2312.01571). It seems that VLMs suffer from image specific when doing ICL, such as short-cut inference (ignoring the image similarity, picking on similar question-answers) and others.\n\n2) The performance of LoRA is intriguing. Although I understand the motivation to completely treat it as baseline and naively apply on the same amount of trainable parameters, I find the comparison forcefully made to fit the story of the paper. The choice of tasks that require external knowledge make it obvious that LoRA can't be a good fit. Here it would be fair to either add another task with knowledge contained or alternatively modify on what LoRA is applied (e.g. synthetic 32-shot ICL examples) to contrast on fair grounds with L-ICV.\n\n[1] https://arxiv.org/abs/2305.09731\n[2] https://arxiv.org/abs/2312.01571", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1 & W1: Task Recognition (TR) and Task Learning (TL)**\n\nThanks for your recommendations of these studies. In our final discussion, we are happy to include these recent works on the interpretation of ICL abilities of LMMs. After thoroughly reviewing [A,B], we think that theoretically, L-ICV is more akin to TL. This is because during the training of L-ICV, we use randomly sampled 32-shot ICDs to encourage L-ICV to learn the common knowledge of the task (Lines 68-71), while overlooking the individual characteristics of the ICDs. In this way, L-ICV does not exploit the individual characteristics of the ICDs to narrow down the label space for predicting as TR mentioned in Sec.3.1 and Figure 1 of [B]. Moreover, Equations (1-4) in Sec.3 of our paper might provide a better mathematical explanation for TL. Specifically, in the latent space, the contextual information of ICDs  helps shift the query representation toward the direction of the correct answer. The role of L-ICV is to learn the most stable shift vector from various ICD combinations. In this way, L-ICV makes LMM less affected by short-cut inference since this often over-exploits the individual characteristics of the ICDs. The analyses in Line 302 \"Hallucinations and Invalid Responses\" show that L-ICV is less affected by yes/no hallucination, suggesting L-ICV is more robust that it does not simply copy yes/no from the ICDs as the answer. \nAdditionally, we follow [B] to deploy the \"new-mapping\" experiment for testing TL ability, where the results are given in Table A. Specifically, when training L-ICV and testing, all \"yes/no\" labels are changed to \"tiger/lion\". When applying ICL, we also change \"yes/no\" to \"tiger/lion\". In this way, if L-ICV has good TL ability, it will learn the mapping between the correct answer to one new label instead of recognizing \"yes/no\". Results in Table A show that L-ICV achieves better results than ICL, suggesting it has good TL ability.\n\n[A] What In-Context Learning \"Learns\" In-Context: Disentangling Task Recognition and Task Learning\n\n[B] How to Configure Good In-Context Sequence for Visual Question Answering\n\n**Q2 & W2: Fair LoRA Comparisons**\n\n1. First, we would like to clarify that in Section 4.3, the term \"external knowledge\" specifically refers to the OKVQA dataset, not to imply that \"since both VQAv2 and OKVQA require external knowledge, LoRA has worse performance.\"  For VQAv2, which does not require external knowledge, LoRA performs reasonably well (as shown in Figure 4, main paper). However, our L-ICV still achieves superior results with a notable improvement of 2.36%.\n2. We followed your suggestion to train LoRA in a fairer setting by using 32-shot ICDs and applying the same KL loss as our L-ICV. As shown in Table B, even with this modified design, LoRA's performance on VQAv2 and OKVQA remains significantly lower than that of L-ICV (49.48% vs. 58.54% in VQAv2 and 37.02% vs. 50.08% in OKVQA). Additionally, we evaluated LoRA with more parameters, as suggested by Reviewer fRTc, and the results are presented in Table B. Similarly, we find that increasing the parameter count does not bridge the performance gap between LoRA and our L-ICV, demonstrating the effectiveness of L-ICV's unique design.\n\n\n**W2: More Extension Tasks**\n\nWe followed your advice to evaluate L-ICV on OcrVQA, which also does not require external knowledge. The results are shown in Table C, and we find that L-ICV outperforms LoRA and 32-shot ICL by 1.6% and 1.2% respectively. Additionally, we evaluated our performance on Image Captioning (Table C) and another multimodal model architecture, IDEFICS2-8B (Table E). These consistent improvements over 32-shot ICL and LoRA demonstrate that our method is highly effective and efficient across various tasks and models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fbTmC1Il3y", "reviewer": "Reviewer_8z8x", "summary": "The study introduces a Learnable In-Context Vector (L-ICV) for Visual Question Answering (VQA) tasks. This approach extracts task information from demonstrations to improve the performance of Large Multimodal Models (LMMs) while reducing computational costs.", "questions": "* Is it worthwhile to increase the training cost of the extra in-context vector for the sake of reduced inference cost? I am curious about the trade-off between reduced inference time and additional training time.\n* How much would performance drop if each layer shared the same trained bias vector?", "limitations": "Yes.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The writing of this paper is excellent, the text is fluent, and the motivation is easy to understand.\n* The proposed method is novel and effective, significantly reducing the cost of reasoning.\n* The experiment is very solid, and the statistics in Figure 5 are impressive, intuitively showing the offset of the in-context vector.", "weaknesses": "please refer to the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: trade-off between reduced inference time and additional training time**\n\nWe believe that performing additional lightweight training to reduce inference time is worthwhile. \nFirstly, the initial one-time training investment is minimal and yields substantial long-term benefits. For instance, in our local machine (3090 GPUs) tests on VQAv2 experiments, training L-ICV takes approximately **11 GPU hours**.  In comparison, inferring 20,000 data samples using 32-shot ICL takes about **13.4 GPU hours**, whereas L-ICV achieves the same inference in just **0.5 GPU hours**.  The time saved for 20,000 samples inference is comparable to the time spent on training. Consequently, the cumulative savings in inference costs can easily exceed this initial investment. With the increasing number of tokens required by scaled LMMs (e.g., IDEFICS v2), L-ICV's efficiency in saving memory and reducing inference costs becomes even more pronounced (Table E). This advantage makes L-ICV better suited for batch inference and enhances the throughput of inference services at scale. \nSecondly, unlike L-ICV, ICL's performance is highly unstable. While various methods exist to stabilize ICL performance—such as selecting ICDs similar to the query—these approaches inevitably incur additional computational and memory costs. In contrast, L-ICV offers stable performance while substantially reducing inference time\nIn the upcoming revision, we will further discuss this issue, emphasizing the feasibility and necessity of lightweight training in exchange for faster inference.\n\n---\n\n**Q2: How much would performance drop if each layer shared the same trained bias vector?**\n\nThank you for your suggestion. We conducted experiments to investigate the impact of using a shared bias vector across all layers (as shown in Table D), and we observed a significant performance drop compared to ours: 20.6% on VQAv2 and 9.01% on OKVQA. This performance drop aligns with the findings in [22] that different Transformer layers play diverse roles in ICL (Lines 70-72). Also,  Equations (1-4) in Sec.3 show that L-ICVs are layer-specific.  Then using a single shared vector fails to capture the distinct information processed at each layer, leading to the observed decline in accuracy.\nAdditionally, since the number of tuned parameters is limited to 32 × 4096 for layer-specific L-ICVs, using a shared 1 × 4096 ICV across layers does not substantially impact training speed. During inference, the computational cost of using a shared ICV versus non-shared ICV remains comparable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xXR0ByioB4", "reviewer": "Reviewer_fRTc", "summary": "This paper proposes a method by introducing the idea of In-Context Vector in the field of NLP, by introducing L-ICV with a smaller number of parameters in LMM to replace in-context demonstrations to introduce external knowledge, and obtain through training The appropriate shift direction is combined in the query. By comparing with some existing methods, the author proves that L-ICV can significantly reduce computational costs while avoiding the bias in the selection of in-context demonstrations and improving the accuracy of VQA tasks.", "questions": "(1) Why 32-shot demonstrations are chosen in L-ICV training? According to Table 3. As the Number of Demonstrations k increases, the performance gap between L-ICV and ICL gradually narrows. When k continues to increase, will the performance of ICL exceed that of L-ICV?\n\n(2) Why is there a need for a unique shift effect for each layer in the L-ICV method, but in the comparative experiment of LoRA, only the LoRA module is added to the token classification head of the last layer?\n\n(3) In the training stage of L-ICV, the k demonstrations used for training are also randomly selected. How to prove that this can eliminate one of the major challenges when applying ICL: the performance is sensitive to the selection of ICDs.", "limitations": "The authors have adequately addressed the limitations in the PaperChecklist.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) The paper is clearly written and easy to comprehend.\n\n(2) The motivation for the proposed method is reasonable. Existing ICL methods are difficult to achieve good results while saving computing resources. Using a shifting vector instead of in-context demonstrations can reduce the computational cost while ensuring the effect.", "weaknesses": "(1) In the comparison of Section 4.2.1, only the inference time consumption of L-ICV and k-shot ICL is listed. How about other methods like TV, FV, etc.?\n\n(2) The method's generalizability is questionable as it has only been tested on the IDEFICS model, with no consideration of other models.\n\n(3) The paper only proves the effect of L-ICV on VQA tasks, which cannot be extended to other task scenarios and has limited application scope.\n\n(4) In the ablation study, there is a lack of a control group using untrained L-ICV to prove the effectiveness of shift direction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Inference Time Comparison of TV and FV**\n\nTV, FV and L-ICV have similar inference time. Specifically, TV takes 57.14 ms, FV takes 57.08 ms, L-ICV takes 56.81 ms for one sample, and Zero-Shot takes 56.69 ms. This similarity in inference times is because TV and FV replace the ICDs with a single vector to intervene in the model’s forward process, just like our L-ICV. However, L-ICV significantly surpasses TV and FV in terms of performance, achieving higher accuracy while also reducing inference costs. We will include these in the final.\n\n**W2: Generalizability of the Method**\n\nTo test the generalizability, we extended L-ICV to IDEFICS v2-8B [A]. Notably, IDEFICS v2 differs in architecture from IDEFICS v1 (used in the paper). Currently, there are two main LMM architectures for ICL: (1) Models like IDEFICS v1 that use cross-attention to fuse image and text tokens, and (2) Models like IDEFICS v2 that employ additional projection layers to concatenate image and text tokens. Testing L-ICV on IDEFICS v2 can further demonstrate the generalizability across architectural designs. \nSince IDEFICS v2 concatenates image and text tokens, it has a longer input sequence than IDEFICS v1 when the same image-text pair is used. For instance, an image is represented by 64 tokens, resulting in 512 vision tokens in an 8-shot scenario, which pushed our GPU to its computational limits. Consequently, we trained L-ICV with an 8-shot ICD sequence and compared it with 1-8 shot ICL. The results in Table E show that L-ICV outperforms 8-shot ICL, achieving 4.1%/0.84% higher accuracy on VQAv2/OKVQA, respectively. Additionally, L-ICV is 8.95 times faster in inference speed compared to 8-shot ICL on IDEFICS v2. \n\n[A] What matters when building vision-language models?\n\n**W3: L-ICV on Other Tasks**\n\nNote that L-ICV is inherently task-agnostic as Equations (1-4) in Sec.3 can be used among various tasks.  We initially focused on VQA because it covers a wide range of vision-language challenges (Lines 56-59), which is a more challenging task for FV/TV. We follow your suggestion to test L-ICV on image captioning, VL-ICL Benchmark and OcrVQA (Table C). It can be found that L-ICV achieves 11.07 higher CIDEr than 32-shot ICL on IC, 3.5 higher ACC than 32-shot ICL on VL-ICL-clevr, 2.5 higher ACC than 32-shot ICL on VL-ICL-textocr, 1.2 higher ACC than 32-shot ICL on OcrVQA, validating the generalizability of L-ICV on various tasks.\n\n\n**W4: Ablation with Untrained L-ICV**\n\nWe evaluated the impact of untrained L-ICV by initializing them randomly with a normal distribution and tested on VQAv2 (Table F), revealing that untrained L-ICV has similar results as zero-shot, which are largely worse than the trained L-ICV. We also visualized the embeddings using t-SNE to see the shift effect (Figure A), showing that the shift effect of untrained L-ICV is similar to zero-shot ICL, while learned L-ICVs align more closely with 32-shot ICL, proving that the training process is important.\n\n---\n\n\n**Q1: Why do we choose 32-shot demonstrations and the performance gap with more shots?**\n\nThe choice of 32-shot demonstrations is influenced by the LMMs’ capacity for understanding long texts, rather than by limitations of our method. Many related studies [6,7] typically use up to 32 shots due to these practical constraints.\nTo investigate the impact of more-shot ICDs, we evaluated L-ICV with 48/64-shot ICDs (Table G), showing that while adding more shots improves performance for both methods, the relative gain for L-ICV remains higher than ICL. For example, with 64 shots, L-ICV achieves a 2.99%/1.85% improvement over ICL on VQAv2/OKVQA. Thus, although using more ICDs improves ICL performance, L-ICV has a larger improvement.\n\n**Q2: Why unique L-ICV for each layer and the comparable design for LoRA?**\n\nSince [22] finds that each layer plays distinct roles during ICL inference (Lines70-72), we use unique shift vectors for each layer where Equations (3-4) are applied to each single layer, making L-ICV layer-specific. Using layer-specific L-ICV requires 32 $\\times$ 4096 parameters. We integrated LoRA into the token classification head of the last layer to minimize the trainable parameters (still 8 times more parameters than L-ICV). \nTo ensure a comprehensive comparison, we also evaluated LoRA when applied to the `O_proj` of all layers, as shown in Table B. This configuration results in an 80-fold increase in parameters, allowing LoRA to achieve decent performance but still falling short of L-ICV by 0.97% in VQAv2 and 5.05% in OKVQA, suggesting that L-ICV is more effective than LoRA.\n\n**Q3: How to address the performance sensitivity to ICD selection with L-ICV?**\n\nAs discussed in Lines 67-70, it is precisely because we aimed to reduce the sensitivity of ICD that we randomly select 32-shot examples during training. Generally, each ICD has individual characteristics and some common knowledge about the task, while the ICL sensitivity stems from the individual characteristics. To encourage the ICV to learn common knowledge, during training, each in-context sequence is set to a random combination of ICDs, which forces ICV to capture the most common knowledge while disregarding these individual characteristics. Please refer to **\"Q1 & W1\" of R.VKDo** for more details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hy7RY9TGQE", "reviewer": "Reviewer_BDyQ", "summary": "This paper aims to improving the in-context learning performance of multimodal models on VQA tasks. The paper proposes a Learnable In-Context Vector (L-ICV) method to distill essential task information from demonstrations into a single vector, reducing computational costs and enhancing accuracy in VQA tasks. Experiments show that L-ICV can achieve better ICL performance compared to random 32-shot demonstrations and significantly reduce the computation cost.", "questions": "Please see the Weaknesses section.", "limitations": "The limitations of this paper are threefold. \n\n1. The experiments are relatively limited regarding the model types and evaluation tasks.\n2. The proposed method requires extra training, optimization, and hyperparameter tuning effort. This added complexity could be a barrier to adoption, especially when there is only a limited set of demo data. \n3. The increased performance is not significant, especially considering the training and tuning effort. This raises questions about the practical benefits of the proposed method relative to the effort involved in its implementation.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The topic is important for the community, and the proposed method is an insightful exploration that can be beneficial. The overall writing of this paper is clear and easy to follow. L-ICV reduces computational costs, requiring only a fraction of the FLOPs and inference time compared to traditional ICL methods.", "weaknesses": "One of the major weaknesses is that the evaluation of L-ICV is limited to a small set of models and datasets. Only one model (IDEFICS-9B) and two datasets (VQAv2, OKVQA) are reported across the paper.  This limitation raises questions about the generalizability of the method to other multimodal models and different dataset configurations. Besides, recent literature has discussed the in-context limitation of these datasets [1, 2, 6, 7] and a broader comparison of tasks that rely more on visual in-context learning ability should be beneficial to showcase the advantage of L-ICV.\n\nThe second major issue is the relatively limited increase of the ICL performance. Although the L-ICV method demonstrates improved efficiency and performance, the reported increase in accuracy is not particularly significant. while L-ICV has been trained on a large dataset, the expected performance gains are not as high as one might anticipate given the extensive training resources utilized. \n\nAdditionally, the paper only compares L-ICV with ICL using randomly chosen demos and does not compare L-ICV with the classical RICE (Retrieval-based In-Context Examples) method [3, 4, 5] or a recently proposed MMICES [6]. RICES and MMICES samples can lead to a direct ICL performance boost, and a direct comparison would help to position L-ICV within the current state-of-the-art. Besides, if L-ICV is trained using RICES samples rather than randomly sampled demos, will the performance be better? This should be an interesting exploration that should be conducted. \n\nFurthermore, the in-context learning ability of large models is practical and useful as it does not need any parameter optimizations and only requires a handful of demonstrations to quickly adapt to new tasks. To enhance this ability, it would be better if the proposed method also requires minimal dataset, tuning, and optimization effort to achieve similar or better performance.\n\nOverall, while L-ICV presents a valuable exploration of the field, addressing these weaknesses through broader evaluations, detailed statistical analyses, comprehensive comparisons, and further optimization efforts would enhance the method's credibility and impact.\n\n[1] Zong, Yongshuo, Ondrej Bohdal, and Timothy Hospedales. \"VL-ICL Bench: The Devil in the Details of Benchmarking Multimodal In-Context Learning.\" *arXiv preprint arXiv:2403.13164* (2024).\n\n[2] Doveh S, Perek S, Mirza M J, et al. Towards multimodal in-context learning for vision & language models[J]. arXiv preprint arXiv:2403.12736, 2024.\n\n[3] Alayrac, Jean-Baptiste, et al. \"Flamingo: a visual language model for few-shot learning.\" *Advances in neural information processing systems* 35 (2022): 23716-23736.\n\n[4] Awadalla, et al. \"Openflamingo: An open-source framework for training large autoregressive vision-language models.\" *arXiv preprint arXiv:2308.01390* (2023).\n\n[5] Yang, et al. \"An empirical study of gpt-3 for few-shot knowledge-based vqa.\" *Proceedings of the AAAI conference on artificial intelligence*. Vol. 36. No. 3. 2022.\n\n[6] Chen, et al. \"Understanding and Improving In-Context Learning on Vision-language Models.\" *arXiv preprint arXiv:2311.18021* 1.2 (2023).\n\n[7] Baldassini, Folco Bertini, et al. \"What Makes Multimodal In-Context Learning Work?.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Generalizability of L-ICV**\n\nWe also follow your suggestion to test L-ICV on the VL-ICL Benchmark[A] and the results are given in Table C. Due to the rebuttal's time constraints, we only tested two subtask performances in the VL-ICL Benchmark. We also tested L-ICV on IDEFICS v2 and some new tasks to validate the generalizability of L-ICV in Table C&E and please refer to **\"W2\\&W3\" of R.fRTc** for more details.\n\n[A] VL-ICL Bench: The Devil in the Details of Benchmarking Multimodal In-Context Learning.\n\n**W2: Performance Improvements**\n\nIt should be stressed that the major motivation of ICV is to accelerate the inference speed instead of improving the ICL performance. To train L-ICV, it costs us 11 $\\times$ 3090 GPU hours on  8000 VQAv2 samples, occupying 1.8% of the VQAv2 dataset, which means that the total cost is not intolerable. Notably, all ICDs are randomly sampled from 8000 samples. After training, the inference speed can be largely improved and please refer to **\"Q1\" of R.8z8x** to see the trade-off between reduced inference time and additional training time.\nFurthermore, if L-ICV is trained by 1-shot ICD, the cost will be much less, 1$\\times$ 3090 GPU hours on 8000 samples, while 1-shot L-ICV achieves 5.45%/6.76% higher accuracy than 1-shot ICL on VQAv2/OKVQA, which is a significant improvement. In this case, even 1-shot L-ICV achieves better results than 32-shot ICL on VQAv2 and 16-shot ICL on OKVQA. In the 32-shot case, the improvement of L-ICV is less significant, while one reason is that modern LMM with ICL ability can not deal with long input sequences. Specifically, these LMMs are trained by 5-shot ICEs, meaning limited capacity for more shot inputs [6,7].\nWe also compared L-ICV with the LoRA method requiring training, and L-ICV consistently outperformed LoRA across various training set sizes, indicating that not all training-required methods achieve such good performance.\n\n**W3.1: Comparison with RICES and MMICES**\n\nRetrieving ICDs based on query sample and learning an ICV represent two major but orthogonal directions in ICL. The former exploits more individual characteristics of the query sample to improve the ICL performance. While the latter exploits more common knowledge to get a single intervene vector to improve the ICL inference speed. Considering their different motivations, we do not directly compare them in the submitted manuscript. Here we follow your suggestion to discuss their relative advantages and disadvantages. \n1.  Table H compares L-ICV with RICE/MMICES, showing that on average, L-ICV is better than RICE/MMICES on VQAv2/OKVQA, while be worse than RICE/MMICES on OKVQA when using more-shot ICDs since this tasks asks for more external knowledge and RICE/MMICES can provide such knowledge when returning similar ICDs.\n2. RICE and MMICES require additional image and text encoders to retrieve ICDs from a database, occupying computational resources and causing significant latency during inference. Moreover, RICE and MMICE require LMMs to deal with many-shot ICDs, thus requiring longer inference time. To sum up, it is less favorable for real-time deployment than our L-ICV.\n3. RICE and MMICES retrieve ICDs based on feature similarity, which makes LMM produce more short-cut learning as demonstrated in [16,17]. This is because that LMM tends to mimic the text description of the most similar samples and then neglect the vision contents. However, since L-ICV tries to capture common knowledge from diverse combinations of ICDs, it is less affected by short-cut learning.\n\n**W3.2: Using RICE samples to learn L-ICV**\n\nActually, at the beginning, we also tried to use RICE samples to learn L-ICV. However, the results were unsatisfactory, with only 57.68% accuracy on VQAv2, while using random samples to train L-ICV got 58.54% accuracy. This may be because using RICE samples will make each training input sequence contain abundant individual characteristics, increasing the difficulty for L-ICV to learn common task knowledge, and even causing L-ICV to learn spurious correlations between the similar ICDs and the query. We will add this experiment and discussion in the revision.\n\n**W4: Why update parameters in a parameter-efficient way.**\n\nIndeed, the advantage of ICL is that it does not require parameter updates, but its sensitivity to ICDs is a drawback. Therefore, many papers attempt to train auxiliary models to select or order ICDs for improving ICL performance [10, 11]. Our L-ICV follows this research direction but uses less data and updates fewer parameters than other parameter-efficient methods like LoRA. \nMoreover, due to the sequence length inference issue, ICL faces significant latency during deployment when applying some retrieval-based methods. However, L-ICV achieves more stable and superior performance with minimal training, striking a trade-off between PEFT and ICL, combining their strengths to enhance inference performance and reduce latency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We gratefully thank all the reviewers for their valuable and constructive feedback. We are pleased to see that the reviewers recognize our motivation: to reduce inference time costs while maintaining the performance of in-context learning (ICL) for LMMs. We are encouraged to see that they find our method novel, insightful and strongly theoretically grounded (Reviewer VKDo, 8z8x, and BDyQ), our visualizations and qualitative assessments comprehensive and in-depth (Reviewer VKDo, 8z8x), our experiments solid and the results strong and effective (Reviewer 8z8x).

We address the concerns and questions in detail below and have appended a PDF file with tables and figures. **Note that to distinguish from the Tables and Figures in the submitted manuscript, the Table and Figure numbers in the rebuttal are marked with A, B, C, etc.**

 Based on these comments, we have summarized the common questions and our responses as follows:

1. **Generalization Ability:** We conducted supplementary experiments on other vision-language tasks and LMM to demonstrate the general applicability of our L-ICV method (Table C and Table E, to Reviewers VKDo fRTc, BDyQ).
2. **More Ablation Studies of L-ICV:** We conducted additional comparisons of L-ICV over the layer-shared, randomized and more-shot-trained ablation to see the effectiveness of our design. (Table D Table F, and Table G, to Reviewer 8z8x, fRTc)
3. **Training Cost and Inference Efficiency:** We provided detailed explanations on how our method can use a slight increase in training costs to achieve significantly faster inference times and better ICL performance, highlighting the significance of this trade-off (Table E, to Reviewer 8z8x, BDyQ).
4. **Comprehensive Comparisons with LoRA:** We show additional explanations and comparison experiments with LoRA,  considering the task does not requiring external knowledge, the different design and parameter count of LoRA (Table B, to Reviewer VKDo, fRTc).

We also address other specific concerns in separate responses.

Meta-review: The paper introduces Learnable In-Context Vector (L-ICV), an approach that enhances in-context learning for multimodal models, specifically in Visual Question Answering (VQA) tasks. Reviewers commend the method for its efficiency and the way it balances performance with reduced computational costs, achieving results comparable to or better than LoRA, particularly on VQAv2 and OKVQA datasets (reviewers 1, 2). The paper is noted for its clear and concise presentation, with strong theoretical support and informative visualizations that help to distinguish L-ICV from other techniques (reviewers 1, 3). However, some concerns were expressed regarding the limited range of evaluations, as the method was primarily tested on one model and two datasets, which may affect its generalizability to other tasks and models (reviewers 3, 4). Additionally, reviewers suggested that the paper could benefit from a broader comparison with other in-context learning methods and a more detailed exploration of the trade-offs between the additional training effort and the savings in inference time (reviewers 2, 4). Overall, the reviewers agree that the paper provides a valuable contribution to the field, justifying its acceptance despite the identified limitations.
Decision: Accept (poster)