LIVE: Learnable In-Context Vector for Visual Question Answering

Authors: yingzhe  peng
chenduo  hao
xinting  hu
jiawei  peng
xin  geng
xu  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 13:31:10

Abstract: As language models continue to scale, Large Language Models (LLMs) have exhibited emerging capabilities in In-Context Learning (ICL), enabling them to solve language tasks by prefixing a few in-context demonstrations (ICDs) as context. Inspired by these advancements, researchers have extended these techniques to develop Large Multimodal Models (LMMs) with ICL capabilities. However, applying ICL usually faces two major challenges: 1) using more ICDs will largely increase the inference time and 2) the performance is sensitive to the selection of ICDs. These challenges are further exacerbated in LMMs due to the integration of multiple data types and the combinational complexity of multimodal ICDs. Recently, to address these challenges, some NLP studies introduce non-learnable In-Context Vectors (ICVs) which extract useful task information from ICDs into a single vector and then insert it into the LLM to help solve the corresponding task. However, although useful in simple NLP tasks, these non-learnable methods fail to handle complex multimodal tasks like Visual Question Answering (VQA). In this study, we propose \underline{\textbf{L}}earnable \underline{\textbf{I}}n-Context \underline{\textbf{Ve}}ctor (LIVE) to distill essential task information from demonstrations, improving ICL performance in LMMs. Experiments show that LIVE can significantly reduce computational costs while enhancing accuracy in VQA tasks compared to traditional ICL and other non-learnable ICV methods.

Main paper: 
 introduction: 
 As language models continue to scale up, Large Language Models (LLMs) [1] [2] [3] have demonstrated emerging capabilities in In-Context Learning (ICL) [4] : these models can solve language tasks when provided with a few similar examples, termed in-context demonstrations (ICDs), as context. Unlike traditional task-specific fine-tuning, ICL, a efficient method to adapt LLM to downstream task [5, 6] , achieves comparable performance without necessitating updates to millions or trillions of model parameters [7] . By prefixing just a handful of data samples to the query input, ICL configures a model's behavior to produce the corresponding output, thus facilitating rapid adaptation across a wide range of downstream tasks. Inspired by these advancements in the language domain, researchers have extended these techniques to develop Large Multimodal Models (LMMs) with ICL capabilities [8] [9] [10] . Employing ICL in LLMs meets two challenges: Firstly, although increasing the number of ICDs typically enhances performance [7] , this practice conflicts with computational efficiency constraints. As ICDs are prefixed to the query, the increase in input tokens severely impacts the Transformer's inference speed, causing a marked slowdown in computational performance. Secondly, the effectiveness of ICL is vulnerable to the selection of demonstrations [4, [11] [12] [13] , particularly when only a limited number are used. It makes the process of choosing demonstrations critical for optimal performance. However, developing selection strategies and measuring the effectiveness of ICDs remain open questions [14] [15] [16] [17] . For LMMs, the challenges above are further exacerbated: 1) The computational complexity is significantly increased due to the integration of multiple data types as ICDs (as shown in Figure 1 (a)). 2) The task of selecting effective multi-modal ICDs becomes more complex and nuanced, as each modality contributes uniquely to understanding the context [18, 19] , further complicating the assessment of their combined effect in demonstration selection.To alleviate these two challenges, recent research on LLMs has introduced In-Context Vector (ICV) to extract the most useful task information from ICDs, and then use it to directly influence the processing in LLMs [20] [21] [22] . For example, [20] proposes that by using multiple demonstrations and a dummy query as inputs, the representation of the last token from a middle layer of the model can be extracted as the vector. This vector is then used to replace the representation of the corresponding token in the same layer during inference, which can achieve performance comparable to ICL. Such in-context vector alleviates the requirement of multiple ICDs during inference, as well as effectively bypasses the complexity of the individual selection of demonstrations by representing the most effective components across many demonstrations.However, these studies apply non-learnable strategies to extract ICVs, although useful in some simple NLP tasks, lose the efficacy in complex multi-modal tasks like Visual Question Answering (VQA). Our preliminary experiments have demonstrated that directly applying these non-learnable ICVs yields unsatisfactory results. The principal reason is the intrinsic complexity of VQA compared to the language tasks addressed by these non-learnable ICVs. For example, the previous methods focus on simple NLP task, such as Antonym [23] and Country-Capital [21] , whose distribution patterns can be easily identified by LLMs. In contrast, as a unified vision-language task, VQA encompasses a diverse array of question types, where each one corresponds to a different vision-understanding task. For instance, questions like "What is this?" or "How many are there?" require classification and counting abilities, respectively. These varied requirements imply that the task information, which non-learnable methods attempt to abstract, cannot be effectively captured by a single ICV.In this study, to make ICVs abstract more useful VQA task information, we try to distill the task information implied in demonstrations into a single Learnable In-Context Vector (LIVE). Our method is motivated by the observation [20] that ICL can be treated as a process of "shifting" the direction of the latent states of query towards the target, i.e., adding this latent state with a shift vector. Then we hope to learn suitable ICVs to replace the ICDs during inference to shift the direction. To achieve this, we train LIVE by minimizing the output distributions of a LMM got by only using LIVE and by inputting a few demonstrations. During training, we use different 32-shot randomly sampled demonstrations for different queries to distill task knowledge. Then LIVE is encouraged to capture the most essential task information from these different combinations by removing the individual characteristics of demonstrations. Moreover, [24] finds that during ICL, different layers of LLM have diverse roles in addressing demonstrations. Then in our method, each layer is assigned with a unique ICV to capture more fine-grained task information.Our LIVE inherits the efficiency of previous non-learnable ICVs, i.e., during inference, under the same performance conditions, LIVE only needs 1/24.97 FLOPs number of 32-shot ICL. Additionally, in VQAV2/OKVQA, LIVE improves accuracy by 2.36/1.6 compared to 32-shot ICL. We also compare LIVE to LoRA [25] that when comparable trainable parameters are used, LIVE requires much fewer training samples than LoRA (500 vs. 8000) to achieve satisfactory performance. Besides, we design lots of analytical experiments to validate whether LIVE can better shift the hidden states of queries to the target direction and analyze why previous non-learnable methods fail to solve VQA.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: We have elaborated on the contributions of our approach in the introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 In-Context Vector: Recently, more and more researchers in NLP have begun to focus on using an In-Context Vector (ICV) to modify the activation values during the forward propagation of LLM to simulate the effect of ICDs in ICL. [20] propose the "Task Vector", which extracts the representation of the middle layer from the LLM during ICL inference as the ICV, and replaces the representation of the same layer during zero-shot inference. Meanwhile, [21] introduced the "Function Vector", which uses attention weight analysis to take the mean of the activation values of the attention heads that most significantly affect the final result in ICL inference as the final ICV. This vector is then directly added to the representation of the middle layer during zero-shot inference to form a new representation. On the other hand, [22] propose "PCA In-Context Vector". They believe that the ICV should be closer to the LLM's representation of the task output and farther from the task input representation. Thus, they extract the input and output representations of several demonstrations and using PCA to find the overall principal direction as the ICV. These efforts mainly focus on using non-learnable methods to find the specific ICV for NLP tasks, achieving effects similar to ICL in various tasks. However, these methods only are tested on some simple tasks in NLP. When LMMs face with more complex tasks, the performance of these methods remains uncertain.Prompt engineering allows LLMs to tackle specific tasks without requiring finetuning [26] [27] [28] [29] [30] [31] [32] [33] . A specific form of this approach, ICL, further improves these capabilities by creating prompts that include several demonstrations. ICL has already demonstrated superior performance and good generalization on many tasks [34, 7, 35, 36] , and can be easily adapted to downstream tasks. However, the use of ICL faces several issues: first, ICL is very sensitive to the selection and arrangement order of demonstrations [4, [11] [12] [13] [37] [38] [39] ; poor demonstrations can severely impact ICL performance. Second, too many demonstrations can significantly slow down the inference speed of LLMs [40] . While ICVs can effectively address these two issues, as it can use only queries as input to the model while preserving ICL performance, without the need for demonstrations as input.As the performance of LLMs continues to improve, an increasing number of researchers begin to adapt LLMs to the multimodal domain [41] [42] [43] [44] [45] [46] . Relying on the powerful inference capabilities of LLMs, some LMMs have started to exhibit ICL capabilities, such as Flamingo [8] and IDEFICS [9] . Moreover, these models have significantly enhanced their ICL capabilities by concatenating multiple samples as contextual information during the training process. Currently, researchers mainly focus on how to configure demonstrations to address the sensitivity of ICL performance in LMMs. [18, 19] have respectively adopted heuristic retrieval methods for selecting demonstrations in Image Captioning and VQA. However, no researchers have yet extracted ICV from LMMs and evalutaed it. Therefore, the effectiveness of ICV in LMMs still needs further exploration. Considering that the IDEFICS model shares the same model structure as Flamingo and possesses stronger ICL capabilities, we primarily focus on valid our method on the IDEFICS model.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [No] Justification: Our method does have limitations, but due to space limitations and our focus on presenting the novelty of the method and the main results, we were unable to discuss these limitations in detail within the paper. However, we recognize that a comprehensive discussion of limitations is crucial for fully understanding the applicability and potential directions for improvement of the method. Currently, there are few open-source models with significant multimodal ICL capabilities. We hope to the release of more models with excellent multimodal ICL capabilities in the future, which will help to validate and improve our method. Additionally, we plan to explore the performance of the method across different application scenarios to provide a more comprehensive and practical solution.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
live: learnable in-context vector: 
 Here we show how to derive the formulation of the shift vector from Self-Attention (SA) mechanism and then introduce how to design LIVE based on this formulation. Generally, to implement In-Context Learning (ICL) using a language or multimodal model (LLM/LMM) M, the input has the following form: X = {X D , x}, where X D = {x 1 , ..., x k } represents the concatenation of k In-Context Demonstrations (ICDs), and x denotes the query input, as shown in Figure 2 . Given X as Key and Value, for each token xi of x, applying Self-Attention (SA) once yields:EQUATIONwhere vector xi X ⊤ D xi x⊤ ∈ R 1×l and l denotes the sequence length of the entire input [X D , x]. Expanding the softmax function, we obtain:EQUATIONwhere l c and l q represent the lengths of the X D and x, respectively. Z 1 and Z 2 are the sum of exponential scores between the query token xi with each token in X D and x: Z 1 = lc exp(x i X ⊤ D ) and Z 2 = lq exp(x i x⊤ ). This leads to the following formulation of SA:EQUATIONwhereµ = Z 1 /(Z 1 + Z 2 ). Let h(z) = SA(x i , z, z).The output of SA(x i ) can then be expressed as:EQUATION) As noted in Equation 4, we observe that h( x) is the representation obtained with self-attention over the query x without appending any ICD; h(X D ) functions similarly to a "shift" vector, altering the attention representation h( x) by incorporating contextual information from the ICDs X D . The coefficient µ quantifies the degree of influence X D has over the original query representation. For a visual demonstration of how ICDs shift the representation space, see Figure 2 (b) . Consequently, once learning a general shift direction to replace the effect of h(X D ), we can employ this shift direction to simulate the ICL process of LMMs without actual demonstrations.We propose a novel method that involves a Learnable In-Context Vector (LIVE) to simulate the ICL process without actual demonstrations. This approach aims to abstract general task information from demonstrations, enabling it to shift the model's representation toward the direction influenced by the ICDs. Figure 2 shows the training pipeline of LIVE. The LIVE training dataset, denoted as D = {d 1 , . . . , d N }, is a subset of the VQA dataset training split, created by randomly selecting N question-answer pairs from it. We use each training sample d i to simulate the query sample x in ICL, and randomly select k demonstrations from D \ {d i } for it. Additionally, [24] shows that during ICL, each layer of an LLM performs a distinct role. Motivated by this, we assume that for LMM, each layer also requires a specific shift direction. We assign a learnable vector v l and a weight factor α l for each layer l to learn the unique shift effect. Our final LIVE comprises of the vector set V and the corresponding weight factor set α as:EQUATIONwhere L is the number of layers. To train V and α, we align the distribution of the model's outputs for the query when shifted by demonstrations, P(x|X D ; M), with that shifted by our LIVE, P(x|V , α; M). This alignment is achieved by minimizing the Kullback-Leibler (KL) divergence:EQUATIONTo obtain the distribution P(x|X D ; M), for each query x, we randomly select k demonstrations to form X D . These are concatenated with the query to form the inputs for the model. The model's output for the query is then considered as the shifted distribution P(x|X D ; M).To obtain the output of x by using LIVE, we follow [20, 22] , we use the vector v l to shift the each layer's output representation M l (x i ) and get: M l (x i ) ′ = M l (x i ) + α l v l , which is shown in Figure 2 (a). After applying LIVE to shift the representations at each layer, we obtain the output distribution P(x|V , α; M). Notably, during training, X D for each query x includes randomly sampled 32-shot demonstrations. This strategy encourages our LIVE to extract the most useful common information from various demonstration combinations and prevents it from being influenced by the individual characteristics of certain demonstrations.In addition, to facilitate the LIVE in acquiring more task-specific information, we also optimize the P(x|V , α; M) with the ground truth by L gt . Thus, the overall loss L is defined as:L = λL gt + L d ,whereEQUATION)where λ is the hyper-parameter to control the importance of ground truth loss.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Justification: we provide complete proof in our Section 3.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
setting and implementation details: 
 Model and Dataset: We evaluate our approach using the IDEFICS-9B model [9] across two datasets: VQAv2 [47] and OKVQA [48] . VQAv2 emphasizes open-ended VQA tasks, encompassing 4, 437, 570 question-answer pairs in its training split, supplemented by an additional 2, 143, 540 pairs in the validation split. OKVQA is a large-scale dataset designed for models that require external knowledge to answer questions. It consists of 14, 055 question-answer pairs, with 9, 009 allocated for training and 5, 046 for validation. For both VQAv2 and OKVQA datasets, We train our LIVE on 8, 000 pairs from each training set. Due to computational resource limitations, we randomly sample 10, 000 question-answer pairs from the VQAv2 validation split for evaluation [18] . For OKVQA, we utilize the entire validation split.During training, we assign 32-shot demonstrations for each query, enabling LIVE to acquire better directions of shifting vectors for VQA tasks. The v i is initialized using a normal distribution with a mean of 0 and a standard deviation of 0.01, and all α i are initialized to 0.1. More detailed training parameters can be found in Appendix.We primarily compare the following methods:Zero-Shot: The model uses only the query as input.k-Shot ICL:The model uses k demonstrations, randomly selected from the VQA dataset training split, along with the query as input. Non-Learnable ICV Methods: We extend three established non-learnable ICV methods from language models to our multimodal settings: (1) Task Vector (TV) [20] uses k demonstrations and a dummy query to extract the representation of the last token from a middle layer of the model as the ICV. During inference, this vector replaces the representation of the last token in the same layer. We conduct evaluations on TV by implementing it across various layers and select the layer where it achieves the highest performance improvement. (2) Function Vector (FV) [21] employs a small subset of the validation data to derive the mean output from critical attention heads, forming the ICV. During inference, this vector is added to the representations of the last token within a specific layer. We conduct evaluations on FV by implementing it across various layers and select the layer where it achieves the highest performance improvement. (3) PCA In-Context Vector (PCA-ICV) [22] computes the ICV by applying PCA to the difference between the question and question-answer representations from k demonstrations. During inference, these vectors are added to the representations of all tokens at each layer,LoRA [25] : This method finetunes the LMMs with the same number of samples of training LIVE. We add the LoRA module in the token classification head of the last layer. In this way, the number of trainable parameters is comparable to that of LIVE.We present performance comparisons with various methods in Table 1 . Certain existing methods show only marginal improvements over Zero-Shot baselines, e.g., FV improves by 0.96/0.48 on VQAv2/OKVQA and PCA-ICV improves by 0.04 on OKVQA. Besides, we observe that all the previous non-learnable ICV methods do not reach the performance of the standard 32-shot ICL, e.g., the best non-learnable method, TV, is still 12.5/15.8 lower than 32-shot ICL on VQAv2/OKVQA. In contrast, our LIVE achieves an accuracy improvement of 2.36 on VQAv2 and 1.6 on OKVQA over 32-shot ICL. These results highlight the inefficacy of non-learnable methods in capturing essential task-specific information for VQA, whereas LIVE, by leveraging diverse 32-shot ICL demonstrations for each query during training, manages to abstract useful task information effectively. We further show that our LIVE outperforms LoRA with less trainable parameters, suggesting LIVE can abstract task information more efficiently.Figure 3 displays the efficiency of LIVE during inference compared to other methods. We average the FLOPs and actual inference time consumption per forward pass over 1000 randomly sampled queries. 1 We observe that LIVE only needs 1/24.97 FLOPs and 1/8.25 inference time of 32-shot ICL per forward pass. Additionally, LIVE maintains almost the same inference speed as Zero-Shot. These comparisons validate the efficiency of LIVE during inference.We use ablation studies to explore the effects of diverse settings, including different training losses, the shot number of demonstrations k used during training, and the number of training data N .Training Loss: Table 2 compares the results of using different losses: only L gt in Eq. ( 7) or L d in Eq. (7) . We find that only using L gt (same as standard fine-tuning) significantly damages the performance, e.g., L gt achieves 16.9/6.12 lower accuracy on VQAv2/OKVQA compared to using the combined loss L; yet using only L d results in a smaller performance drop -3.78/3.14 lower VQAv2/OKVQA than when using L. This suggests that with a limited number of trainable parameters, LIVE trained with L d is more robust and capable of capturing essential We compare the performance of LIVE trained with k demonstrations per query and the corresponding k-shot ICL in Table 3 .The result shows that an increase in the number of demonstrations enhances the performance of ICL and LIVE, indicating that more demonstrations can provide each query with a richer context to help train LIVE. Additionally, LIVE consistently surpasses the performance of k-shot ICL across different training sizes, showcasing the robustness of our LIVE in utilizing demonstrations. Notably, when the number of demonstrations is limited, the performance gap between LIVE and ICL becomes more pronounced. This is because ICL is highly sensitive to the choice of demonstrations; with insufficient demonstrations, the model may shift the query representations in an incorrect direction. In contrast, LIVE continuously by learning the main shift direction of the query representations from the demonstrations, reduces the negative impact of poor demonstrations on the query during training and is more robust that can extract essential task information. To better demonstrate the shifting effect of LIVE on query samples, we randomly select 200 query samples and conduct different methods of inference in LMMs. We extract the representation vector of the first answer token for T-SNE dimensionality reduction, shown in Figure 5 . Additionally, to quantitatively evaluate the effect of shift directions of different ICV methods, we calculate the following metrics. Given a query x, we use r icl , r zs , r * to denote the representation of the first answer token obtained by 32-shot ICL, Zero-Shot, specific ICV methods, respectively. Then we calculate the standard shift direction as s gt = r icl -r zs and the shift direction of specific ICV as s * = r * -r zs . Finally, we define the shift direction similarity as the cosine similarity between s * and s gt , indicating how closely the shift direction of the ICV method aligns with the standard shift direction. The results are presented in Table 4 . From Figure 5 , we can find that 32-shot ICL exhibits a significant shift compared to Zero-Shot, visualizing the shift effects given in Eq. 4.Considering both Table 4 and 1, we find that the shift direction similarity has positive correlation to the accuracy: if a method has large direction similarity, it also has better performance. For example, among non-learnable methods, TV has higher shift direction similarity than other ones, then it has better accuracy in Table 1 . Furthermore, for LIVE which has the best accuracy in Table 1 , its shift direction similarity is also the highest, which is 0.742/0.829 on VQAv2/OKVQA, validating that LIVE can produce shifts in query samples similar to 32-shot ICL, as visualized in Figure 5 . Decoding ICV To Tokens: We follow previous studies [49] [50] [51] to analyze the parameters of Transformers by directly decoding them into vocabulary tokens. Specifically, given a vector v ∈ R 1×d , it can be projected using the unembedding matrix E ∈ R d×N of LMMs to obtain the corresponding token probability distribution p , where N is the vocabulary size:EQUATIONWe calculate the p of the vectors got from different methods in VQAv2 and select the top-10 tokens with the highest probabilities in p shown in Table 5 . We can see that the tokens got from FV are not highly relevant to the VQA task, which proves that FV does not capture the task information of VQAv2. On the other hand, the frequency of "yes" and "no" tokens is relatively high in the decoding results of PCA-ICV and TV, suggesting that they prefer to capture the simple patterns from demonstrations, e.g.yes/no, but struggle to grasp the overall task information of complex VQA. In contrast, the tokens of LIVE decoding are not biased to specific answers like yes/no, suggesting it abstracts more summary task knowledge of VQA.Hallucinations and Invalid Responses.VQA contains various answer types and for convenience, VQAv2 divides them into three categories: "yes/no", "number", and "other". After delving deeper into the answer details, interestingly, we find that TV frequently answers "yes or no" to number/other questions as shown in Figure 6 (1)(2)(3). We term this phenomenon the yes/no hallucination and count the frequency of the yes/no hallucination over all test data samples for different methods in Table 6 . We can find that TV appears 111 times of yes/no hallucination, being consistent with the observations in Table 5 , suggesting TV is biased to yes/no type question. We also observe that non-learnable methods tend to respond meaningless text (e.g."\n") when responding to "number/other" questions as shown in Figure 6 (1)(2)(4). Table 6 shows the number of meaningless answers. We find that PCA-ICV and TV have more chance to return meaningless answers for "other" questions, suggesting these methods do not capture the overall task information of VQA and are not able to answer some less frequently appeared questions. However, for LIVE, it has less yes/no hallucination and meaningless responses, validating that LIVE captures more robust task information of VQA.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We show all hyperparameters and detailed experimental procedures for all experiments in Appendix. 
conclusion: 
 To address the two major drawbacks of ICL in LMM-long computation time and sensitivity to demonstration selection-we try to apply non-learnable ICV methods from NLP to solve VQA. However, due to the complexity of VQA and the significant biases often inherent in non-learnable methods, the performance is unsatisfactory. A Implementation Details.A.1 LIVE Hyperparameters Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We submit all code for reproducing all experiments of our paper in supplementary material.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We specify all the training and test details in our paper.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We conduct a large number of experiments, and the cost of repeating them is relatively high.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our approach is primarily used to enhance the reasoning capabilities of the model without modifying the parameters of the large model. Therefore, it does not involve any negative societal impacts. Additionally, the method is focused on improving technical aspects and does not directly interact with sensitive data or applications that could lead to ethical concerns or misuse. As such, the potential for both positive and negative societal impacts is minimal, making the discussion of broader impacts not applicable in this context.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our approach is not related to this.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We respect all licenses of the model and datasets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators.  
 
 Appendix: In our ICV inference process, we employ the following hyperparameters: For ICV model inference, the maximum number of new tokens is set to 5, the number of beam searches is set to 3, the length penalty is set to 0, and the minimum number of generated tokens is set to 0. During the inference process, we utilize two Xeon Silver 3414 CPUs, one RTX 3090 GPU, and 384 GB of memory.This subsection provides a detailed presentation of some experimental data, primarily including forward propagation FLOPs and runtime cost, as well as a comparison of specific data between LIVE and LoRA. Table 9 presents the detailed results of the runtime and the FLOPs shown in Figure 3 .For a forward operation LIVE compared to k-shot ICL. The average token length during the forward inference is 38 tokens for Zero-Shot and LIVE, 107 tokens for 4-shot ICL, 187 tokens for 8-shot ICL, 330 tokens for 16-shot ICL, and 633 tokens for 32-shot ICL. Task Vector: This section examines the test results of the task vector utilized in the experiment across various layers of the IDEFICS model. Table 13 displays the test results for VQAv2, while Table 14 shows the test results for OKVQA. The task vector is derived using 32 question-answer pairs. The best result of the task vector on VQAv2 is at the 10th layer, reaching 43.68, while the best result on OKVQA is at the 12th layer, reaching 32.68.PCA-ICV: PCA-ICV employs a weighting factor α to regulate the degree of interference ICV has on the model. We test various values of α to assess performance. Table 15 shows the results of PCA-ICV on VQAv2 and OKVQA datasets with different α value and extract samples. It is evident that the optimal alpha for the VQAv2 dataset significantly differs from that for OKVQA. The optimal result of PCA-ICV on VQAv2 is 34.75 when alpha is set to 1e-2, while the best result for PCA-ICV is 30.59 when alpha is set to 1e-5. LIVE is essentially a shift vector, allowing us to control the shift directions of query representations, which means we can control the shift direction of LMMs by adding or subtracting different shift vectors. Therefore, we can use several LIVE trained on different VQA dataset to get the General LIVE. Specifically, given the diverse VQA task set T = {t 1 , . . . , t n }, where t i is a VQA task and n is the number of tasks, we train the task-specific LIVE V i = {v i 1 , . . . , v i L } and the weight factor α i = {α i 1 , . . . , α i L } on each VQA dataset. Then, we have the LIVE set V set = {V 1 , . . . , V n } and its weight factors set α set = {α 1 , . . . , α n }. The General LIVE has the same shape of task-specific LIVE. For the l-th layer, it is defined as v l = i α i l v i l . During inference, we use the vector v l to shift the original representation, resulting in M l (x i ) ′ = M l (x i ) + v l . We average the LIVE trained in OKVQA and VQAv2 to get the general ICV and evaluate the performance of the general ICV V g = {v 1 , . . . , v L } on OKVQA and VQAv2, with the results presented on Table 16 . The results indicate that while the performance of the general ICV is somewhat reduced compared to task-specific LIVE; it decreased by 2.37 on VQAv2 and by 0.56 on OKVQA. However, its performance is very close to that of 32-shot ICL. Most importantly, it offers significant advantages in real-world scenarios where the distribution of test data is unknown. This makes the general ICV more suitable for practical applications, providing a robust solution that can be effectively utilized across varying environments. Furthermore, the findings highlight the scalability of LIVE: whenever a new VQA dataset is used to train LIVE, we only need to simply recalculate the mean shift vectors of all previously trained task-specific LIVEs and the new LIVE to generate a new general ICV, thus creating a more general VQA LIVE. This approach not only simplifies the adaptation process for diverse tasks but also ensures that the model maintains a high level of performance across different applications. The results, therefore, underscore the potential of general ICVs to enhance the flexibility and applicability of VQA systems in real-world settings.D More exploratory experiments.We conducted supplementary experiments on other vision-language tasks and LMMs to demonstrate the general applicability of our LIVE method.In Table 17 , we utilized IDEFICSv1-9B as the baseline model to evaluate the performance of LIVE against ICL and LoRA on the COCO caption dataset [54] , OcrVQA dataset [55] , and VL-ICL dataset [56] . Experimental results demonstrate that our method consistently maintains advantages over both ICL and LoRA across various tasks. To test the generalizability of our method across different architectural models, we extended LIVE to IDEFICSv2 model [57] . It is noteworthy that IDEFICSv2 (8B-base) differs from IDEFICSv1 in its handling of image-text interactions; while IDEFICSv1 uses a cross-attention mechanism for the fusion of image and text tokens, IDEFICSv2 utilizes an additional projection layer to map image information into the text space for LMM processing. These two architectures represent the primary approaches within LMM. Conducting experiments with this model can validate the generalizability of LIVE.Due to the longer input sequence of IDEFICSv2 compared to IDEFICSv1, we reached the limits of our GPU memory during 8-shot ICL inference. Therefore, we compared the LIVE trained on 8-shot data with 1-8 shot ICL. As shown in Table 18 , LIVE improves upon 8-shot ICL by 6.1% and 0.84% on VQAv2 and OKVQA, respectively. In terms of inference speed, it surpasses 8-shot ICL by a factor of 8.95. These results demonstrate that the LIVE method is both generalizable and effective within LMM, excelling in both inference accuracy and speed compared to ICL. We conducted experiments to investigate the impact of using a shared bias vector across all layers (as shown in Table 20 ), and we observed a significant performance drop compared to LIVE: 20.6% on VQAv2 and 9.01% on OKVQA. This performance drop aligns with the findings in [24] that different Transformer layers play diverse roles in ICL. Also, Eq 5 show that LIVE vectors are layer-specific.Then using a single shared vector fails to capture the distinct information processed at each layer, leading to the observed decline in accuracy.To provide a more detailed comparison with LoRA, we made the following modifications to LoRA. First, based on the loss function formula in Eq. 7 we applied the same loss function to LoRA. As shown in Table 21 , even after modifying the loss function, LoRA's performance remains significantly lower than LIVE on both VQAv2 (49.48% vs. 58.54%) and OKVQA (37.02% vs. 50.08%). Additionally, we fine-tuned LoRA with more parameters, but similarly, increasing the parameter count did not close the performance gap between LoRA and our LIVE, further demonstrating the effectiveness of LIVE's unique design. We evaluated the impact of untrained LIVE by initializing them randomly with a normal distribution and tested on VQAv2 (Table 22 ), revealing that untrained LIVE has similar results as zero-shot, which are largely worse than the trained LIVE. We also visualized the embeddings using t-SNE to see the shift effect in Fig 7 (A) , showing that the shift effect of untrained LIVE is similar to zero-shot ICL, while learned LIVEs align more closely with 32-shot ICL, proving that the training process is important. In previous works [8, 19, 18] , the importance of selecting ICDs for ICL in VQA and other visionlanguage tasks has been demonstrated. To reasonably compare the performance differences between LIVE and ICL, we adopted two methods for selecting ICL samples: RICE [43] , which retrieves samples based on image similarity to the query, and MMICES [58] , which combines both language and image similarity for ICD retrieval. The results comparing LIVE and ICL with different ICD selection strategies are presented in the Table 23 , showing that even with optimized ICD selection, ICL still lags behind LIVE in terms of accuracy.We also tried to use RICE samples to learn LIVE. However, the results were unsatisfactory, with only 57.68% accuracy on VQAv2, while using random samples to train LIVE got 58.54% accuracy. This may be because using RICE samples will make each training input sequence contain abundant individual characteristics, increasing the difficulty for LIVE to learn common task knowledge, and even causing LIVE to learn spurious correlations between the similar ICDs and the query.NeurIPS Paper Checklist• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Corresponding author. 38th Conference on Neural Information Processing Systems (NeurIPS2024).For detailed hardware information, refer to Appendix. 


Decision: The paper introduces Learnable In-Context Vector (L-ICV), an approach that enhances in-context learning for multimodal models, specifically in Visual Question Answering (VQA) tasks. Reviewers commend the method for its efficiency and the way it balances performance with reduced computational costs, achieving results comparable to or better than LoRA, particularly on VQAv2 and OKVQA datasets (reviewers 1, 2). The paper is noted for its clear and concise presentation, with strong theoretical support and informative visualizations that help to distinguish L-ICV from other techniques (reviewers 1, 3). However, some concerns were expressed regarding the limited range of evaluations, as the method was primarily tested on one model and two datasets, which may affect its generalizability to other tasks and models (reviewers 3, 4). Additionally, reviewers suggested that the paper could benefit from a broader comparison with other in-context learning methods and a more detailed exploration of the trade-offs between the additional training effort and the savings in inference time (reviewers 2, 4). Overall, the reviewers agree that the paper provides a valuable contribution to the field, justifying its acceptance despite the identified limitations.

Reviews: review: paper_summary: paper_summary: This paper introduces learnable in-context vector (L-ICV) that extends the standard ICV method by training it on task samples. The method achieves performance benefits similar to ICV and the training methodology enables the L-ICV to outperform while still preserving the benefits of few-shot in-context learning. This is achieved through the use of a KL-loss term, that preserves the behavior rather close to ICV, but allowing adaptation as necessary. 

The evaluation of the method is done on OKVQA and VQAv2 datasets. For both of datasets the performance of L-ICV outperforms that of LoRA fine-tuning. Authors provide an explanation and namely that due to need to access external knowledge, preserving the in-context learning behavior is advantageous compared to performing LoRA fine-tuning that disrupts the behavior, particularly when using few samples.
strength_weakness: strength_weakness: There are a few weaknesses, when contrasting it in-depth with methods focused on ICL for visual-language models.

1) Ignoring some of the recent work on interpretation of in-context learning abilities (whether few-shot or zero-shot). It has been discussed as to whether the learning process performs task recognition (superficial format recognition, for example) or task learning (learning to map inputs to outputs) [[1]](https://arxiv.org/abs/2305.09731). The concept was extended to VLMs, by extending it to visual task recognition, studied particularly on VQA [[2]](https://arxiv.org/abs/2312.01571). It seems that VLMs suffer from image specific when doing ICL, such as short-cut inference (ignoring the image similarity, picking on similar question-answers) and others.

2) The performance of LoRA is intriguing. Although I understand the motivation to completely treat it as baseline and naively apply on the same amount of trainable parameters, I find the comparison forcefully made to fit the story of the paper. The choice of tasks that require external knowledge make it obvious that LoRA can't be a good fit. Here it would be fair to either add another task with knowledge contained or alternatively modify on what LoRA is applied (e.g. synthetic 32-shot ICL examples) to contrast on fair grounds with L-ICV.

[1] https://arxiv.org/abs/2305.09731
[2] https://arxiv.org/abs/2312.01571
questions: questions: Questions directly map to weaknesses discussed at previous point.

1) Could authors ground the work in this line of interpretation of in-context learning and explain how L-ICV improves upon task recognition or task learning angles?

2) Could the authors extend their analysis to such a task (e.g. OcrVQA, DocVQA, ChartQA, or InfoVQA)? Alternatively, why is comparing naively LoRA with L-ICV tuned to the output behavior of few-shot in-context demonstrations a fair comparison at all?

I'm willing to raise my score if the authors help me better understand these angles.
limitations: limitations: Yes.
paper_summary: paper_summary: The study introduces a Learnable In-Context Vector (L-ICV) for Visual Question Answering (VQA) tasks. This approach extracts task information from demonstrations to improve the performance of Large Multimodal Models (LMMs) while reducing computational costs.
strength_weakness: strength_weakness: please refer to the questions.
questions: questions: * Is it worthwhile to increase the training cost of the extra in-context vector for the sake of reduced inference cost? I am curious about the trade-off between reduced inference time and additional training time.
* How much would performance drop if each layer shared the same trained bias vector?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes a method by introducing the idea of In-Context Vector in the field of NLP, by introducing L-ICV with a smaller number of parameters in LMM to replace in-context demonstrations to introduce external knowledge, and obtain through training The appropriate shift direction is combined in the query. By comparing with some existing methods, the author proves that L-ICV can significantly reduce computational costs while avoiding the bias in the selection of in-context demonstrations and improving the accuracy of VQA tasks.
strength_weakness: strength_weakness: (1) In the comparison of Section 4.2.1, only the inference time consumption of L-ICV and k-shot ICL is listed. How about other methods like TV, FV, etc.?

(2) The method's generalizability is questionable as it has only been tested on the IDEFICS model, with no consideration of other models.

(3) The paper only proves the effect of L-ICV on VQA tasks, which cannot be extended to other task scenarios and has limited application scope.

(4) In the ablation study, there is a lack of a control group using untrained L-ICV to prove the effectiveness of shift direction.
questions: questions: (1) Why 32-shot demonstrations are chosen in L-ICV training? According to Table 3. As the Number of Demonstrations k increases, the performance gap between L-ICV and ICL gradually narrows. When k continues to increase, will the performance of ICL exceed that of L-ICV?

(2) Why is there a need for a unique shift effect for each layer in the L-ICV method, but in the comparative experiment of LoRA, only the LoRA module is added to the token classification head of the last layer?

(3) In the training stage of L-ICV, the k demonstrations used for training are also randomly selected. How to prove that this can eliminate one of the major challenges when applying ICL: the performance is sensitive to the selection of ICDs.
limitations: limitations: The authors have adequately addressed the limitations in the PaperChecklist.
paper_summary: paper_summary: This paper aims to improving the in-context learning performance of multimodal models on VQA tasks. The paper proposes a Learnable In-Context Vector (L-ICV) method to distill essential task information from demonstrations into a single vector, reducing computational costs and enhancing accuracy in VQA tasks. Experiments show that L-ICV can achieve better ICL performance compared to random 32-shot demonstrations and significantly reduce the computation cost.
strength_weakness: strength_weakness: One of the major weaknesses is that the evaluation of L-ICV is limited to a small set of models and datasets. Only one model (IDEFICS-9B) and two datasets (VQAv2, OKVQA) are reported across the paper.  This limitation raises questions about the generalizability of the method to other multimodal models and different dataset configurations. Besides, recent literature has discussed the in-context limitation of these datasets [1, 2, 6, 7] and a broader comparison of tasks that rely more on visual in-context learning ability should be beneficial to showcase the advantage of L-ICV.

The second major issue is the relatively limited increase of the ICL performance. Although the L-ICV method demonstrates improved efficiency and performance, the reported increase in accuracy is not particularly significant. while L-ICV has been trained on a large dataset, the expected performance gains are not as high as one might anticipate given the extensive training resources utilized. 

Additionally, the paper only compares L-ICV with ICL using randomly chosen demos and does not compare L-ICV with the classical RICE (Retrieval-based In-Context Examples) method [3, 4, 5] or a recently proposed MMICES [6]. RICES and MMICES samples can lead to a direct ICL performance boost, and a direct comparison would help to position L-ICV within the current state-of-the-art. Besides, if L-ICV is trained using RICES samples rather than randomly sampled demos, will the performance be better? This should be an interesting exploration that should be conducted. 

Furthermore, the in-context learning ability of large models is practical and useful as it does not need any parameter optimizations and only requires a handful of demonstrations to quickly adapt to new tasks. To enhance this ability, it would be better if the proposed method also requires minimal dataset, tuning, and optimization effort to achieve similar or better performance.

Overall, while L-ICV presents a valuable exploration of the field, addressing these weaknesses through broader evaluations, detailed statistical analyses, comprehensive comparisons, and further optimization efforts would enhance the method's credibility and impact.

[1] Zong, Yongshuo, Ondrej Bohdal, and Timothy Hospedales. "VL-ICL Bench: The Devil in the Details of Benchmarking Multimodal In-Context Learning." *arXiv preprint arXiv:2403.13164* (2024).

[2] Doveh S, Perek S, Mirza M J, et al. Towards multimodal in-context learning for vision & language models[J]. arXiv preprint arXiv:2403.12736, 2024.

[3] Alayrac, Jean-Baptiste, et al. "Flamingo: a visual language model for few-shot learning." *Advances in neural information processing systems* 35 (2022): 23716-23736.

[4] Awadalla, et al. "Openflamingo: An open-source framework for training large autoregressive vision-language models." *arXiv preprint arXiv:2308.01390* (2023).

[5] Yang, et al. "An empirical study of gpt-3 for few-shot knowledge-based vqa." *Proceedings of the AAAI conference on artificial intelligence*. Vol. 36. No. 3. 2022.

[6] Chen, et al. "Understanding and Improving In-Context Learning on Vision-language Models." *arXiv preprint arXiv:2311.18021* 1.2 (2023).

[7] Baldassini, Folco Bertini, et al. "What Makes Multimodal In-Context Learning Work?." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.
questions: questions: Please see the Weaknesses section.
limitations: limitations: The limitations of this paper are threefold. 

1. The experiments are relatively limited regarding the model types and evaluation tasks.
2. The proposed method requires extra training, optimization, and hyperparameter tuning effort. This added complexity could be a barrier to adoption, especially when there is only a limited set of demo data. 
3. The increased performance is not significant, especially considering the training and tuning effort. This raises questions about the practical benefits of the proposed method relative to the effort involved in its implementation.

Comments: title: Two more questions
comment: Thank you for the insightful comments and constructive suggestions you provided during the review process. Your expertise and attention to detail have significantly enhanced the quality of our manuscript. We greatly appreciate the time and effort you invested in helping us improve our work.
Dear Authors,

Thanks again for the newly added experiments and analysis. It is good to see the L-ICV can also attain comparable or slightly better performance even with a small set of training samples. Also thanks for analyzing the transferability capacity of the learned context vectors. It is recommended that the authors add these two parts to the paper for a more thorough version. 

I am glad the authors have addressed most of my concerns and I am happy to increase my score by 1. 

Best
We appreciate your response. We conducted additional experiments to show how L-ICV performs with a few demonstrations and to explore its transferability. We also include more discussion of these questions.

**Q1: A few demonstrations to train L-ICV**.

L-ICV only uses 800 samples (the total number of samples in the training split) in the VL-ICL Clevr and Textocr tasks. 

To further explore how much data L-ICV requires to be effective for those tasks, we conducted additional experiments. Theoretically, even with a small number of ICD samples, they can generate a large number of ICD sequences. For example, there are  $A_{300}^{32}$ ICD sequences for 300 data samples. Therefore, by extending the training steps of L-ICV, the model can learn from more randomly arranged samples. We trained for 5 additional epochs on the original basis.
In the TextOcr task, only 50 samples were needed to achieve **an accuracy of 27.5% (3.5% higher than 32-shot ICL)**. In the Clevr task, 200 samples were required to achieve **an accuracy of 34.5% (1% higher than 32-shot ICL)**．Additionally, we conducted further tests on VQAv2. Due to time constraints, we used 4-shot training on 300 samples for L-ICV. After 15 epochs, L-ICV achieved **an accuracy of 53.75% (0.23% higher than 4-shot ICL)**. These results indicate that L-ICV can be effective even with a relatively small amount of data. Given time, as we continue to increase the number of ICD sequences and training epochs, we may get satisfactory L-ICVs with fewer samples.

**Q2: The transferability for L-ICV.**

We followed your good suggestion to evaluate the transfer performance of L-ICV and ICL.  Specifically, for ICL, we used VQAv2/OKVQA samples as ICDs during inference when testing on OKVQA/VQAv2 (VQAv2->OKVQA/OKVQA->VQAv2). Similarly, for L-ICV, we used L-ICV trained on VQAv2/OKVQA for inference on OKVQA/VQAv2 (VQAv2->OKVQA/OKVQA->VQAv2). The results are shown in the table below:

| Methods | VQAv2->VQAv2 | OKVQA->OKVQA | OKVQA->VQAv2 | VQAv2->OKVQA |
|---------|--------------|--------------|--------------|--------------|
| ICL     | 56.18        | 48.48        | 50.45        | 45.02        |
| L-ICV   | **58.54**    | **50.08**    | **51.73**    | **45.98**    |


It can be seen that L-ICV achieves better transferability compared to ICL. For example, in the OKVQA->VQAv2 experiment, L-ICV outperformed ICL by 1.28%, and in the VQAv2->OKVQA experiment, L-ICV outperformed ICL by 0.96%. One possible reason is that when using ICL, the individual characteristics of the ICDs from another task may have more damage than the general task summary of another task, which is captured by L-ICV. This demonstrates that the L-ICV we trained can be directly transferred to similar tasks while maintaining relatively good performance. It can also be found that the transferred L-ICV is worse than the original L-ICV, e.g., OKVQA->VQAv2 L-ICV is worse than VQAv2->VQAv2  L-ICV. This aligns with intuition, because when using samples from a task to train L-ICV, L-ICV will learn the Task Learning ability (refer **Q1&W1 of R.VKDo**) and learn the common knowledge of the task, which may have a distribution shift with anohter task and lead to the performance decline in other task. 

To compensate this effect, we further propose a general ICV, it enables us to combine knowledge from multiple datasets. As shown in Appendix C, Table 16, we simply evaluated that by using the mean vector of L-ICV trained on VQAv2 and OKVQA, and it can achieve results on both datasets that are comparable to 32-shot ICL. We believe this could be a promising direction for future research, as a general ICV learned from similar tasks could effectively address scenarios where a new task has only a few demonstrations and no large annotated dataset.

However, it is out of scope of the discussion of this paper and we hope to leave it as future work.
Thank you for the feedback. 

I understand that the main goal of this paper is to accelerate inference time. I agree that learning ICVs and retrieving ICDs based on a query can be orthogonal. I also appreciate the newly added experimental results and findings during training. I have two further questions below. 

In a realistic setting, where a new task has only a few demonstrations and no large annotated dataset for training, how can LICV be better utilized? I believe this is more practical compared to VQAv2 with 8000 samples in applications. Also, how many images are used as training data for the VL-ICL Clevr and Textocr tasks? 

Additionally, I am curious about the transferability of the context vector trained on one task's data to another relatively similar dataset, such as being trained on VQAv2 but tested on OKVQA. Do you have the experimental results and analysis regarding this matter?