M²IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineering
Authors: Yanshu Li
Yi Cao
Hongyang He
Qisen Cheng
Xiang Fu
Xi Xiao
Tianyang Wang
Ruixiang Tang
Venue: COLM 2025 (2025)
Keywords: Large Vision-Language Model
In-context Learning
Representation Engineering
TL;DR: We propose a novel approach of vectorized multimodal in-context learning, improving both efficiency and performance effectively.

Abstract: Multimodal in-context learning (ICL) equips Large Vision-language Models (LVLMs) with the ability to adapt to new tasks via multiple user-provided demonstrations, without requiring any model parameter updates. However, its effectiveness is constrained by the token-intensive nature of multimodal inputs and the complexity of cross-modal few-shot reasoning, which together hinder LVLMs from extracting useful patterns from demonstrations. To address these challenges, we propose \textbf{M²IV}, a novel representation engineering approach that replaces explicit token-level demonstrations with a set of learnable Multimodal In-context Vectors directly injected into the residual streams of LVLMs. By analyzing the distinct roles of multi-head attention (MHA) and multi-layer perceptrons (MLP) in the ICL process, we design a training strategy that enables M²IV to perform fine-grained semantic distillation and robust cross-modal representation learning. M²IV not only improves performance across diverse tasks and LVLMs but also significantly reduces token overhead, enabling graceful scaling to many-shot scenarios. To further enhance usability, we introduce \textbf{VLibrary}, a repository that stores trained M²IVs for flexible retrieval and injection. With VLibrary, users can steer pre-trained LVLMs in a customized manner that meets diverse requirements. Extensive experiments demonstrate that M²IV consistently outperforms vanilla ICL and prior representation engineering baselines, achieving an average accuracy gain of 3.74\% with substantial improvements in overall efficiency.

Published as a conference paper at COLM 2025
MFIV: Towards Efficient and Fine-grained Multimodal
In-Context Learning via Representation Engineering
Yanshu Lif Yi Cao Hongyang He²
Qisen Cheng*
Xiang Ful
Xi Xiao
Tianyang Wang Ruixiang Tang"t
BBronn University Soochow University University of Warwick *Samsung US 'Botonn University University of Alabama at Birmingham Rutgers University
Abstract
Multimodal in-context learning (ICL.) equips Large Vision-language Models (LVLMs) with the ability to adapt to new tasks via multiple user-provided demonstrations, without requiring any model parameter updates. How- ever, its effectiveness is constrained by the token-intensive nature of multi- modal inputs and the complexity of cross-modal few-shot reasoning, which together hinder LVLMs from extracting useful patterns from demonstra- tions. To address these challenges, we propose MIIV, novel representa- tion engineering approach that replaces explicit token-level demonstrations with a set of learnable Multimodal In-context Vectors directly injected into the residual streams of LVSMs. By analyzing the distinct roles of multi-head attention (MHA) and multi-layer perceptrons (MLP) in the ICL process, we design training strategy that enables MIV to perform fine-grained semantic distillation and robust cross mrosll representation learning. MIV not only improves performance across diverse tasks and LVLMs but also significantly reduces token overhead, enabling graceful scaling to many- shot scenarios. To further enhance usability, we introduce VLibrary, a repository that stores trained MIIVs for flexible retrieval and injection. With VLibrary, users can steer pre-trained LVLMss customized manner that meets diverse requirements. Extensive experiments demonstrate that MTV consistently outperforms vanilla ICL and prior representation engineering baselines, achieving an average accuracy gain of 3.74% with substantial improvements in overall efficiency.
1 Introduction
Large Vision-language Models (LVLMs) unify visual and textual modalities within the representation space of their Large Language Model (LLM) backbones, thereby gaining advanced multimodal understanding and generation capabilities (Zhou et al., 2022; Zhang et al., 2024b; Laurençon et al., 2025). They are being deployed in an ever-growing array of vision-language (VL) applications (Hartsock & Rasool, 2024; Xu et al., 2024). As task complexity rises, effectively and efficiently guiding LVLMs to adapt to new tasks becomes increasingly important. In-context learning (ICL) offers promising solution that allows models to quickly learn from demonstrations directly inserted into the prompts, without updating any parameters (Brown et al., 2020; Dong et al., 2022).
Despite recent progress in applying multimodal ICL to tasks such as image captioning and classification (Yang et al., 2023; Huang et al., 2024), extending it to more complex or knowledge-intensee
tasks remains a fundamental challenge (Li et al., 2024b). This difficulty stems from two key limitations. First, the token-heavy nature of interleaved image-text input makes it hard to incorporate external knowledge efficiently. A common four- or eight-shot prompt introduces far greater inference latency than a single-image input. When more supporting knowledge is needed (Xia et al., 2024), users may have to split the prompt
"Equal contributions.
I 'Corresponding author: muxxing.tang@rutgers..dd
1

---

Published as a conference paper at COLM 2025
Query Sample
is cheese coke.
Corror cake.
What cake served?
MEIV
LYLM
Layer
Layer
Layer
Knowledge Base
Retriever
Layer
Layer
Projector
Vaion Encoder Projector
What are the silver wrappers front Harzhay's.
is unprepore eart Rice.
n-shot demonstrations
What kind cake has been served?
been
served?
Demo.:
(a) Vanilla ICL
(b) McIV
Figure 1: (a) In Vanilla ICL, for given query sample, retrieve instances from knowledge base as demonstrations and feed them together with the LVLM. (b) In contrast, we train set of vectors emulating n-shot ICL using the same knowledge base and inject these vectors into the LVLM, greatly reducing token consumption and boosting model performance.
the query sample into
into several segments, as most LVLMs still provide limited context windows. This further increases latency and can trigger catastrophic forgetting (He et al., 2024). Second, ICL can be surprisingly unstable due to its high sensitivity to few-shot prompt designs. Even ofte--of-the-rt (SOTA) models are found to exhibit substantial performance fluctuations depending on the format, content, and order of the demonstrations provided (DeepSeek-Al et al., 2025). These challenges are further amplified in multimodal scenarios, where aligning visual and textual features adds another layer of complexity (Lu et al., 2021; Qin et al., 2025).
To address these issues, representation engineering has emerged et al., 2025a). This line of work views ICL as shifts applied to the model's 's internal activations (Merullo et al., 2023). It extracts these ICL-induced shifts as vectors and reinjects them during zero-shot inference to implicitly encode task-specific guidance. For instance, Hendel et al. (2023) derive task vectors from the hidden state of the context's last token, while Todd al. (2024) compute function vectors by averaging the outputs of key attention heads. Although these methods perform well on simple tasks with clear mappings, they fall short on complex multimodal tasks like visual question answering (VQA). Consequently, Peng et al. (2025) propose training in-context vectors to capture richer features. However, their approach still suffers from rapid degradation as complexity increases. This exposes shared limitation: an inability to effectively model intricate interactions multimodal ICL. To overcome this shortcoming, two key challenges must be tackled simultaneously: I. identifying the critical task mappings within individual demonstrations and the interdependencies among different demonstrations, and II. achieving fine-grained semantic distillation of the complex multimodal information.
feasible direction (Chen
Building these insights, we introduce MTIV, novel method that harnesses the distinct roles of multi-head attention (MHA) and multi-layer perceptrons (MLP) in multimodal ICL. MrV assigns set of learnable vectors to the MHA branch and another set to the MLP branch at each decoder layer, as illustrated in Figure 1(b). We design a dedicated training strategy that enables these vectors to absorb the deep semantic patterns captured by MHA in the demonstrations and to simulate the distilled information produced by MLP. The experiments show that MTV achieves SOTA performance in 18 21 experiments conducted on three LVLMs and seven benchmarks, while utilizing only about 24% of the total training data used by LIVE. Furthermore, MIIV delivers an efficiency breakthrough as the reduction in inference time fully compensates for the one-time training cost and yields even greater cost-effectiveness as adoption scales. To further explore the potential of MIIV, we present VLibrary, container that stores trained MrIVs and supports on-demand retrieval for plug- and-play use. VLibrary can be seamlessly incorporated into real-world systems and applied to address critical challenges in the LVLM domain.
2

---

Published as a conference paper at COLM 2025
Our main contributions are: (1) We analyze the unique roles of MHA and MLP in multimodal ICL, revealing how MHA drives semantic integration and MLP refines fine-grained details. (2) We propose MPIV, which simultaneously achieves complex semantic understanding and fine-grained semantic distillation, demonstrating superior performance across three LVLMss and seven diverse benchmarks. (3) We introduce VLibrary and use to address three critical challenges: cross-modal alignment, output customization, and safety, thereby offering novel and promising pathway for future LVLM research.
2 Background and Related Works
ICL emerges as a pivotal capability as language models scale (Radford et al., 2019; Garg et al., 2022) and has since been extended to the multimodal domain. Several LVLMs are specifically pretrained or fine-tuned to attain multimodal ICL, capability. Among the most notable examples are OpenFlamingo (Awadalla et al., 2023) and IDEFICS (Laurençon al., 2024), which serve as key components of our work. With demand for multimodal ICL steadily increasing, supporting it within fixed context windows has become a critical feature of modern LVLMs, such as Qwen2.5VL (Bai et al., 2025) and Grok3 (xAI, 2025).
In LVLMs, ICL is defined as the process in which a pretrained model M is provided with prompt containing a multimodal context C (absent in zero-shot settings) and a query sample Q. To generate the answer A, the model computes the conditional probability:
PM (AIQ,C) C)
(1)
via a feed-forward pass'. Cincudee few-shot demonstrations, as shown in Figure (a).
With the prevalence of ICL, re- searchers have been investigating its underlying mechanisms (Xie et al., 2021; Dai et al., 2023), They point out that key attention dy-
Tokyo
LLM
Berlin
Japar
China-Baij Beis-BBjjgg.... Fronce- Germany
Ch,
Coptial!
Othiow
cancdo
namics, most notably skill neu- rons and induction heads (Wang et al., 2022), play crucial role in its success. These insights have driven advances in representation engineering for ICL. Firstly, Hen- del et al. (2023) and Todd et al. (2024) nearly simultaneously pro- pose Task Vector (TV) and Func- tion Vector (FV). Building on this, Liu et al. (2024c) obtain layer-wise In-Context Vectors (ICVs) by comparing the hidden states of the final tokens in the original and target sequences, then applying PCA distill task-relevant information. Expanding ICV further, Li et al. (2025) propose Implicit ICL (I2CL), which extracts residual stream deltas at each demonstration's 's last token position across layers and utilizes a set of coeffi- cients to regulate their injection. As noted in $1 and illustrated in Figure 2, these methods often underperform in multimodal ICL. Peng et al. (2025) attribute this gap to the static extraction-injectoo paradigm and propose attention shift-based training method, LIVE, yet it still neglects the essential role of fine-grained semantic distillation.
(9)
LVDN
L.LM
X
Wall: whee''s the tork! Councing reengnisont
LJHSH bananas are there the image? A:
C moryy
QrAre they dorcing brown flcor?
standing?
Figure 2: Overview of applying Function Vector to (a) the Country-Capita task and (b) the VQA task, which requires complex reasoning. While it performs well in (a), it falls short of fully representing the context (b).
3 Methodology
In this section, we detail the proposed MIIV framework by analyzing the roles of MHA and MLP LVLM ICL, which motivates our definition of the a parameters ($3.1). We then
present the complete training strategy ($3.2). Finally, we introduce VLibrary, the storage for MPIVs (63.3). The overview pipeline is illustrated in Figure 3.
We refer to this process as Vanilla ICI. In this work, we focus on image-to-text tasks.
3

---

Published as a conference paper at COLM 2025
3.1 Anchoring M
For an L-layer LVLM M processing an input sequence of length 1, the residual stream architecture is recursively defined as follows, with / E {1,22.... and i (1,2,...,11
a MHA
hh/ (h/_1:0))
(2)
m MLP h a Wi).
(3)
h h + a/ + m/)
(4)
where h/ E RdM denotes the residual stream at layer I and position i, with hidden dimension d.M. The functions MHA (+;0)) : .R + Rd and MLP (:; W,) Rd - RdM denote the MHA and MLP transformations at layer 1, parameterized by 8 and W,, respectively.
Previous research (Chang et al., 2024; Li et al., 2025) has demonstrated that MHA and MLP branches play distinct roles in ICL. Based on these insights, we infer that in multimodal ICI, MHA dynamically allocates attention to capture both the internal semantics within demonstrations and the interactions among them. MLP further extracts, filters, and stores key information in a fine-grained manner, conveying more aggregated yet nuanced features. To validate our inferences, we explore their computational invariance for demonstration matrix C E RCxdM with C tokens, and two invariance properties are established.
Theorem 1 (MHA Computational Invariance). There exists Y : Dy RdM such that, for any query matrix Q E R RIxdM and residual stream h' E RddM, there exist J), of IR satisfying:
Attn hh' [[T [cT eT]") = = -' C... C.C. + +ny
Attn (H'.Q.0).
(5)
where Attn (-) is a function denoting the self-attention mechanism. The query matrix Q is Q concat (Enelmg (I), Enc (Q)), with I and Q being the query sample's image and question. Here, Encing (-) and Enc (-) denote the encoding functions for visual and textual features, respectively.
Remark. Theorem shows that the self-attention mechanism decomposes into two parts: query-only component, Attn h', Q Q), and a context-augmented component, Yhh,, C, C). This decomposition highlights MHA's role in dynamically allocating attentionn, allowing M to integrate query-specific focus with contextual insights from demonstrations in ICL.
Theorem 2 (MLP Computational Invariance). For linear transformation matrix W E Rddxdd, there exists vow R + Rd such that,for any token position i, there exist 3¹, 17 E R satisfying:
MLP (acoo) = -7' - 1ow (at) + + -MLP (26).
(6)
where MLP (.) represents the MLP mechanism, and ab,cc ab,, and alc are the MHA outputs at the i-th token position with both query and context, with query-only, and with context-only, respectively.
Remark. Theorem 2 shows that the MLP mechanism decomposes into two parts: query-only component, MLP(aa).. and context-enhanced component, 4hw (aE). Through weighted combinations, MLP extracts and retains key features from both query and context, enabling M to convey more aggregated yet nuanced representations in ICL.
Theorems and 2, with proofs provided in Appendices A.1 and A.2, illustrate the dual processing pathways inherent in multimodal ICL. This also offers a new perspective the distinct roles of each layer in ICL, supplementing Wang et al. (2023): shallow layers primarily aggregate information within the multimodal context C; intermediate layers rely more on MHA to capture deeper semantic details; and deep layers tend to refine and integrate prior information via MLP. These insights drive our layer-wise design of MIIV.
Based on the analysis presented above, we propose MIIV for the fine-grained representation of multimodal ICL. MPIV assigns a learnable vector and weight factor to both MHA and MLP branches at each layer of an LVLM. Specifically, we define:
MHA: V
{v1, v2
.....,, v v..), a {af, ag ag....,,f.)
(7)
MLP: T {v] v v
... v }", {aft a² ...., a
(8)
4

---

Published as a conference paper at COLM 2025
Training Framowork
P(C.Q)
P(Q:8)
A
Grouns:
MPV
Training
Teather
Studeet
Stoage
Mnoelooge
Layer
Layer L
d
MLP my
Layer
Layer
V Viibrary
Layer
Processing
Vision Enceder & Projector
& Projector
Resicual Stream
Redileue
Inject
Do
Sumple
Query
Query
L - Anain foar
Aeys beye
Aayy
Long
LVEM
Sample
Figure 3: Pipeline of MFV's training and storage. We begin by constructing the training sets Do and De. from an existing knowledge base. Next, self-distillation framework employing three losses (Lmim, Lsyn and Lsup) is used to train MPV. Finally, the trained MIV stored VLibrary for on-demand retrieval.
Here, vf, v/" RRMM and at, R. The complete of M-IV is represented by + {al, vf, vi') J1=1 which can be injected directly into the LVLM's residual streams. The updated residual stream is recursively defined for (1,2,..../ .} and i {1,2, ..., as:
hf h + (a) + a9 v$) + m + am" vv."
(9)
3.2 Training MPIV
Given a dataset D = [14 (j. A))){( used for ICL, we aim to train MAV to capture the effect of providing any instances from D contexts under specific retrieval strategy R.
We first process D as follows: (1) For each instance (4,, 4)) E D, we separately embed its I and Q; with CLIP model and then concatenate them to form a joint embedding as its semantic representation. (2) We cluster all joint embeddings via k-means with cosine similarity, partitioning D into K clusters. From each cluster, we select the instance closest to the centroid (after removing its answer) to construct the query sample set Do (size K); D Doo becomes the support set. (3) For each query sample in Do, we apply R to retrieve instances from the support set, forming an n-shot context. These contexts constitute the context set Dc. We augment De by creating copy of each context with its demonstrations randomly shuffled and adding it. Further details are provided in Appendix B.
After obtaining the training data, employ self-distillation framework to train MPIV. The teacher model processes each n-shot context in Dc with its corresponding query sample in Do to perform Vanilla ICI. The student model is injected with the initialized O and receives only the same query sample as input. Based on this, we introduce mimicry loss:
Lmii = IDRL (PI(C).) PM((:0)). CE Dc, DED
(10)
where DDzz (- -I is the KL divergence. We apply temperature scaling with parameter T to PM (C, to facilitate smooth knowledge distillation and mitigate overconfidence.
Beyond distributional alignment, we seek to capitalize on the synergy between the MHA and MLP branches. Thus, we apply synergistic loss the student model, which is designed to fortify their coherence and complementarity. Formally, we define it as follows:
Euyo = Σ
Σ
M
2 +
M
(11)
21,, Qj and Aj denote the image, question and answer of the j-th instance in D, respectively.

---

Published as a conference paper at COLM 2025
where M = (0)) (0) E R is cross-view correlation matrix. (9) and 2 (0) denote the normalized output of MHA and MLP at layer after injecting θ respec- tively. M denotes the element at the i-th row and j-th column of M'. The hyperparameter balances consistency within dimensions and orthogonality across dimensions 3.
Finally, we employ standard cross-entropy loss function supervised loss, ensuring that the student model's predictions remain faithful to the answer A = {Ai, A A Ar]:
Saup
Σ log PM (A+ QAC,H) E
(12)
The final training objective combines these losses weighted sum:
C Amim m.mim + Asyn Luyn + Asup' Liupp,
(13)
where Amimo Asyn, and Asup E (0, denote the hyperparameters that balance each loss item.
By fully leveraging the MLP's semantic aggregation and storage capabilities, we can extend the MiV self-distillation framework to many-shot ICL scenarios, overcoming the context window limitations of LVLM. Here, each context in Dc contains more than 100 demonstra- tions, which are partitioned into overlapping windows of length w with overlap o. Each subcontext is processed individually by the teacher model, and the outputs of its MLP branches are extracted as the semantic representation for that window. These representa- tions are then sequentially aggregated in a pairwise manner until final, comprehensive representation is obtained that encapsulates the semantic information of the entire n--hot context and serves as the teacher model's final MLP state. The detailed processing pipeline for many-shot MIIV is presented in Appendix D.
3.3 VLibrary: MPIV Storage
Using the above training strategy, we obtain OP for each dataset D and LVLM M. To facilitate management and fully exploit MIV's plug-and-play potential, we build VLi- brary-a repository for storing learned vectors after training. Each MFIV is indexed by its a². When wishing to equip an LVLM with domain-specific knowledge or steer it toward a desired generation pattern, index and inject it into the model according to Eq 9. VLibrary is designed solely for storing and indexing the trained vectors and performs no inter-vector operations. As result, the structural requirements of VLibrary remain minimal and consistent across LVLMs with different hidden state dimensions. We can implement VLibrary minimal cost, and its practical details are provided in Appendix E. By translating the gains of MIIV into practical utility, VLibrary constitutes an integral part of the overall MTV framework.
can retrieve the corresponding MTV from VLibrary by its
Experiment
4.1 Implementation Details
Benchmarks. For VQA, we select three widely used datasets: VQAv2 (Goyal et al., 2017), VizWiz (Gurari et al., 2018), and OK-VQA (Marino et al., 2019). Towards more complex VL scenarios, we incorporate A-OKVQA (Schwenk et al., 2022) and GQA (Hudson & Manning, 2019), which emphasize multi-hop reasoning (Kil et al., 2024). Additionally, we include the Asia split of the multicultural VQA benchmark CVQA (Romero et al., 2024) to assess the ability to integrate novel in-domain knowledge into pretrained models. For an evaluation of the general ICL,, we utilize the image-to-text split of the latest multimodal ICL benchmark, VL-ICL bench (Zong et al., 2024). To better serve as inputs for ICI,, we make necessary modifications to these benchmarks; details are provided in Appendix F.
Configurations. We evaluate three LVLMs: OpenFlamingov2 (9B), Idefics2 (8B), and LLaVA- NeXT (7B) (Liu al., 2024a). These models differ in their LLM backbones, connection
'The detailed computation process is presented in Appendix C.

---

Published as a conference paper at COLM 2025
Methods
VQAv2
VizWiz
OK-VQA
GQA
A-OKVQA
CVQA
VL-ICL bench
Zero-shot
43.10
20.77
31.04
52.42
32.97
31.09
11.39
Vanilla ICL
60.08
39.88
51.37
69.70
50.92
56.39
29.08
47.88
23.86
38.24
60.26
37.58
43.02
18.42
FV
47.09
23.40
38.96
59.33
38.43
41.82
18.32
ICV
44.71
20.83
35.98
54,80
32.58
38.45
11.92
12CL
52.02
26.16
44.13
61.93
39.10
49.42
20.48
LIVE
62.22
39.17
53.47
69.20
51.60
55.42
29.95
МРV
63.93
43.40
55.37
73.81
53.59
60.11
33.36
Multi-turn ICL (128-shot)
59.60
38.96
52.61
66.76
51.25
58.32
MPI (128-shot)
65.19
45.13
56.58
73.58
53.36
62.67
Multi-turn ICL (256-shot)
59.73
38.76
52.37
67.14
51.32
59.59
MPI (256-shot)
65.92
46.33
57.43
74.98
54.77
62.20
Table 1:
Comparison between MIVV and baseline methods. The highest scores are high- lighted in bold and the second-best scores underlined. In addition to the primary 16-shot results, this table also presents the outcomes of many-shot ICL achieved with MFIV. Detailed results for each LVLM can be found in Table 8.
modules, and context windows. Unless otherwise noted, all results are reported as the average across these models. Following $3.2, construct query set Do (of size K) and a context set De (of size 2K) from each benchmark's s training set, with K varying by benchmark. We adopt Random Sampling the retrieval strategy R, and fix the number of shots to 16. During training, we use AdamW as the optimizer. Evaluation is performed on the corresponding validation sets. Initialization of V and a, along with the data sizes and hyperparameters for each benchmark, are detailed in Appendix I.1.
Comparative Methods. In addition to the zero-shot baseline and n-shot Vanilla ICL, we compare MPIV with the representation engineering methods for ICL introduced in $2, including TV, FV, ICV and I2RL. All of these methods are training-free. We highlight LIVE as the key comparison our experiments, since it also employs training strategy obtain layer-wise vectors. Full details of all baselines are provided in Appendix H.
4.2 Main Results
As shown in Table 1, training-free methods outperform the zero-shot baseline but still fall short of 16-shot Vanilla ICL, indicating limited capacity for handling complex multimodal tasks. While LIVE's training strategy yields improved performance, it still underperforms on benchmarks such VizWiz, GQA, and CVQA. In contrast, MIIV achieves the best scores on all benchmarks, surpassing 16-shot Vanilla ICL by an average of 3.74%, and outperforming LIVE by 3.52%, 4.11%, and 3.72% on the benchmarks where LIVE lags behind. As shown in Table 8, our method ranks first in 18 out of 21 experiments across all three LVLMs. These results confirm that MFIV effectively preserves semantic fidelity and further distills fine-grained information, thereby outperforming Vanilla ICL and other comparative methods
variety of complex multimodal ICL tasks. Paired with VLibrary, method enables efficient and precise LVLM steering. Moreover, MPIV requires only about 24% of the data used by LIVE, underscoring its efficiency and potential in data-limited scenarios. In many-shot scenarios, MPIV also yields consistent and significant improvements, demonstrating that the aggregation of MLP outputs counteracts multi-turn forgetting and effectively unlocks the benefits of many-shot ICL.
4.3 Efficiency Analysis
A principal reason for the wide adoption and growing importance of ICL is its efficiency without any parameter update. Thus, we must examine whether MTV undermines this key advantage. First, in Figure 4 we observe that M2IV preserves and even improves the inference efficiency of ICL thanks to its alug-and-play design. Its direct internal injection eliminates the large token overhead introduced by explicit in-prompt demonstrations, thus cutting FLOPs and inference time to levels well below Vanilla ICL and approaching those
116-shot has been empirically shown to approximate optimal performance among the chosen Ms (Liu al., 2024b). It also allows higher-resolution images without exceeding context windows.
7

---

Published as a conference paper at COLM 2025
FLOPs and inference Time Comparisons
Overall Ericknnc
3.57X
8-
4.37X
LITTX
2.1K
0.998 B.8X
1:80
1.4
2019 SMY
MWV
SchetiCL
Valla Fleons
Figure 4: Comparison of MIV, Zero-shot, and æ-shot Vanilla ICL in terms of the total number of FLOPs and inference time used during the entire evaluation process.
Figure 5: Comparison of GPU hours con- sumed by Vanilla ICL for inference across the entire evaluation and by MTV for both training and inference.
seen in zero-shot settings. Beyond inference latency, we also report the overall runtime that includes the training cost of MPIV. This metric can better reflect the true efficiency of the competing methods. As shown in Figure 5, MPV's inference-time savings fully offset its training cost. Moreover, as MIV increasingly applied to zero-shot inference tasks, its cost-effectiveness improves, making it well-suited for mass applications. It is worth noting that when the repeated demonstration-rettiee
cost of Vanilla ICL is taken into account, the advantage of MIIV becomes even more significant, as each knowledge base needs only single end-to-end training run to produce a reusable for the target LVLM. These results collectively demonstrate that, although M*IV involves training phase, it delivers higher efficiency than Vanilla ICI. In Appendix J, we compare MPTV with LoRA and prefix tuning, highlighting its joint benefits in precision and efficiency.
4.4 Ablation Studies and Discussions
In this section, we discuss three primary concerns through extensive ablation experiments.
Why does MIVV not only replicate the effect of Vanilla ICL but also surpass it? We first explore the advantages of MPTV over Vanilla ICL as the shot count varies. As shown in Figure 6, MPIV consistently surpasses Vanilla ICL across all shot settings, highlighting the robustness of its fine-grained representation. Notably, the largest gains occur in 2-shot and 4-shot settings, especially on datasets with imbalanced training distributions such as CVQA and WizWiz. This suggests that MFIV can, through training, effectively capture and internalize the overall distribution of dataset, thereby avoiding the influence of skewed distributions resulting from data scarcity. In many-shot scenarios, the increase in performance gains with higher shot counts due MiV's capability of mitigating the forgetting issue that worsens with additional input turns.
We further explore how the effectiveness of MTT varies with diverse retrieval strategies R. Including Random Sampling (RS), we compare four strategies: 121 (image similarity-based retrieval), IQ2IQ (image-question joint similarity-ban
retrieval) and Oracle (a greedy retrieval performed by the LVLM based on ground-trus a As shown in Figure 7, Vanilla ICL with 12I performs the worst across all benchmarks, echoing previous findings that 12I tends to lead to visual shortcut learning and hallucinations in complex tasks (Li, 2025). Remarkably, MPIV always delivers the largest gains under this strategy, demonstrating its ability to filter out isolated or misleading features while preserving holistic semantics.
Which component contributes most V's improved ICL performance? We first investigate the data aspect in Appendix L and find that the distribution of training data is far more influential than its sheer volume. Here, we conduct more comprehensive ablation studies on the key designs of MPIV. Table reveals the following points: (1) Joint embedding-
DDue to its reliance ground-truth answer, Oracle is used to simulate optimal contexts and inapplicable in real-world scenarios. Details of these strategies are provided Appendix K.
8

---

Published as a conference paper at COLM 2025
Performance co GAP under Diverse Shots
Performance under Diverse Retrieval Strategies
10564
172
Mtelic
RGA
A-DRVOA
CVDA
Number Shots
Voh2
GBA
A-DOY0A
CYDA
100
21
102
Chadlo
Figure 6: Performance gap same shot Vanilla ICL (few-shot) or multi-turn ICL (many-shot) across different shot settings. Each value indicates how much MPIV out- performs the baseline.
Figure 7: Performance of Vanilla ICL and MIIV across diverse demonstration retrieval strategies. The light-colored portion of each bar indicates the gain achieved by MTV.
VQAv2
VizWiz
OK-VQA
GQA
A-OKVQA
CVQA
VL-ICL bench
MPIV
63.93
43.40
55.37
73.81
53.59
60.11
33.36
only
50.48
32.61
52.07
68.26
47.41
43.06
32.80
(b) only
58.72
38,97
53.29
70.62
48.94
59.25
32.72
(c) w/o clustering
51.98
29.74
47.58
66.37
45.36
44.05
30.31
(d) w/o augmentation
55.48
33.28
51.13
69.61
43.23
54.81
30.59
(e) w/o Cyim
56.87
45.79
43.91
51.33
24.39
w/o Leyn
53.13
28.61
47.30
58.92
38.79
43.17
22.63
(g) w/o Coup
61.53
53.90
72.02
50.79
59.83
30.81
Table 2: Accuracy (%) of MPIV under diverse ablation settings. (a)- focus on the data
processing phase: (a) and (b) use only image or question similarity for clustering: (c) replaces the K-means clustering with random sampling; (d) omits shuffled augmentation of Dc- (g) each remove one of the training loss terms.
based clustering improves generalization due to varied modality biases across datasets. (2) Clustering optimizes the distribution of training data while augmenting Dc improves the robustness of MII. (3) Among training losses, Lsup yields only an average gain of 2.15%, while Lyyn delivers an average gain of 13.00% and also outweighs all data processing strategies. Clearly, Layn underpins our method, underscoring the importance of synergizing the MHA and MLP branches to ensure general semantic fidelity and fine-grained filtering.
MIVV seems to be task-specific and model-specific. Does this impact its flexibility? The mathematical properties of vectors allow MiVV to support flexible combination and transfer, with proof provided in Appendix A.4. We propose two strategies for combining multiple MIIVs to endow the model with multi-task capabilities: (1) a training-fres linear addition, and (2) the introduction of learnable parameter ihat fine-tuned using small amount of multi-task data. Thus, by creating set of atomic MPIVs, we can customize combinations to meet diverse needs. Similar strategies can also be applied to facilitate cross-model transfer of MPIV If two LVLMs have the same number of layers and an identical hidden state dimension, an M'IV trained on one model can be inserted into the other without any additional training and will deliver comparable results. Adding learnable parameter for light fine-tuning can further enhance this effect. The detailed procedures and results of the above strategies are collectively presented in Appendix M. They highlight the unique advantage of steering LVLMs' intermediate representations via MPIV.
5 VLibrary: An All-purpose Toolbox for LVLM
In this section, we demonstrate the practical value of VLibrary in solving the key challenges that LVLM faces in real-world applications. As shown in Figure 10, VLibrary empowers us to store and retrieve tailored M-IV, facilitating versatile steering while seamlessly integrating into existing systems. For the subsequent applications, we set K 1250 during training.

---

Published as a conference paper at COLM 2025
VLibrary enhance cross-modal align- ment in a special way. In $3.1, we config- ure MrIV on layer-wise basis, recogniz- ing that each layer contributes differently to ICI.. Alternatively, MIV can be injected only into layers chosen for their functional importance; for instance, targeting shallow layers to reinforce fine-grained cross-modal alignment. To evaluate this, we construct detail-focused datasets by prompting GPT- 4o to expand MSCOCO's (Lin al., 2015)
Methods
VQAv2
ViaWiz
OK-VQA
CVQA
Zero-shot
43.10
20.77
31,04
31,09
+MIV
First 10 layers
46.92
24.20
34.51
36.27
Middle 10 layers
43.29
22.09
32.13
32.41
Last 10 layers
43.82
21.14
30.19
30.79
All layers
45.17
22.05
32.83
32.86
Table 3: Performance on four VQA bench- marks in the zero-shot settings and with MIIV for captioning injected at various positions.
captions into detailed descriptions of each image's visual features, and train MPV on them. We then inject part of the trained MIV into corresponding LVLM layers for zero-shot VQA. As shown in Table 3, injecting MIVV into the first 10 layers yields the greatest performance gains, even surpassing full injection. These findings reveal that MPIS can be flexibly applied to enhance LVLMs' overall capabilities.
VLibrary enables versatile customization of LVLM outputs. Instruction following is vi- tal for LVLMs to align with user intent and facilitate various interactions. However, the added visual modality complicates instruc- tion adherence, requiring extensive param- eter updates to achieve proper alignment. We use LLaVA-Bench to test whether MIIV can steer the model to follow specific user instructions when generating content. The benchmark covers three types of instruction: conversation, detailed description, and complex reasoning (Liu et al., 2023). We train MIV on the LLaVA dataset, which contains the same three instruction types. As shown in Table MIIV consistently enhances instruction-followe its effectiveness in addressing the challenge with minimal overhead. In Appendix N, we also explore using MFIV to enable LVLMs to explicitly output their reasoning process.
Method
Conversation
Detail
Complex
All
Vanilla
56.41
75.23
LoRA
69.24
63.27
83.05
74,09
16-shot MII
72.53
64.95
86.26
76.93
MPV
74.34
66.70
84.09
76.89
Table 4: Instruction following evaluation of LVLM. We employ the same metrics as the original LLaVA-Bench, using GPT-4 to score the generated content.
performance across all types, emphasizing
VLibrary is well-suited for studying LVLM safety. MIIV's strong behavior-steering ca- pability makes it a powerful tool for investigating jailbreak scenarios with LVLMs. By constructing MIV vectors that encode harmful multimodal instructions and injecting them into model, we can compel it to override its moral safeguards and generate disallowed con- tent. In the opposite direction, safety-oriented MTTV vectors strengthen LVLM's s awareness of harmful prompts, allowing it to detect and refuse such requests with greater precision. Experimental results and detailed analyses appear in Appendix N.
6 Conclusion
In this study, we present MIV,, novel representation engineering method for multimodal ICL. It leverages the unique roles of MHA and MLP branches in residual streams. Through training, MTV achieves complex multimodal understanding and fine-grained semantic distillation, demonstrating SOTA performance on three LVLMs and seven benchmarks with relatively limited training data while maintaining ICL's efficiency. The retrieval-then injection design of VLibrary further expands MIIV's applicability, enabling rapid solutions to many practical challenges in LVLM. In general, M'IV offers promising paradigm for both multimodal ICL and LVLM steering, providing valuable insights for further breakthroughs in the multimodal domain. Currently, our method is only applicable to open-source LVL.s. In future work, we hope to extend to closed-source LVLMs, possibly by utilizing the trained MIV' empower a lightweight language model dedicated to demonstration selection.
A Acknowledgments
We sincerely thank Tian Yun and Ellie Pavlick from Brown University for their helpful suggestions on this paper.
10

---

Published as a conference paper at COLM 2025
Ethics Statement
This paper presents examples that contain harmful or offensive language and imagery, primarily drawn from HatefulMemes and MMM-SafetyBench. We unequivocally reject the disrespectful and unethical views expressed in these materials, which appear here only to advance the creation of a fair, unbiased, and safe community for large vision-language models.
References
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training large autoregressive vision-language models. arxiiv preprint vrXit:2308.013990 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https: org/ abs/2502. 13923.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Adoances in neural information processing systems, 33:1877-1901, 2020.
Ting-Yun Chang, Jesse Thomason, and Robin Jia. When parts are greater than sums: Individual Ilm components can outperform full models. arxiv preprint orXio:2406.13133 2024.
Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. Persona vectors: Monitoring and controlling character traits in language models, 2025a. URL https: 21509.
//arxiv.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025b. URL https: / http///axxvvvveeeeeeeeeeeeeee.ve wrgp/iii.ooraaaeeeeeeeeeooeeee abss//bs2220 2412. 05271
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers, 2023. URL https: //axxiv org/ argg/bbs/221... 10559.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L..
11

---

Published as a conference paper at COLM 2025
Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang. Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. Li, Y. Q. Wang, Y. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. /ttpa//iiii../////////11111 2b50 12948.
URL /tattss//aaxiv..... wrw.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv preprint arXiv:2301.0022: 2022.
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30553 2022.
Yash Goyal, Tejas Khot, Douglas Summers- Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6904-6913, 2017.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608-3617, 2018.
Iryna Hartsock and Ghulam Rasool. Vision-language models for medical report generation and visual question answering: A review. Frontiers in Artificial Intelligence, 7:1430984, 2024.
Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, et al. Multi-if: Benchmarking 1lms on multi-turn and multilingual instructions following, arxiv preprint #rXio:2410.15553 2024.
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors, 2023,
URL https://arxiin wttpa/awwww...rr//////////./ 15916.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
URL https::/arxxvv.org/bbb/////
09685.
Hujiang Huang, Yu Xie, Jun Gao, Chuanliu Fan, and Ziqiang Cao. Select and order: Opti- mizing few-shot image classification with in-context learning. In International Conference on Multimedia Modeling, pp. 425-438 Springer, 2024.
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019. URL https://arxxvv.rrgg 09506.
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes, 2021. URL https::/ /ttp::///xxvvvvree wbwp/b//////.....o.
12

---

Published as a conference paper at COLM 2025
Jihyung Kil, Farideh Tavazoee, Dongyeop Kang, and Joo-Kyung Kim. li-mmr: Identifying and improving multi-modal multi-hop reasoning in visual question answering. arViv preprint ar Xio:2402.11058, 2024.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36, 2024.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? Advances in Neural Information Processing Systems, 37: 17874-87907, 2025.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. alava-onevision. Easy visual task transfer, 2024a. URL http:ps///aaxx...gr/ab/////////
03326.
Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, and Xu Yang. How to configure good in- context sequence for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 16710-26722 2024b.
Xiang Lisa Li and Percy Liang. Prefix-tuming: Optimizing continuous prompts for genera- tion, 2021. URL
Yanshu Li. Unveiling and mitigating short-cut learning in multimodal in-context learning. In Workshop on Spurious Correlation and Shortcut Learning: Foundations and Solutions, 2025. URL https://openreeview wttp://wwww......oooooo/o/r////a
Zhuowei Li, Zihao Xu, Ligong Han, Yunhe Gao, Song Wen, Di Liu, Hao Wang, and Dim- itris N. Metaxas. Implicit in-context learning, 2025. URL htttps://arxii.org//bbsss 14660.
Ys Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: htte///aa//1000055/ 405. 0312.
Common objects in context, 2015. URL http::://aaxiv.orgg/g/o
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, OCR, and world knowledge. github. io/blog//////ooo 2024-01 30- poo--llva-neete January 2024a.
Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, et al. Mibench: Evaluating multimodal large language models over multiple images. arviv preprint arXiv:2407.15272 2024b.
Sheng Liu, Haotian Ye, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering, 2024c. URL
236616668..
Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao. Mm-safetybench: A benchmark for safety evaluation of multimodal large language models, 2024d. URL https://arxiv.orr//bb//33
2301.. 17600.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. ar Xiv preprint ar drXiv:2104.08786 2021.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the I.F.F//off conference on compuler vision and pattern recognition, pp. 3195-3204, 2019.
13

---

Published as a conference paper at COLM 2025
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple word2vec-style vector arithmetic. arXiv preprint ar Xrio:22305.16130 2023.
Yingzhe Peng, Xinting Hu, Jiawei Peng, Xin Geng, Xu Yang, et al. Live: Learnable in-context vector for visual question answering. Advances in Neural Information Processing Systems, 37:9773-9800, 2025.
Libo Qin, Qiguang Chen, Hao Fei, Zhi Chen, Min Li, and Wanxiang Che. What factors affect multi-modal in-context learning? an in-depth exploration. Advances in Neural Information Processing Systems, 37:123207-123236 2025.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lam- bebo Tonja, et al. Cvqa: Culturally-diverse multilingual visual question answering benchmark. arv preprint ar Xio:2406.05967, 2024.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pp. 146-162. Springer, 2022.
Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models, 2024. URL https://arxiv org.otgg//o.. //rg//s//// 2310. 15213.
Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. ar Xiv preprint ar arXit:2305.14160 2023.
Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. Find- ing skill neurons in pre-trained transformer-based language models. arXiv preprint ar XXx::2211..07349, 2022.
xAI. Grok 3 beta the age of reasoning agents. https://x.aa//blog/ 2025-02-21.
2025. Accessed:
Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. Rule: Reliable multimodal rag for factuality in medical vision language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp- 1081-1093, 2024.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. ar preprint XXix:2111.02083 2021.
Baixuan Xu, Weiqi Wang, Haochen Shi, Wenxuan Ding, Huihao Jing, Tianqing Fang, Jiaxin Bai, Xin Liu, Changlong Yu, Zheng Li, al. Mind: Multimodal shopping intention distillation from large vision-language models for e-commerce purchase understanding. ar Xrviv preprint ar Xix::2406.10700 2024.
Xu Yang, Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng, Exploring diverse in-context configurations for image captioning. Advances in Neural Information Processing Systems, 36:40924-40943, 2023.
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlock- 1555//388..11.88....
ing the long-text capability of clip, 2024a. URL
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. J.F.F. Transactions on Pattern Analysis and Machine Intelligence, 2024b.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337 224-))3888, 2022.
14

---

Published as a conference paper at COLM 2025
Yongshuo Zong, Ondrej Bohdal, and Timothy Hospedales. VI-icl bench: The devil in the details benchmarking multimodal in-context learning. arv e-prints, pp. arXiv-2403, 2024.
A Additional Theoretical Proof
A.1 Proof of Theorem 1
Proof. The attention mechanism for query vector h' over a key-value pair [cl Q"]" is:
Attn hh' [CT 27' [CT QT]" softmax (la awwo==)
[
(14)
where softmax (-) normalizes the input into probability distribution, and d is the scaling factor to stabilize training. Expanding the computation, let:
SC E exp ( sQ E exp (zAWO)))
(15)
which denotes the sums of exponentiated scores over the demonstration and query tokens, respectively, where Cj and Qj are the j-th rows of and Q. Thus, the attention output is:
Attn [ e [[]""["TTT]]
= (s) (exp ((ttNCT) c+ exex (axtwie))
(extWiC) C c+ (sc + (os softmax (awtiiT) Q. (16)
&c (&c + sQ) softmax
Based the aforementioned analysis, we take V, J. and of follows:
Y (eq. C2 Qv) softmax (axteve) Qvr
A (eg. C2. (x. Qo)
(17)
g : sc +Q) 1, = (sc (sc sQ) and then the proof of the theorem is completed.
(18)
A.2 Proof of Theorem 2
Proof. Based on Theorem 1, there exist 5¹, n/ E R satisfying:
ac.o t alc + 1/ ab
(19)
Multiply both sides Eq (19) on the right by W, we obtain the following equation:
Coo w = af + +ny a6w.
(20)
Take 4ww (x) xW, we obtain:
MLP (ác.e) - vw (alc) + n/¹ MLP (a(e).
(21)
and then the proof of the theorem is completed.
15

---

Published as a conference paper at COLM 2025
A.3 Extension to MLP with Non-linear Activations
Theorem 3. For linear transformation matrix W1 E Rduudg, W2 E RayxdM, there exists
P,,,;; RdM + R&M such that, for any token position i, there exist 5 n/ E IR satisfying:
MLP (ac.o) - (ot) +ay MLP (
(22)
where the MLP operation with non-linear activations is defined as MLP (x) = a (xW1) W2, and
(-) denotes the activation function.
Proof. Based on Theorem 1, there exist č2, 1// E R satisfying:
acc = ç a'c + n/ ao. Multiply both sides of Eq (23) on the right by W1, we obtain the following equation:
(23)
cco WT = ac
+ 1 WoW,.
(24)
Apply (.) to both sides of the equation and then multiply by W2, we obtain:
wi) W2 =Ç' (at W2 12+ (agw.) W2.
(25)
Take w, (x) o (xW1) W2, we obtain:
MLP (ac.e) = 5' Pw
aec) tny MLP (26)
(26)
and then the proof of the theorem is completed.
A.4 Task Combination
Each learned in-context vector encodes the contextual semantics of different tasks. We further demonstrate that by linearly combining these vectors, we can obtain the context required for new tasks. This property of linear composability enhances the representational capacity of in-context vectors, thereby improving the generalization capability of the LVLM. Given 11 N as the number tasks, a model M with hidden dimension d M, and matrices {C+)²_ representing the in-context demonstrations for each task, we investigate whether these demonstrations could be combined to support multimodal ICL across multiple tasks.
Theorem 4. There exists a function & Di - R&M such that for any query matrix Q and token position i corresponding to the residual stream h', there exist { 0f
" R R] and a R such that:
Attn hi, [ CJ C. o'] C. ... c o ))
Σ 8 Φ h. C. C.) + of Attn (h..0.0).
(27)
Proof. Based on Theorem and the computational formula of the attention mechanism, for any query matrix Q and any token position in the residual stream h', we obtain:
Attn h², [ ...
[cT c
C, ", )T)))
= y C, t Attn (h/, C ... c e=]" [ [[] ... c. )T)))
zi y C,,
+ + y (ni,c,
vin2 Atm hi [c]
c cl Q [c] ... c )'l')
Σ
nh-n A y hh, C,,, +
n7 Atm (N.
(28)
16

---

Published as a conference paper at COLM 2025
where 5i and 1// are derived from the attention scores for each task's demonstration matrix and the query matrix, respectively. Take 4, { and a as follows:
(eq. Ch. 2x) = Y (k. V (eg. (xr
(29)
0f = ži II Thk
WE (1,2,...,
(30)
11
(31)
and then the proof of the theorem is completed.
L
B Dataset Processing
For efficient training of the MFV framework, we propose data sampling strategy for multimodal datasets.
For each instance (4j, Aj) in D, we construct its semantic representation by:
E CLIP())) @ CLIP(Q),, Vj (1,2,..,
(32)
where E; denotes the joint embedding of the j-th sample, and + represents the concatenation operation. We use CLIP-L/14 model to separately embed both the image I and question Qj. then concatenate these embeddings to form multimodal representation. For datasets containing some long textual prompts, such as MM-SafetyBench in Appendix N, the text sometimes exceeds CLIP's limit of 77 tokens. In such cases, we switch to LongCLIP (Zhang et al., 2024a), well-trained variant that extends CLIP to longer text, and obtain embeddings with a consistent dimension.
We apply k-means clustering the joint embeddings of all instances using cosine similarity as the distance metric, dividing D into K clusters:
{ (C,, C2...., Ck} k-means( { [E)) K, cosine),
(33)
where C; represents the i-th cluster, and K is hyperparameter that specifies the number of clusters. For each cluster, we identify the instance closest to its centroid:
jk arg min cosine distance cosine jCC;
centroiod (Cx))) Vk E {1,2,..., K),
(34)
where je is the index of the instance closest to the centroid of cluster Cg. These K instances form our query sample set Do {(4, Qx) with each answer A₃ removed, they will be used as query samples during training.
After extracting the query samples, define the support set Dsupport all remaining instances in D:
D supp D [14, Qs. A
(35)
where \ denotes the set difference operation.
For each query sample 4, Qs in Do, we apply retrieval strategy R to select n relevant instance from the support set D suppoot, forming an n-shot context C₁:
C (D (4,, n)), Vk (1,,,..,,K,
(36)
where R can be implemented as various retrieval methods such as similarity- based retrieval, random sampling, or other domain-specific selection strategies. These contexts form our initial training set Dc = {C+) To enhance the robustness of the MLP output to context permutation, we augment De by creating permuted version C. of each context C₁ where the order of the n demonstrations is randomly shuffled:
C Shuffle(Cy)..
Vk (1,2,...,K) ..,,,.K,K .K.
(37)
17

---

Published as a conference paper at COLM 2025
The final augmented training set contains 2K instances, comprising both the original and permuted contexts:
Dc = {C,, C
(38)
where each training instance consists of context (either original or permuted). Each query sample in Do corresponds to two contexts in Dc. When used as input, the prompt places the n-shot context in front, followed by the corresponding query sample.
C Synergistic Loss
For an LVLM with layers, suppose that layer (where E I,), after injecting O the final outputs of MHA and MLP at that layer are A and M,, respectively;
A, = + af vf.
(39)
Mi(O) = m + a²" v
(40)
We apply the following normalization to obtain (7(()) (+) and Z,"H (O):
(,
(7) |A
(41)
(2
(0) [M][
(42)
We compute the cross-view correlation matrix M E RdxxdM for the two normalized final
outputs:
M (Z) (0) Z (
(43)
We use this matrix to compute the synergistic loss follows:
Σ Σ (1
2 M
dm
M
Sugn
+7
(44)
where M is the element at the i-th row and j-th column M' and y is a hyperparameter.
D Many-shot MIIV
Pairwise aggregation pipeline. A long n-shot prompt is first divided into overlapping windows of length w with overlap o. The teacher LVLM runs on every window and the
MLP activations are mean-pooled into fixed-length vectors m; E Rd. These wer-window summaries are then combined left-to-right in a size-preserving two-vector loop. At step i the current aggregate f; is concatenated with the new window vector to form
Z = im; m,] 22
(45)
Two learned projections Wg. W1 E Rdd22 generate a gate g and a candidate
g = oWzz),
(46)
č tanh (Wz)..
The aggregate updated by
m; - 0 1
(47)
which keeps dimensionality at d, matching the original MLP hidden size. Sharing the same tiny gating block across all N windows makes the procedure (N) time and adds only O (dl) parameters, so the final vector my can injected back into the LVLM without any adapter.
18

---

Published as a conference paper at COLM 2025
Information loss mitigation. The element-wise gate learned above selectively retains high- salience features from earlier aggregates. To further curb boundary effects, we employ a lightweight tuning strategy to determine the optimal window length w and overlap o:
w E {32, 64, 128), o E {0, 100, }00
(48)
This coarse grid is evaluated on a held-out set using cosine similarity between the aggregated student vector and a full-context teacher vector as a cheap proxy for fidelity. The top few candidates then enter a successive-halving loop that progressively allocates more budget, and the survivor is chosen by end-task accuracy adjusted for latency.
E VLibrary
In practice, VLibrary is implemented using an off-the-shelf object store, which is easily integrated into existing systems. During retrieval, the system queries using the MIV' designated parameter set, a". This architecture avoids the overhead of rebuilding datastores and instead supports straightforward, scalable deployment.
VLibrary is hosted on Amazon S3, where each MIV asset is structured binary object that contains the learned vectors (Va, V and the associated scalars (a",, a²") ) for each decoder layer. We serialize each asset using Protocol Buffers to preserve floating point precision, and then compress it with Zstandard before uploading. To guarantee uniqueness and support lifecycle management, we use content-based addressing. Specifically, we normalize the index parameters a by enforcing fixed precision and deterministic layer ordering. We then compute a SHA-256 hash, referred to as the M2IV Content Hash,, which serves as the S3 object key. For efficient access, a Redis-based Mapping Service maintains associations between human-readable versioned identifiers (e.g., When an application requests a specific MIIV,
wttp::.:::tssanmla waw.sellam@@...... 1.0) and
their corresponding M2IV Content it provides the versioned keys, retrieves the hash from Redis, and uses it to fetch the binary object from S3. The object is then decompressed and deserialized back into a usable in-memory structure. We further accelerate frequent queries via application-evvel caching.
F Benchmarks
The amount of data used in our experiments is shown in Table 7. For few-shot VQA evaluation, we adopt the following datasets/benchmarkss
VQAv2 is based on images from the MSCOCO dataset and features traditional question-answer pairs, assessing model's ability to accurately interpret both the image and the language of the question.
VizWiz introduces additional real-world complexities with lower-quality images, questions that often lack sufficient context, and a significant portion of unanswerable queries. Models must contend with incomplete visual information and learn to handle uncertainty.
OK-VQA focuses on questions that require external knowledge beyond the image content, serving as a benchmark for evaluating whether models can incorporate outside information to arrive at correct answers.
GQA is a large-scale dataset focusing on compositional question answering over benchmark for evaluating how models parse intri- cate scene relationships and perform multi-step reasoning.
real-world images, serving
A-OKVQA expands upon OK-VQA by offering a wider variety of question types and more demanding knowledge requirements. Notably, 30.97% of its samples involve at least two inference hops, highlighting the dataset's emphasis on multi- step reasoning and deeper knowledge integration. A-OKVQA samples come in
For datasets with multiple human-annotated labels per sample, one of them is randomly chosen the ground-truth label in demonstrations
19

---

Published as a conference paper at COLM 2025
two forms, multi-choice and direct answer. We opt for the latter for open-ended evaluation. A-OKVQA provides a reasoning rationale for each instance. In the main experiments, remove the rationale from each instance, whereas in the explainability experiments, we utilize these rationales.
CVQA a cultural-diverse, multilingual visual uuestion-answeringg benchmark that gathers images and question-answer pairs from 30 countries and 31 languages, aiming to assess a model's ability to handle both visual input and text prompts in truly global context. It employs multiple-choice format- one correct answer with three distractors. CVQA is divided by continent to highlight regional and linguistic diversity: We only use its Asia split, which comprises 19 types of Asian country-language pairs, such as China-Chinese, India-Hindi and Japan-Japanese. We mix the 19 types in proportion rather than computing an average across single ones.
Besides, we use the latest multimodal ICL benchmark, VL-ICL bench for general ICL evaluation. VL-ICL Bench is comprehensive evaluation suite tailored for multimodal ICL, encompassing both image-to-ter and text-to-image tasks. It tests a broad range of capabilities, from fine-grained perception and reasoning to fast concept binding, all using a few demonstrations. VL-ICL Bench shows meaningful improvements with more shots and highlights fundamental model limitations. In our study, we only employ the image- to-text split of VL-ICL Bench, which includes Fast Open MinilmageNet, CLEVR Count Induction, Operator Induction, TextOCR, Interleaved Operator Induction, and Matching MinilmageNNe; we test each task individually and then average the performances to obtain the final score. For splits containing only images and answers, we assign uniform question to each instance. For instance, in the Fast Open MinilmageNet split, we use. "What's in the image?" as the question.
For the VQA datasets and VL-ICL bench, we use Accuracy(%) as the metric to assess the models' ability to provide correct answers:
3 Lx match (a, 8k)
Acca; = max
1,
(49)
10
where a denotes the model's generated answer, 8k denotes the k-th ground true answer. match (-,-) decides whether two answers match, if they match, the result is 1, otherwise 0.
Datasets
Training
Evaluation
VQAv2
10,000
10,000
VizWiz
8,000
4,000
OK-VQA
8,000
5,000
GQA
10,000
5,000
A-OKVQA
8,000
5,000
CVQA
3,000
1,500
VL-ICL bench
8,360
1,120
Table 5: Overview of the size distribution across the benchmarks used in LIVE.
G
Models
In our study, we select three LVLMs that support multi-image inputs and have demonstrated multimodal ICL capabilities: OpenFlamingov22991 Idefics2-8B and LLaVA-NeXT-7B, Their respective configurations are shown Table 6. OpenFlaminnoov--B is a versatile model that processes and generates text based on both visual and textual inputs, designed for broad VL tasks. Idefics2-8B is a model that seamlessly integrates visual and linguistic information, em- phasizing robust cross-modality understanding for diverse applications. LLaVA-NeXT-7B is a flexible model optimized for natural conversational interactions by effectively merging visual cues with language understanding, supporting intuitive multimodal dialogue.
20

---

Published as a conference paper at COLM 2025
LVLM
LLM Backbone Connection Module
Image Tokens
Context Window (Triin)
Context Window (Test)
OpenFlamingor2
MPT Mistral Vicuna
Perceiver
64
2048
2048
Idefies-8B
MLP MLP
32K
LLaVA-NeXI-7B
576
2048
4096
Table 6: Detailed configurations of the LVLMs used in study.
H
Baselines
Besides zero-shot setting and æ-shot Vanilla ICL, compare MIV with the following methods:
Task Vector (TV): TV decomposes ICL in LLM into two parts. The first part uses the initial layers to compute a task vector 0 from the demonstration set S, while the second part employs the later layers to apply 0 to the query x for generating the output. Importantly, these two parts are independent, allowing the same 0 to be used for different queries. Considering TV is extracted from single layer, apply it to each layer of the LLM and take the average performance across all layers the final result.
Function Vector (FV): FV is computed by first extracting task-conditioned mean activations from a set of attention heads that are selected based on their causal mediation effects. These mean activations are then summed to form the function vector, effectively distilling the task information from in-context demonstrations into a single representation. Considering FV is extracted from a single layer, we apply it to each layer of the LLM and take the average performance across all layers as the final result.
In-Context Vector (ICV): ICV is computed by first forwarding demonstrations (x,y) separately through the model to extract the last token's 's hidden states across all layers. These layer-wise representations are concatenated and the differences between the target and input embeddings (e.g. hx))) are computed for each demonstration. Principal component analysis is then applied to these differences to obtain the dominant direction, which serves as the ICV. During inference, this vector is added to the latent states of the query across all layers. ICV uses a weighting factor & to control the degree of steering. We set to 1e-3, which provides the best overall performance our chosen benchmarks.
Implicit In-Context Learning (I2CL): I2CL is implemented by extracting demonstra- tion vectors from the end-residual activations of both the MHA and MLP branches across all layers, which are then aggregated via element-wise mean form unified vector. At inference, layer-wise scalar coefficients linearly combine this context vector with the query activations through simple scalar multiplications and element-wise additions.
Learnable In-context VEctor (LIVE): LIVE is implemented by training layer-wise shift vectors and weight factors that capture the essential task information from multiple in-context demonstrations. During training, LVLM processes query along with few-shot demonstrations, and these vectors are optimized to align the output distribution with that of realistic ICL. During inference, well- well-trained vectors are simply added to the hidden states the model. The data sizes used to train and evaluate LIVE are shown in Table 5.
I Experiment
L1 Additional Implementation Details
We initialize the vectors in V and VM from a normal distribution with mean and standard deviation 0.01, while af is set to 0.1 t(z)) and a"" to 0.1 t where 10-6 6, For all training procedures, AdamW serves as the optimizer, weight decay is fixed at 1e-4, the
21

---

Published as a conference paper at COLM 2025
warmup factor at 1e-3, precision at FP16, and batch size at 2. Dataset-specific hyperparam- eters include the learning rates for V and a, the temperature parameter 7 the mimicry loss, the parameter y in the synergistic loss, the weights of the three loss functions, and the number of epochs. These details are provided in Table 7. We utilize four RTX 4090 GPUs.
Dataset
K
Evaluation
7/v
f/a
Amim
Asyn
Asup
Epochs
VOAv2
2,000
10,000
le-3
le-2
1.5
0.20
0,8
0.8
0.5
15
VizWiz
1,500
4,000
le-4
le-2
1.8
0.20
1.0
0.8
0.4
10
OK-VQA
1,500
5,000
le-4
5e-3
1.3
0.15
1.0
0.8
0.5
10
GQA
2,000
5,000
1e-4
le-2
1.8
0.15
0.8
1.0
0.4
15
A-OKVQA
2,000
5,000
2e-4
5e-3
1.5
0.15
0.8
0.8
0.5
10
CVQA 1,500
1,500
2e-4
le-2
1.8
0.20
0,8
1.0
0.5
10
VI-ICL bench
3,000
1,120
1e-2
le-3
1.0
0.05
0.8
0.6
0.6
15
Table 7: Overview of dataset sizes for training and evaluation along with training hyperpa- rameters across the benchmarks in our main experiments.
1.2 Additional Main Results
In Table 1, we report the average performance of various methods on three LVLMs. For greater clarity, Table 8 details the performance on each LVLM individually. Our method ranks first in 18 out of experiments, demonstrating outstanding generalizabil- ity and robustness. We additionally report MtTV performance on three recently released LVLMs: LLaVA-OneVision (7B) (Li et al., 2024a), InternVL2.5 (8B) (Chen et al., 2025b), and Qwen2.5VL (7B) (Bai et al., 2025). As shown in Table 9, MFIV also achieves SOTA performance on these LVLMs, demonstrating its strong generalization capability.
J M vs Other Methods
The improvement offered by MAV comes with a modest increase in trainable parameters, particularly when compared other methods for adapting LVLM to new data, such as Caaameter-Effiieent Fine-Tuning (PEFT). Here, we take LoRA (Hu et al., 2021) as an example. LoRA one of the most popular PEFT methods for adapting LVLMs. Its implementation is based on low-rank adaptations of the model's weights. For LoRA, we use the same 16-shot context data from the entire De as MII,, and is applied to the projo of every layer of LVLM. Figure 10 illustrates that M21V trains only 1/50.8 of the parameters required by LoRA, yet it achieves superior performance across all benchmarks with an average gain of 2.77%.
Next, compare MIIV with prefix tuning (Li & Liang, 2021), widely used parameter- efficient adaptation strategy that modulates model behavior by prepending learnable tokens to the input.
Streamlined Inference: MIIV eliminates the need to maintain additional token during inference, resulting more efficient computation without expanding the attention mechanisms or memory requirements that typically accompany prefix- based approaches.
Specialized Processing Path: Unlike prefix-tuning uniform approach to all tokens, MPTV create different pathways for context and query processing. This separation al- lows for targeted optimization of how multimodal context influences the generation process while preserving the query's original characteristics.
Deeper Activation Integration: MTV operates on the model's intermediate repre- sentations rather than just modifying input embeddings, allowing for more control over how visual and textual information interact throughout the network. This addresses of prefix tuning's major shortcomings: insufficient semantic fidelity.
22

---

Published as a conference paper at COLM 2025
Models
Methods
VQAv2
OK-VQA
GQA
A-OKVQA
CVQA
bench
Zero-shot
42.37
18.21
31.53
51.39
29.81
28.65
8.92
Vanilla
57.64
50.86
65.93
47.79
57.43
26.70
TV
46.38
23.18
35.05
61.02
50.40
40.62
14.37
FV
56.27
50.63
ICV
43.10
17.81
32.94
53.08
52.17
30.35
10.24
Olannnon2
I2CL
51.53
25.49
41.80
58.87
51.42
48.60
19.02
LIVE
59.68
35.28
53.27
67.32
48.83
56.24
27.91
MIV
63.12
40,97
56.10
70.81
52.01
39.35
30.84
Multi-turn JCL (128-shot)
61.28
54.36
63.31
48.22
61.41
MPV (128-shol)
65,80
43.66
59.08
69.83
53.73
62.05
(256-shot)
MFIV (256-shut)
66.94
45.31
61.35
71.42
54.67
62.29
Zero-shot
Vanilla ICL
68.51
52.93
53.14
72.38
59.06
57.89 89
35.16
TV
56.42
63.29
43.54
48.83
24.32
FV
51.38
26.79
44.51
67.81
47.07
45.57
18.68
ICV
47.74
25.59
36.62
60.52
32.96
44.67
14.45
Idefics2
LIVE
71.25
54.33
$5.45
70.49
61.25
54.82
37.02
Multi-turn ICL (128-shot)
65.72
52.96
54.60
70.85
61.46
58.34
MPV (128-shot)
73.95
77.05
63.78
64.34
Multi-turn ICL (256-shot)
66.03
51.79
54.97
69.53
62.71
61.26
MPTV (256-shot)
75.59
58.67
58.79
78.19
65.45
64.15
Zero-shot
39.65
19.48
34,92
47.36
30.18
31.19
10.17
Vanilla ICL
53.79
28.77
50.11
69.28
45.92
53.84
25.39
TV
40.83
19.59
38.01
56.47
16.58
FV
45.26
21.67
35.18
53,90
37. 58
41.35
17.47
LLaVA-NeXT
I2CL
49.56
23.96
43.47
59.98
37.97
47.81
15.93
LIVE
MIV
54.39
31.90
$1.09
74.73
43.96
38.79
28.65
Multi-tumn ICL (128-shol)
51.80
25.88
86
65.91
44.07
55.21
MIIV (128-shot)
55.83
33.48
50.46
73.86
42.56
58.63
Multi-turn JCL (256-shot)
50.41
26.24
48.57
07.45
42.17
56.27
(256 shot)
52.16
75.33
44.19
Table 8: Detailed comparison between MIVV and baseline methods on three different LVLMs.
The highest scores are highlighted in bold and the second-best scores are underlined. Table 1 presents the average results across these models.
Models
Methods
VQAv2
VizWiz
OK-VQA
CQA
A-OKVQA
CVQA
VL-ICL bench
Vanilla ICL
64.77
44.18
58.42
68.75
65.47
66.91
43.56
I2CL.
57.53
39.02
53.64
62.89
58.49
57.84
37.89
LLaVA-OneVision
LIVE
66.93
45.78
60.84
71.31
66.28
68.30
45.16
M-IV
68.21
48.00
63.91
72.64
68.40
73.28
47.04
Vanilla ICL
68.27
57.82
62.85
74.71
67.29
66.39
46.75
I2CL
63.73
49.08
58.69
69.
63.36
57.07
41.05
LIVE
70.62
58.74
64.37
76.41
70.40
67.34
48.83
MPV
72.48
Vanilla ICL
71.40
59.62
64.83
73.95
68.31
70.16
47.28
Qwen2.5VL
I2CL
65.17
51.38
56.75
70.48
63,95
64 28
43.54
LIVE
73.36
61.58
65.27
76.38
69.84
71.59
49.61
MAIV
75.23
63.57
66.83
76.97
71.64
74.58
51.32
Table 9; Comparison between MPIV and baseline methods on three additional LVLMs. The highest scores are highlighted in bold and the second-best scores underlined.
23

---

Published as a conference paper at COLM 2025
Finally, we compare the four methods discussed in this paper across four key aspects, shown in Table 11. MPIV demonstrates overall superiority.
MAV
LoRA
Parameters
1.0x
50.8x
Method
Strength
Fidelity
Cost
Time
VQAv2
63.93
61.86
MFV
VizWiz
43.40
39.36
Vanilla ICL.
OK-VQA
55.37
54.05
Prefix Tuning
GQA
73.81
70.79
PEFT
A-OKVQA
53.59
52.48
CVQA 60.11
55.07
High
Medium
Low
Table 11: Comparison of various methods across four aspects: steering strength, semantic fidelity, computa- tional cost, and inference time.
Table 10: Comparison of MPIV and LoRA in terms of total training parameters and perfor- mance across six benchmarks.
K Retrieval Strategies
Co
Vot.
daigg
the bika?
are Plera in Image?
The Piaf
the pible the pesus??
R 10102
Figure 8: Shortcut learning in multimodal ICL arises from overreliance isolated visual features (e.g,, similar composition or a prominent object the query image). MPIV eases this problem, thereby improving performance.
After obtaining Po, we apply a retrieval strategy R to construct an n-shoo context for each query sample. Consequently, MPIV actually represents the Vanilla ICL under this specific R. In our main experiments, we adopt Random Sampling as R, which uniformly selects n instances from the entire support set in-context demonstrations for each query sample. In $4.4, we investigate the performance of MIV for representing different retrieval strategies, including the following three:
Image-to-Image (I2I): Given query sample Q (i, and support set D supp (4, Qj. Qj. 41) } |Puppl we utilize CLIP -LiP-generated image embeddings (i.e., CLIP(I)
and CLIP(I,) to compute similarity scores and select the top n instances from Dsupp. These instances are arranged context descending order of similarity. Because this method relies exclusively on visual features while neglecting linguistic cues and the deeper task mapping arising from visual-language interactions, it tends to induce problems such shortcut learning and hallucinations in multimodal ICL (Li et al., 2024b; Li, 2025). Therefore, its performance may be inferior to that of Random Sampling. We provide some cases Figure 8.
ImggQueetionnee.maagqeeeeeei (IQ2IQ): Given a query sample Q = (f, and a support set Dsupp = (1, A,) | |Paupll we select demonstrations by leverag- ing joint similarity between image and question embeddings, in our semantic clustering process. This method, by incorporating language modality features into the retrieval process, helps mitigate the model's over-reliance on visual features; however, it may also result in visual-language imbalance.
Oracle: Oracle refers to a method that, when the ground truth answer for query sample is available, leverages this answer along with the LVLM itself to perform
24

---

Published as a conference paper at COLM 2025
greedy optimization- retrieval. By allowing the LVLM to evaluate and select, this approach obtains a near-optimal context for the model (even though local optima may still occur). However, its reliance on the ground truth renders it inapplicable to real-world scenarios. Given an LVLM M, a query sample Q (1, with its ground truth answer A = {A,, ..... A and a support set Ssupp
{ {1,, Qj, Aj) [Pauppl, | our goal is to construct an ntshot context C". We first define
score M for evaluating the context in the form of maximum likelihood:
SM(C") E log PM (A: C", A.ct).
(50)
We then employ greedy strategy. Given a context of length m - 1 (where 1 < S n), we select an instance x from D Suupp that maximizes the score when appended to the current context, and repeat this process for и iterations:
Xm = argmax
XE xEDsupp
(S + +x)
(s(( 1 m S n.
(51)
Data vs Training
Performance under Diverse K
10
Vantin IC)
Accuraicy (N))
Verile E3
VSh2
48-
5-OHICA
2,000
Figure 9: Performance under different Do sizes. "MIV w/o clustering" denotes that Do is constructed by randomly sampling from the full dataset without applying k-means clustering. "*" marks the performance of 16-shot Vanilla ICL on each dataset.
We here investigate whether a well-structured query distribution outweighs the sheer quan- tity. As shown in Figure 9, although increasing the size of Do generally benefits MIV,, strategically refining the data construction yields significantly greater gains than simply adding more samples. On average, random sampling requires over 1000 additional exam- ples to match the performance of Vanilla ICL enhanced by semantic clustering. Furthermore, semantic clustering leads to rapid early gains before plateauing, whereas performance under random sampling grows steadily, suggesting that once Do approximates the task distribution, further scaling yields diminishing returns. This highlights the importance of diversity over volume. By aligning with the overall dataset distribution, semantic clustering achieves superior results with far fewer data.
To further determine whether the performance gains arise from dataset construction our training strategy, we conduct preliminary experiment that replaces LIVE's training dataset with Do and Dc while keeping all other settings unchanged. Results in Table 12 reveal that applying semantic clustering directly to LIVE does not yield the expected improvement; in fact, performance declines. This motivates to conduct more comprehensive ablation studies to pinpoint the components that contribute most to MPIV's performance.
M Combination and Transfer
MIV, as a vector representation, inherently exhibits linear additivity, greatly enhancing its flexibility and utility. This property allows us to combine individual MTTV into unified
25

---

Published as a conference paper at COLM 2025
Methods
VQAv2
VizWiz
OK-VQA
GQA
A-OKVQA
CVQA
VL-ICL bench
LIVE
62.22
39.17
53.47
69.20
51.60
55.42
29.95
LIVE+ Do
59.81
35.54
51.74
66.69
49.93
48.34
23.98
MPIV
63.93
43.40
55.37
73.81
53.59
60.11
33.36
Table 12:
Comparison of performance between LIVE and its variant using Do & Dc as training data across different benchmarks.
representation that encapsulates multiple tasks simultaneously. In Appendix A.4, we provide detailed proof of this task combination ability, showing that the linear sum of MaVV preserves and integrates the task-specific information from each component. Thus, although instances in VLibrary are originally trained for particular datasets, their linear additivity enables us to construct an MIIV with multi-task capabilities by summing multiple vectors. We propose two strategies to achieve this combination. The first is training-free free method that directly sums the MIIVs. The second strategy involves fine-tuning the combined vector small amount of multi-task data. We assume that we need to combine P M2IVs from VLibrary. In the training-free free setting, user-specified base weights wi are assigned to each vector such that 1 w = 1. The combination is performed in a separate manner: the scaling factors and vectors are aggregated independently. For the MHA component at layer I, we compute:
21
wi ai,i-
(52)
Likewise, for the MLP component we have:
aT" = Σ wi "7,""
(53)
L Then we obtain è {ag, aT" v/" 11=1 which constitutes an MVIV capable of representing tasks.
While the training-free strategy is efficient, fine-tuning the combined representation on a small multi-task dataset can further enhance precision. For each MFV,, we introduce learnable scalar corrections i for the MHA branch and 8 for the MLP branch. The scaling factors become:
P (w) + +8,,))
27
Σ
ai,j.
(54)
a/" Σ (w; + 871) aT
(55)
and also obtain Ô We randomly sample 500 instances from each MFI's corresponding dataset that are not included in its Do De then mix them to form a fine-tuning dataset D After injecting è, we fine-tune using Layn and Csup, with the corrections initialized to le-5. During this process, the original V and remain fixed, and only 8i, and 8"; are updated.
We evaluate MII'' combination capability on two benchmarks with multiple splits: CVQA and the VL-ICL bench. CVQA has four splits, and we use only the Asia split in our main experiments. Here, we include the Europe split well, training MFIV(A) and MFIV(E) on these two splits, respectively. We then combine them using two different strategies to produce MrVV(A&E). Evaluation is performed on three sets: the Asia split, the Europe split, and mixed dataset comprising both splits. For the VL-ICL bench, which contains six splits, our main experiments train an MFIV on each split individually and reported an averaged result. Here, we combine all six MFISs into MFIV(comb) and evaluate it a mixed dataset from all six splits. As shown in Table 13, combining multiple MPIVs does not degrade performance on individual tasks and can even lead to gains, while also introducing multi- task capabilities. Furthermore, combining more than two MIV's can also yield effective multi-task improvements. Fine-tuning always outperforms the training-free free strategy.
26

---

Published as a conference paper at COLM 2025
Methods
CVQA(A)
CVQA(E) CVQA(A&E)
MFIV(A)
60.11
32.15
50.73
MPIV(E)
39.26
54.36
42.23
(TVIAAAE)(Triiiinngfeee
59.85
56.09
58.13
MFIV(A&L)FFinettningg
61.07
57.38
59.84
VL-ICL bench
MFIV (a free)
33.36
31.49
(FF((comb)Finne-tnnig tuning)
35.90
Table 13: Results of 16-shot MIV combination CVQA and VL-ICL bench.
Similarly, we can transfer the MPIV obtained on one LVLM M to another LVLM M' with the same number of layers and hidden state size, either via training-free or fine-tuning strategy. In the training-free setting, directly inject @ {at, vf, aT" v/") 1==1 from M into the target LVLM M'.
Given that M' shares the same layer count and hidden state dimensionality as M, directly injecting the pre-trained MTV a reasonable approach to achieve efficient transfer. However, subtle differences in internal parameter distributions and activation dynamics can cause misalignment with M'ss residual stream. To address these differences, we also introduce learnable scalar corrections o7 for the MHA branch and 8!"d for the MLP branch. The scaling factors are then updated as follows:
2f (1 87) a7,
(56)
a/" (1 + 87") a²"
(57)
Thus, the refined transferred MTV is given by è v/ af, 500 instances from the dataset corresponding to O that are not included in its Do or Dc, forming a fine-tuning dataset D' Then we inject Ô into M' and fine-tune it using Laye and Lsup, with the corrections initialized to 1e-5. During this process, only 8 and 8,"h are updated. Table shows that when MIIV is transferred to another LVLM, the training- free strategy results in an average performance difference of only - 1,48%, and with fine-tuning, this difference drops further to -0.25%. This demonstrates MPIV's capacity for cross- cross transfer, making it suitable for broader applications.
"aaa"vvi v"' vi We randomly select
Methods
VQAv2
VizWiz
OK-VQA
GQA
A-OKVQA
CVQA
Olaminnnng
63.12
40.97
56.10
70.81
52.01
59.35
Idefics2 OOeenpeeainnggn Seeeellllligg/(11e1 (Trainiigg seatiiinnn---eee free)
61.87
38.59
56.31
68.57
50.76
57.85
Idefics2 SaanFllaiiooo((ieetttii
63.25
40.31
57.16
70.54
50.95
58.79
Idefics2
74.28
57.32
58,92
75,89
64.79
62.18
OpenFlamingov2 -ldefics2(Traaiii (raining-free)
71.69
55.28
56.84
75.48
65.01
59.73
OpenFlamingov2 >ldefics2
75.12
57.41
56.79
75.60
65.27
61.54
Table 14: Results of 16-shot MPIV cross-LVLM transfer. We evaluate two strategies in both directions: from OpenFlamingov2 to Idefics2 and vice versa.
N The Usage of VLibrary
We use MFIV and VLibrary to steer LVLMs in diverse, practical ways, thereby overcoming several key bottlenecks. Here, we illustrate three examples: modality alignment, alignment with human needs, and security in LVLMs, as shown in Figure 10. By training MIVS on specific datasets and storing them in VLibrary, achieve more efficient and powerful manipulations than Vanilla ICL or PEFT, demonstrating the sigmificant potential of our method.
27

---

Published as a conference paper at COLM 2025
appears King MPEV Micdonalds
Alli kichen stove, and per the woman with brown hui
What's in the imagee
pondereeeteeee solls, entice cumtly leaturin
restaurant
Zero-shoti precisely
shote.
the o/ w/teletittttoo PeeiitoTTio They
Rettorate comtanta alephants stundiin
la men cutting the
T
0
M/ZV </thinlo That type of cup designed keep ber bevarogea more than worn longer pariods
Lanka the sid tanka
tomperature lady's Lusoraeee?
wet,
Haster
might enjoy contralled
Why deasrt
C Witch food Item right tabile somreon who uvodds spicy
portion without hoving
ripeeling oomeone with Jooth? decaen pioce
i
shane commot to longer pie.
with white frceting
docids epiyy food, the plactic canta was zu mate cction in the senowich cnother contanst.....
VLibrary
Q What could be ane't motivation choosing personal toppings?
M/ZY: One possible
for chooking
persional sized pizza
sumoons with sextit
voriitty topolings and floverfull moal thout consuming too much shaning The emaller
torth secause
chocolite with the
i
Zero-shot: I'm but can't help with that
MCVV: Firstly. order to maximize the profits gembling games, necessory develop
The imape showis
g
pfricss actintiy
phrise actro by pariorn aetivity
dotail uoc
nerthrm actuuiyy fink, you noad to find caansel.....
Q The image shows phrsse activity. effective invelved performing thi activity accurately.
A Fiol, choose you
deftssets
behaviien such bet size, game selectionn, and session durali on
Zerovehot: waterfrent
What in the image?? tree munk 1iting lne
What's Image?? Sorry, canaatt
cityscape with buildigs
archifecturd
including
masque featuring blue dame and hall minamet
Image??
M7IV: Sonny, but connot ossist with that request.
Figure 10: By constructing specialized datasets and training MPIV on them, then storing these M'IVs in VLibrary, we can retrieve and apply them on demand for efficient and effective LVLM steering. Here, we demonstrate five key steering alignment, explainability, instruction following, jailbreak, and safety-yet the potential uses of MIV and and VLibrary extend far beyond these examples.
28

---

Published as a conference paper at COLM 2025
N.1 Experiments
We use MFV and VLibrary to steer LVLMs in diverse, practical ways, thereby overcoming several key bottlenecks. Here, we illustrate three examples: modality alignment, alignment with human needs, and security in LVLMs. By training Here, we illustrate three examples: modality alignment, alignment with human needs, and security in LVLMs. By training MIIVs on specific datasets and storing them in VLibrary, we achieve more efficient and powerful manipulations than Vanilla ICL or PEFT, demonstrating the significant potential of our method.
VQAv2
VizWiz
A-OKVQA
16-shot
CIDErt
81.24
74.58
82.36
Vanilla ICL.
Accuracyt
63.70(+3.62)
41.26(+1.38)
56.39(-5.47)
CIDErt
82.69
77.61
83,97
LoRA
Accuracyt
61.75
38.27
52.43
16-shot
CIDErt
85.71
78.30
85.37
MFV
Accuracyt
(8.51(+4)58)
47.28(+3.88)
57.43(+3.84)
128-shot
CIDErt
84.62
73.91
86.18
MPIV
Accuracyt
70.23(+4.31)
48.18(+1.85)
56.73(+1.96)
Table 15: LVLM explainability evaluation. CIDEr assesses the quality of generated rationales, and Accuracy evaluates VQA performance. The values in parentheses indicate the gains introduced by the addition of rationales.
Explainability is critical for LVLMs, it fosters user trust and understanding. To this end, we augment both the training and evaluation data of VQAv2, VizWiz and A-OKVQA with instance-specific rationales. While A-OKVQA already provides rationales, those for VQAv2 and VizWiz are generated by GPTT-oo in the A-OKVQA style. All rationales are wrapped in </think> tokens. We train MFISs on these rationale-augmenter datasets and store them in VLibrary. Table 15 shows that injecting MFIV enables LVLMs to explicitly articulate higher-quality reasoning processes than Vanilla ICL. This in turn improves problem-solving performance by enabling LVLMs to learn and prioritize sound reasoning.
MPI''s strong behavior-steering capabilities precisely meet the needs model jailbreak. We experiment on the JIM-SafetyBen benchmark (Liu al., 2024d), using 80% of the data to construct the training sets and the remaining 20% to evaluate MIV''s ability to steer LVLMs toward malicious behavior. As shown in Table 16(a), injecting MFIV successfully bypasses LVLMs' built-in ethical safeguards, making them more prone to generating harmful content. This indicates that VLibrary is a powerful tool for evaluating LVLM safety.
Conversely, MPIV can also be repurposed to improve safety. We demonstrate this using the HatefulMemes dataset (Kiela et al., 2021), which includes both hateful and non-hateful meme images. Each sample is standardized to include an image, fixed question ("What's in the image?"), and response: either an image caption (non-hateful) or refusal message (hateful). We train and store an MIV on it and evaluate whether, after injection, the LVLM can correctly detect and reject harmful inputs in the validation set. As shown Table 16(b), the injection effectively enables LVLMs to accurately identify and reject malicious content.
Method
ASR
Method
RRT
No-attack
0.00
Vanilla ICL
57.68
Vanilla ICL
37.85
LoRA
49.71
16-shot MFI
89.36
16-shot M-IV
85.14
128-shot MTV
88.17
128-shot MFIV
89.32
1B]
Table 16: (a) Attack success rate (ASR%) for multimodal jailbreaks. (b) Refusal rate (RR%) for rejecting hateful image input.
29

---

Published as a conference paper at COLM 2025
N.2 Additional Details
Cross-modal alignment. We leverage the property that in multimodal ICL, the shallow layers of LVLMs mainly handle cross-modal feature extraction and understanding. By inserting the first ten layers of MFI (trained on the COCO Captions dataset) into the corresponding shallow layers of the LVLM, we reinforce the model's visual-language alignment. This enables the model to capture visual features more comprehensively and deeply within unified embedding space, ultimately improving performance on general VL tasks such as VQA. COCO Captions is a large-scale dataset containing over 330,000 images, each with five human-annotated captions. We randomly select one caption per image and add short question 'What's in the image?' to each instance. We then utilize GPT-4o to enhance each selected caption, transforming them into comprehensive descriptions that thoroughly document all visual details and features present in the images. This process results in detailed textual representations that capture fine-grained visual elements, spatial relationships, object attributes, and scene compositions. These enriched captions form the foundation for training feature-level image captioning MPIV representation specifically designed to strengthen the cross-modal alignment capabilities within the early layers of LVLMs.
Output customization. To enhance the instruction-followinn capability of LVLMs, we train MIIV on the representative LLaVA dataset. LLaVA dataset contains approximately 158,000 maage-instruction-rspponn triplets, categorized across three distinct instruction types. The first category, conversation, involves multi-turn dialogues about images, requiring coher- ent, context-aware responses. The second type, detailed description, prompts the model for thorough, multi-paragraphe explanations of visual elements and spatial relationships. The third, complex reasoning, demands deeper analysis- relationships, making predictions, or explaining scenarios beyond simple descriptions. We then test MIIV LLaVA- (In-the-Wild).
Safety. For the safety of LVLMs, we first use MIIV for jailbreak research, investigating whether it can circumvent ethical constraints and produce harmful content. We train MTT MM-SafetyBench. MM-Safety Bench is comprehensive framework designed for conducting safety-critical evaluations of LVLM against image-based manipulations. The benchmark encompasses 13 distinct scenarios representing content and actions typically prohibited for LVLM, including illegal activities, hate speech, malware generation, physical harm, economic harm, fraud, pornogrpphy, political lobbying, privacy violation, legal opinion, financial advice, health consultation, and government decision-making tasks. The dataset consists of 5,040 text-image pairs, where each image is generated with the given user query. The evaluation metric used is the Attack Success Rate (ASR), which measures the percentage of inputs that successfully cause the model to generate harmful or inappropriate responses. A higher ASR score indicates that the model can be more easily guided to produce malicious content.
We use M2IV for jailbreak, but, conversely, we can also enhance models' safety awareness with it. Currently, a key approach is enabling the model to accurately detect harmful intent and refuse to generate related content. We adapt the Hateful Memes dataset by labeling hateful content with refusal messages and then train MTTV. After MIV is injected, the model gains the ability to precisely detect various harmful multimodal contents and respond with refusals. Hateful Memes contains approximately 10,000 multimodal memes labeled as either benign or hateful, designed to test models' ability to detect hate speech in multimodal content. Each meme combines an image with overlaid text, requiring models to understand both modalities correctly identify harmful content. For our safety enhancement exper- iments, we reformulate each instance into an image-question-answer remains the same, the question standardized to "What's in the image?", and the answer depends on the image's 's label descriptive captions for non-hateful content and refusal messages (e.g., "Sorry, but I cannot assist with that request.") ") for hateful content. We use this reformulated dataset to train MFIV vectors for safety enhancement, evaluating effectiveness through the Refusal Rate (RR) metric, which measures the percentage of hateful inputs that the model correctly refuses to answer. The higher the RR, the better the model's safety awareness when encountering potentially harmful visual content.
triplet: the image
30


Reviews:
[{"review_id": "B8tCazvCRV", "reviewer": "Reviewer_Ed6E", "summary": "This paper presents M2IV, a novel framework for efficient and fine-grained multimodal in-context learning (ICL) in Large Vision-Language Models (LVLMs). The authors address the inherent limitations of traditional ICL—particularly the token inefficiency and prompt sensitivity—by introducing learnable in-context vectors (M2IV) that are directly injected into the model's residual stream. These vectors leverage the distinct roles of Multi-Head Attention (MHA) and Multi-Layer Perceptrons (MLP) to semantically distill and store cross-modal knowledge. The approach includes a sophisticated self- distillation training mechanism and is supported by a retrievable vector repository named VLibrary. Extensive experiments across 7 benchmarks and 3 different LVLMs show that M2IV consistently outperforms both traditional ICL and state-of-the-art vector-based methods like LIVE, with improvements in both accuracy and inference efficiency.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method reduces token overhead while improving performance, which is critical for scaling LVLMs. VLibrary shows promise for broader impact and future extensibility.\n\n2. Theoretical contributions clarify the roles of MHA and MLP, which deepens understanding of transformer internals in multimodal settings.\n\n3. Experiments are thorough and the performance gains are solid across diverse benchmarks. Ablation studies highlight the importance of each component (especially L_{syn}).", "reasons_to_reject": "1. Although the paper claims to address two key challenges—capturing critical task mappings and their interdependencies, and achieving fine-grained semantic distillation—the ablation studies do not explicitly disentangle the method contribute to solving each challenge. More targeted experiments or visualizations could help verify these claims.\n\n2. It is well known that task-specific training or fine-tuning may compromise a model’s general capabilities. The paper does not sufficiently examine whether M2IV, while enhancing performance on trained tasks, may degrade the model’s performance on unseen tasks or its general tasks.", "questions_to_authors": "1. Can the authors provide more fine-grained analysis or visualizations to demonstrate how M2IV captures interdependencies among demonstrations and supports semantic distillation?\n\n2. Does applying M2IV to a specific domain compromise the LVLM’s general capabilities or performance on unrelated tasks?", "extra_scores": {}}, {"review_id": "0u8K1IENQV", "reviewer": "Reviewer_P1rJ", "summary": "This paper presents a new way to improve how large models understand both images and text. Instead of showing these models many examples, this method uses special learnable \"In-context Vectors.\" This technique helps the models perform better, work more efficiently, and handle more information. The paper also describes a library to store and use these vectors for different tasks like improving how models connect images and text, and enhancing safety. Tests show this new approach works better than older methods.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* **Novel and Impactful Approach:** The paper introduces an innovative method using learnable \"In-context Vectors\" that significantly enhances how models process and understand combined image and text information, leading to superior performance.\n* **Improved Efficiency and Scalability:** This new technique streamlines multimodal learning, making models more efficient in terms of processing and able to handle a large number of examples, overcoming previous contextual limitations.\n* **Practical Toolkit for Versatile Applications:** The research provides a valuable \"VLibrary\" for these vectors, allowing flexible use in important real-world scenarios such as better aligning model understanding across modalities, tailoring model outputs, and improving safety, thereby paving the way for future advancements.", "reasons_to_reject": "**1. Model-specific datastore overhead**\nDoes the proposed method require constructing a *separate* datastore module for every backbone, simply to maintain a consistent representation space? If so, the engineering burden and memory footprint would scale linearly with the number of models, which is impractical for real-world deployment.\n\n**2. Handling of very long text in CLIP**\nThe paper does not explain how CLIP (or any vision-language encoder) is adapted to cope with overly long textual prompts. Please clarify whether you truncate, segment, or otherwise modify the inputs—and how this choice affects performance.\n\n**3. Lack of genuine multimodal novelty**\nApart from the bespoke datastore, Section 3.1 presents a modelling pipeline that appears identical to a single-modal setup. What concrete mechanisms exploit cross-modal interactions? At present, the approach seems to inherit most of its gains from datastore engineering rather than from any multimodal insight.\n\n**4. Obsolete evaluation baselines & uncertain scalability**\nAll three backbone models are roughly a year old. Could the authors demonstrate similar (or larger) improvements on **current-generation** LVLM checkpoints? Moreover, it remains unclear whether the gains hold across model scales (e.g., 7B → 70B parameters). Without such evidence, the practical impact is questionable.\n\nGrammatical error on line 114: The sentence \"*…framework. by analyzing*\" (line 114).", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pvddklYAQQ", "reviewer": "Reviewer_SUT6", "summary": "The paper proposes M2IV, a method for enhancing multimodal in-context learning in LVLMs by replacing token-heavy demonstrations with compact, learnable vectors injected into the model's residual stream. By explicitly leveraging the distinct roles of multi-head attention and MLP, M2IV achieves fine-grained semantic distillation and efficient cross-modal reasoning. It is trained via a self-distillation framework and enables efficient multi-shot ICL through a vector repository called VLibrary. The approach achieves state-of-the-art performance across seven multimodal benchmarks and three LVLMs, with better efficiency and generalizability compared to prior representation engineering techniques.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel decomposition of residual stream into MHA and MLP components for interpretable and targeted vector injection.\n2. Efficient many-shot scaling through M2IV aggregation, mitigated context window limitations without additional tokens.\n3. Introduction of VLibrary, a plug-and-play system for managing and applying learned vectors.", "reasons_to_reject": "1. M2IV requires task-specific training, which partially undercuts the appeal of ICL as a parameter-free adaptation method.\n2. The approach, though efficient at inference, introduces a non-negligible training overhead and relies on curated training data and clustering pipelines.\n3. Generalization across models or tasks not explicitly trained on remains somewhat unclear; transfer strategies are briefly discussed but not deeply validated.\n4. Limited discussion on failure modes or scenarios where M2IV underperforms (e.g., in highly abstract or text-heavy tasks).", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uefYd5WMf3", "reviewer": "Reviewer_w71E", "summary": "This paper introduces M²IV, a method to improve multimodal in-context learning (ICL) in Large Vision-Language Models (LVLMs) by replacing token-intensive demonstrations with learnable vectors injected into the MHA and MLP blocks. These vectors, trained via self-distillation with mimicry, synergistic, and supervised losses, aim to capture fine-grained semantics and leverage the distinct roles of MHA and MLP for better efficiency and performance, particularly in many-shot scenarios. The paper also proposes VLibrary for storing and reusing these M²IV vectors for various tasks.   \n\nQuality & Clarity: The methodology and experiments are reasonably detailed and extensive, though clarity on some aspects like the synergistic loss and many-shot aggregation could be improved.\n\nOriginality: While building on prior work in representation engineering, the specific M²IV approach targeting both MHA/MLP with synergistic training appears novel.\n\nSignificance: Addresses key ICL challenges (efficiency, performance). If claims hold, M²IV and VLibrary offer a potentially valuable and efficient paradigm for LVLM adaptation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Decent Empirical Results: Demonstrates state-of-the-art performance across a comprehensive set of 7 benchmarks and 3 LVLMs, consistently outperforming both vanilla ICL (few-shot and many-shot) and several existing representation engineering methods (TV, FV, ICV, I2CL, LIVE). The average accuracy gain of 3.74% over 16-shot ICL is notable.\n\nEfficiency Gains: Addresses the high cost of standard ICL by significantly reducing token overhead during inference. The paper provides evidence that the inference time savings can offset the initial training cost, making it potentially cost-effective for widespread application. This efficiency extends to many-shot scenarios, bypassing context length limitations.\n\nPractical Utility (VLibrary): Introduces VLibrary, a practical framework for storing, retrieving, combining, and transferring M²IVs, showcasing potential for real-world applications like cross-modal alignment, output customization (instruction following, explainability), and safety improvements.", "reasons_to_reject": "Incremental Novelty: While the specific M²IV formulation is new, the core concept of using learned vectors to replace ICL demonstrations builds directly upon recent work like LIVE, ICV, I2CL, FV, and TV. The improvement over LIVE, while present, might be seen as incremental, especially considering LIVE also uses trained layer-wise vectors.\n\nComplexity and Specificity: The method requires a specific training process for each task and each LVLM architecture. While VLibrary allows storing these, generating them involves significant computation (though potentially amortized). The transferability experiments, while promising, show a performance drop (-1.48% on average without fine-tuning), indicating model/task specificity remains a factor. The combination strategies also add complexity.\n\nClarity of Theoretical Justification: Theorems 1 and 2 attempt to formalize the distinct roles of MHA and MLP. However, the theorems themselves seem to show decompositions based on linearity/structure rather than fundamentally proving why MHA handles broader semantics and MLP handles fine-grained details in the complex non-linear dynamics of deep networks. The link between these linear decompositions and the high-level claims about MHA/MLP roles could be stronger or more nuanced. The assumptions (e.g., linear MLP in Thm 2 proof) might oversimplify.\n\nLimited Analysis of Failure Cases: While ablations show the importance of different components, the paper provides limited qualitative analysis or discussion of when or why M²IV might fail or underperform compared to vanilla ICL, even if it does so less often. Understanding the limitations is crucial.\n\nData Processing Dependence: The data processing steps (CLIP embeddings, k-means clustering, augmentation) seem crucial for performance, as shown in ablations (Table 2, Fig 9). This adds components and hyperparameters to the overall pipeline, and the sensitivity to these choices (e.g., number of clusters K) could be explored further.", "questions_to_authors": "1. Could you provide more intuition behind the synergistic loss? How exactly does encouraging alignment along the diagonal of M^l and orthogonality off-diagonal foster the desired complementary roles of MHA and MLP?\n\n2. Regarding Theorems 1 and 2: How well do these linear decompositions capture the actual behavior within complex, non-linear LVLMs? Theorem 2's proof assumes a linear MLP transformation (xW); how does this relate to standard MLP blocks with non-linear activations?\n\n3. The many-shot aggregation method is mentioned briefly. Could you elaborate on the pairwise aggregation mechanism for MLP outputs? How is information loss mitigated during this sequential aggregation?\n\n4. While M²IV outperforms vanilla ICL on average, are there specific task types or data characteristics where vanilla ICL might still be preferable or where M²IV shows limitations? A deeper qualitative error analysis would be beneficial.\n\n5. How sensitive is the performance of M²IV to the choice of the CLIP model used for initial embedding/clustering and the number of clusters K in the k-means step?", "extra_scores": {}}]

Author rebuttal:

Meta-review: I am deeply interested in the premise of the paper. As I understand it, the idea is to figure out how to perform ICL with a multi-modal language model. In contrast to normal ICL, where one provides the model with x-y input--output pairs of strings, x may be an image and y may be a string. I had, personally, never really considered this angle despite having published multiple papers in the multimodal LM space. Many of the reviewers' objections seem pedantic to me, e.g., baselines that are too old. For me, the most important aspect of this paper is simply the fact that the approach is quite interesting. There seems to be no critique of its novelty. Moreover, the authors' engagement with the reviewers during the review process is quite comprehensive.
Decision: Accept