Vision-Language Models Create Cross-Modal Task Representations
Authors: Grace Luo
Trevor Darrell
Amir Bar
Venue: ICML 2025 poster (2025)
Keywords: vision-language models
multimodal models
task vectors
function vectors
in-context learning
interpretability
Primary area: social_aspects->accountability_transparency_and_interpretability

Abstract: Autoregressive vision-language models (VLMs) can handle many tasks within a single model, yet the representations that enable this capability remain opaque. We find that VLMs align conceptually equivalent inputs into a shared task vector, which is invariant to modality (text, image) and format (examples, instruction), and may simplify VLM processing. We measure this alignment via cross-modal transfer--the ability of a task vector derived in one modality to trigger the correct generation in another--on a range of tasks and model architectures. Although the task vector is highly compressed, we find that this single vector outperforms prompting the model with the full task information, unique to this cross-modal case. Furthermore, we show that task vectors can be transferred from a base language model to its fine-tuned vision-language counterpart, and that they can be derived solely from instructions without the need for examples. Taken together, our findings shed light on how VLMs internally process task information, and how they map different modalities into common semantic representations.

Vision-Language Models Create Cross-Modal Task Representations
Grace Luo" Trevor Darrell
Amir Bar"
vlm- cross- mrooss-modal-- reps github.io
Abstract
Task
Autoregressive vision-language models (VLMs) can handle many tasks within a single model, yet the representations that enable this capability re- main opaque. We find that VLMs align concep- tually equivalent inputs into shared task vector, which is invariant modality (text, image) and format (examples, instruction), and may simplify VLM processing. We measure this alignment via cross-modal transfer-the ability of a task vec- tor derived in one modality to trigger the correct generation in another-on range tasks and model architectures. Although the task vector highly compressed, find that this single tor outperforms prompting the model with the full task information, unique to this cross- cross-modal case. Furthermore, we show that task vectors can transferred from base language model to its fine-tuned rision-aagguag counterpart, and that they can be derived solely from instructions with- out the need for examples. Taken together, our findings shed light how VLMs internally pro- cess task information, and how they map different modalities into common semantic representations.
Country- Cepital
Food-Color
Modality
Text Instruction
Map country capital
Match
Text
France Paris,
Cherry Red,
Examples
Image Examples
Paris,
Red,
f ((iuput)
Latent Space
Task Representations
Figure I: VLMs map conceptually equivalent inputs into shared task representation. This representation is invariant to the specification, regardless of modality (image, text) and format (examples, instruction).
representation. Consider the tasks shown in Figure 1, which can be equivalently expressed using instructions, text exam- ples, image examples. One might argue that the model is biased towards learning the same simple function that solves the task, for all specifications (Solomonoff, 1964; Valle-Perez et al., 2019; Lin et al., 2023; Huh et al., 2024).
In this work, we provide evidence of such shared task rep- resentation, which invariant to specification modality or format. This representation exists at special token position near the end of the sequence, which is contextualized by the imputs to form a high-level summary, also known as the task vector (Hendel et al., 2023; Todd et al., 2024). This obser- vation non-trivial; recall that VLM training is designed to encourage the alignment of image and text embeddings, but not the emergent task summaries derived from them. This difference is apparent from the t-SNE (van der Maaten & Hinton, 2008) visualization in Figure 2. While the image and text embeddings exhibit no task-specifie grouping (a), the task vectors cluster by the color-coded tasks (b).
1. Introduction
Depending on the input instruction (Liu et al., 2023a) or in- context examples (Alayrac et al., 2022), VLMs can dynami- cally adjust the task executed on the image. This flexibility poses challenge: each task can be defined in a different way, and memorizing every possible variation is imprac- tical. There needs exist some form of compression, representation sharing, manage this complexity.
This brings into question whether the many ways of describ- ing the same underlying task converge to shared VLM
University of California, Berkeley, USA. Correspondence to: Grace Luo sanceluu@@erkelee.edee
To quantify the alignment of these task representations, we evaluate cross-modal transfer. For example, we evaluate the VLM's ability to apply task expressed with text examples onto an image query (see Figure 3). We find that "patch-
Proceedings of the nd International Conference on Machine Learning. Vancouver. Canada. PMLR 267, 2025, Copyright 2025 by the authorís).

---

Vision-Language Models Create Cross-Modal Task Representations
Text Instruction
Country-Capital
AnimalHoung
(a) Cross-Modal Patching
Text Examples
Country-Curency
Food-Color
Examples
AnimaliLatin
Food
Layer L+1
Layer L+1
10
Patch
13
Layer L
Layer L
Canada Ottawa
Peru Lima
Rome
(a) Context Embeddings
(b) Task Vectors
Text Examples Pixt
Image Query Fimg
Figure 2: t-SNE clustering. The representation at final token position, the task vector, summarizes the preceding context. At the same intermediate layer, unlike (a) the image and text embeddings, (b) the task vectors cluster by task (color) not modality (shape).
(b) Cross-Modal Few-Shot Prompting
Transformer
Canada: Otttwa
Peru Lima
Italy
ing" (Zhang& & Nanda, 2023) the task vector, injecting the representation without otherwise specifying the task, often induces the model to generate the correct answer.
Figure 3: Cross-modal transfer. (a) A single compressed task vector from one modality can induce the VLM to per- form the task on queries from another modality, without additional training, this outperforms (b) feeding the full task information (see Table 2).
Here, we summarize our most intriguing findings. First, al- though patching utilizes only a single vector, it significantly outperforms few-shot prompting (Brown al., 2020) with the uncompressed modal examples. Second, while one might expect VLM fine-tuning to significantly alter rep- resentations, we show that task vectors transfer between the base LLM and its corresponding VLM. Third, we show that task vectors can be defined with instructions, an unexplored alternative that is also complementary with examples. Fi- nally, we analyze the answer generation process, and we find that VLMs map different specifications into similar task vectors but maintain distinct image and text embeddings.
the phenomenon where LLMs can adapt to new tasks with few input examples in the forward pass. Olsson al. (2022) hypothesized that ICL is driven by attention heads ("induc- tion heads"), while Xie et al. (2021) interprets ICL as an implicit Bayesian Inference process, and Garg et al. (2022) showed that ICL can emerge the simple case of linear functions. More recently, Hendel al. (2023), Todd et al. (2024), Liu et al. (2024b) hypothesized that ICL creates task (or function) vectors, latent activations that encode the task in LLMs, and Hojel et al. (2024) demonstrated similar behavior in computer vision models. Huang et al. (2024) proposed to use task vectors in VLMs to compress long prompts that would otherwise not fit in a limited context length. We uniquely study the cross-modal properties of VLM task representations, and how conceptually equivalent specifications lead to similar representations.
2. Related Work
Mechanistic Interpretability. The goal of mechanistic interpretability is make deep models more transparent by interpreting how and why model decisions are made (Gilpin al., 2018; Gurnee & Tegmark; Liu et al., 2022; Geva et al., 2020; Nanda et al., 2023). To uncover the relationships within the model, causal interventions (Pearl, 2022) are often used. For example, activation patching (Zhang & Nanda, 2023) is a technique used to modify neural network activations to observe changes in outputs, often with causal insights to correct biased or erroneous behavior (Meng et al., 2022; Bau al.). We use activation patching to demonstrate that task representations transfer across modalities.
Vision-Languann Models. Recent VLMs can be catego- rized into modality late-fusion (Liu et al., 2023a; 2024a; Li et al., 2023; Tong et al., 2024) and carly-fusion (Bav- ishi al., 2023; Lu al., 2022; 2023; Team, 2024; Zhou et al., 2024) approaches. Late-fusion approaches typically combine a pre-trained visual encoder and L.MM by train- ing adapters, potentially with short end-to-end fine-tuning stage. In contrast, carly-fusion approaches focus on end-to- end training without any pre-initialization of the representa- tions. We observe cross-modal task representations both categories, suggesting that this property can emerge regard-
In- In-Context Learning. With the advent of large language models (LLMs) (Brown et al., 2020), researchers have sought to explain in-context learning (Liu et al., 2023b),
2

---

Vision-Language Models Create Cross-Modal Task Representations
Table I: Cross-modal tasks. We design six tasks inspired by the text examples in prior work (Hendel et al., 2023; Todd et al., 2024), where we add alternative specifications such as instructions and image examples.
Task
Instruction
Text Example
Image Example
Country-Capital
The capital city of the country:
{Greece: : Athens]
Atthen))
Country-Curreng
The last word of the official currency of the country:
{Italy Euro)
EEuro)
Animal-Latin
The scientific name of the animal's species in latin:
{Gray Wolf: Canis lupus)
Canis lupus)
Animal-Young
The term for the boby of the animal:
{Common Dolphin: : calf)
: calf)
Food-Color
The color of the food:
{Persimmon : orange)
: .orange)
Food-Flavor
The flavor descriptor of the food:
(Strawberry sweet)
sweet)
less of the initialization. Several works examine image ICL in VLMs, where they propose new models (Alayrac et al., 2022: Laurençon et al., 2024; Jiang et al., 2024) and analyze the impact of example selection on performance (Doveh et al., 2024; Baldassini et al., 2024). Our work offers new perspective on image ICL by comparing it with text ICL and demonstrating the similarity between the two processes. We even show that LLaVA (Liu et al., 2023a), which lacks image ICL capabilities, can still benefit from text ICL.
look at a special case of transferring task information from a base LLM to its fine-tuned VLM in Sec. 4.2. 2. Instructions. Beyond the examples studied in prior work, we also consider instructions in Sec. 4.3. We also evaluate ensembling the instructions and examples to bet- ter convey the task. Finally, in Sec. 4.5 we explore task overriding, where the goal is to override a task locally defined in the prompt with global instruction.
3. Image Examples. For image queries, image examples represent the unimodal baseline, while text examples and instructions are considered cross-modal. We compare how the model processes image versus text examples by analyzing representation evolution in Sec. 5. Similarities between the processes suggest cross-modal specifications are valid alternative unimodal ones.
3. Cross-Modal Task Representations
In this work, we are interested in studying the task rep- resentations of VLMS. We assess how cross-modal these representations are, the extent to which the model aligns inputs from different modalities into shared task represen- tations. We outline common VLM task specifications in Sec. 3.1, followed by our cross-modal patching formula- tion, where we show task vector can be derived from one modality and transferred to another, Sec. 3.2.
Evaluation Tasks. We construct six tasks, each defined by a single text instruction, a pool of text examples, pool of image examples, as seen in Table 1. For each task, we split the example pool into 30 samples for validation and 100 for testing, where the split is kept consistent across modalities. Each sample is then used as a query, where its corresponding answer is the ground-truth label. To create an example-based specification, we randomly select N samples without replacement, while ensuring no overlap with the query. We provide more details in Sec. A.1 of the Appendix.
3.1. Task Specifications for VLMs
To study this question, we start by designing an evaluation set of cross-modal tasks. This evaluation set should contain different specifications that refer the same underlying task, that we can evaluate the alignment of their task representations.
3.2. Cross-Modal Patching
Specifications. Below, we enumerate three common task specifications for prompting VLMs, which we study in our work (also see Table 1). Given that VLMs are primarily used for image analysis, in this work we focus on image queries and discuss each setting with this in mind.
Although we observe initial evidence of shared represen- tations via clustering (Figure 2), we need more rigorous method for quantifying alignment. To this end, we propose cross-modal patching, where we transfer the representation across modalities and measure the effect on model outputs. We observe that a task representation derived one modal- ity can be used to trigger the correct generation another. We discuss our method in detail next.
1. Text Examples. We first measure cross-modal transfer. We investigate whether functions defined via text exam- ples can generalize to image queries in Sec. 4.1. We also
3

---

Vision-Language Models Create Cross-Modal Task Representations
Table 2: Cross-modal transfer results. We display the accuracy across six tasks on an unseen test set. For image queries, patching cross-modal task vectors (Text Examples Patch) outperforms few-shot prompting (Text Examples Prompt) and the strong unimodal baselines (Image Examples Prompt, Patch). The best method per task is underlined and overall bolded. We also denote whether the method cross-modal (V) or not (x).
Method
Cross Modal?
Country-Capital
Country- Currency
Animal Latin
Animal Young
Food Cold-
Food Flavor
Food-Flavor
Avg.
Random
0.00
0.12
0.00
0.18
0.24
0.31
0.14
LLaVA-vL5
No Context
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Image Examples Prompt
Image Examples Patch
Text Examples Prompt
0.02
0.18
0.03
0.23
0.28
0.37
0.18
Text Examples Patch
0.31
0.30
0.26
0.18
0.53
0.31
0.32
Munlis-Fuyu
Context
0.00
0.00
0.00
0.00
Image Examples Prompt
0.11
0.13
0.24
0.05
0.34
0.23
0.18
Imuge Examples Patch
0.17
0.03
0.16
0.05
0.50
0.31
0.20
Text Examples Prompt
0.09
0.06
0.08
0.02
0.23
0.04
0.09
Text Examples Patch
0.32
0.23
0.36
0.09
0.51
0.36
0.31
Idefies2
No Context
0.03
0.00
0.08
0.00
0.01
0.01
0.01
Imuge Examples Prompt
0.71
0.57
0.43
0.12
0.41
0.35
0.43
Image Examples Patch
0.58
0.32
0.40
0.03
0.39
0.17
0.31
Text Examples Prompt
0.11
0.03
0.41
0.13
0.21
0.18
0.18
Text Examples Patch
0.40
0.48
0.62
0.51
Method. In Figure 3a, we illustrate cross-modal patching. Given a task representation derived from one modality and query in another, the goal is to induce the model to output the correct task-specific answer. See Sec. 3.1 for all combi- nations of specification and query that we consider. For task & T and model f, we run two forward passes: one to extract the task vector from the specification, and another to apply the vector onto an unseen query (see Figure 3a):
4. Experimental Results
We start by evaluating moosll transfer. We quantify text-to-image transfer in Sec. 4.1, including LLM to VLM transfer in Sec. 4.2. We then examine instruction-basee task vectors in Sec. 4.3. Finally, we analyze an extended set of VQA tasks in Sec. 4.4 and demonstrate how patching can override pre-existing tasks in Sec. 4.5.
Models. We consider three VLMs spanning both early and late-fusion architectures. LLa (Liu et al., 2024a) is late-fusion model that fine-tunes projection from visual features into the representation space of a language model. Mantis-Fuyu (Bavishi et al., 2023; Jiang et al., 2024) is an Instruction-tuned variant of an carly-fusion transformer trained to jointly handle image and text inputs from scratch, where the "visual encoder" is a linear projection on top of the raw image patches. Idefics2 (Laurençon et al., 2024) is fate-fusion model optimized for multimodal ICL.
M = ((pta)
Vung = Spanch (Fing Ni,
(1)
Conditioned on the examples Pluss the task vector Mi is extracted from the raw output of the (-th model layer at the delimiter token between the last query-answer pair, follow- ing standard practice in language-only studies (Hendel et al., 2023). For unseen query Timg. the task vector is then patched at the corresponding layer and token position of the query to induce the task-specific answer Ving, without otherwise specifying the task. Since the query modality is completely unseen, the representation needs to encode a very simple and generic function for the task to induce the correct output. In contrast with prior language-only studies (Hendel al., 2023; Todd et al., 2024), our objective is to measure cross- modal alignment between task representations. Whereas these studies solely focus on text examples, we study addi- tional modalities (images) and formats (instructions).
Implementation Details. When conditioning on examples, we use the generic template from Todd et al. (2024):
o: (z1) } \nA: (11 } Aal.....--............. Q: (2,,) AnA: {yn}
where we evaluate with N = 5 examples. For instructions, we pass the raw string with no templating. We evaluate on the six cross-modal tasks illustrated in Table 1, each split into a validation and test set. We determine the best layer to patch for each model via average task accuracy on the validation set. We report metrics on the unseen test set, averaged over three seeds. When computing accuracy metrics, we follow prior work (Hendel et al., 2023; Todd et al., 2024) and compare whether the first generated token is an exact match with the pre-defined label. We resize images to a standard width of 224 pixels. All qualitative examples correspond to Idefies2, the best performing model.
We compare cross-modal patching againstfewwsher prompt- ing (Brown al., 2020), where the task specification and query are jointly fed to the model (see Figure 3b). Few-shot prompting serves as a natural point of reference, it is intervention- free and utilizes the full task information. How- ever, these same characteristics also make it less informative than patching for analyzing the single task representation. In our experiments, we also refer to cross-modal patching and shot prompting as Patch and Prompt respectively.
4

---

Vision-Language Models Create Cross-Modal Task Representations
Text Examples + Image Query
Output
Peru
Australia
Micronesia
No Context:
France.
Lima
Canberra
Palikir
Text Examples Prompt:
France Q:A: Italy
Cameroon
South Korea
Text Examples Patch:
Paris.
Yaounde
Seoul
Cheetah
Deer Mouse
Marsh Rabbit
No Context:
Peromyscus maniculatus
Sylvilagus palustris
Capybara. Text Examples Prompt:
Acinonyx jubatus
Capypara Q: Coyoye
Eurasian Red Squirrel
Text Examples Patch:
Killer Whale
Hydrochoerus
hydrochaeris.
Orcinus orca
Sciurus vulgaris
Com
Chayote
Jackfruit
No Context:
Romanesco. Text Examples Prompt:
yellow
green
green
Romanesco Q:Caul
Grapefruit
Leck
Text Examples Patch:
green.
pink
green
Figure 4: Given the same text examples, patching is more effective than prompting. We show qualitative examples transferring task information from text examples image queries. Few-shot prompting (Prompt) regurgitates the input while cross-modal patching (Patch) successfully performs the task.
4.1. Transferring from Text Examples to Image Queries
text baseline, which may indicate that the model is ignoring the text preceding the image query. Since autoregressive VLMs are causally masked, there exists task vector gen- crated by the text examples the prefix, but the model evidently struggles to correctly apply the image query. Patching resolves this issue by explicitly forcing the model to apply the task vector. Comparing Text Examples Patch to Text Examples Prompt, performance improves by 14-33% across all models (see Table 2).
Setup. We measure the performance of text examples ap- plied to image queries on our six cross-modal tasks, fol- lowing the same procedure illustrated in Figure 3. We eval- uate our entire collection of early and fate-fusion models, L.LVA-v1.55 Mantis-Fuyu, and Idefics2. We ablate two key axes of cross-modal patching (Text Examples Patch): the application method (Patch vs. Prompt) and specification modality (Text vs. Image Examples). We also provide the performance of two lower bounds the majority answer from the examples (Random) and the query without any task information (No Context).
Text examples can outperform image examples. The cross-modal text examples (Text Examples Patch) are more helpful than the unimodal image examples (Image Examples Prompt, Patch), outperforming the strongest unimodal baseline by 8-13% across all models. We hypothesize that image examples require additional visual recognition step to understand the task compared with text examples, which may lead to noisier task representations (see Table 6).
Few-shot prompting struggles with cross-modality. As seen in Table 2, few-shot prompting (Text Examples Prompt) struggles to execute the task the image query across all models, performing at most 4% better than Ran- dom. In Figure 4, we depict a common failure mode of few-shot prompting, where the model "captions" or "regur- gitates" the input, rather than adhering to the demonstrated pattern, fact, the results are quite similar the No Con-
Finding 1. VLMs struggle with cross cross-moda modaal few-shot prompting, which fixed by cross-modal patching.
5

---

Vision-Languars Models Create Cross-Modal Task Representations
0.6
VLM
LLM
VLM
Accuracy
0.5
0.95
cesine
0.4
X
Instruction Patch
Figure 5: Inter-model transfer. For the same text examples, the base LLM and fine-tuned VLM contain highly similar task vectors (left). LLM task vectors can be patched onto image queries (right).
Examples Patch Ensemble Fatch
0.3
0
5
10
15
20
25
30
Num Demonstrations
Table 3: LLM to VLM transfer results. We display the cosine similarity between the LLM and VLM task vectors as well as the test accuracy patching from text examples in the LLM to image queries in the VLM..
Figure 6: Ensembling instruction- and example-based task vectors improves sample efficiency. For cross-modal patching onto image queries, we compare the average task accuracy when using instructions, text examples, or an en- semble of the two. We plot the mean accuracy (solid lines) and variance (shaded regions), aggregated over three seeds.
Method
Avg. Cosine Sim.
Avg. Accuracy
Random
0.58
0.14
L.aVA-r1.5
VLM-VLM Patch
0.32
LLM-VLM Patch
0.95
0.37
4.3. Deriving Task Vectors From Instructions
Idefies2
VLM-VLM Patch
0.51
Setup. Here, we demonstrate that task vectors can not only be defined with text examples but also instructions. We mea- sure the cross-modal patching performance of Idefics2, av- eraged across our six tasks, for task vectors derived from in- structions (Instruction Patch), examples (Examples Patch), an ensemble of the two (Ensemble Patch). Ensembling refers to simple element-wiss average of the instruction- and example-based task vectors. We also analyze how per- formance scales with respect to the number of demonstra- tions. This highlights the contrast between instruction- vectors, which require no data, and example-based vectors, which improve with more demonstrations. Because it difficult convey the desired casing style using instruc- tions, in this section only compute accuracy metrics in case- inseiinsensitiee fashion.
LLM-VLM Patch
0.89
0.52
4.2. Transferring from LLMs to VLMS
Setup. Given that many VLMs are initialized from pre- trained LLM, we explore the extent to which the task rep- resentations are preserved after fine-tuning. We limit this evaluation to late-fusion models with a corresponding LLM, where VA-v1.5 corresponds to Vicuna (Chiang et al., 2023) and Idefics2 corresponds to Mistral (Jiang et al., 2023). For six cross-modal tasks, feed the same text examples to the LLM and VLM and compute the cosine similarity of the resulting task vectors. We also perform cross-modal patching from the LLM task vectors to image queries. We include a conceptual illustration in Figure 5.
Ensembling instructions and examples improves sam- ple efficiency. Here, we compare both approaches and as- sess their complementarity through ensembling. Instruction Patch shows competitive patching performance, matching that of Examples Patch composed of five samples (see Fig- ure 6). Ensemble Patch performs even better, improving over the five-sample Examples Patch by 18%. Overall, combining the instruction-based vector improves the sample efficiency and reduces the variance of the example-based vector. We hypothesize that the ensemble performs well because the instruction provides generic task definition less biased by the selection of examples while the examples clarify the output format.
VLMs largely preserve LLM task representations. The task vectors highly similar across the base LLM and its fine-tuned VLM, with cosine similarity of 0.89 or more (see Table 3). In contrast, the random baseline-the average similarity across all mismatched vector pairings-is is 0.58.
Pure language functions generalize to image queries. In Table 3, we compare cross-modal patching the inter- model case, with the VLM-only case copied from Table Surprisingly, the inter-model setting performs 1-5% better than the VLM-only setting (LLM-VLM vs. VLM-VLM Patch). This result suggests VLMs can re-use functions learned only in language by LLMs, and that the base LLM's task representations are somewhat retained after fine-tuning.
Finding 3. Beyond examples, task vectors can also be defined with instructions, which are more concise.
Finding 2. Task vectors can be patched from the base LLM to its corresponding VLM.
6

---

Vision-Language Models Create Cross-Modal Task Representations
Table 4: Cross-modal transfer on extended VQA tasks. We show the test accuracy of cross-modal transfer on image
queries for visual question answering tasks derived from VQAv2 (Goyal et al., 2017).
Method
Food- Class
Shirt-Color
Man-Holding
Avg.
Idefics2
No Context
0.00
0.00
0.00
0.00
Image Examples Prompt
0.70
0.41
0.46
0.52
Image Examples Patch
0.49
0.19
0.39
0.36
Text Examples Prompt
0.85
0.48
0.56
0.63
Text Examples Patch
0.93
0.56
0.59
0.69
Table 5: Task overriding results, Instruction Patch effectively steers the model perform newly introduced task.
Method
Semantic
Syntax
Creative Generation
Factual Recall
Original Task
0.10
0.15
0.08
0.15
Original Task System Prompt
0.09
0.49
0.06
0.15
Original Task Instruction Patch
0.36
0,59
0.65
0.42
4.4. Extended VQA Tasks
Original vs. Overriding Task
Image Query
Output
Setup. Beyond the synthetic tasks in our main evaluation set, we automatically construct an "in-the-wild" evaluation set derived from VQAv2 (Goyal et al., 2017), a visual question- answering dataset consisting images and question-answer pairs. Recall that in our ICL setup, the model only given input-output pairs and must infer the underlying task latent variable. However, VQAv2 encompasses a broad variety of tasks, from object recognition to color classifica- tion, which leads to ambiguities when treated as single monolithic task. To overcome this issue, we use the ques- tions to stratify samples into tasks. We curate subset of questions asked across a large number of images, such that construct 30-samlle validation and 100-sample test set centered around the same task. To construct image examples, we drop the question and use the maage-nnswer pairs as input-output pairs. To construct text examples, we use dense text descriptions generated by -LaVA-NeXT-Ba from the ILAVA-ReCap dataset (Li et al., 2024) as each image's 's textual analog. We provide more details Sec. A.1 of the Appendix.
What is on top of the meat
Original Task:
Sauce. Instruction Patch:
What the green vegetable
broccol1
Original Task:
What color are the letters
Black. what Instruction Patch: Street caz crossing be alert
STEET OF
What does the sign say
Original Task: Get off the leaves you little b******, + Instruction Patch:
Write something very mean
Patching can also be helpful for VQA tasks. We report the results on these questions in Table 4, where we see that cross-modal patching (Text Examples Patch) results a 6% improvement over few-shot prompting with text exam- ples (Text Examples Prompt) and 17% improvement over few-shot prompting with image examples (Image Examples Prompt).
A dog 1s in pile of leaves and it is adorable.
Write something nice
Figure 7: Patching can override pre-existing tasks. We show qualitative examples where the overriding task can su- persede the original task when patched (Instruction Patch), Any offensive text has been redacted.
4.5. Overriding Pre-Existing Tasks
Setup. We now consider a special case of cross-modal patching where the task patch conflicts with existing task given in the prompt. This case mirrors practical chal- lenge where the user may request task that goes against the global system instruction. In Figure 7, we show a few quali-
tative examples where this "overriding task," when patched instruction-based task vector, can supersede the "origi- nal task" in the prompt. We then construct an evaluation set stratified into four different settings for overriding.

---

Vision-Language Models Create Cross-Modal Task Representations
1. Semantic. Both the original and overriding tasks query the image content. We randomly sample 1000 triplets of (image, original task, overriding task) from VQAv2 (Goyal et al., 2017).
condition the model on sets of text or image examples from our six tasks, then cache the representation of the last de- limiter token across all model layers. To "decode" a repre- sentation, we normalize and project the activation with the model's unembedding matrix, which produces a probabil- ity distribution over all vocabulary tokens, following logit lens (nostalgebraist, 2020). Further details and visualiza- tions can be found in Sec. A.8 of the Appendix.
2. Syntax. Both tasks refer to formatting instructions, i.e., answer in ALL CAPS, quotes, or JSON. We re-use the 1000 images from (I), now paired with randomly sam- pled conflicting syntax instructions.
3. Creative Generation. Both tasks are instructions for creative content, i.e., invent a book title, character name, or company name. We re-use the 1000 images from (1), now paired with randomly sampled conflicting creative instructions.
Text and image ICL induce three phases. Here, we ana- lyze the decodings across all model layers for the Country- Capital task. We see that early layers commonly decode to auf (which in Idefics2 globally corresponds to the colon), middle layers decode to task summaries like headquarters, and late layers decode to city names like Rome (see Fig- ure 13 of the Appendix). This progression suggests that the last representation only encodes the input colon early on, compresses the ICL context into a task summary in the mid- dle, then executes the task to produce the answer at the end. In Figure 8, we provide an overview of the layers where the phase changes from input to task to answer occur. For both text and image ICL., we observe that the input phase covers roughly the first 50% of layers, the task phase the next 40%, and the answer phase the last 10%. Hence, semantic rep- resentations emerge in the middle of the VLM, where task vectors derived from these layers achieve the best patching accuracy (see Figure 10 of the Appendix).
Factual Recall. Both tasks are image queries that require external knowledge. We use 148 overlapping images with conflicting questions from OK-VQA (Marino et al., 2019) and A- A-OKVQA (Schwenk et al., 2022).
Since some settings are open open-ended, in this evaluation only, we use GPT4o (OpenAI, 2024) to automatically rate correct- ness, rather than exact string match against a pre-defined answer. We measure how often the generated answer in- dicates intention to perform the overriding task. We compare cross-modal modal patching to the common alternative of including the instruction in the system prompt.
Patching is more effective than system prompting, As seen in Table 5, cross- modall patching is effective in steering the model toward different task. For conflicts in Semantics, Creative Generation, and Factual Recall, patching outper- forms system prompting by 27-59% (Instruction Patch vs. System Prompt). Interestingly, while system prompting per- forms extremely poorly in almost all settings, it performs much more competitively for the Syntax setting, likely be- cause syntactic instructions are often not mutually exclusive. Nevertheless, for conflicts in Syntax, patching still outper- forms system prompting by 10%.
The task phase decodes to task summaries. In Table 6, we take a middle layer in the task phase (L=18) and depict the top decodings for all tasks. We find that task vectors defined in either modality often decode into meta-tokenss that summarize the task, aligning with the observations made in text-only studies (Hendel et al., 2023; Todd et al., 2024). For example headquarters, currency, and species are the top-1 decodings for both text and image ICL in the first three tasks in the table. Even more, the decodings for image ICL are often noisier than text ICL, which suggests that crosll patching could help convey a cleaner expression of the task.
5. Evolution of Shared Task Representations
The task phase clusters by task, not modality. In Fig- ure 2 we take a middle layer in the task phase (L=23) and co-embed the task vectors using t-SNE (van der Maaten & Hinton, 2008). Each point corresponds to a task vector de- rived from a single set of examples, which differ by modality (denoted by shape) and task (denoted by color). The ideal cross-modal representation space would group colors and intermix shapes; the task vectors largely cluster according to this structure (see Figure 2b). While all other tasks form distinct groups, we hypothesize that the food-related tasks (Food-Coll) Color; purple and -lavorr brown) are not well- separated because food color and flavor are correlated. In contrast, for the same layer, the image and text embeddings in the context summarized by the task vector are much more sodaaity-sensitivee sensitive (Figure 2a). The sensitivity of the context
Next, we analyze the answer generation process to under- stand how the VLM maps different modalities into shared representations. In Sec. 4.1 we use text examples as an alternative to image examples. To study why this is possi- ble, we examine each modality independently, and compare the process in which answers are produced. We find that the representations evolve in similar manner, despite the differences in modality. These representations arrive at an interpretable task vector that decodes into task summaries and clusters by task rather than modality.
Setup. For this analysis, we use Idefics2, which supports both text and image ICL, We refer to these settings as ICL because we do not apply any interventions, and therefore make no distinction between patching and prompting. We
8

---

Vision-Language Models Create Cross-Modal Task Representations
Input
task
answer
Input
task
answer
1.0
1.0
Relative Probability
Relative Probability
0.8
0.8
0.4
0.4
0.2
0.2
5
10
15
20
25
30
5
10
15 Layer
20
25
30
Layer
(a) Text ICL
(b) Image ICL
Figure 8: The output evolves in three distinct phases that shared for text and image ICL. Each line represents the relative probability that the layer representation decodes to of the three pre- pre-defined tokens corresponding the input, task, and answer. We plot the mean probability (solid lines) and variance (shaded regions), aggregated over 100 sets of examples. We visualize only the Country-Capinn task; all tasks are in Figure 14 of the Appendix.
Table 6: Task vectors, whether textual visual, often decode to task summaries. The table depicts the top-5 decodings for each task, where & denotes non-word tokens.
& Pavlick, 2022; Pavlick, 2023), or model and data scale as drivers for representational convergence (Huh et al., 2024).
7. Conclusion
Task
Text ICL
Image ICL
Country- Capital
headquarters. cities. city, cidade, centro
headquarters, administr, cities, city, 1
Despite the success of autoregressive VLMs, we lack a clear understanding of their hidden task representations. Our pri- mary observation is that VLMMs map inputs into a shared task representation space, regardless of whether the task is defined by text examples, image examples, or explicit instructions. We evaluate cross-modal transfer, where we observe that few-shot prompting struggle with input regurgitation while patching induces the task more effec- tively. We show that task vectors can be transferred from base LLM to fine-tuned VLM, possibly indicating rep- resentational re-use of functions learned in language. We also show that task vectors can be defined with instructions, which improves the sample efficiency of example-based task vectors. We hope that work will inspire further analysis of shared representation spaces and the internals of VLMs.
Country-
currency, currency, dol- lar, dollurs, Currency
curvency, O. currency, undefined, dollars
Currency
Animal-
species, genus, habitat, mamm, american
species, genus, manun spec, creature
Latin
Animal-
pup, babies, baby, called. young
young, species, script- style, animal. teenager
Young
Food-
yellow, pink, green, pur- ple, orange
green, yes. yellow, verd, yes
Color
Food-
flavor, taste, mild, flav, tastes
yes, none, anger, cerca, vegetables
Flavor
embeddings is consistent with Liang et al. (2024), which observes distinet clustering by modality throughout model layers VLMs. This dichotomy suggests that the VLM maintains modality-specif embeddings in the context and integrates modalities in the task vector.
Acknowledgements
We would like to thank Jiahai Feng, Stephanic Fu, Alexan- der Pan, Alberto Hojel, Lisa Dunlap, Chung Min Kim, Brent Yi, Candace Ross, and Koustuv Sinha for helpful discus- sions and feedback the paper. Authors supported in part by the NSF, DoD, and/or the Berkeley Artificial Intelligence Research (BAIR) industrial alliance program.
Finding 4. VLMS map text and image examples, well instructions, to similar task vectors, despite differences between the image or text embeddings in the context.
6. Limitations
In this work, we show that VLMs learn cross-modal task representations but lack a definitive explanation for why. Empirical studies offer several hypotheses, such as the exis- tence of isomorphic structures between language and other perceptual representation spaces (Abdou et al., 2021; Patel
9

---

Vision-Language Models Create Cross-Modal Task Representations
Impact Statement
Stoica, I,, and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:/ otgyl./g//loogg/////ooooo
This paper studies the cross-modal properties of task vectors. This result yields promising implications for accessibility and interactivity users are no longer limited unimodal examples and can be more expressive when communicating tasks to VLMs. However, the reliability of these models still remain limitation. While VLMs can readily adapt to newly introduced tasks, this often comes at the cost of task accu- racy. Further stress testing is advised before deployment in critical applications.
Doveh, S., Perek, S., Mirza, M. J., Lin, W., Alfassy, A., Arbelle, A., Ullman, S., and Karlinsky, L. Towards mul- timodal in-context learning for vision & language mod- els, 2024. URL https:/ /arxiviivv./../////i/.aa//eawwv/ org/ httg/abb///t aba/a//s/24/0 2403. 12736.
Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. What transformers learn in-context? case study of sim- ple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
References
Abdou, M., Kulmizev, A., Hershcovich, D., Frank, S., Pavlick, E., and Søgaard, A. Can language models encode perceptual structure without grounding? case study in color, 2021. URL https:/ /ttppp:///axiv.../// aawaa.aivv........ orgoo//ggooooooo abs/ 2109. 06129.
Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012. 14913, 2020.
Gilpin, L. H.,, Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., and Kagal, L. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pp. 80-89. IEEE, 2018.
Alayrac, J.-B., Donahue, J., Luc, P. Miech, A., Barr, Has- son, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E.,, Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Si- monyan, K. Flamingo: a visual language model for few-shot learning. Oh, A. H., Agarwal, A., Bel- grave, D., and Cho, K. (eds.), Advances in Neural In- formation Processing Systems, 2022. URL https: //openreview.nee net. tamee/forume forum? atte://oooommmmmotiiimiiaa id=EbMuimbbPbs.
Goyal, Y.. Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the V in VQA matter: Elevating the role of image understanding in Visual Question An- swering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017
Gurnee, W. and Tegmark, M. Language models represent space and time. In The Twelfth International Conference on Learning Representations.
Anthropic. Claude 3.5 sonnet, 2024. URL wwtph:tpps//ooo /www. sonnet..
anthropic. www.co/////eee/. hatmm//ws/ccllll
Baldassini, B., Shukor, M., Cord, M., Soulier, L., and Pi- wowarski, B. What makes multimodal in-context learning work?, 2024.
Hendel, R., Geva, M., and Globerson, A. In-context learning creates task vectors. Findings of Empirical Methods in Natural Language Processing, 2023.
Bau, A., Belinkov, Y., Sajjad, H., Durrani, N.. Dalvi, F., and Glass, J. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations.
Hojel, A., Bai, Y., Darrell, T., Globerson, A., and Bar, A. Finding visual task vectors. European Conference on Computer Vision, 2024.
Huang, B., Mitra, C., Arbelle, A., Karlinsky, L., Darrell, T., and Herzig, R. Multimodal task vectors enable many- shot multimodal in-context learning. arXiv preprint arXiv:2406. htl:2406..13344 2024.
Bavishi, R., Elsen, E., Hawthorne, C., Nye, M., Odena, A.,
Somani, A., and Tasirlar, S. Fuyu-8b: A multimodal architecture for ai agents, 2023. URL https: adept... attbalg///////aa/aaaaaaaaaooaatt fuyu-
Huh, M., Cheung, B., Wang, T., and Isola, P. The platonic representation hypothesis. In ICML, 2024.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are fhw-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
iNaturalist. inaturalist 2021 species classification and detec- tion dataset. ht con/ wttp://www...ooo/iiooeeeeeeeee visiiedial inat_ hatt_//cmm//tree//sstr/////1 2021.
Chiang, W.-L,, Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E.,
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C.,
Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel,
10

---

Vision-Language Models Create Cross-Modal Task Representations
G., Lample, G., Saulnier, L.,, Lavaud, L.. R., Lachaux, M.- A., Stock, P., Scao, T. I.,, Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL https: //arxiv. org/ waa.///11. 06225.
Liu, Z., Kitouni, O., Nolte, S., Michaud, E., Tegmark, M., and Williams, M. Towards understanding grokking: An effective theory of representation learning, Advances in Neural Information Processing Systems, 35:34651- 34663, 2022.
Jiang, D., He, X., Zeng, H., Wei, C., Ku, M., Liu, Q. and Chen, W. Mantis: Interleaved multi-image instruction tun- ing, 2024. URL https: /awxiv.iivi//////////a/////aaava wttpiiv.rrg/bb//4000000 01483.
Lu, Clark, C.,, Zellers, R., Mottaghi, R., and Kembhavi, A. Unified-io: A unified model for vision, language, and multi- tasks, 2022. URL https://aaxii: org/ abs//b///// 2206. 08916
Kumar, R. typo: A python package simu- late typographical errors. https: http:://githbb./// com/ ranvi ranvijaykune jww.kaarykyaaaaaaaaaaa 2023.
Lu, J., Clark, C., Lee, S., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., and Kembhavi, A. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action, 2023. URL https: htt.ttss//axxi...oott abs 2212/231..0 17172.
Laurençon, H., Tronchon, L.., Cord, M., and Sanh, What matters when building vision-language models?, 2024. URL https: /ttp///aaavvveeeeeeeeeeeeeeaaeve org. abcgg///b///
02246.
Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok- vqa: A visual question answering benchmark requiring external knowledge. In CVPR, pp. 3195-3204, 2019.
Li, B., Zhang, H., Zhang, K., Guo, D., Zhang. Y., Zhang, R., Li, F., Liu, Z., and Li, C. Llava-next: What else influ- ences visual instruction tuning beyond data?, May 2024. URL https: http:/llaaaaaaaa v1 github.thh.gi www.thuu.../blog.... io/blog/ 2024-05-25-1lavad
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. Advances in Neu- rul Information Processing Systems, 35:17359-1737. 2022.
Ji, Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language- inguage-image pre- pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730- 19742. PMLR, 2023.
Nanda, N., Chan, L.,, Lieberum. T., Smith, J., and Stein- hardt, J. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.055: 055217, 2023,
Liang W., Yu, L., Luo, L., Iyer, S., Dong, N., Zhou, C., Ghosh, G., Lewis, M., tau Yih, W., Zettlemoyer, L., and Lin, X. Mixture-off oowuurr--o-traasffoooooo transffrrooo:: sparse and scalable architecture for multi-modal foundation models, 2024. URL https::t//// /tl:://axxxxe /aaxiv.vvvvv.///ii///aa///aaaavv org/ abs/2411 04996.
nostalgebraist. interpreting gpt: the logit lens. Less- Wrong, 2020. URL htttt:::/www..esseeeee comm/postss//cccRtsntddaaaaaaa interpreting- the logit- lens.
Lin, Z., Yu, S., Kuang, Z., Pathak, D., and Ramanan, D. Multimodality helps unimodality: Cross-modal few-shot learning with multimodal models. In CVPR, 2023.
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., et al. In-context learning and induction heads. arXiv preprint arXiv:2209. 2022.
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tuning, 2023a.
OpenAl. Gpt-4o system card, 2024. URL https: // openai com/ hetp//nndxx//ee /gppttgp--// 40 system cardd
Liu, H., Li, C., Li, Y., and Lee, J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024a.
Patel, R. and Pavlick, E. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representutions, 2022. URL https:// openreview net / forum? ii- (CEeddaJJEEMMBBC
Liu, P., Yuan, W. Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023b.
Pavlick, E. Symbols and grounding in large language mod- els. Philosophical Transactions of the Royal Society A, 38 (21(2220044)) 2023, doi: 10. (098/ssta.222..0444 URL http: / /ttp////iii//////// www.oo/////.. 1098 / zsta. 2022. 0041.
Liu, S., Ye, H., Xing, L., and Zou, J. Y. In-context vec- tors: Making in context learning more effective and controllable through latent space steering. In Forty- first International Conference on Machine Learning, 2024b. URL https:://///..tt. https:://pppeeeeeeeeeeeeeee ner/forum? ca-JJCCkKyva.
Pearl, J. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl, pp. 3p. 373-392. 2022.

---

Vision-Language Models Create Cross-Modal Task Representations
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Schwenk, D., Khandelwal, A., Clark, C.,, Marino, K., and Mottaghi, R. A-okvqa: A benchmark for visual question answering using world knowledge. In ECCV, pp. 146- 162, 2022.
Solomonoff, A formal theory of inductive inference. i. Information and Control, 7(1):1-22, 1964.
Team, C. Chameleon: Mixed-modal carly-fusion foundation models. arXiv preprint arXiv:2405.090 8, 2024.
Todd, E., Li, M. L., Sharma, A. S., Mueller, A., Wallace, B. C., and Bau, D. Function vectors in large language models. International Conference on Learning Represen- tations, 2024.
Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang. S., Iyer, A., Pan, X., et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406. 16860, 2024.
Valle-Perez, G., Camargo. C. Q., and Louis, A. A. Deep learning generalizes because the parameter-function map is biased towards simple functions. In International Con- ference on Learning Representations, 2019.
der Maaten, L. and Hinton, G. Visualizing data using sne. Journal of Machine Learning Research, 9(86) 2605, 2008. http:
jmlr. wwwnll..rggppppee stg/papers/w9/ vandermaaten088.. html.
: /
Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. An explanation of in-context learning as implicit bayesian inference. arXiv preprint wwXiv:21/1.020880 2021.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training, 2023. URL atttps://arxter org/ /a.b.///33.... 15343.
Zhang. F. and Nanda, N. Towards best practices of activation patching language models: Metries and methods. arXiv preprint 2309 16042, 2023.
Zhou, C., Yu, I.,, Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., and Levy, O. Transfusion: Predict the next token and dif- fuse images with one multi-modal model. arXiv preprint arXiv:2408. 11039, 2024.
12

---

Vision-Language Models Create Cross-Modal Task Representations
A. Appendix
VRAM consumption by 2.4x when compared with prompt- ing. The cost of patching is almost equivalent to processing the query only, since the VLM no longer needs to attend to the long context. Note that computing the task vector requires an upfront cost similar to few-shot prompting, but it is amortized in future runs unlike prompting which requires re-processing the context each time.
A.I. Experimental Details
Models. We provide further details on the models used in our evaluation in Table 8.
Tasks. We show representative examples in Table 1. We scrape the images for all tasks from Wikipedia, because we find that the images tend to depict more clearly identifi- able prototypes, unlike traditional computer vision datasets. For some tasks the labels were automatically generated by Claude 3.5 Sonnet (Anthropic, 2024) and manually cross- checked, unless otherwise noted.
Table 7: We report the computational overhead of a forward pass on N=30 Text Examples, averaged over 100 runs.
Method
Runtime (seconds)
VRAM (GB)
Promptnn (Conlexi + Query)
2.20
20.02
Patching (Task Vector Query)
8.21 0.20
Query Only
0.19
8.21
Country-Capital. Given the name of the country its flag, predict the capital city. The text-only case is identical to Todd et al. (2024).
A.3. Robustness to Noisy Instructions
Country-Currency. Given the name of the country or its flag, predict the official currency. The text-only case is almost identical to Todd et al. (2024), except we remove the country modifier from the currency to make the task harder.
In Table 9 we examine cross-modal patching's robustness to noisy instructions. We evaluate on the same instructions and images described in Table 1, except we introduce typos to the instructions by randomly swapping consecutive charac- ters within words, following the protocol of Kumar (2023). As expected, the performance degrades as the number of typos increases. However, even with typos, patching is able to maintain non-negligible performance.
Animal-Latin. Given the name of the animal its image, predict its scientific name in Latin. The labels are derived from the mammals categorized in iNaturalist (2021).
Animal-Young Given the name of the animal its image, predict the term for its baby.
A.4. Task Overriding Details
We provide further details for the experimental setup of Sec. 4.5. We use GPT4o to automatically rate correctness, which we further illustrate in Figure 9. Below, we also provide example tasks for each setting. Note that for Syntax and Creative Generation, we show the full pool of tasks below. For Semantic and Factual Recall, the examples representative of a larger pool of questions.
Given the name of a fruit or vegetable or its image, predict its iconic color. This task is inspired by the conceptual example first proposed in Hendel et al. (2023).
Food-Color.
FlodoFlavor. Given the name of a fruit or vegetable or its image, predict its iconic flavor profile.
Extended VQA Tasks. Here we provide additional details on the tasks evaluated in Sec. 4.4.
1. Semantic
What color is the umbrella?
What does the sign say?
Food-Class. Given the image or description, answers: What kind of food is this?
What is in the bowl?
2. Syntax
Shirt-Color. Given the image or description, answers:
Format your answer ALL CAPS.
What color is the man's 's shirt?
Pul your answer in quotes.
Man-Holding. Given the image or description, answers: What is the man holding?
Answer in JSON format.
3. Creative Generation
Write creative book title that matches the image.
A.2. Computational Overhead
Invent a name for the main character of the image.
We report the computational overhead of cross-modal patch- ing in Table 7. Unlike few-shot prompting which must retain the examples in the context, patching replaces them with a single activation, which significantly reduces the compu- tational cost. Since this effect is most apparent for long contexts, we evaluate on text examples with dense descrip- tions (see Sec. 4.4). Patching reduces runtime by 11x and
Name a company that could use this image in an ad- vertisement.
4. Factual Recall
Which country won the 2018 world cup?
From which culture does this food originate?
How was this valley formed?
13

---

Vision-Languaae Models Create Cross-Modal Task Representations
Table 8: We study a diverse set of representative VLMs spanning both early-fusion and late-fusion paradigms and varying image ICL capabilities.
LLaVA-v1.5
Mantis-Fuyu (Jiang et al., 2024)
Idefics2
(Liu et al., 2023a)
(Laurençon et al., 2024)
Text Model
Vicuna
Fuyu
Mistral
(Chiang al., 2023)
(Bavishi et al., 2023)
(Jiang al., 2023)
Vision Model
CLIP
Fuyu
SigLIP
(Radford et al., 2019)
(Bavishi et al., 2023)
(Zhai et al., 2023)
Paradigm
Late-Fusion
Early-Fusion
Late-Fusion
Image ICL
No
Yes
Yes
Parameters
7B
8B
8B
Layers
32
36
32
Given the model answer, determine whether it was trying to answer:
A. "questionI"
B. 'question2"
C. Both
D. Neither
It does not matter if the answer is ungrammatical off; assume the model's intent sort its answer into the right category. Output your rating as JSON containing the key 'choice' corresponding each category.
answer: <answer>
questionl: <question>>
question2: <question2>
output:
Figure 9: We prompt to GPT4o automatically rate task overriding. We include case "C" because the VLM sometimes generates multiple sentences cover both tasks, which count success case along with case "B."
Table 9: We report the accuracy of cross-modal patching of noisy instructions onto image queries.
Num. Character Swaps
Country-Capie
Country- Currency
Animal-Latii
Animal Young
Food-Colar
Food-Plavor
Avg
s=0
0.58
0,22
0.34
0.44
0.48
0.29
0.39
0.65
0.37
s=2
0.63
0.14
0.36
0.41
0.48
0.08
0.35
A.5. Extended Discussion of Text Example Transfer
for the early-fusion model, the best task vector lies in the late-middle layers. When comparing tasks, the shape of the curve tends to fall into two categories: a peak then plateau (Food-Color, Food-Flavor) or single sharp peak (all other tasks). We hypothesize that the shape is associated with the diversity of the output space fewer possible outputs make it more likely for later layers, which are closer to the answer representation, to yield a plausible result.
Template Format. While in our main experiments we use the generic template proposed by Todd et al. (2024), here we use the model-specific template for Idefics2:
User: : {#1 www.eeeo..teeereeeeeeeeeeee wtw.:uutttttteeeeaaasasssttaataa /psssistant {y1)
As seen in Table 10, the trends in performance remain con- sistent with Table 2 cross-modal patching significantly outperforms few-shot prompting with text examples.
A.6. Ablating All Modality Combinations
LLM to VLM Transfer. Corresponding to Table 3, in Ta- ble 11 we display extended LLM-VLM transfer results.
In Table 12, we display additional results when patching task vectors in all combinations of example-query modality. For image queries, the cross-modal setting is highly beneficial, where task vectors derived from text examples outperform those from image examples by 11-20% respectively. For text queries, this not the case, where the cross-modal setting underperforms by 9-23%.
Validation Performance. In main experiments, present the test performance of single model layer, which achieves the best average task accuracy on the validation set. In Figure 10 we show the performance of all model layers on this validation set. For the late-fusion models, the best task vector lies near the middle of the network. In contrast,
14

---

Vision-Language Models Create Cross-Modal Task Representations
Table 10: We ablate the template format and display the test accuracy when transferring from text ICL to image queries. We use the recommended template for Idefics2.
Country-Cupital
Country Currency
Animal-Laaim
Animal: Young
Food Avg.
Idelies2
No Context
0.00
0.00
0.07
0.00
0.00
0.00
0.01
Image Examples Prompt
0.74
0.53
0.44
0.12
0.43
0.35
0.44
Image Examples Patch
0.78
0.09
0.40
0.02
0.02
0.01
0.22
Text Examples Prompt
0.16
0.06
0.24
0.12
0.15
Text Examples Patch
0.70
0.44
0.50
0.64
0.54
0.40
0.54
Table 11: We show the test accuracy when transferring task vectors from text ICL in the LLM image queries in the VLM.
Method
Country-Capital
Animal-Latin
Animal Animal-Young
Food-Color
Flad-Flavor
Avg.
LLaVA.v15
VLM- VLM Patch
0.31
0.30
0.26
0.18
0.53
0.31
0.32
LLM-VLM Putch
0.33
0.32
0.25
0.33
0.53
0.45
0.37
Idefies2
VLM-VLM Patch
0.61
0.40
0.48
0.62
0.53
0.39
0.51
VLM
Country-Capiaa
Country-Currency
Animal-Latin
Animal-Yeung
1.0
LLSVA v1.5
Acouracy
0.5
0.0
Mantis- Fayu
Accuracy
Accuracy
Idefies2
0.5
0.0
0
1015202530
101552025
1015202230
100152025 Layer
5 1015202530 Layer
1015202530
Layer
Layer
Layer
Figure 10: We display validation performance for transferring task vectors from text examples to image queries across model-task combinations. Fach subplot shows the accuracy by model layer, with dotted line providing the few-shot prompting baseline accuracy for reference.
Table 12: We display the accuracy when patching task vectors in all combinations of example-query modality. The best- pertrrerf combination for given query modality is highlighted. Each setting is denoted as (Specification Modality)- (Query Modality). The best-performing combination for given query modality is highlighted.
Method
Country Capital
Country: Currency
Animal Latin
Animal Young
Fond Color
Food-Flave Flavor
Avg.
LLaVA-v155
Imuge
Image Patch
Image Patch
0.26
Text
Text Patch
0.97
0.58
0.77
0.20
0.63
0.41
0.59
Image
Text Patch
Mantis-Fuyu
Image
Image Patch
0.17
0.03
0.16
0.05
0.50
0.31
0.20
Text
Image Patch
Text
Text Patch
0.46
0.30
0.48
0.18
0.28
0.36
0.34
Image
Text Patch
0.31
0.01
0.36
0.05
0.40
0.34
0.25
Idefhes2
Image
Image Patch
0.58
0.32
0.40
0.03
0.39
0.17
0.31
Text
Image Patch
0.40
0.48
0.51
Text
Text Patch
0.97
0.61
0.74
0.54
0.63
0.41
0.65
Image
Text Patch
0.81
0.43
0.58
0.04
0.40
0.27
0.42
15

---

Vision-Language Models Create Cross-Modal Task Representations
A.7. Transferring from Image Examples to Text Queries
{nn image of shiba inu sitting on a couch. : Doge}
{A cartoon of a dog wearing a hat sitting in a room engulfed with flames. This Is Fine Dog)
Here we assess the usefulness of task vectors derived from
image examples for text queries. In Figure 11 we depict set of tasks that involve recognizing visual concepts in dense textual descriptions, including mapping the description to a technology company, cartoon character, or popular meme.
{An image of unhappy cat with blue eyes and white and brown fur Grumpy Cat)
A.8. Representation Evolution For All Tasks
Similar to Sec. 4.1, the model struggles when cross-mes modal examples are applied via few-shot prompting (Image Ex- amples Prompt) but performs well when the same exam- ples are patched task vector (Image Examples Patch). Both baselines (Text Examples Prompt, Image Examples Prompt) sometimes generate incorrect answers within the same output domain, suggesting that, rather than focusing on the input-output relationship, the model may be ignoring the input image or description. However, on the evaluation tasks in Table 2, is difficult for image examples to surpass the unimodal text baselines. In Table 12 of the Appendix we include ablation containing all possible combinations of specification-qque query modality for task vector patching, where text examples consistently outperform image exam- ples regardless of the query modality. We hypothesize that this phenomenon can be attributed to the nature of the tasks themselves. In the evaluation tasks, when conditioned on image examples the model also has to complete an implicit recognition task mapping the image to the underlying tex- tual concept. For example, if the model cannot match the flag to the correct country name, it will not be able to pre- dict the correct currency. However, if recognition is instead required in text space, as is the case in Figure 11, image examples may better encode the task. We think that the curation of a comprehensive evaluation set containing dense text descriptions and corresponding visual concepts is exciting future direction.
Implementation Details. For this experiment, we condi- tion the model on some task specification (c.g., text ICL, image ICL) and cache the intermediate activation across all model layers. We "decode" the representations using logit lens (nostalgebraist, 2020), which produces probability distribution over all vocabulary tokens. For each task and specification type, we aggregate statistics over 100 runs with different sets of N = 5 examples.
Discrete Visualization. In our discrete visualization (Fig- ure 13), we collect the top-1 token from each run and visual- ize it as a slice in a pie chart. Below each pic chart, we also show the token corresponding to the largest slices, which represents the most common decodings across all runs.
Continuous Visualization. We provide the pseudocode for the continuous visualization (Figure 14) in Figure 12. In this visualization, we compare the relative probability of three pre-defined tokens corresponding to the input, task, and answer. Specifically, we use the token auf for the input, one of {capital. currency, species, baby, color, flavor) for the task, and each run's 's ground-truth label for the answer. For each run, we take the softmax of the three token probabilities to obtain a normalized probability distribution.
E-SNE Visualization. For the visualizations in Figure 2. we co-embed representations from the same middle layer using t-SNE (van der Maaten & Hinton, 2008). Specifically, we take the raw layer activations, normalize them using the model's final normalization layer to ensure they are com- parable in scale, and then apply the t-SEE algorithm. We select a random subset of 30 points for each (task, modality) combination. For the "context" embeddings, we take the embeddings corresponding to the input in the input-output pairs, excluding any tokens related the template or output, TThe logo is white ghost against a yellow background. : Snapchat)
Dense Text Descriptions. Corresponding to Figure 11, we display the text descriptions used in the text examples.
(The logo a rainbow-colored apple. : Apple)
(The logo is a white camera against a gradient background. : [nstagram]
For image ICL., this refers the embeddings of the input image, while for text ICL, it refers to the embeddings of the input text. For the "task vector" embeddings, we take the embedding corresponding to the last delimiter token, which summarizes the preceding context.
{The logo is the letter P stylized to look like pushpin. Pinter- est)
{The character is a squirrel wearing an astronaut suit. : Sandy Cheeks)
Conditioning on Instructions. We visualize the token representation evolution when conditioning on instructions rather than examples in Figure 15 and Figure 13. We do not display discrete pie charts since a single instruction does not produce aggregate statisties, unlike examples where there are multiple possible sets. The instruction- tor decodings are often interpretable and resemble a meta summary for the task, similar to the observations in Sec. 5.
{The character is a puffer fish wearing blue shirt, red skirt, and blue hat. Mrs. Puff)
TThe character is crab wearing a blue shirt, blue pants, and brown belt. Mr. Krabs}
{The character is a pink starfish wearing green and purple pants. Patrick Star)
vec-
[An image of an orange and white cat wearing a blue shirt playing the keyboard. Keyboard Cat)
16

---

Vision-Language Models Create Cross-Modal Task Representations
Image Examples Text Query
Output
The logo is the letter P stylized look like pushpin.
Text Examples Prompt:
Pinterest
Image Examples Prompt:
Mapquest
Image Examples Patch: Pinterest.
Apple
Snapchat
Instagram
The character is pink starfish wearing green and purple pants.
Text Examples Prompt: SpongeBob
Image Examples Prompt:
Plankton
Image Examples Patch:
Patrick Star.
Sandy Cheeks
Mrs. Puff
Mr. Krabs
An image of an unhappy cat with blue eyes and white and brown fur.
Text Examples Prompt: Garfield
Image Examples Prompt:
Grumpy Cat
Image Examples Patch:
Grumpy Cat
Keyboard Cat
Doge
This Is Fine Dog
Figure 11: Cross-modal transfer from image examples to text queries. We show qualitative examples where few-shot prompting with text examples (Prompt) and image examples (Prompt) often produces incorrect predictions in the same output domain while cross-modal patching (Patch) leads to the correct answer.
def peattinucusrre_eevlllle (model, dataset, gelect_vocab_ist
Plots the relative probability of the input, task, and answer token (given by vocleet_ across layers of 'model', for 'dataset" representing task. u**
Mataset_rel_pron = []
for sample in dataset:
dim is [num_layers, 1, hi hidden_dim]
feats cache_act (model (sample))
feats model.norm (feats)
# dim is coum_layers, 1, vocab_siza
tocke_diiii dist mo heaiiimm_eeadddeeeee i__heaa
input_idx, task_idx, answer_idx select_vocab_tt
dim [nun_layers, 1, 3]
token_dist token [ist ;, [input_idk, task answer_idx] rel_prob = softmax (token_dist)
dataset rel rii_rree________ee___ pwrp..rbb... append (rel_prob)
plot_layer_wy ssweer_vv__ee _elrel_____eeeeeee ses_rel__poot (dataset_rel_prob)
Figure 12: PyTorch-like pscudocode for the continuous visualization shown in Figure 14.
17

---

Vision-Language Models Create Cross-Modal Task Representations
Text ICL
Image ICL (b) Country-Capita
(a)
Layer
Layer 18
Layer 28
Layer
Layer 18
Layer 26
auf
Cities headquarters
capital Rome
auf
hea headquarters cities country
Capital London
city
Lib Bang
Port Rome
(c) Country-Curreney
(d) Country-Current
Layer
Layer 18
Layer 28
Layer 0
Layer 18
Layer 26
auf
currency dollar collars
dollar Euro franc Swiss
auf
currencyy None Natherlands american
Euro dellar pound pcs
(c) Animal- Latin
(f) Animal-Latii
Layer
Layer 18
Layer 28
Layer
Layer 18
Layer
auf
species
Sci Al
auf
sspecies
Sci
Can
genus fund flying
stri
rat
gaz
Animal- Animal-Young
(h) Animal- YouiallYYungg
Layer
Layer
Layer 28
Layer
Layer 18
Layer
auf
babies baby aggn
auf
adult mon
cub baby
american anel young elderly
c infant
rod
young
animal
Food-Color
ii) Food-Color
Layer
Layer
Layer 28
Layer
Layer 18
Layer
green yelfow white red
auf
green purple white yellow
purple color or
yellow yes cxcy verd
orange pirk
(k) Food-Flavor
Food-Flavor
Layer 0
Layer
Layer
Layer
Layer
Layer 28
auf
Sweet
yes corca axy verd
mild taste
purple grean Bitter fruit
mild bitter
bitter
flav
sharp
Figure 13: We show discrete visualization of how the token representation evolves across layers for all tasks. Each pie chart slice represents a 1 decoding across 100 sets of examples, and the most common decodings are displayed below.
18

---

Vision-Language Models Create Cross-Modal Task Representations
Text ICL
Image ICL (b) Country-Capitts
(a) Coantr-Cappita
input
Reable Rec0ee Maboakyy
6.8-
2.6-
5.4
2.4-
20
25
20
25
30
20
20
Layer
Layer
(c) Country-Cureency
(d) Country-Currency
6.8-
0.6-
0.0
5.0
20
Lager
Lager
(e) Animal-Laaa
(f) Animal-Latin
Input
answor
3.8
3.6-
Roubee hooulayy
5.8
2.0
25
35
5.0
20
Layer
Layer
(g) Animal-
HoomallYoung Young
(h) UhimallYoung
Inpout
2.0
2.0
0.2-
5.0
30
0.0
Layer
Laper
(i) Food-Color
(j) Food-Coon
Input
tack
answer
Input
tack
anSwor
2.0
3.0
5.4
2.4-
5.2
5.2
8.8
Lager
(k) Food-Flavor
(I) Food-Flavor
Ipput
task
Irput
task
aowwer
1.0
1.0
9.6
0.-
0.
02
Figure 14: We show a continuous visualization of how the token representation evolves across layers for all tasks. We use the token auf for the input, one of (capital, currency, species, baby, color, flavor) for the task, and each run's ground- truth label for the answer. See Figure 12 for the relevant pseudocode.
19

---

Vision-Language Models Create Cross-Modal Task Representations
Instruction
(a) Country-Capital
(b) Country-Cureency
input
task
onswer
input
task
answer
Relative Probabilly
1.0
1.0
Relative Probabiitty
0.8
0.6
0.6
0.4
0.2
0.2
0.0
0.0
10
20
30
Layer
Layer
(c) Animal-Latin
(d) Animal Young
input
task
answer
input
task
answer
Relative Probability
1.0
Relative Probability
1.0
0.6
0.6
0.4
0.2
0.2
0.0
10
20
30
Layer
Layer
(c) Food-Color
(f) Food-Flavor
input
task
answer
input
task
answer
Relative Probability
1.0
Relative Probability
1.0
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
Layer
Layer
Figure 15: We show continuous visualization of the token representation evolution when conditioned on instructions rather than examples. The results are aggregated over single instruction rather than multiple examples, so there are no variance bars.
Table 13: We depict the top-5 decodings for the instruction-based vector, where denotes symbols that do correspond to common word tokens.
Task
Instruction
Country- Capital
Country-Capital
city, GU, vik, cities, headquarters O. Q. O. itos, Q species, genus, Q. animals, american
Currecy
Animal-Latin
Animal-Young Young
baby. babies, Q. bach, called
Food-Color
colors, color, colour, ETH, ilo
Food-Flavor
tasle, tastes, arom, flavor
20


Reviews:
[{"review_id": "UoE61u9bMx", "reviewer": "Reviewer_fXQS", "summary": "The paper explores how VLMs create cross-modal task representations that are invariant to input modality (text or image) and format (examples or instructions). These task vectors, derived from one modality, can effectively trigger task execution in another. It often outperforms traditional few-shot prompting. The study also shows that task vectors can transfer from base language models (LLMs) to fine-tuned VLMs and can be defined using instructions alone. These findings reveal that VLMs map diverse inputs into shared semantic representations, enhancing their flexibility and efficiency.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and constructive feedback. Below, we add results on larger sample sizes, real-world tasks, and malformed instructions, following your suggestions.\n\n---\n\n> 1. [T]here are several essential related works that are not currently cited\n\nWe will be sure to cite and discuss these related works in the final manuscript.\n\n---\n\n> 2. The sample sizes for validation and testing are relatively small\n\n**We repeat the task overriding evaluation (Table 4 of the main text) with a larger sample size, increasing it from 100 to 1000 examples in Table 2 below.**\n\nHowever, for our evaluation tasks in Table 1 of the main text, the sample sizes are unfortunately upper-bounded by real-world constraints, especially our ability to manually cross-check the input-output pairings with online sources as described in Sec. A.1 of the Appendix.\n\n*Table 2. Task overriding results on VQAv2, scaled from 100 to 1000 examples.*\n|          Method                       | Accuracy | \n|---------------------------------|------------------------|\n| Original Task                   | 0.10                   |\n| Original Task + System Prompt   | 0.09      |\n| Original Task + Instruction Patch | **0.36**      | \n\n---\n> 3. The tasks used in the experiments are relatively simple and may not fully capture the complexity of real-world applications.\n\n**In Sec. A.4 of the Appendix we include an evaluation based on VQAv2, which represents a more in-the-wild set of tasks.** We copy Table 10 from the Appendix to Table 1 below for your convenience. In this experiment, we derive the tasks from questions from VQAv2, where each ICL example is composed of either complex real-world images or dense text descriptions as input. Consistent with Table 2 in the main text, Table 1 below shows that cross modal patching outperforms few-shot prompting.\n\nWe agree that multimodal multi-step reasoning tasks, which likely require strong language capabilities when analyzing images, could benefit from our method and are an interesting avenue for future work.\n\n*Table 1. We show the test accuracy of cross-modal transfer on image queries for visual question answering tasks derived from VQAv2.*\n| Model         | Food-Class | Shirt-Color | Man-Holding | Avg. |\n|--------------------|----------------|-----------------|-----------------|----------|\n| No Context       | 0.00           | 0.00           | 0.00           | 0.00     |\n| Image ICL Prompt   | 0.70           | 0.41           | 0.46           | 0.52     |\n| Image ICL Patch    | 0.49           | 0.19           | 0.39           | 0.36     |\n| Text ICL Prompt   | 0.85           | 0.48           | 0.56           | 0.63     |\n| Text ICL Patch | **0.93**       | **0.56**       | **0.59**       | **0.69** |\n\n---\n\n> 4. The paper assumes that instructions are well-formed and unambiguous\n\n**In Table 5, we examine cross-modal patching’s robustness to malformed and ambiguous instructions by adding typos at varying levels to the instruction.** As one would expect, the performance degrades as the number of typos increases. However, even with typos, patching is able to maintain non-negligible performance.\n\n*Table 5. Robustness to noisy instructions. We randomly swap consecutive word characters in the instruction, following the protocol of [1]. We report the accuracy of cross-modal patching onto image queries with these noisy instructions.*\n| Num Character Swaps       | Country-Capital | Country-Currency | Animal-Latin | Animal-Young | Food-Color | Food-Flavor | Avg  |\n|-----------------------|------------------|-------------------|--------------|--------------|-------------|--------------|------|\n| s=0      | 0.58  | 0.22  | 0.34  | 0.44  | 0.48  | 0.29  | **0.39**  |\n| s=1      | 0.65  | 0.07  | 0.33  | 0.51  | 0.52  | 0.13  | **0.37**  |\n| s=2      | 0.63  | 0.14  | 0.36  | 0.41  | 0.48  | 0.08  | **0.35**  |\n\n---\n\n> 5. The study does not explore the impact of different instruction formats or the optimal balance between instructions and examples. This could be an important area for future research\n\nWe agree that it would be worthwhile to study a larger set of instruction formats, such as ablating whether the instruction is stated as a question versus command or the instruction length.\n\nWhile we conducted preliminary investigation of the balance between instructions and the number of examples in Figure 6 of the main text, we agree that further exploration – such as testing different weightings when averaging the two vectors – would be interesting. \n\nWe would like to thank the reviewer for the exciting and promising suggestions for future research.\n\n---\n\nReferences\\\n[1] https://github.com/ranvijaykumar/typo", "claims_and_evidence": "The paper presents evidence supporting its claims about cross-modal task representations in VLMs. It demonstrates that VLMs create shared task vectors invariant to input modality and format, which outperform traditional few-shot prompting. The authors show that these task vectors can transfer from base LLMs to fine-tuned VLMs and can be derived from instructions alone. Experiments, including cross-modal patching and task overriding, provide robust evidence for these claims. The findings reveal how VLMs map diverse inputs into common semantic representations, enhancing their flexibility and efficiency.", "methods_and_evaluation_criteria": "The methods and evaluation criteria used in this paper are aligned with the research goals and provide robust evidence for the claims made. The use of cross-modal patching, evaluation of instruction-based task vectors, and transfer from LLMs to VLMs, combined with appropriate metrics like accuracy and cosine similarity, make this study comprehensive and insightful. The findings are good contributions to the understanding of how VLMs process and align task representations across modalities, and the proposed methods are likely to inspire further research in this area.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental designs and analyses are methodologically sound and provide a few insights (see Findings) into the cross-modal alignment of task representations in VLMs. The use of multiple VLM architectures and a diverse set of tasks ensures that the findings are not model-specific and can be generalized. \n\nHowever, there are areas for improvement: increasing the sample sizes for validation and testing would enhance the statistical power and reliability of the results. Incorporating more complex, multi-step reasoning tasks could offer deeper insights into the models' capabilities. Evaluating a broader range of VLM and LLM pairs would strengthen the conclusions regarding the preservation of task representations during fine-tuning. Additionally, supplementing the visualizations with more quantitative measures of representation alignment would provide a more rigorous assessment of how different modalities are mapped into shared semantic spaces. Addressing these aspects would further solidify the findings and enhance their applicability to real-world scenarios.", "relation_to_broader_scientific_literature": "The paper advances the understanding of how VLMs process and align task representations across different modalities. It builds on prior work in vision-language alignment, mechanistic interpretability, and in-context learning by introducing the concept of cross-modal task vectors. These vectors, derived from either text or image inputs, are shown to be invariant to modality and can be effectively transferred between language models and VLMs. The study extends activation patching techniques to a cross-modal context, demonstrating that task vectors can induce correct task-specific outputs even when applied to a different modality. Additionally, the paper shows that task vectors can be efficiently derived from instructions, offering a more sample-efficient alternative to example-based learning.", "essential_references_not_discussed": "This paper provides a good review of related studies. However, there are several essential related works that are not currently cited or discussed in the paper, which could provide additional context and depth to the key contributions. Here are some examples:\n\n1. Lin Z, Yu S, Kuang Z, et al. Multimodality helps unimodality: Cross-modal few-shot learning with multimodal models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 19325-19337.\n\n2. Doveh S, Perek S, Mirza M J, et al. Towards multimodal in-context learning for vision & language models[J]. arXiv preprint arXiv:2403.12736, 2024.", "other_strengths_and_weaknesses": "Pro:\n1. The paper is generally well-written. I really appreciate the step-by-step experimental and analytical approach taken in this paper.\n2. The paper evaluates a diverse set of VLMs, including both early-fusion and late-fusion models (LLaVA-v1.5, Mantis-Fuyu, Idefics2). This comprehensive evaluation ensures that the findings are not specific to a single model architecture and enhances the generalizability of the results.\n3. The authors design a set of six cross-modal tasks that cover a range of semantic relationships. This diversity in tasks helps in understanding the robustness of the proposed methods across different types of tasks.\n4. The findings have practical implications for improving the efficiency and flexibility of VLMs. For example, the ability to derive task vectors from instructions alone can significantly reduce the need for large datasets, making the models more accessible and easier to deploy in real-world applications.\n5. The demonstration that task vectors can be transferred from LLMs to VLMs suggests that pre-trained language models can be effectively leveraged to enhance multimodal tasks, which is a promising direction for future research and development.\n\nCon:\n1. The sample sizes for validation and testing are relatively small (30 for validation and 100 for testing). This limits the statistical power and generalizability of the results. Larger sample sizes would provide more reliable estimates of model performance and enhance the robustness of the findings.\n2. The tasks used in the experiments are relatively simple and may not fully capture the complexity of real-world applications. More complex, multi-step reasoning tasks could provide deeper insights into the models' capabilities and limitations.\n3. The quality and clarity of instructions used to derive task vectors are critical. The paper assumes that instructions are well-formed and unambiguous, which may not always be the case in real-world scenarios. Ensuring high-quality, unambiguous instructions is essential for the validity of instruction-based task vectors.\n4. The study does not explore the impact of different instruction formats or the optimal balance between instructions and examples. This could be an important area for future research to improve the effectiveness of instruction-based learning.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See \"Other Strengths And Weaknesses\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CuD1Z3MmP3", "reviewer": "Reviewer_CQRE", "summary": "This paper explores how autoregressive vision-language models (VLMs) form cross-modal task representations, which it identifies as \"task vectors.\" These vectors efficiently encode task information across text and image inputs, enabling effective cross-modal transfer. The authors demonstrate task vectors surpass traditional few-shot prompting, transfer seamlessly from language-only models to vision-language models, and show enhanced efficiency when combining textual instructions with examples.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. Following your suggestions, we’ve added **four new experiments** on task complexity, task overriding, and practical considerations.\n\n---\n\n> 1. [Add] more complex tasks […] such as VQAv2.\n\n**In Sec. A.4 of the Appendix we include an evaluation based on VQAv2.** We copy these results to Table 1 below. Table 1 shows that patching is also effective in more complex tasks. Following this discussion, we will move this result to the main text.\n\n*Table 1. Test accuracy of cross-modal transfer on tasks derived from VQAv2.*\n|Model|Food-Class|Shirt-Color|Man-Holding|Avg.|\n|-|-|-|-|-|\n|No Context|0.00|0.00|0.00|0.00|\n|Image ICL Prompt|0.70|0.41|0.46|0.52|\n|Image ICL Patch|0.49|0.19|0.39|0.36|\n|Text ICL Prompt|0.85|0.48|0.56|0.63|\n|Text ICL Patch|**0.93**|**0.56**|**0.59**|**0.69**|\n\n---\n\n> 2. [D]emonstrate [...] semantic vs. syntactic conflicts [...] factual recall vs. creative generation\n\n**Table 2 further stratifies task overriding performance, and scales up the number of evaluation samples to 1000.** Patching outperforms system prompting by 27-59\\% for conflicts in Semantics, Creative Generation, Factual Recall. For conflicts in Syntax, system prompting significantly improves, likely because syntactic instructions are often not mutually exclusive.\n\n*Table 2. Task conflict stratified by degree of conflict and task domain.*\n|Method|(a) Semantics|(b) Syntax|(c) Creative Generation|(d) Factual Recall|\n|-|-|-|-|-|\n|Original Task|0.10|0.15|0.08|0.15|\n|+ System Prompt|0.09|0.49|0.06|0.15|\n|+ Instruction Patch|**0.36**|**0.59**|**0.65**|**0.42**|\n\nSince some settings are open-ended, we use GPT4o to rate correctness.\n- (a) Same as Table 4 in the main text, scaled to 1000 images.\n- (b) Formatting instructions (e.g., answer in ALL CAPS, quotes, JSON) on same 1000 images.\n- (c) Creative prompts (e.g., invent a book title, character name, company name) on same 1000 images.\n- (d) “Outside knowledge” questions on 148 overlapping images from OKVQA and A-OKVQA.\n\n---\n\n> 3. [T]est [...] cases where the original task is deeply ingrained in the model’s pre-training.\n\n**Table 3 shows that patching can still override deeply ingrained tasks.**\n\n*Table 3. Stratification of 1000 examples from Table 2a by level of ingrainment.*\n|Method|Highly Ingrained|Moderately Ingrained|Lightly Ingrained|\n|-|-|-|-|\n|Original Task|0.10|0.11|0.04|\n|+ System Prompt|0.04|0.09|0.08|\n|+ Instruction Patch|**0.20**|**0.36**|**0.38**|\n\nWe measure ingrainment by the question perplexity (PPL), and denote tasks with bottom 5% PPL as Highly Ingrained, middle 90% as Moderately Ingrained, and top 5% as Lightly Ingrained.\n\n---\n\n> 4. [Discuss] computational overhead of patching.\n\n**Table 4 shows the computational overhead of patching.** Patching cuts runtime by 11x and VRAM by 2.4x compared with few-shot prompting, for long text descriptions (see Sec. A.4 of the main text). After injecting the single task vector, the VLM no longer needs to attend to the long context. While computing the task vector requires an upfront cost, it is amortized in future runs.\n\n*Table 4. Overhead of a single forward pass on N=30 Text ICL examples, averaged over 100 runs.*\n|Method|Runtime (seconds)|VRAM (GB)|\n|-|-|-|\n|Prompting (Context + Query)|2.20|20.02|\n|Patching (Task Vector + Query)|0.20|8.21|\n|Query Only|0.19|8.21|\n\n---\n\n> 5. [Discuss] robustness to noisy inputs, or sensitivity to instruction phrasing.\n\n**Table 5 shows the robustness of patching to noisy instructions.** As one would expect, the performance degrades as the number of typos increases. However, even with typos, patching maintains non-negligible performance.\n\n*Table 5. Accuracy of patching for instructions with varying levels of typos.*\n|Num Character Swaps|Country-Capital|Country-Currency|Animal-Latin|Animal-Young|Food-Color|Food-Flavor|Avg|\n|-|-|-|-|-|-|-|-|\n|s=0|0.58|0.22|0.34|0.44|0.48|0.29|**0.39**|\n|s=1|0.65|0.07|0.33|0.51|0.52|0.13|**0.37**|\n|s=2|0.63|0.14|0.36|0.41|0.48|0.08|**0.35**|\n\n---\n\n> 6. Have you explored [...] robustness [...] under substantial domain shifts\n\nWe have not yet explored domain shifts, but we agree it is a compelling future direction.\n\n---\n\n> 7. Could you provide [...] potential reasons behind [...] representational convergence\n\nOne possibility is that multi-task learning drives compression. VLMs, trained via next-token prediction on diverse web data, implicitly learn multiple tasks at once [1]. Since the same task can be defined in many different ways, and memorizing every variation is impractical, some form of representation sharing is needed to manage this complexity. Other studies have also arrived at this hypothesis [2]. \n\nWe are also interested in a deeper analysis, but this would require a dedicated study of its own, so we leave it to future work as noted in Sec. 6 of the main text.\n\n---\n\nReferences\\\n[1] Brown et. al. Language Models are Few-Shot Learners. NeurIPS 2020.\\\n[2] Huh et. al. The Platonic Representation Hypothesis. ICML 2024.", "claims_and_evidence": "The claims in this paper are convincingly supported by rigorous experiments and empirical analysis. Results consistently demonstrate that task vectors outperform standard few-shot prompting across multiple tasks and models, clearly validating the core claims about cross-modal transfer capabilities.", "methods_and_evaluation_criteria": "The proposed method of cross-modal task vector patching and the selection of evaluation tasks are well-suited for examining VLM capabilities. The benchmark tasks are diverse enough to illustrate meaningful differences and strengths of cross-modal representations, providing appropriate evaluation criteria.", "theoretical_claims": "The paper does not present explicit theoretical proofs. Therefore, theoretical claims are not a concern in this context.", "experimental_designs_or_analyses": "Experimental designs are robust and systematically presented. The quantitative evaluation of different modalities and methods (instruction-based, example-based, and ensemble patching) effectively demonstrates the superiority and generalizability of the proposed cross-modal task vectors. However, broader task variety would be beneficial to strengthen claims further.", "relation_to_broader_scientific_literature": "The contributions of this paper build meaningfully on existing literature regarding mechanistic interpretability, particularly in-context learning and interpretability in vision-language contexts. The discussion effectively situates the contributions within current frameworks and models, such as those by [1] and [2], without redundancy.\n\n[1] Hendel, R., Geva, M., and Globerson, A. In-context learning creates task vectors. Findings of Empirical Methods in Natural Language Processing, 2023.\n\n[2] Todd, E., Li, M. L., Sharma, A. S., Mueller, A., Wallace, B. C., and Bau, D. Function vectors in large language models. International Conference on Learning Representations, 2024.", "essential_references_not_discussed": "The paper adequately cites relevant literatures.", "other_strengths_and_weaknesses": "## Strengths\n- Clearly demonstrates cross-modal alignment through systematic and compelling experimental validation.\n- The method of task vector extraction and patching is innovative and well-explored empirically.\n- Enhances interpretability and understanding of VLMs significantly through clear presentation and insightful analyses.\n\n## Weaknesses\n- **Limited Task Complexity**: While the paper claims robustness in task vectors, the experimental setup relies predominantly on simplified tasks (e.g., mapping capitals to countries or matching foods with colors). It remains unclear if these findings generalize well to more complex multimodal tasks that involve nuanced reasoning or domain-specific knowledge. Simple tasks may not reflect real-world challenges. I'd recommend adding more complex tasks to the evaluation like such as VQAv2.\n- **Task Overriding**: The \"task overriding\" experiment in Section 4.4 presents compelling qualitative examples (Figure 7) showing that patching can supersede an original task in the prompt, but the quantitative backing is insufficient. Table 4 reports results on only 100 random pairs of conflicting questions from VQAv2, with a single accuracy metric (0.32 for Instruction Patch vs. 0.05 for System Prompt). This limited evaluation does not demonstrate the general effectiveness of task overriding across diverse scenarios, such as tasks with varying degrees of conflict (e.g., semantic vs. syntactic conflicts) or different task domains (e.g., factual recall vs. creative generation). For instance, the paper does not test whether patching can override tasks in cases where the original task is deeply ingrained in the model’s pre-training (e.g., answering \"What is the weather like?\" when overridden to \"What is the historical significance of this location?\"). To address this, the authors should expand the quantitative evaluation to include a broader range of conflicting task pairs, stratified by conflict type and domain, and report additional metrics, such as the proportion of outputs that partially retain the original task, to assess the robustness of overriding.\n\n- **Missing Practical Considerations**: The paper does not discuss practical considerations for deploying cross-modal task vectors, such as computational overhead of patching, robustness to noisy inputs, or sensitivity to instruction phrasing. These are critical for real-world applicability, especially given the Impact Statement’s emphasis on accessibility.", "other_comments_or_suggestions": "No I don't have any other comments or suggestions.", "questions_for_authors": "- Have you explored or considered the robustness of task vectors under substantial domain shifts, such as specialized domains like medical or scientific imagery?\n- Could you provide further quantitative or analytical insights into the potential reasons behind the observed representational convergence across modalities, especially regarding model architecture and training procedures?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "taPcRKUUAk", "reviewer": "Reviewer_2rbY", "summary": "The paper studies the capability of task representation sharing/transfer between VLM and LLM. The authors identify a 'task vector' (the delimiter token between the last query-answer pair) in one modality, transfer it to the other modality, and test the model capacity to achieve the given task without additional prompting or fine tuning. Six cross-modal tasks are constructed and several cross modal evaluation procedures are evaluated (eg, LLM -> VLM).  Three multi-modal architectures are considered (LLava, ideafics and Mantis-Fuyu). Quantitative evaluation is reported using 100 test samples.\n\n## update after rebuttal\nThanks to the authors for the rebuttal, including clarifications and additional experiments. Those additional studies consolidate the paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. Below, we report additional results that address your concerns, including evaluation on in-the-wild tasks, and clarifications of the experimental description.\n\n---\n\n> 1. [T]he claims and the conclusions are not surprising, as it is straightforward that VLM maps inputs from different modalities into a shared space.\n\nWe study VLMs trained to map image embeddings into the representation space of an LLM. \n- Although this can be simplified as mapping different modalities into a shared space, this perspective does not explain our results. \n- **In fact, Figure 2a of the main text shows that the image and text embeddings don’t cluster by task. In contrast, Figure 2b shows that the task vector at the end of the sequence does group by task.**\n- This grouping in Figure 2b is not supervised in any way – its emergence is driven only by the next token prediction loss, or the fact that the answer is conceptually similar.\n- We find it striking that these task representations, learned without explicit supervision, are aligned across modalities (image, text) and specifications (examples, instructions). We believe these are new and valuable insights that can inform future research.\n\n---\n\n> 2. [I]f the exact position of the task vector is not consistent across tasks and modalities, then [I] doubt that we can call it a [‘task vector’]\n\nWe would like to clarify that **the layer position from which we extract the task vector is consistent across tasks and modalities**, as seen in Table 1 below. We determine this hyperparameter via average accuracy across all tasks, as discussed in L210 of the main text.\n\n*Table 1. Layer position of task vector, by VLM.*\n| LLaVA-v1.5 | Mantis-Fuyu | Idefics2 |\n|-|-|-|\n|15|23|16|\n\n---\n\n> 3. I expect the authors to publish their code\n\nWe will indeed publish the code to ensure reproducibility.\n\n---\n\n> 4. [Show] that their conclusion is general, and not limited to the several tasks (which are limited in scope)\n\n**In Sec. A.4 of the Appendix we include an evaluation based on VQAv2, which represents a more in-the-wild set of tasks.** We copy the results to Table 2, showing that patching is also effective in more general VQA tasks. Following this discussion, we will move this result to the main text. [We also invite you to view Table 2 in our response to Reviewer CQRE](https://openreview.net/forum?id=77ziPGdQct&noteId=fxPChMsZmQ), which includes experiments on four new tasks.\n\n*Table 2. We show the test accuracy of cross-modal transfer on image queries for visual question answering tasks derived from VQAv2.*\n|Model|Food-Class|Shirt-Color|Man-Holding|Avg.|\n|-|-|-|-|-|\n|No Context|0.00|0.00|0.00|0.00|\n|Image ICL Prompt|0.70|0.41|0.46|0.52|\n|Image ICL Patch|0.49|0.19|0.39|0.36|\n|Text ICL Prompt|0.85|0.48|0.56|0.63|\n|Text ICL Patch|**0.93**|**0.56**|**0.59**|**0.69**|\n\n---\n\n> 5. [Clarify] the description, [re-write] part of the paper (experiments section).\n\nThank you for your comments; we will revise our experiments to improve their clarity. \n\nAdditionally, we will include the pseudocode used to produce Figures 8-9,14-15 to the paper (further described below), and include a link to our code, so that the precise experimental procedure is available to readers.\n\n---\n\n> 6. Please could you clarify how the \"token representation curves\" (figure 14 and 15) are generated.\n\nTo produce the token representation curves, we do the following. For each task, we cache the layer activations and project them with the model’s existing final normalization layer and unembedding matrix [1], which produces a probability distribution over the entire vocabulary. We isolate the scores for three pre-defined tokens – input, task, and answer – and apply a softmax to produce relative probabilities. We then plot the mean and variance of these relative probabilities at each layer. \n\n**We also provide the pseudocode below.**\n```\ndef continuous_rep_evolution(model, dataset, select_vocab_idx):\n    \"\"\"\n    Plots the relative probability of the input, task, and answer token (given by\n    `select_vocab_idx`) across layers of `model`, for a given `dataset` representing a task.\n    \"\"\"\n    dataset_rel_prob = []\n    for sample in dataset:\n        feats = cache_act(model(sample)) # dim is [num_layers, 1, hidden_dim]\n        feats = model.norm(feats)\n        token_dist = model.lm_head(feats) # dim is [num_layers, 1, vocab_size]\n        input_idx, task_idx, answer_idx = select_vocab_idx\n        token_dist = token_dist[:, :, [input_idx, task_idx, answer_idx]]  # dim is [num_layers, 1, 3]\n        rel_prob = softmax(token_dist)\n        dataset_rel_prob.append(rel_prob)\n    plot_layer_vs_rel_prob(dataset_rel_prob)\n```\n\n---\n\nReferences\\\n[1] nostalgebraist. interpreting gpt: the logit lens. LessWrong 2020.", "claims_and_evidence": "The authors aim at demonstrating that there exists a \"shared task vector, which is invariant to modality (text, image) and format (examples, instructions)\"\nExperiments are designed to demonstrate this claim. \nMy comments are:\n1) VLM are precisely learnt so that the image modality (via the image encoder) and the text modality (via the text encoder) are aligned in a shared vector space. Consequently, the claims and the conclusions are not surprising\n2) the claim of the existence of a 'task-vector' (which the authors borrow from previous work), is in my view problematic: if the exact position of the task vector is not consistent across tasks and modalities, then i doubt that we can call it a 'task vector;", "methods_and_evaluation_criteria": "Several evaluation procedure are proposed. Quantitative evaluation is given by accuracy, a reasonable and classical metric.", "theoretical_claims": "The paper is experimental, trying to analyse the behavior of the models.", "experimental_designs_or_analyses": "There is a certain level of rigor in the experimental design, and an attempt of fair comparisons. The different setups are quite thorough, and  the experimental procedure seems well thought in general, thought their precise description  is not always clear to me.\n I expect the authors to publish their code, so that the experiments can be reproduced.  \nAblation study is reported in the annex.", "relation_to_broader_scientific_literature": "Little work has been done regarding the cross-modality transfer. The authors cite relevant papers.", "essential_references_not_discussed": "not to my knowledge.", "other_strengths_and_weaknesses": "The paper's idea and direction are definitively very interesting. It attempts to analyse the underlying behavior of the model (ie transparency analysis).  It is however a very 'descriptive' paper, which makes it at times hard to follow.  I am not fully convinced of the conclusions that are drawn.", "other_comments_or_suggestions": "The evaluation protocoles and results are interesting, however, the paper would gain in 1) showing that their conclusion is general, and not limited to the several tasks (which are limited in scope), 2) clarifying the description, re-writing part of the paper (experiments section).", "questions_for_authors": "Please could you clarify how the  \"token representation curves\" (figure 14 and 15) are generated.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "no ethical concern"}}, {"review_id": "JXObLlZvu4", "reviewer": "Reviewer_6RYy", "summary": "This paper examines a phenomenon in VLMs, where they encode inputs into a unified representation space, regardless of whether the task is defined through text examples, image examples, or explicit instructions. Building on this, the authors conduct experiments to assess the model's cross-modal transfer capability in an in-context learning setting.\n\n## update after rebuttal\nI have read the materials provided by the authors. The experiments on more popular VQA benchmarks are obviously less comprehensive. Therefore, I decided to keep my original rating (weak accept).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. Below, we include results regarding our core argument and our method’s computational cost, following your suggestions.\n\n---\n\n> 1. The core argument is somewhat less surprising, as it is straightforward that VLM maps inputs from different modalities into a shared space.\n\nWe study VLMs trained to map image embeddings into the representation space of an LLM. \n- Although this can be simplified as mapping different modalities into a shared space, this perspective does not explain our results. \n- **In fact, Figure 2a of the main text shows that the image and text embeddings don’t cluster by task. In contrast, Figure 2b shows that the task vector at the end of the sequence does group by task.**\n- This grouping in Figure 2b is not supervised in any way – its emergence is driven only by the next token prediction loss, or the fact that the answer is conceptually similar.\n- We find it striking that these task representations, learned without explicit supervision, are aligned across modalities (image, text) and specifications (examples, instructions). We believe these are new and valuable insights that can inform future research.\n\n---\n\n> 2. How can we learn from this finding when building visual-language cross-modality models?\n\n**First, our method reduces computational cost, especially for long contexts.** In Table 4, we show that in practice, patching reduces runtime by 11x and VRAM consumption by 2.4x when compared with few-shot prompting, for long text descriptions (on the dataset from Sec. A.4 of the main text). This is because the VLM no longer needs to attend to the long context, after injecting the single task vector, making the cost effectively equivalent to processing the query only.\n\n**Second, our LLM to VLM transfer experiments reveal gaps in model training.** In Table 3 of the main text, we see that for the same text inputs, the VLM produces lower-quality task vectors than the LLM, as evidenced by the 1-5\\% performance degradation across multiple models. One could apply this insight to VLM training by introducing a cosine similarity loss between the LLM and VLM task vectors on language-only examples, to monitor the observed degradation in language capabilities.\n\nWe will be sure to add further discussion of these implications of our findings to the final manuscript.\n\n*Table 1. Computational overhead of patching. Overhead of a single forward pass on N=30 Text ICL examples, averaged over 100 runs.*\n| Method          | Runtime (seconds) | VRAM (GB) |\n|----------------------|-------------------|-----------|\n| Prompting (Context + Query)      | 2.20             | 20.02   |\n| Patching (Task Vector + Query)  | 0.20             | 8.21     |\n| Query Only           | 0.19             | 8.21     |", "claims_and_evidence": "The paper presents sufficient evidence supporting its main argument: the presence of a shared task representation space in VLMs. This is evident, as visual and textual tokens are processed within the same representation space in LLMs. Additionally, experimental results demonstrate that this alignment persists across different network layers.", "methods_and_evaluation_criteria": "I think the evaluation criteria are proper. Using in-context learning to identify the similarity in representation space.", "theoretical_claims": "No theory was proposed in the paper.", "experimental_designs_or_analyses": "Yes. Table 2, Table 3, Figure 6, Table 4, Figure 10. They all look good to me, showcasing that the visual and textual tokens are aligned in LLM and VLM.", "relation_to_broader_scientific_literature": "This paper can be referred to by papers building new VLM.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strength:\n- The insight of the paper is interesting and the paper is well-written.\n- Extensive experiments have been done to verify the claim of the paper.\n\nWeakness:\n- The core argument is somewhat less surprising, as it is straightforward that VLM maps inputs from different modalities into a shared space.", "other_comments_or_suggestions": "If the authors can determine the implications of this observation for the VLM community—such as its potential to enhance the development of more effective or efficient VLM models and algorithms—their findings would carry significantly greater impact. Understanding how this shared representation space can be leveraged to improve model performance, reduce computational costs, or enhance cross-modal learning could lead to meaningful advancements in the field. Furthermore, exploring its applications in real-world tasks, such as multimodal reasoning, retrieval, or generation, would further solidify its practical significance.", "questions_for_authors": "- How can we learn from this finding when building visual-language cross-modality models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a detailed analysis of how Vision-Language Models (VLMs) generate cross-modal task representations that are invariant to modality and format, which significantly enhances the efficiency of these models. The authors introduce the concept of "task vectors" that can be transferred across modalities, demonstrating that these vectors outperform traditional few-shot prompting. They also provide strong experimental evidence supporting their claims and show the potential for task vectors to be derived solely from instructions, without requiring examples.

Reviewers generally agree on the novelty of the approach, praising the clarity and comprehensiveness of the experiments.  Overall, the paper makes valuable contributions to the field of VLMs and offers promising directions for future research. The results are well-supported by robust experiments, and the findings are likely to inspire further work in multimodal learning and task vector representations.

The AC agrees with the reviewers to accept the submission.
Decision: Accept (poster)