True Multimodal In-Context Learning Needs Attention to the Visual Context
Authors: Shuo Chen
Jianzhe Liu
Zhen Han
Yan Xia
Daniel Cremers
Philip Torr
Volker Tresp
Jindong Gu
Venue: COLM 2025 (2025)
Keywords: Multimodal In-Context Learning
Multimodal Large Language Models
Vision Language Models
TL;DR: We propose both a Dynamic Attention Reallocation tuning algorithm and a dedicated dataset to improve the true MICL ability of MLLMs

Abstract: Multimodal Large Language Models (MLLMs), built on powerful language backbones, have enabled Multimodal In-Context Learning (MICL)—adapting to new tasks from a few multimodal demonstrations consisting of images, questions, and answers.
Despite showing noticeable improvement on standard vision-language datasets, current MLLMs struggle to leverage visual information in the demonstrations.
Specifically, they tend to neglect visual cues and over-rely on textual patterns, leading to mere text imitation rather than genuine multimodal adaptation.
This behavior makes MICL still unimodal and largely restricts its practical utility. 
More importantly, this limitation is often concealed by the improved performance on tasks that do not require understanding the visual context.
As a result, how to effectively enhance MICL ability and reliably evaluate the MICL performance remains underexplored.
To address these issues, we first introduce Dynamic Attention Reallocation  (DARA), an efficient fine-tuning strategy that encourages models to attend to the visual context by rebalancing attention across visual and textual tokens.
In addition, we present TrueMICL, an MICL-dedicated dataset with both support and test sets that explicitly requires the integration of multimodal information—particularly visual content—for correct task completion.
Extensive experiments demonstrate the effectiveness of our holistic solution, showcasing substantial improvements in the MICL capabilities.
Code and datasets are available at [here](https://chenxshuo.github.io/true-micl-colm/).

Published as a conference paper at COLM 2025
True Multimodal In-Context Learning Needs Attention to the Visual Context
Shuo Chen 1,3,5,6* Jianzhe Liu Zhen Han', Yan Xia*, Daniel Cremers Philip Torr?, Volker Tresp's Jindong Gu?t
LMU Munich, Technical University of Munich, Siemens AG UUniversity of Science and Technology of China, MMunich Center for Machine Learning (MCML), Konrad Zuse School of Excellence in Reliable Al (relAI),
University of Oxford
Abstract
Multimodal Large Language Models (MLLMs), built on powerful language backbones, have enabled Multimodal In-Context Learning (MICL)-adaptie to new tasks from a few multimodal demonstrations consisting of images, questions, and answers. Despite showing notice- able improvement on standard vision-language datasets, current MLLMs struggle to leverage visual information in the demonstrations. Specifically, they tend to neglect visual cues and over-rely on textual patterns, leading to mere text imitation rather than genuine multimodal adaptation. This behavior makes MICL still unimodal and largely restricts its practical util- ity. More importantly, this limitation is often concealed by the improved performance on tasks that do not require understanding the visual con- text. As a result, how to effectively enhance MICL ability and reliably evaluate the MICL performance remains underexplored. To address these issues, we first introduce Dynamic Attention ReAllocation (DARA), an efficient fine-tuning strategy that encourages models to attend to the vi- sual context by rebalancing attention across visual and textual tokens. In addition, we present TrueMICL, an MICL-dedicated dataset with both support and test sets that explicitly requires the integration of multimodal piformation--artiulllal
visual content-for correct task completion. Ex- tensive experiments demonstrate the effectiveness of our holistic solution, showcasing substantial improvements in the true multimodal in-context learning capabilities. Code and datasets are available at here.
1 Introduction
Multimodal Large Language Models (MLLMs) have extended the emergence of in-context learning (Brown et al., 2020; Dong et al., 2022) from Large Language Models to multimodal domains and enabled Multimodal In-Context Learning (MICL) (Alayrac et al., 2022; Jiang et al., 2024). These pre-trained models can rapidly adapt to vision-language (VL) tasks, given few-shot multimodal demonstrations (demos) consisting of images, questions, and answers, without heavy parameter adaptations (Ferber et al., 2024), as shown in Figure 1. Compared with zero-shot evaluation, MICL has shown noticeable improvement on standard VL tasks such as image captioning (Alayrac et al., 2022; Awadalla et al., 2023; Laurençon et al., 2024b).
However, various studies have identified a key limitation of current MLLMs: they struggle to effectively utilize visual information in the demonstrations (Baldassini et al., 2024; Chen et al., 2023b; Jia et al., 2024; Zong et al., 2024). Concretely, they tend to overlook visual context in the multimodal demonstrations and over-rely on textual patterns in the context,
Equal contribution.
t Correspondence:

---

Published as a conference paper at COLM 2025
Demonstrations (2-shot)
Query
MLLM's Response
A labby and while cat sitting the ground near beick looking
right.
Caption: A small white dog sitting inside car, looking out of the window:
Capeion: A lsing on Nonket,
wearing rod knitted hat with busny
Caption:
11
11
11
2
2
10
2
9
3
9
3
9
3
96
8
4
4
4
7
What the result of this image? What the result of this image?
What is the result of this
33
image?
Figure 1: Examples of using MICL to solve image captioning from MSCOCO (Chen et al.,
2015) (top) and Clock Math from our proposed dataset, TrueMICL (bottom). Generating captions relies more on the ability of task recognition, and a correct caption can be answered based the text in the demos (eg., mimicking the caption style in the demos), without deep understanding of the demo images. However, our task requires task learning where the model needs to learn the relationship between the text and images in the demos (i.e., multiplying the two numbers pointed by the clock arrows) to correctly respond to the query.
which results in textual pattern imitation rather than true multimodal adaptation. This behavior renders multimodal ICL still unimodal and largely limits both its applications and practical utility. Moreover, this disadvantage is hard to discern as can be concealed by the improved performance tasks where reasonable responses can be generated from text pattern following, without a deep understanding of visual context. For example, Chen et al. (2023b) found that omitting the demo images during image captioning still yields comparable performance, indicating that models can generate reasonable captions for the query image even without referencing the demo images. Hence, these datasets are not suitable for evaluating MICL. To address these challenges, this study explores the following two essential research questions: 1) How to efficiently alleviate the overlooking of visual modality to truly advance MICL? 2) What kind of datasels are more suitable for improving and evaluating the true MICL. ability, especially the ability understand the visual information in the demos?
For the first question, we introduce Dynamic Attention ReAllocation (DARA) to mitigate visual context neglect and reduce overreliance on textual modality. DARA introduces a set of learnable attention-balant balancing parameters that dynamically regulate the influence of visual and textual tokens in demos during attention computation. DARA multiplies the columns of the attention score matrices corresponding to visual tokens by these learned parameters, thereby encouraging the model to emphasize the visual information in the demos. More importantly, DARA is lightweight, introducing only a small number of learnable parameters for rapid adaptation. Our experiments show that with around 100 parameters, DARA can yield up 10% improvement on downstream tasks. Consequently, DARA preserves the advantages of in-context learning while still achieving substantial performance gains.
For the second question, we propose that MICL datasets should prompt models to leverage visual context, quickly infer intent, and adapt to queries, rather than merely replicating textual answer styles or learning textual label spaces. We are motivated by the ICL disentan- glement framework (Pan et al., 2023), which disentangles ICL into two components: task a known task from the demonstrations and applying pre-trained priors-and and task learning-aqquriing new input-label mappings from the demos. Existing MICL datasets focus primarily on task recognition, requiring models to identify pre-learned tasks such as image captioning. However, they largely overlook task learning, where models learn an unseen mapping relationship between images and texts from the demos (Zong et al., 2024; Jia et al., 2024; Wang et al., 2024a). As shown in the bottom of Figure 1, the correct response needs the learn the task strategy of identifying numbers and doing the math calculation. To address this problem, we introduce TrueMICL, MICL-dedicated dataset
2

---

Published as a conference paper at COLM 2025
designed with a critical principle: correct responses must rely comprehensive understanding of the multimodal context, especially the visual information. In other words, the dataset should focus more on multimodal task learning, and correct response must rely the presence and understanding of demo images. Besides, TrueMICL is designed to be scalable and configurable for different levels of difficulty and number of samples, making it practical and flexible for MICL evaluation. Specifically, we generate a dataset with both support and evaluation splits with 860 samples in total, comprising 4 types and 7 distinct tasks, covering mathematical reasoning, pattern finding, and novel visual concept learning etc.
Through extensive experiments across range of MLLMs and tasks, we demonstrate the effectiveness of both DARA and TrueMICL. Our results show that current MLLMs find the evaluation tasks from TrueMICL. quite challenging, and DARA significantly im improves the MICL performance on both our evaluation tasks and standard VL. tasks. This highlights the potential of DARA to advance MICL ability and the importance of TrueMICL for more suitable MICL evaluation. Our contributions can be summarized as follows:
To enhance the MICL ability and especially to cure the visual context neglect, we propose DARA, an effective and efficient fine-tuning approach, to boost the influence of visual modality in the context. With only a few additional learnable parameters, DARA achieves substantial MICL performance improvements.
Towards more reliable MICL improvement and evaluation, we curate TrueMICL, a MICL-specialized dataset generation pipeline focused on emphasizing the critical role of visual information in multimodal contexts to improve MICL performance.
Comprehensive experiments across diverse MLLMs show the effectiveness of both DARA and TrueMICL, providing valuable empirical insights into the effective strategies for true multimodal in-context learning ability.
Related Work
Multimodal In-Context Learning. Built on powerful language backbones, some MLLMs start to show MICL ability, such as Flamingo (Alayrac et al., 2022), Qwen2-VL (Wang et al., 2024b), Idefics3 (Laurençon et al., 2024a), and Phi-3.5-Vision models support interleaved multiple image-text pairs as input and can generate responses to the query image-question pair conditioned on the previous image-text demonstrations. Another line of work focuses on understanding the working mechanism of MICL (Chen et al., 2023b; et al., 2023c; Yang al., 2024b; Baldassini et al., 2024; Qin al., 2024; Luo et al., 2024; Xu et al., 2024; Zhang et al., 2024; Li et al., 2025; Huang et al., 2025). Some focus on exploring better demonstration configurations for VQA (Li et al., 2023c) and Image Captioning (Yang et al., 2024b). Some (Baldassini et al., 2024; Luo et al., 2024; Chen et al., 2023b) examine the essential factor that contributes to MICL and also find that current MICL relies primarily on text-driven mechanisms, showing little to no influence of the visual modality. Zong et al. (2024) points out the limitations of using conventional vision-language tasks to evaluate MICL. Their benchmark indicates that most models face substantial challenges performing MICL tasks without providing alleviation methods. Given such limited performance, we propose a holistic solution to advance the MICL ability by encouraging the model to see the demo images with the help of our fine-tuning method DARA and dataset TrueMICL.
(Abdin et al., 2024), etc. These
Fine-tuning MLLMs to enhance MICL ability. Some studies (Zhao et al., 2023; Doveh et al., 2024; Li et al., 2023b;a; Gao et al., 2025; Li et al., 2023b) have tried fine-tuning the MLLMs for better MCIL ability mainly by adjusting the dataset format. For example, Zhao et al. (2023) transforms interleaved data into a unified context and Doveh et al. (2024) extends the instruction tuning dataset into the form of multi-image conversation. However, the models still suffer from over-dependency on the text modality, and the evaluation datasets are mainly traditional VL tasks that can be answered based solely on the query images. Jia et al. (2024) replaces the original text answers in the demos with semantically irrelevant strings to encourage the model to focus more on the images. However, the proposed method requires heavy direct reference optimization. Different from these, our work proposes a holistic solution consisting of both a lightweight tine-tuning method, DARA, and a dedicated MICL dataset, TrueMICI, to improve and evaluate the MICL. ability.
3

---

Published as a conference paper at COLM 2025
3 Methodology
We begin by briefly formulating MICL and introducing DARA in Section 3.1, followed by the introduction of TrueMICL in Section 3.2.
3.1 Dynamic Attention Reallocation
In multimodal in-context learning, an input query q, consisting of an image 1g and a question or instruction Tq. follows context prompt Cg are passed to a pre-trained MILM f. The context prompt Cg includes N task demonstrations drawn from a support set S, where each demonstration comprises an image I,, instruction Tú, and response Rj. The MLLM f generates response Rg to the query q-for instance, answering ba-based on both the query components I, and qq. and the multimodal context Cg. Formally, the process can be represented as: Rg- = f(CCg. 4)), where 9 = (Ig, TT), C {[, (;), JN.
Previous studies (Chen et al., 2023b; Baldassini et al., 2024; Luo et al., 2024) have identified the limited influence of context images {4;} N on MICL performance. Chen et al. (2023b) analyzed attention patterns and suggested that this lack of sensitivity to visual information in the context may stem from the model's imbalanced attention allocation between images and texts. To address this imbalance, we propose Dynamic Attention Reallocation (DARA), which effectively amplifies the influence of visual tokens within the attention mechanism.
Considering one decoder layer in the language backbone of an MLLM, it includes the crucial self-attention module (Vaswani et al., 2017), which maps input hidden representations X into queries Q keys K, and values V via linear projections using weights Wo, Wx, Wy. The decoder layer's 's output is then calculated as O softmax (S,,, where S QK" represents the attention score matrix. The jth column S, in the attention score matrix S indicates the relative influence of the jth token over other tokens in the input sequence. To encourage the model to focus more on context images, we directly introduce a set of learnable parameters to dynamically adjust the attention scores for columns corresponding to images. Specifically, the attention scores for visual tokens from context images are scaled by these learnable parameters. We implement this by introducing an attention balancer factor F to the softmax operation, represented as S' SF, where F diag(f) R/xi and visual
f f R' is a vector of learnable parameters. Only the positions in f corresponding tokens from I are non-zero, while other positions remain zero. The modified attention output is then computed as O' = softmax(S') V. Parameters in F can be trained by normal cross-entropy loss and all the other parameters in MLLMs are frozen.
DARA is simple to implement, with PyTorch-style pseudocode provided in Al- gorithm 1 on the left. After calculating the attention score matrix (Line 2) in one of the attention heads, each score column corre- sponding to the input images I is scaled by the learned weights for those images (Lines 4-6). The adjusted score matrix is then used to compute the final attention output of the modified attention head. Different heads can have different DARA modules for more flexible adaptation.
Algorithm 1 Pseudocode of DARA.
scale: divide by the square of dimension.
+ Compute attention scores and apply anplification scores = staln(t natmul (query, key.
+ Adjust the attention scores of each input inage for ind_ ing_ind
mage__indexes:
# nultiply learned weights for each inage with the corresponding columss
scores[: ing_pos_index] == innn_eeggttingg.in]]]
# Final attention scores output
scores = softnax(scores mask)
attantion_output = torch. value)
Relationship between DARA and Low-Rank Adaptation. Low-Rank Adaptation (LoRA) (Hu et al., 2021; Mao et al., 2025; Yang et al., 2024a; Liu et al., 2024b; Gu et al., 2023; Chen et al., 2023a) is a popular approach for parameter-efficient fine-tuning, which uti- lizes low-rank decomposition matrices to update parameters. Appendix mathematically proves that, for any f, an equivalent low-rank decomposed update can achieve the same output. In other words, DARA can be interpreted as a concise version of LoRA for MICL with many fewer trainable parameters. Experimental in Appendix D.1 also show that, given the same scale of trainable parameters, DARA can achieve better results compared to LoRA. Besides, DARA can be applied to LoRA to improve MICL performance further.
4

---

Published as a conference paper at COLM 2025
3.2 TrueMICL, MICL-dedicated Dataset
To design dataset with indispensable visual context, we are motivated by the framework from Pan et al. (2023), which disentangles in-context learning (ICL) ability into two com- ponents: task recognition and task learning. Task recognition involves identifying known task from demonstrations and applying pre-trained priors, such as performing general visual question answering or image captioning. In contrast, task learning requires learning a new input-label mapping from demonstrations. Task learning is scale-dependent and exhibits significant discrepancies across models of varying sizes (Pan et al., 2023).
already
Several studies (Zhao et al., 2023; Doveh et al., 2024; Li et al., 2023b;a; Liu et al., 2024a) have proposed evaluation and fine-tuning datasets for MICL. Examples include collecting more data in unified formats (Zhao et al., 2023) and converting data into multi-image conversations (Doveh et al., 2024). However, these datasets do not sufficiently encourage models to acquire new mapping relationships from demonstrations. Instead, they focus on tasks similar to pre-training or instruction tuning datasets, such as image captioning and general visual question answering. This leads to a critical limitation: query questions can often be answered by either inspecting only the query image or imitating the answeer style in the demo text, without utilizing the demonstration images. This approach falls short of achieving the true multimodal in-context learning, as it neglects the core task learning ability. In this setup, new image-text pair knowledge is required to answer the query question. Instead, the model merely needs to recognize the task and align the response with the desired text label space. As a result, fine-tuning on such catasetss can make the model tend to ignore the demo images, as it is unnecessary for the correct final response to the query. More importantly, without a proper evaluation dataset, this limitation hard to detect. Because such an issue can be concealed by the improved performance on evaluation datasets, where the performance is mainly from textual logic imitation (i.e., task recognition) rather than truly understanding the novel multimodal context (i.e., task learning).
To comprehensively evaluate and incentivize MICL ability, besides evaluating task recogni- tion, another core but usually missing principle is that correct responses must depend on an accurate understanding of the multimodal context, particularly the visual information. In other words, the demos should present image-text pairs with relations that are less unknown to the model so that the model can conduct task learning given the context. Additionally, the task should become unsolvable if the context images are removed. To this end, we have designed a novel MICL-dedicated dataset, TrueMICL, guided by the following principles: 1) Context dependency: The task must be unsolvable without the context images. 2) Novelly: The task should introduce novel image-text relationships that are uncommon in pre-training or instruction tuning, to effectively challenge task learning ability. 3) Perceivable Visual Information: The necessary information extracted from the images should not be overly complex, ensuring that the visual encoder can perceive it accurately. This allows us to focus on MICL ability rather than visual perception challenges. 4) Compatibility with Backbone: The task should push the boundaries of multimodal in-context learning without exceeding the language backbone's capabilities. 5) Configurabilily and Extensibility; This pipeline should be easily configured to generate more data samples with different levels of difficulty. These guidelines ensure that TrueMICL effectively evaluates and enhances the model's ability to learn new tasks from multimodal contexts. In the end, we have curated 867 samples in total spanning 4 different categories, consisting of 7 distinct tasks as listed in Table 1 and more detailed information is present in Table 4 and Appendix B
4 Experiments
4.1 Experimental Setup
MLLMs. We utilize 3 popular open-source MLLMs capable of MICL with varying sizes, architectures and pre-training datasets: Qwen2-VL (Wang et al., 2024b), Idefics3 (Laurençon et al., 2024a), Phi-3.5-Vision (Abdin et al., 2024). We also report the performance on GPT- 4o (OpenAl et al., 2024) to showcase how current closed-source models perform on our datasets. More detailed information about MLLMs is in Appendix C.1.
5

---

Published as a conference paper at COLM 2025
Task Type
Task Full Name
Task Short Name
#Support
# Test
Operator Induction
Operator Clock
30
100
Math Roasoning
Clock Math
30
100
Concept Binding
Outlier Detection
Outlier CLEVR
30
100
CLEVR Count
30
100
Pattern Findding
Sudoku
Sudoku
30
100
Palindrome Number
Palindromme
Novel Concept
Character Classification
Character
Table 1: TrueMICL contains 4 different categories, including a total of 7 distinct tasks. More detailed information and examples are shown in Table 4 and Appendix B
Datasets and Evaluation Metrics. The evaluation datasets include the test split from TrueMICL and standard VL datasets. For TrueMICL, demos for each query are selected from the support set. Standard VL datasets used in our experiments include VQ (Antol et al., 2015), GQA (Hudson Manning, 2019) and A-OKVQA (Marino et al., 2019), and MSCOCO (Chen et al., 2015). The demos for query samples from these datasets are from their corresponding training split. The evaluation metric for TrueMICL. is accuracy. For numerical answers, only strict matching is considered correct, while textual answers are evaluated through keyword-basee
matching. As for standard VL datasets, we follow the default metric for each dataset, such as accuracy for VQAv2. A more detailed introduction on datasets and metrics used in this study is present in Appendix C.2.
Baselines.
We have conducted experiments over the following 5 baselines: 1) Zero-shot: The model receives only the query image and text prompt, without context. 2) No-image: The model receives the query image and text prompt, with randomly selected 4-shot demos including only the text information. 3) Random: The 4-shot demos are randomly selected, including both images and text. 4) RICES: The demos are selected by Retrieval-based In- Context Example Selection (Alayrac et al., 2022; Yang et al., 2022; Zebaze et al., 2024; Wang et al., 2023). For given query, we chose the most similar images from the support set and used the image-label pairs as demos. 5) LoRA: We also fine-tune LoRA on the support set for each task while controlling the number of learnable parameters to the same scale as DARA. For more detailed settings, please refer to Appendix C.3.
DARA. We insert the DARA module into the first transformer layer of the language back- bone in the MLLM, specifically, attention score matrices in all attention heads in the first layer will be regulated by DARA's learnable parameters. The number of learnable param- eters is decided by the number of demo images and the number of heads to modify. For instance, given images including 4-shot and the query image, and 32 attention heads to adjust, the total number of learnable parameters is x 32 160. These learnable parameters are trained using Adam on the small support set with learning rate of 0.001. More detailed hyperparameters are present in Table 6 in the Appendix.
4.2 Result Analysis on TrueMICL
The main experimental results of the models' performance on TrueMICL are shown in Table 2. We report the accuracy of three models across various tasks in TrueMICL, where each row represents a different inference method. Except for the Zero-shot, all other methods use a 4-shot setting. Additionally, the methods labeled LoRA and DARA represent the performance of models trained under the finetuning settings described in Section 4.1.
Table first shows that these MLLMs perform poorly on TrueMICL in zero-shot scenarios and when demonstrations contain only text without images. Some models even achieve accuracy under 10% on certain tasks, such as Phi-3.5-Vision on Operator Induction. With randomly selected 4-shot demonstrations, the models demonstrate limited improvement in accuracy across tasks. However, applying the RICE method for selecting more relevant demonstrations yields minimal additional performance gains. This result highlights two key observations: 1) Tasks in TrueMICL truly require both visual and textual information from the context to be solved correctly; 2) Due to the model's limited ability to effectively make use of the visual information in demonstrations, simply improving the relevance of the shots (as done in RICE) is insufficient to enhance model's performance further.

---

Published as a conference paper at COLM 2025
Model
Method
Operator
Clock
Outlier
CLEVR
Sudoku
Palindrome
Character
Average
Zero-Shot
13.674057
24.00=2x5
41.33425
34.33+1.3
88.00=175
30.33-115
23.00
36.38
No-image
18.67
27 27.67:153
46.33x208
41.67a506
90.00=179
36.33=251
24.33
41.00
Qwen2-VL
Random
33
31.00=1m
46.67405
00:000
93.33
96.00=1.
83.33+115
77.67
RICES
67.67
31.33:155
87.33
87.33
95.33
95.00=1.0
94,00+200
79.71
LoRA
66.33445s
32.67-0m
87.33
00+1.
92.67=4..
-1.67-158
96.00
80.24
DARA
72.67a11s
37.33
91.67
90.00
95.33
96.00=1
96.00
83,00
Zero-Shot
6.67=1.3
0.67
50.00=1.0
19.00 19.00=2.00
53 63.33=2.0
18.33=1115
23 L355
25.95
No-image
10.334206
2.00
52.67
23.00
39,33=218
28.67
27.33
29.05
Idefies3
Random
16.00
9.33
59.67
25.00
86.67 L1.5
29.35
67.67
41.95
RICES
17.33+1.3
9.67
59 00anne
24.67
87 =7.00
29
77.00=100
43.43
LoRA
18.67
9.67
59.67 183s
24.33 1us
87.67 1c:66
28.33 1.15
89,3331155
45.38
DARA
21.33
14.67
64.00:200
25.00:1.01
91.33:115
33.67
90.67:1.33
Zero-Shot
9.33
3.37
39.00.200
8.33
46.67 -::5
32.67
21.33
23.00
No-image
11.33zuss
6.33
45.33
13.00a2.M
52.67
36.00.175
28. 28.33
27.86
Phi-3.5-vision
Random
RICES
14.33
15.00
17.67.4.6
17:33
61.67 12:08
61.33:uu
24.3
1433
84.00
84.67
42.00
42.33.00.
84.67:220
68.00
44.62
47.05
LoRA
14.00
16.67
65.33
27.67
84.33
41.33.6.
91.33
48.67
DARA
17.33:153
20.00.1.75
70.67
32.00
89.33-1.55
47.33:244
93.00
52.81
Table 2: Performance of MICL from 3 different MLLMs using different methods on TrueMICL. Each column demonstrates the performance from each task in TrueMICL, with the task abbreviation as the column title. The best performance for each setting is in bold, and the second- -best is underlined. DARA (rows in light red) achieves the best performance in most scenarios and outperforms baselines by a large margin. Experiments are averaged over different seeds, and the performance is in percentage with the standard deviations.
However, as shown in the last two rows for each model, the performance on TrueMICL significantly improves after finetuning, particularly with the use of DARA. Compared to LoRA, which introduces several thousand parameters but yields limited gains and remains almost the same performance as random 4-shot, DARA leads to further improvements across nearly all tasks. For tasks involving math reasoning and concept binding, each model achieves an average improvement of 3 to 5 percent per task when using DARA. Meanwhile, in tasks such as pattern finding and concept learning, models like Dwen2-VLL-which already achieve good accuracy using Random baseline due to their strong capabilities show an additional stable gain in accuracy when using DARA. Besides, the performance of the other two models on these tasks also demonstrates stable and noticeable gain when DARA is applied, further supporting its effectiveness.
Overall, the results demonstrate that the proposed dataset, TrueMICL. serves as reliable benchmark for evaluating true MICL capabilities, as every task category requires the model to combine both visual and textual information to provide the correct answer. Furthermore, the proposed method, DARA, consistently achieves the highest average performance across nearly all TrueMICL tasks for each model tested. This highlights the effectiveness of DARA in enhancing the model's ability to perform multimodal in-context learning.
4.3 Visualization of the DARA's Reallocation Effects
To better understand how DARA affects attention, we present both qualitative and quantita- tive visualizations as shown in Figure and 3. The spatial attention heatmap (Zhang al., 2025) of input images (Figure 2(a)) shows that, without DARA, both demonstration and query images do not receive much attention, as indicated by predominantly blue regions in the top row. After tuning with DARA, attention over image tokens increases markedly (more red/yellow areas the bottom row), indicating enhanced attention to the visual input. While not all focus aligns with the specific object regions, the overall shift indicates that DARA encourages greater incorporation of image information.
We also quantitatively compare the attention allocation ratio over different modality tokens with and without using DARA. Figure 2(b) highlights the shift of attention on these two modalities. Without DARA, the model allocates only 28% of attention to image tokens, focusing primarily on text. With DARA, this rises to 46.7%, indicating substantial shift toward visual content during response generation.
7

---

Published as a conference paper at COLM 2025
Demo Image
Query Image
Nommalzzed Attention Proportion
10
Image Attention Text Attention
7
8 8
5
88
53.3%
71.4%
45.7%
7 3
7
02-
28.5%
(a) Attention heatmaps over input images with- out (top) and with (bottom) applying DARA.
(b) Normalized attention ratios over image and text tokens without and with applying DARA.
Figure 2: DARA enhances visual attention both qualitatively (left) and quantitatively (right).
To further understand the value distribution of learned factors, Fig- ure 3 visualizes these factors over four demos and one query image from the first transformer layer (8 heads) of Qwen2-VL. Values larger than 1 indicate attention amplifi- cation on visual tokens. DARA induces clear redistribution: (a) demo images consistently receive factors larger than 1, encourag- ing stronger reliance on context; and (b) different heads special- ize-for for example, Head 1 empha- sizes Demo (1.27), while Head emphasizes Demo (1.32). These patterns confirm that DARA en- ables selective, context-aware vi- sual attention during MICL.
071
1.23
1.27
1.15
1.20
1.19
1.28
1.27
1311
14
276
1.27
1.17
1.23
1.16
1.19
1.24
1.15
1.19
12
1.30
1.
1.19
1.21
1.18
1.32
1.21
1.17
117
8?7
0.87
1.02
0.97
1.01
1.00
0.96
1.04
0.93
1ead1
head7
head3
head4
head5
haad6
head7
headB
Figure 3: Learned attention amplification factors across 8 heads and 5 images in the 1st layer of Qwen2- VL. DARA introduces structured visual emphasis on demonstration images.
4.4 Result Analysis on Standard VL Datasets
We also analyzed the performance of DARA standard VL datasets, following the same baseline configuration as described in Section 4.1. The results are summarized in Table in the Appendix. The results show that, for standard VL. tasks, model performance lies primarily on the model's own ability to understand the query image and associated textual information. By comparing the first three settings, no demonstrations (Zero-shot), demonstrations without images (No-image), and random 4-shots, we observe almost no differences in performance. Taking Qwen2-VL on VQAv2 as an example, the accuracy under these settings is 78.6%, 78.7%, and 79.1%, respectively. These results align with previous findings (Chen et al., 2023b) and indicate that correct responses for these standard vision tasks do not depend on the visual context provided in the demonstrations. Once the model has extracted sufficient information from the query image and integrated with the textual context, high-quality response can already be generated.
Moreover, by examining the performance of DARA, we observe that DARA DARA-designed to encourage the model to better focus on visual content in the iinoonstrtioos-acciee results comparable to the 0-shot and 4-shot settings. This finding confirms, on the one hand, that standard VL benchmarks are indeed suboptimal for evaluating true multimodal in-context learning, as they do not even require visual context; and on the other hand, that the use of DARA does not lead to performance degradations these standard VL. tasks.
8

---

Published as a conference paper at COLM 2025
Qwen2-VL
Operator
Clock
Outlier
CLEVR
Sudoku
Palindrome
Baseline
67.33%
31.00%
86.67%
86.00%
93.33%
96.00%
Operator
31.67% (+0.67%)
90.33% (+3.66%)
89.33% (+3.33%)
95.67% (+2.34%)
97.00%
1-1.00%)
Clock
72.00%
(+4.67%)
88.00% (+1.33%)
88.67% (+2.67%)
95.00% (+1667)
96.00% (+0.00%)
Outlier
68.33%
(+1.00%)
32.00%
(+100%)
87.67% (+1.67%)
94.00%
(+0.67%)
95.33% (+1.33%)
CLEVR
67.00%
(-0.33%)
31.00% (+0.00%)
87.67% (+1.00%)
95.33% (+2.00%)
97.00% (+1.00%)
Sudoku
72.00%
(+4.67%%)
31.33% (+0.33%)
89.33% (+2.666%)
89.00% (+3.00%)
95.67% (-0.23%)
Palindrome
71.00% (43.67%) 31.67% (+0.67%)
89.67% (+3.00%)
89.67% (+3.67%)
97.00% (+3.67%)
Table 3: Transferability performance of DARA across tasks. Diagonal cells (self-transfer) are omitted for clarity. Each column reports the performance when DARA is trained on other tasks and evaluated on the column task. DARA shows good transferability and consistently achieves transfer gains in most settings, with improvements up to 4.67%.
Q-Vn2-VL
Idefics3
Phi-3.5-vision
80-
20
20
19-
Accuracy (96)
Accuracy (%)
16-
18
Accuracy
17
35-
16
12-
14.
13
2
1'6 Number shots
32
4
à
2
8
16
32
Numbee of shots
Number shots
Operator
Operator
(DARA)
Figure 4: Performance comparison on Operator and Clock tasks across three models with different numbers of shots, ranging from to 32. Compared to the baseline Random method, DARA consistently improves the MICL performance across different numbers of shots.
4.5
Ablation Study
We further conduct various ablation studies on DARA to verify the effectiveness of our design. A more detailed discussion is in Appendix D
Transferability on untuned tasks. We evaluate the transferability DARA by testing whether improvements on task transfer to untrained ones. As shown in Table training on single task yields 2-5% higher accuracy on unseen tasks compared to standard 4-shot baseline inference. While this does not reach the performance of models trained di- rectly on each target task, it demonstrates that DARA can introduce transferability, enabling better generalization across related MICL tasks.
Impact of Increasing Shots. We also evaluated the performance of models fine-tuned with TrueMICL under different shot settings. As shown in Figure 4, we present comparison between the performance of models trained with varying numbers of shots and the corre- sponding baseline at that shot level. Within a certain range of shot numbers (for example, up to 10 shots for Qwen2-Vi, VI. determined by the maximum number of images the model can reliably process at once), applying DARA consistently brings performance improvements across all shot settings. However, when the number of shots becomes too large, neither DARA nor LoRA can consistently guarantee further gains. This finding suggests that as more capable models emerge in the future, DARA will still prove to be effective.
Prompt Design. To test whether DARA's improvements originate from prompt engineering, compare our original minimal prompt-wii which avoids leaking visual cues through text-with an enhanced version that includes detailed instructional cues (Sahoo et al., 2024). As shown in Appendix D.4, while such prompts help standardize response formatting, they

---

Published as a conference paper at COLM 2025
do not lead to significant accuracy gains. This supports that DARA's effectiveness arises from improved reasoning over demonstrations, rather than prompt tuning.
Human Evaluation. To assess whether TrueMICL tasks truly require MICL ability, we conducted a human study involving 20 participants, In 0-shot setting, where no demos were provided, participants generally failed to answer. However, their performance improved once demonstrations were included. These results suggest that the tasks cannot be solved using prior knowledge or superficial cues alone. Instead, successful completion requires learning from the multimodal context, thereby validating the design of our benchmark.
Layer Design DARA. We further investigated applying DARA to multiple transformer layers beyond only the first layer. While extending DARA to deeper layers yields compa- rable accuracy, leads to increased computational and parameter overhead. We attribute this to the first layer's 's unique role in initiating early cross-modal fusion, as later layers process already entangled representations, limiting the impact of targeted amplification. Thus, restricting DARA to the first layer achieves a better balance between effectiveness and efficiency. The result of applying different layers is shown in Appendix D.3
Hard-coded Attention Adjustment. As an additional baseline, we experimented with a hard-coded attention amplification strategy. Specifically, we modified attention logits to completely ignore text tokens for half of the heads, forcing attention exclusively onto image tokens. This rigid masking led to unstable and incoherent outputs, likely due to disrupted modality balance. In contrast, DARA employs a learnable amplification factor that softly increases attention to image tokens, allowing the model to dynamically adjust during training. This preserves output fluency while improving performance.
Evaluation on GPT-4o. We also tested TrueMICL on GPT-4o, state-of-the-art closed-source model. In the 0-shot setting, GPT-4o fails on most tasks except few logic-based ones like Sudoku. With 4-shot demonstrations, it achieves significantly better accuracy (full results in Appendix D.2), highlighting the indispensable role of context images in TrueMICL. However, despite GPT--4'ss strong performance, certain fundamental MICL challenges persist; for example, GPT-4o's accuracy significantly decreases when facing more challeng- ing tasks such as our specially designed harder Sudoku variant. Furthermore, due to the closed-source nature of GPT-4o, it remains unclear whether these performance gains are driven purely by scaling or by undisclosed architectural and training strategies. Importantly, the lightweight design of DARA complements these powerful models, providing further improvements even when combined with parfieter-efficient fine-tuning approaches like LoRA. Consequently, DARA remains valuable in addressing attention imbalance, especially benefiting openly accessible, resource-oosstrinee models widely used in diverse appli- cations. Moreover, TrueMICL serves as a robust benchmark for accurately evaluating the MICL capabilities of both open-source and proprietary MLLMs.
5
Conclusions
Despite recent advances in multimodal large language models, our study reveals a critical limitation in current approaches to Multimodal In-Context Learning (MICL): the tendency to overlook visual information in demonstrations and over-rely on textual patterns. This undermines the fundamental promise of MICL, reducing it to predominantly unimodal process and limiting its real-world applicability. To tackle this issue, we propose DARA, a lightweight yet effective fine-tuning method that dynamically reallocates attention to enhance the influence of visual tokens in the in-context examples. Complementing this, we introduce TrueMICI., challenging and diagnostic dataset that emphasizes true multimodal adaptation by requiring visual understanding for task success. Our extensive experiments across diverse MLLMs demonstrate that DARA consistently improves performance on both standard and TrueMICL tasks, validating its effectiveness in enhancing genuine multimodal adaptation. Furthermore, TrueMICL exposes existing blind spots in MLLMs and serves as a valuable benchmark for evaluating true MICL capabilities beyond textual-level imita- tion. Together, DARA and TrueMICL offer a holistic framework for truly advancing and evaluating MICL, paving the way for more faithful multimodal ICL in future MLLMS.
10

---

Published as a conference paper at COLM 2025
Acknowledgment
This paper is supported by the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry Research, Technology and Space.
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arx preprint arXio:2404.1421, 2024.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 11223766-23733 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. Xiv preprint arXiv:2308.01339 2023.
Folco Bertini Baldassini, Mustafa Shukor, Matthieu Cord, Laure Soulier, and Benjamin Piwowarski. What makes multimodal in-context learning work? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1539-1550, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Shuo Chen, Jindong Gu, Zhen Han, Yunpu Ma, Philip Torr, and Volker Tresp. Benchmarking robustness adaptation methods on pre-trained langunn models. Advances in Neural Information Processing Systems, 36:51758-51777 2023a.
Shuo Chen, Zhen Han, Bailan He, Mark Buckley, Qin Yao, Philip Torr, Volker Tresp, and Jindong Gu. Can multimodal large language models truly perform multimodal in-context learning? arXiv preprint ar orXi::2311..80.1 2023b.
Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. ar Xiv preprint ar trXiv:1504.00325 2015.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning, ar preprint ar Xrxi::2301. 000244 2022.
Sivan Doveh, Shaked Perek, M Jehanzeb Mirza, Amit Alfassy, Assaf Arbelle, Shimon Ullman, and Leonid Karlinsky. Towards multimodal in-context learning for vision & language models. arXiv preprint ar Xxx::2403..12366 2024.
Dyke Ferber, Georg Wölflein, Isabella C. Wiest, Marta Ligero, Srividhya Sainath, Narmin Ghaffari Laleh, Omar S.M. El Nahhas, Gustav Müller-Franzes, Dirk Jäger, Daniel Truhn, and Jakob Nikolas Kather. In-context learning enables multimodal large language models to classify cancer pathology images. arviv preprint arXio:2403.074000 2024. URL https://doi. org/10. 48550 http://axivv.v... 2403. 07407. 40 pages, 5 figures.
11

---

Published as a conference paper at COLM 2025
Jun Gao, Qian Qiao, Tianxiang Wu, Zili Wang, Ziqiang Cao, and Wenjie Li. Aim: Let any multimodal large language models embrace efficient in-context learning. In Proceedings of the AAAI Conference on Artificial Intelligence, number in 39, pp. 3077-3085, 2025.
Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, and Philip Torr. A systematic survey of prompt engineering on vision-language foundation models. ar preprint arXiv:2307.12986 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Alu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint ar Xiv::1066 0968,, 2021.
Chengyue Huang, Yuchen Zhu, Sichen Zhu, Jingyun Xiao, Moises Andrade, Shivang Chopra, and Zsolt Kira. Mimicking or reasoning: Rethinking multi-modal in-context learning in vision-language models. Xrv preprint ar arXio:2506.07938 2025.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 6700-6709, 2019.
Hongrui Jia, Chaoya Jiang, Haiyang Xu, Wei Ye, Mengfan Dong, Ming Yan, Ji Zhang, Fei Huang, and Shikun Zhang. Symdpo: Boosting in-context learning of large multi- modal models with symbol demonstration direct preference optimization. ar arv preprint arXi::241.111909 2024.
Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H. Chen, and Andrew Y. Ng. Many-shot in-context learning in multimodal foundation models. Xiv preprint ar arXto:2405.09798, 2024. URL https: //doi. org/ 48550/ /a150/ 09798. Benchmarking GPT-4o and Gemini 1.5 Pro across 14 multimodal datasets.
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision-language models: insights and future directions. In Workshop on Responsibly Building the Next Generation of Multimodal Foundational Models, 2024a.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? ar Xiv preprint ar Xii::2405.022444 2024b.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arv preprint arXiv:2306.05425 2023a.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arv preprint arXiv:2305.03726, 2023b.
Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, and Xu Yang. How to configure good in-context sequence for visual question answering. arXiv preprint 2rXio:331..0157,, 2023c.
Yanshu Li, Tian Yun, Jianjiang Yang, Pinyuan Feng, Jinfa Huang, and Ruixiang Tang. Taco: Enhancing multimodal in-context learning via task happing-guider sequence configuration. Xiv preprint Sttt:2505.17999 2025.
Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, et al. Mibench: Evaluating multinodal large language models over multiple images. ar Xiv preprint ar Xiv:2407.112222 2024a.
Yilun Liu, Yunpu Ma, Shuo Chen, Zifeng Ding, Bailan He, Zhen Han, and Volker Tresp. Perft: Parameter-efficient routed fine-tuning for mixture-of-expert model. arx preprint ar arXie:2411.08212 2024b.
Yang Luo, Zangwei Zheng, Zirui Zhu, and Yang You. How does the textual information affect the retrieval multimodal in-context learning? ar Xiv preprint wxXiv22404.12866 2024.
12

---

Published as a conference paper at COLM 2025
Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. A survey on lora of large language models. Frontiers of Computer Science, 19(7):176005, 2025.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
OpenAT, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, and et Gpt-4o system card. ar Xiv preprint ar 21122421006 2024. URL ht / //doi. org/100 48550/arXiv. 2410. 21276. Cov- ers architecture, multimodal input/ output capabilities, and safety evaluations of GPT-4o.
Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning "learns" in-context: Disentangling task recognition and task learning. In Findings of the Association Toronto, Canada, July 2023, As- sociation for Computational Linguistics. doi: 10.18653/v1. /t////11222//// htt//2222...nnnng.... findings-acl.527. URL org/2023, findings-ade ac1. 527.
for Computational Linguistics: ACL 2023, pp. 8298-8381111
https://aclanthonn
Libo Qin, Qiguang Chen, Hao Fei, Zhi Chen, Min Li, and Wanxiang Che. What factors affect multi-modal in-context learning? an in-depth exploration. arxiv preprint arXim:2410.20480 2024.
Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering in large language models: Techniques and applications. ar Xiv preprint ar arXio:2402.07927, 2024. URL https:://oil http:///ooooor/////////// / arXiv. 07927. 12 pages, 2 figures; provides taxonomy, strengths/limitationn,, datasets and models for prompt engineering.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural informa- fion processing systems, 30, 2017.
Jinyin Wang, Haijing Zhang, Yihao Zhong, Yingbin Liang, Rongwei Ji, and Yiru Cang. Advanced multimodal deep learning architecture for image-text matching. arXiv preprint ar htte:2406.153066 2024a. URL https: / //ooi. org/10 48550/ / 2406. 15306. Proposes novel architecture with cross-modee attention and hierarchical fusion.
Liang Wang, Nan Yang, and Furu Wei. Learning retrieve in-context examples for large language models. arXiv preprint ar Xiv::230 .01644077444444 2023. URL https://dol http::///ooooo////// 48550/ arXiv. www.2238.000 071 64. Accepted by EACL 2024.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. ar preprint arXiv:2409.12 121991, 2024b.
Nan Xu, Fei Wang, Sheng Zhang, Hoifung Poon, and Muhao Chen. From introspection to best practices: Principled analysis of demonstrations in multimodal in-context learning. arXiv preprint w-Xio:2407.00900 2024.
Menglin Yang, Jialin Chen, Yifei Zhang, Jiahong Liu, Jiasheng Zhang, Qiyao Ma, Harshit Verma, Qianru Zhang, Min Zhou, Irwin King, et al. Low-rank adaptation for foundation models: A comprehensive review. arXiv preprint XXir:2501.00366 2024a.
Xu Yang, Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng. Exploring diverse in-context configurations for image captioning. Advances in Neural Information Processing Systems, 36, 2024b.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI conference on artificial intelligence, number in 36, pp. 3081-3089, 2022.
Armel Zebaze, Benoît Sagot, and Rachel Bawden. In-context example selection via similarity search improves low-resource machine translation. arv preprint Xi0o22408.00397, 2024.
URL https:://oft org/10. /8500..48001//8/////// /a.50///riv..... 2408 00397.
13

---

Published as a conference paper at COLM 2025
Haowei Zhang, Jianzhe Liu, Zhen Han, Shuo Chen, Bailan He, Volker Tresp, Zhiqiang Xu, and Jindong Gu. Visual question decomposition on multimodal large language models. aiv preprint ar wxxx:2449.193399 2024.
Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip llievski. Milms know frrei where to look: Training free perception of small visual details with multimodal llms. International Conference Learning Representations, 2025.
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang, Mmicl: Empowering vision-language model with multi-modal in-context learning. Xrv preprint ar arXix:3309.0791 2023.
Yongshuo Zong, Ondrej Bohdal, and Timothy Hospedales. VI-icl bench: The devil in the details of benchmarking multimodal in-context learning. arXiv preprint arXim:2403.11 131644 2024.
A DARA as a constrained version of LoRA
Consider transformer-based attention mechanism with query, key, and value transfor- mations defined by Wo. WK. Wy, and let S OK be the attention score matrix, where Q = X and K = XWx, where X is the input tensor to the transformer attention module. Suppose we introduce an attention reallocation factor to the softmax operation by defining
va
= diag(f) E IRLXI, where f E R- a vector of learnable factors. The modified scores are S' := SF. On the other hand, consider the update (e.g., from gradient descent algorithm) on the module weights Wo, WK, defined as AWQ, AWK. After updating the weights, we have We == We + AWQ, W == WK + AWx, the updated Q' XW K' XW and the updated attention score matrix S" O'K"
Prove that for Vf, E AWQ and 3 AWW. o. so that Vi
S"
Proof. We want:
S" Q'K"T
QK Vd
F SF
(1)
Vd
Equivalently:
Q'K = =K F.
(2)
We need to express QK Fas Q'K"T with Q' = -XW and K' XW
There is no unique solution. In fact, there family of solutions parameterized by an invertible matrix R RRixd_ Consider:
Q' = QR and ='= T
(3)
Then:
Q'K'T = (Q = Q
Since Q = XWo and K = XWx, we have:
Q' = XW XWoR = W = WoR,
(5)
full rank or using suitable inverse peeudoivveeddooeeeeeeeee.
K' = -X FXW
W x F
(6)
Here, x denotes a left-inverse pseudoinverse of X. Since R is arbitrary and invertible, there are infinitely many such solutions. Hence, for any given F, we can always find AW W We and ÁWx W Wg to achieve S" S'.
14

---

Published as a conference paper at COLM 2025
Prove that each corresponding AW Q, AWx can be decomposed into low-rank matrices multiplica- tions, such as AWo = AB, AW K CD
Proof. Any matrix admits factorization into two matrices of potentially lower rank. For given AWQ, consider its singular value decomposition (SVD):
Q UEV
where U and V are orthonormal matrices and Σ is diagonal with nonnegative singular values. By retaining only the nonzero singular values (or even a subset to achieve a rank reduction), we obtain low-rank factorization:
AWo = U,I.V,
(8)
with U, E Rdaa I, E R and V, Rxxd Define A = U,E, E Rda and B = V E
R Thus:
AWQ AB.
(9)
similar argument applies AWk. K. Hence:
AWK CD
(10)
for some C E R x and E R possibly with the same or different rank
To conclude, the updates AWo and AWx obtained to achieve s" S' can always be decomposed into products lower-rank matrices. This establishes that these parameter updates can be implemented in a low-rank form, aligning with practical low-rank adaptation methods. It is necessary to clarify that DARA can be seen as a constrained variant of it requires input-dependent updates and token-
LoRA rather than technical equivalent, position-specffic factors that deviate from the standard LoRA assumptions. Specifically, the weight updates LoRA are input-independent and fixed after learning, whereas the desired low-rank update AWx depends on the specific input X as shown in the Formula However, such constraints actually can provide practical benefits to MICL. While LoRA is more general, it can be indirect and suboptimal to rebalance the attention allocation for better MICL. In contrast, DARA's constrained formulation enables targeted visual attention modulation with far fewer parameters, leading to more efficient MICL. improvement.
B TrueMICL
Guided by the principles in Section 3.2, we designed and compiled four categories encom- passing total seven distinct tasks, as illustrated in Table 1 and 4. TrueMICL encompasses wide range of MICL capabilities, including mathematical reasoning, novel concept binding, and pattern interpretation, craatiion,rovvidiii providing comprehensive dataset for multimodal in-context learning. Each task is designed with adjustable difficulty levels, such as more diverse con- cepts in novel concept binding, more complex visual patterns in pattern interpretation, etc. Each task in TrueMICL is divided into support set containing 30 samples and different test set containing 100 queries (except for Character Classification, due to the limited number of character images sourced from movie scenes).
Operator Induction is derived from Zong et al. (2024). The dataset consists of simple arithmetic operations formed by three operators: addition, subtraction, and multiplication. Since the operator used in each image is not explicitly provided, the model must analyze the relationship between the image content and the answer to respond to the query.
Clock Math is an original task inspired by the arithmetic pattern. The model must first perceive the number in the image, then learn the relationship between the number and the answer (e.g,, addition), to correctly answer the query.
For arithmetic tasks, considering the current capabilities of existing models, our dataset adopts the most straightforward form, using actual digits to perform the whole operations. In fact, replacing digits with arbitrary symbols to express implicit operations can all be seen as an extension of this type of task in higher levels of difficulty.
15

---

Published as a conference paper at COLM 2025
Calegory
Task
Demo
Demo
Query
Label
Explanation
0 ? 1
2 6
1 ? 8
Multiplying the two numbers in the image
Math Reasoning
Operator Induction
8
Adding the two numbers in clock.
Clock Math
20
AAAAA
AAA
AA
The outlier color in the image
Concept Binding
Outlier Detection
Green
Black
Red
CLEVR Count
2
The number of sphere
916
917
216
468
489
missing number
Pattern Finding
Sudoku
918
217
470
in between
Palindrome
9 4 2
9 6
9
To form palindrome number
Character
The same character
Novel Concept
Flora
Flora
Classification
in demo
Table 4:
An overview of task examples in TrueMICL. The label to the query requires the model to learn the relationship between images and text in the demos.
Outlier Detection originally designed to evaluate the model's binding capability. Each image in this dataset contains varying numbers of two different shapes and colors. The model needs to identify the desired outlier feature based on the demo text.
Similarly, in concept binding tasks, features such shape, color, and size can be replaced with less explicit attributes, such as visual appearance or texture. This substitution will make the task significantly more challenging.
CLEVR Count is derived from Zong et al. (2024). Each image contains objects with varying attributes such as colors and shapes. The model will be asked to count the number of objects with specific attributes based on the demonstrations. We simplified the samples to match the model's visual perception capabilities. Specifically, we removed hard-to-distinguiie attributes such as material and size, kept only color and shape, and redesigned the prompts to align with our intended focus on concept binding in the dataset.
Sudoku and Palindromic Number are originally designed for pattern finding, They contain images of incomplete Sudoku grids or partially missing sequences of palindromic num- bers. To solve these tasks, the model must recognize the underlying patterns behind the multimodal demos and apply this learned relation to correctly respond to queries.
For pattern finding tasks, taking Sudoku digits or introducing additional implicit rules for number placement can substantially raise the task difficulty. High-level reasoning problems commonly used in IQ logic tests also fall into this category as more advanced tasks of pattern finding.
example, increasing the number of missing
Character Classification is originally designed to test the ability of novel concept learning from multimodal demos. We collect movie character images after the model's cutoff date and assign previously unseen names. This setup requires the model to associate novel names with unfamiliar appearances through the demonstrations in order to correctly answer the queries.
16

---

Published as a conference paper at COLM 2025
For this unique type of classification task, any similarly constructed dataset that uses images collected after the release the evaluated models can be considered valid.
C
More Details of Experimental Setup
C.1 MLLMs
Model
Qwen2-VL-7B-Instuud
Idefics3-BB-Llama3
Phi-3.--Viiion-nsttuul
Params Visual Encoder
7B Custom Patch Encoder
~8.55
4.2B CLIP ViT-L/14 Phi-3.5-mini Autoregressive Dynamic (block concat)
SigLIP-SO400M
Language Model
Owen2-7B-Instruet Autoregressive Up to 16,384
LLaMA3-8B-Instruct Autoregressive 169 (364x364)
Architecture
Visual Tokens
Table 5: 3 MLLMs used in this study, with various architectures, sizes, and number of visual tokens per image.
Table 5 shows more details about the models used in this research. The following a brief introduction to all MLLMS.
Qwen2-VL a taate-offthhe-rr multimodal model excelling in visual understanding. It supports multilingual OCR and dynamically adapts to arbitrary image resolutions for Instruct.
human-like visual processing. The link to the model on Hugging Face: Qwen2-VI--7B-
Idefics3 is an open multimodal model that processes arbitrary sequences of images and text to generate text outputs. It supports tasks like visual question answering, image-based storytelling, and pure language modeling, with improved OCR, document understanding, and visual reasoning over previous versions. The link to the model on Hugging Face: Idefics3-SB-Llamaal
Phi-3.5-Vision is a lightweight, state-of-the-art multimodal model from the Phi-3 family, trained on high-quality text and vision data. It is enhanced via supervised fine-tuning and direct preference optimization to ensure strong instruction following and safety. The link to the model on Hugging Face: Phi-3.--Vision-Insttue
C.2 Datasets and Evaluation Metrics
To construct the dataset, each task type is first divided into four subsets: the query set, the support set, the training-query set, and the training-support set. These subsets are mutually exclusive and contain no overlapping samples. Depending whether the setting inference or fine-tuning, we then organize the data into either an inference file or a training file accordingly, with the desired number of shots. While choosing demos, only those demos of the same type of query will be chosen. This structure ensures clear separation between evaluation and training data, and allows for flexible adaptation to different experimental settings.
To assess the performance of our model on our proposed dataset, we primarily adopt accuracy (in percentage) as the evaluation metric. For numerical answers, only strict matching considered correct, while textual answers are evaluated through keyword- matching. We additionally perform manual verification on subset of the predictions to ensure the reliability of the scoring. The final accuracy is computed as:
Ncorect
Accuracy
x 100%
N
where Neorrect denotes the number of correctly predicted answers and Nootal represents the total number of evaluated samples.
17

---

Published as a conference paper at COLM 2025
C.3 Baselines
To successfully apply DARA, we first initialize new nn. Module inside the model, with dimensions [number of attention layers, number of attention heads, number of images]. Then, for each input, we extract the position of each image from the input ids. By progressively passing the new parameters into the attention layer, we are able to apply scalar factor to each image token in the attention matrices of the specified layers and heads. All of the factors are first initialized to 1. As the model is finetuned using the settings described in Table 6, this parameter matrix is updated automatically.
Parameter
Value
Batch size per device Gradient accumulation steps
4
Epochs
5
Learning rate
1e-3
Warmup steps Optimizer
5
A (8-bit)
Weight decay
0.01
Learning rate scheduler
Linear
Precision
fp16 / bf16 (auto)
Max sequence length
2048
Random seed
3407
Output directory
"outputs"
Table 6: General Supervised Fine-tuning Configuration
To ensure fair comparison in parameter scale between LoRA and DARA, we restrict LoRA fine-tuning to only the first attention layer. Specifically, we disable LoRA in all other layers and retain only the two projection layers, q.proj and k_proj, in the first attention layer. Furthermore, to make the number of trainable parameters linearly controllable and scalable, freeze proportion of the gradients in the initialized lora and 1ora_B matrices based on predefined ratio. This allows us to gradually vary the effective amount of the parameter of LoRA.
Parameter
Value
Target layers
layers. 0. self-attn at...pttn....rjj
layers. 8, self_atltttt k_proj
Trainable modules
Attention only
LoRA rank (r)
2
LoRA alpha
LoRA dropout
0.0
Bias setting
None
Seed
3407
Table 7: LoRA Configuration
D More Experimental Analysis
D.1 Discussion on the Performance between DARA and LoRA
In this section, we provide detailed comparison between LoRA and DARA. We focus both parameter efficiency and performance across various tasks in the TrueMICL.
DARA is specifically designed to enhance model's ICL ability with minimal parameter overhead. As shown in Figure 5, taking the result of Qwen2-VL on the task Operator Induction example, when both methods modify the same target- the and q projection
18

---

Published as a conference paper at COLM 2025
Model
Method
VQAv2 GQA A-OKVQA COCO
0.800
Qwen2-VL Operator Induction
7ero-shot
78.6
69.9
87.8
120.1
DARA
No-image
78.7
69.9
87.8
119.9
0.775
4shots baseline
Qwen2-VV.
Random
71.6
89.1
120.2
LoRA
79.3
71.3
89.9
120.1
0.750
DARA
79.1
71.3
89.7
119.9
Accuracy
0.725
Params:140
Zero-shot
62.9
65.5
79.7
117.6
No-image
62.7
65.6
80.0
116.9
0.700
Idefics3
Random
61.1
66.4
80.8
117.6
LoRA
66.7
80.8
116.8
0.675
DARA
64.2
66.5
80.7
117.3
0.650
Zero-shot
63.2
62.9
80.3
115.7
No-image
63.5
62.7
80.4
115.7
0.625
Phi-3.5-Vision Random
64.1
116.1
2252
4505
6758
9011
11264
13516
15769
18022
20275
22528
LoRA
65.3
63.9
81.5
115.7
DARA
65.3
63.8
81.5
115.7
Num of Params
Table 8: Results on four classical vision- language tasks. Accuracy used for VQAv2, GQA, and A-OKVQA and CIDEr used for COCO. The performance is similar across all methods, suggesting that classical tasks benefit little from in-context learning, and DARA shows signs of overfitting.
Figure 5: (green point) and LoRA (blue curve). DARA outperforms the 4-shot baseline using only 140 parameters, whereas LoRA requires tens of thousands to reach similar performance, highlighting DARA's parameter efficiency.
Comparison between DARA
Model
Method
Operator
Clock
Outlier
CLEVR
Sudoku
Palindrome
Owen2-VL
LoRA
93.33
49.67
87.33
98.00
97.67
99.00
LoRA+DARA
94.67
(+1.34) 51.33 (+1.66)
89.33 (+2.00)
99.00 (+1.00)
99.00 (+1.33)
99.67 (+0.67)
Idefics3
LoRA
67.67
34,00
81.33
63.00
95.00
79.67
LoRA+DARA
70.00
(+2333)
37.00 (+3.00)
83.00 (+1.67)
64.33 (+1.33)
96.00 (+1.00)
80.33 (+0.66)
Phi-3.5-vision
LoRA
65,33
45.67
82.00
56.33
92.67
85.33
LoRA+DARA
68.00 (+2.67) 45.00 (-0.67)
84.00
(+2.00)
38.00 (+1.67) 94.33 (+1.66) 96.33 (+1.00)
Table 9: Accuracy (%) of different models on six TrueMICL tasks, comparing full-parameter LoRA and LoRA with DARA. While full-parameter LoRA already achieves strong perfor- mance across all tasks, the integration of DARA still provides modest but consistent gains.
layers in the first attention layer-DARA significantly outperforms LoRA at low parameter scales. Specifically, LoRA with 2,252 parameters shows no improvement over random 4-shot inference. In contrast, DARA achieves better results using fewer than 225 parameters, matching the performance of LoRA configurations with 11,264 parameters. When applied to the same layer, LoRA needs 40-50 times more parameters than DARA to achieve similar performance. These results demonstrate DARA's effectiveness in low-resource settings while maintaining exceptional parameter efficiency.
We further explore DARA's behavior when combined with full-scale LoRA train- ing. As shown in Table 9, full-parameter LoRA-where millions of parameters are trained-und sained-undouubedlly leads to a performance boost across all TrueMICL tasks. How- ever, when DARA is applied on top of full LoRA, we still observe small but consistent improvement of 1 to 2 percent in accuracy. While this gain may appear small, it is achieved with parameter cost that is less than one ten-thousandth of that used in full LoRA, once again demonstrating the efficiency and practicality of DARA.
To summarize, DARA shows clear advantages over LoRA in low-parameter regimes, achiev- ing better performance with significantly fewer parameters. It is particularly effective for tasks that truly rely on in-context learning, as shown in TrueMICL. Moreover, DARA remains beneficial even when combined with full-parameter LoRA. These findings demon- strate that DARA serves as both an effective standalone solution and a valuable complement to current adaptation approaches for multimodal in-context learning.
19

---

Published as a conference paper at COLM 2025
D.2 Closed-source model performances
We evaluate GPT-4o on the six TrueMICL tasks under both 0-shot and 4-shot settings, As state-of-the-art multimodal model, GPT-4o's performance can serve as an approximate upper bound. As shown in Table 10, GPT-4o struggles in the O-shot setting, achieving low accuracy except on Sudoku. Once demonstrations are added, performance improves dramatically across all tasks. This further confirms that TrueMICL. requires models to effectively integrate and reason over visual-textual demonstrations, validating its role as multimodal in-context learning benchmark.
Model
Method
Operator Clock Outlier CLEVR Sudoku Sudoku (hard) Palindrome
GPT-4o
Zero-shot
8.00
2.00
26.00
4.00
94.00
0.00 91.00
33.00
4-shot MICL
100.00
87.00
99.00
96.00
100.00
97.00
Table 10: Accuracy (%) of GPT-4o on the six TrueMICL tasks under different settings. Demonstrations greatly improve performance, indicating strong multimodal in-context learning requirements.
D.3 Amounts of params in DARA
Qwen2-VL Operator Induction
74
Accuracy (%)
72
70
head
heads
69
14 heads
21 heads
28 heads
20 Number of Attention Layers
Figure 6: Comparison of performances when applying different settings changed attention layers and heads.
Figure 6 illustrates the model's performance under different design choices of DARA, where we vary the number of modified layers and attention heads. Taking the n-shot setting as an example, the total number of trainable parameters introduced by DARA is calculated as (n + +1) (number heads) (number of layers). shown in the figure, within the range of few hundred parameters, different configurations have no significant difference in the impact on the final performance.
D.4 Effect of Prompt Design
We provide further analysis on the effect of prompt formulation. Our default prompt is intentionally minimal to avoid leaking visual information into the text and to better test models' ability to reason from image demonstrations alone. To evaluate whether DARA's improvements could instead be attributed to better prompt wording, we compare this with more explicit instructional prompt.
Original Prompt (Minimal and Neutral)
System: Learn from the demos and give only the answer
User: <demo1> Question: what is the result of the following
20

---

Published as a conference paper at COLM 2025
mathematical expression? Answer: 12
(repeated for demos)
<query> Question: What is the result of the following mathematical expression? Answer:
Instructed Prompt (With Explicit Task Description)
System: The following image-text pairs will show you how to answer the questions.
From these demos, learn the desired operation the numbers shown in each image.
User: Here is the demo image 1: <demol> The question is: What is the result of the following mathematical expression?
The answer is: 12
(repeated for demos)
Here is the query where you need to answer, following the previous answers:
<query> The question is: what is the result of the following mathematical expression? The answer is:
Model
Prompt
Operator
Clock
Outlier
CLEVR
Sudoku
Palindrome
Qwen2-VL Original
67
31
87
86
94
96
Instructed
67
30
87
87
95
96
Table 11: Performance comparison under two prompt settings (accuracy %). Instructional prompts offer marginal formatting consistency, but little accuracy gain.
Performance Comparison Table 11 shows the performance of Qwen2-VL under both prompt settings. The instructional version yields slightly more consistent formatting, but does not improve overall accuracy. This suggests that the performance gain brought by DARA is not due to prompt wording, but due to its stronger ability to leverage visual support examples.
21


Reviews:
[{"review_id": "vp4Oiz6gI4", "reviewer": "Reviewer_Lzs5", "summary": "The paper argues that current MLLMs do not make full use of the visual information in demonstrations and that prior MICL test sets fail to stress this aspect. To address this gap, it introduces a new MICL dataset, PureMICL, which evaluates an MLLM’s ability to integrate multimodal information. In addition, it proposes Dynamic Attention Reallocation (DARA), a simple and efficient fine-tuning strategy that pushes the model to pay more attention to image cues.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper’s discussion of the weakness in current MLLM and MICL benchmarks is convincing and meaningful. The PureMICL benchmark provides several fresh directions for assessing MICL, which can guide future MLLM development toward stronger multimodal skills. The proposed fine-tuning method is simple and effective, and its comparison with several baselines soundly supports the authors’ claims.", "reasons_to_reject": "The paper may omit some experimental details. Here are the main issues I observed:\n\n1. Prompt design is under-specified. All models used in the paper are dialogue-style, yet the paper does not show the prompts used when running PureMICL tasks. I assume the experiments feed the models plain image-text pairs and then a stand-alone query image. If so, the chosen MLLMs may not be familiar with such an MICL format and could struggle to output answers that exactly match the dataset’s required wording. Under this setup, DARA may have an extra advantage because it is pre-exposed to support data.\n    \n    For a fair comparison, here are some possible prompt design for reference:\n    - add an initial instruction stating that several image-text pairs will follow\n    - add a final reminder asking the model to produce an answer in the same style as the context\n    - optionally state the concrete task at the very start, which would form another strong baseline\n\n2. Definition of the \"No-image\" baseline is unclear.  Does it supply only the answer text for every demo, or does it also include textual descriptions of the demo images?\n    \n3. In the main results table (Table 2), you may wang to add a final “Overall” column that reports the average performance across the seven tasks, so that the gaps between baselines are easier to see.\n    \n4. Are LoRA and DARA evaluated on top of the Random baseline? You may want to state more clearly in the paper.", "questions_to_authors": "Another paper[1] is similar in topic but focuses more on text-image pairs, which may be a useful reference.\n\n[1] Zeng, Yuchen, Wonjun Kang, Yicong Chen, Hyung Il Koo, and Kangwook Lee. \"Can MLLMs Perform Text-to-Image In-Context Learning?.\" In First Conference on Language Modeling.", "extra_scores": {}}, {"review_id": "vNOhJZFxnf", "reviewer": "Reviewer_gEKm", "summary": "This paper identifies a critical issue in visual language understanding, where Multimodal Large Language Models (MLLMs) over-rely on textual patterns in demonstrations and neglect visual cues (termed the \"unimodal trap\"). To address this, the authors propose two contributions: 1) Dynamic Attention ReAllocation (DARA), a lightweight fine-tuning method to rebalance attention towards visual tokens , and 2) PureMICL, a new benchmark dataset specifically designed to require understanding of visual context in demonstrations for successful task completion (focusing on \"task learning\" rather than just \"task recognition\"). Experiments show current MLLMs struggle with PureMICL, but DARA significantly improves performance on both PureMICL and standard VL tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper addresses a significant and widely recognized limitation (visual neglect) in current MLLMs' capabilities.\n- This study proposes DARA, a parameter-efficient fine-tuning technique that demonstrably improves MICL performance by encouraging visual attention. It outperforms comparable LoRA configurations significantly in low-parameter regimes.\n- PureMICL is contributed as a diagnostic dataset specifically designed to test and necessitate true multimodal understanding in context, moving beyond limitations of standard VL benchmarks for MICL evaluation.\n- Comprehensive experiments are provided, across multiple MLLMs (Qwen2-VL, Idefics3, Phi-3.5-Vision), showing the shortcomings of existing models on PureMICL and the consistent effectiveness of DARA.", "reasons_to_reject": "- While DARA aims to induce stronger attention to visual tokens via learnable reweighting, the paper lacks sufficient evidence that this actually occurs. To validate DARA's effectiveness, the authors should provide visualizations or quantitative analyses of attention maps/statistics, demonstrating the shift in attention towards visual tokens after tuning.\n- The paper lacks a baseline that would provide a clearer understanding of DARA's advantage: a hard-coded setting where specific attention heads are forced to attend only to visual tokens. This simple comparison would isolate the benefit of DARA's learned, dynamic reweighting versus a static constraint.\n- The paper does not provide performance data for state-of-the-art VLMs (ChatGPT Vision, Claude, Gemini). This raises the question of whether scaling model size or pre-training data alone could solve some of the PureMICL tasks, potentially negating the need for a novel method like DARA.\n- PureMICL Task Diversity: While PureMICL introduces tasks requiring visual context, the task types (math, concept binding, patterns, novel concepts) might not fully represent the breadth of real-world multimodal applications.\n- Comparison with Other Mitigation Strategies: The paper compares DARA primarily to LoRA and baseline MICL. Discussion or comparison with other potential strategies for mitigating visual neglect (e.g., different model architectures, prompt engineering techniques) is limited.", "questions_to_authors": "- Does DARA primarily amplify existing relevant visual signals, or could it potentially introduce biases by forcing attention onto less relevant visual tokens in some cases?\n- Beyond the tasks in PureMICL, what other complex, real-world scenarios do you foresee benefiting most significantly from overcoming the \"unimodal trap\" using methods like DARA?", "extra_scores": {}}, {"review_id": "eshq8UatXk", "reviewer": "Reviewer_Ro7V", "summary": "As MLLMs rapidly advance, there is growing interest in their Multimodal In-Context Learning (MICL) capabilities. However, existing MLLMs tend to ignore visual information during MICL and rely heavily on textual cues.\nTo address this, the paper proposes a fine-tuning method,  called Dynamic Attention ReAllocation (DARA), that introduces additional learnable parameters to rescale the attention weights associated with visual context, encouraging MLLMs to attend more to visual cues. \nMoreover, the authors identify a key limitation in existing MICL benchmarks: models can perform well by relying solely on textual information, bypassing the need to interpret visual inputs. To mitigate this, they introduce PureMICL, a new benchmark specifically designed to require integration of both visual and textual modalities. The dataset ensures that successful task completion is only possible through genuine multimodal reasoning.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses an important and timely aspect of MLLMs, specifically focusing on the existing limitations of MICL.\n2. The paper is well written and easy to follow.\n3. The introduction of the new dataset, PureMICL, is a significant contribution to the community and will likely become a valuable benchmark for evaluating MICL performance in future MLLM research.", "reasons_to_reject": "1. The paper claims that for any attention scaling factor matrix $F$, there exist corresponding LoRA-style updates $W'_Q$ and $W'_K$ as shown in Eq (5) and Eq (6). However, Eq (6) involves the input matrix $X$, making $W'_K$ dependent on the input. This contradicts the standard LoRA setting, where weight updates are fixed and input-independent. Furthermore, matrix $F$ varies based on the positions of visual tokens in the input sequence, which again violates the assumptions of typical LoRA configurations. Please correct me if I am misunderstanding the formulation.\n2. The paper argues that DARA outperforms LoRA in terms of both performance and parameter efficiency. However, the LoRA setup used in the experiments is overly restricted, as it only applies LoRA to the first layer’s  $W_Q$  and  $W_V$  matrices. While this constraint helps match the number of learnable parameters with DARA, it deviates from how LoRA is commonly used in practice. In standard usage, LoRA is applied across all layers and includes $W_V$ with a typical rank of 8, which still offers good parameter efficiency. To make the comparison more fair and informative, the authors should report results for this more representative LoRA configuration. Additionally, comparing DARA against full fine-tuning results should help establish its effectiveness relative to a stronger baseline.\n3. Although DARA is intended to boost attention toward visual tokens, the paper does not include any empirical analysis or visualization of the learned attention scaling parameters. Including evidence such as the distribution of these parameters or attention heatmaps would strengthen the claim that DARA effectively encourages the model to focus more on visual information.", "questions_to_authors": "Please refer to the Reasons to Reject section above.", "extra_scores": {}}, {"review_id": "cnxkIrGhWl", "reviewer": "Reviewer_1vtb", "summary": "This paper investigates the limitation of current Multimodal Large Language Models (MLLMs) in Multimodal In-Context Learning (MICL) where they tend to prioritize textual information over visual cues in demonstration examples (\"unimodal trap\"). This can hinder genuine multimodal adaptation and task learning for new scenarios. \n\nThe authors introduce two contributions aimed at addressing this:\n1. Dynamic Attention ReAllocation (DARA): A lightweight fine-tuning strategy (around 160 parameters) designed to encourage the model to attend more to visual tokens within the context images by applying learnable scaling factors to their attention scores.\n2. PureMICL: A new benchmark dataset comprising 7 tasks across 4 categories (math, concept binding, pattern finding, novel concept). \n\nThese tasks are designed such that solving the query ideally requires understanding the multimodal relationships presented in the demonstrations (task learning), making them purportedly more sensitive to true MICL ability than tasks solvable by text imitation alone. The authors suggest these tasks become difficult if demonstration images are absent.\n\nThe paper presents experimental results on several MLLMs (Qwen2-VL, Idefics3, Phi-3.5-Vision, and GPT-4o). The performance on PureMICL for baseline models appears low, suggesting the tasks are challenging. DARA shows some improvement on PureMICL compared to baselines, including parameter-matched LoRA, and demonstrates parameter efficiency. The paper also indicates DARA does not negatively impact performance on standard Vision-Language benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Addresses a Relevant Problem: The paper tackles the issue of MLLMs potentially neglecting visual information in MICL, which is a recognized challenge in the field.\n\n2. Presents Novel Approaches: The DARA fine-tuning method and the design principles behind the PureMICL benchmark offer new ways to address and evaluate MICL capabilities.\n\n3. Demonstrates Some Empirical Improvements: Experiments show that DARA can lead to performance gains on the proposed PureMICL benchmark compared to certain baselines, suggesting it has some potential to improve visual attention.\n\n4. Highlights Parameter Efficiency: DARA achieves its results with a relatively small number of parameters, which is a desirable characteristic for practical application.\n\n5. Generally Clear Presentation: The paper is reasonably well-written and the concepts are generally understandable.", "reasons_to_reject": "1. Marginal Performance Gains: While DARA shows improvements, the magnitude of these gains, especially on certain tasks and compared to the potential of full fine-tuning, might be considered relatively modest, placing the contribution marginally above the acceptance threshold.\n\n2. Limited Exploration of DARA: The application of DARA is primarily focused on the first transformer layer. The lack of extensive exploration across different layers might leave questions about its full potential and optimal usage.\n\n3. Convincingness of PureMICL: While the design principles of PureMICL are plausible, further analysis might be needed to definitively demonstrate that the tasks truly and uniquely require deep multimodal understanding of the demonstrations and are not solvable through other means or are significantly impacted by visual perception limitations.\n\n4. Comparison to Other Methods: The comparison to existing methods, while including LoRA, could potentially be broadened to include other recent techniques aimed at improving multimodal understanding in MLLMs.", "questions_to_authors": "1. Could you provide a more detailed analysis of the specific instances where DARA leads to the most significant improvements on PureMICL? What characteristics of these tasks might explain this?\n\n2. Have you considered or experimented with applying DARA to transformer layers beyond the first one? What were the preliminary results or what are the challenges in doing so?\n\n3. To further validate the design of PureMICL, could you provide more in-depth ablation studies, perhaps showing the performance drop when specific visual cues in the demonstrations are altered or removed (beyond just removing the entire image)?\n\n4. How sensitive is DARA to the choice of hyperparameters, such as the learning rate or the magnitude of the learnable attention weights? Were these extensively tuned?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a new method and a benchmark to address the "unimodal trap" where multimodal models ignore visual information in context. The reviewers all agreed that this is an important problem. They praised the new PureMICL benchmark as a valuable contribution. Initially, the reviewers expressed some reservations. They questioned if the proposed DARA method actually shifted the model's attention and asked for more experiments to validate the benchmark's design. The preliminary ratings were generally positive but reflected these concerns.

The authors provided a comprehensive response with many new experiments. They included visualizations that showed DARA successfully redirects attention to visual information. They also added human studies and further ablation experiments to strengthen the claims about their benchmark. These additions satisfied the reviewers' concerns. All reviewers found the rebuttal convincing and updated their reviews to recommend acceptance. From the AC's perspective, while the methodology is interesting, the current examples may not be comprehensive or robust enough, especially against large-scale models. However, the novel methodology and important problem setting are interesting enough to warrant acceptance.
Decision: Accept