Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answering
Authors: Akash Gupta
Amos Storkey
Mirella Lapata
Venue: ICLR 2026 Poster (2026)
Keywords: Large Multimodal Models
Meta-learning
Soft Prompts
Test-Time Adaptation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A meta-learning method using soft prompts and an attention-mapper boosts few-shot performance in Large Multimodal Models

Abstract: Large Multimodal Models (LMMs) often rely on in-context learning (ICL) to perform new visual question answering (VQA) tasks with minimal supervision. However, ICL performance, especially in smaller LMMs, does not always improve monotonically when increasing the number of examples. We hypothesize that this happens because the LMM is overwhelmed by extraneous information in the image embeddings that is irrelevant to the downstream task. To address this, we propose a meta-learning approach that induces few-shot capabilities in LMMs through a fixed set of soft prompts distilled from task-relevant visual features, which are adapted at test time using a small number of examples. We facilitate this distillation through an attention-mapper module that can be easily integrated with any LMM architecture and is jointly learned with soft prompts. Evaluation on the VL-ICL Bench shows that our method successfully achieves task adaptation in low-data regimes with just a few gradient steps, outperforming  ICL by 21.2%.  Comparisons with parameter-efficient finetuning methods demonstrate that meta-learning further enhances this adaptation by 7.7% for various VQA tasks.

Published as a conference paper at ICLR 2026
META- PROMPT DISTILLATION FOR FEW- SHOT VISUAL QUESTION ANSWERING
Akash Gupta, Amos Storkey, Mirella Lapata School Informatics, University of Edinburgh fakash. gupta, a. wtw.sstorryyedo.... ac. .u, mlap@inf ac.....feddc...u.
ABSTRACT
Large Multimodal Models (LMMs) often rely on in-context learning (ICL) perform new visual question answering (VQA) tasks with minimal supervision. However, ICL performance, especially in smaller LMMs, does not always improve monotonically when increasing the number of examples. We hypothesize that this happens because the LMM is overwhelmed by extraneous information the image embeddings that is irrelevant to the downstream task. To address this, we propose a meta-lcarning approach that induces few-shot capabilities in LMMs through a fixed set of soft prompts distilled from task-relevant visual features, which are adapted at test time using a small number of examples. We facilitate this distillation through attention-mane mapper module that can be easily integrated with any LMM architecture and is jointly learned with soft prompts. Evaluation on the VL-ICL Bench shows that our method successfully achieves task adaptation in low-data regimes with just a few gradient steps, outperforming ICL by 21 21.2%. Comparisons with purameter-efficiin finetuning methods demonstrate that meta-learning further enhances this adaptation by 7.7% for various VQA tasks.'
INTRODUCTION
Humans have the remarkable ability to quickly learn new tasks in multimodal environments with just few trial-and-error attempts. Extensive research in cognitive science suggests that this ability arises from learning hierarchical abstractions and maintaining shared structural priors across related tasks based on past experiences (Griffiths et al., 2019; Finn, 2018: Kirsch & Schmidhuber, 2022). Drawing on this prior knowledge enables rapid learning in new situations and reduces the need for large amounts of task-specific demonstrations (Finn et al., 2017).
Large Multimodal Models (LMMs) are able to perform a multitude of tasks ranging from reasoning to fine-grained image understanding and visual question answering (Liu al., 2024: Li al., 2023a; Laurençon et al., 2024). They are typically built on top of a base Large Language Model (LLM) by supplementing it with vision encoder and a connecting module that acts as bridge for different modalities to interact. When (pre)trained at sufficient scale and finetuned on wide range of multimodal tasks (with natural language instructions), LMMs can learn new tasks by virtue of in-context learning (ICL), i.e., by being prompted with few input-output examples, without requiring any updates to model parameters (Zhao et al., 2024; Zong et al., 2025; Coda-Forno et al., 2023). Although the training- frree nature of ICL has led to its rapid adoption across tasks and domains, its underlying mechanism remains ill-understood (Hendel et al., 2023; Huang et al., 2024) and its empirical behaviour can be inconsistent.
Zong et al. (2025) demonstrate that ICL is most effective for large-scale LMMs (~72B parameters), while smaller models (<7B parameters) struggle with increasing in-context examples and their performance either plateaus or deteriorates, even when extending the context length or giving detailed instructions. They attribute this limitation the fact that smaller models struggle with the large number of image tokens in long sequences. They become confused and perform the task haphazardly or default their parametric knowledge, effectively ignoring the in-context examples. Figure I shows a failure case from the Fast Open-Ended MinilmageNet dataset (T (T'simouueell et al., 2021), using
We release our code here- https: / //gi //gggtuub...////// cww/cm/m/aa..oooo cat.uu..cm/aaaaeeeeee http///kasguupp.a9..m 97/MPPD

---

Published as a conference paper at ICLR 2026
LMM
Gold Answer:
Blicket
In-Context Examples
Query
OO
0-shot: This an image of
lion. The lion appears to be resting.
1-shot: Lion
2-shot: Slation
3-shot: Slation
Question: This is a? Answer: Blicket
Question: This is a? Answer: Slation
4-shot: Slation
Question: This is a?
5-shot: African
Figure 1: Failure case of LLaVA- -aeeAA-Oneiiiiii Vision-7B (Li et al., 2025) on an example from the Fast Open- Ended MinilmageNet classification task (Tsimpoukelli al., 2021). When no in-context examples are provided (0-shot), the model generates a generic description of the image. As more examples (shots) are added, it begins to learn the answer format (single word), but still fails to grasp the task, producing incorrect or irrelevant predictions. We only show the in-context examples (left) for 2-way I-shot setting for the sake of brevity but provide model predictions (in red) for up to 5 shots.
LLaVA-OneVisiont (Li al., 2025). The task is framed in 2-way N-shot format where a support set with N labeled examples of two classes is provided. The model uses ICL with the support set to classify new query examples from the two classes. Without any support set or in-context examples (0-shot), the model outputs a generic description about the image based on parametric knowledge and ultimately fails to answer correctly, despite being prompted with a few examples.
Building on this observation, we hypothesize that effective few-shot adaptation at test time may be compromised by the information added by the im- age embeddings. Figure 2 compares maaee-to-Text (I2T, red) and Text (T2T, bluc) performance of LLLaVA-OneVision-77 on the Operator Induction and CLEVR Count Induction tasks (see Appendix A.2.2), The results reveal a significant performance gap: T2T ICL consistently outperforms 12T and im- proves monotonically with additional shots. More- over, adding detailed task instructions to 12T (green; see Appendix A.2. 1) actually degrades performance, suggesting that naively increasing the number of im- age embeddings in context impairs the model's inher- ent ICL ability. While a set of more precise image embeddings would be preferable, their continuous nature makes it challenging to distill task- in- formation from them. As alternative, propose to learn a fixed set of new embeddings that can be easily finetuned at test time.
Text (12T)
75
Image Text (ET) Fost (121)
65
Moun Accuracy (%)
55
40
35
25
Number of Shots
Figure 2: 12T and T2T performance with VA-OOe Vision-- 7 on Operator Induction and CLEVR Count Induction tasks.
This idea of task adaptation has gained significant traction in the literature through prompt tuning (Lester et al., 2021) which finetunes a set of continuous soft prompts while keeping the underlying language model frozen; the prompts are prepended in the context at test time, effectively steering the model toward the desired task. Our approach learns new tasks using soft prompts that receive task information from the LLM in the form of loss gradients during finetuning. These gradients update the soft prompts which when fused with the image embeddings are able to distill relevant features from them. To facilitate this fusion, we propose an attention mappeer that uses a multi-head attention (Vaswani et al., 2017) architecture for extracting relevant task-specife image information and can be substituted the projection layer of any LMM architecture.
Our approach relies on rapidly adapting to new tasks at test time using only few examples, which is not addressed by traditional finetuning methods. Prior work (Finn et al., 2017: Ravi & Larochelle, 2017) addressed this challenge by training meta-learner that can infer an optimal learning strategy for new task after being exposed to a distribution of tasks. We apply this procedure to our multimodal prompt distillation setting by employing the widely known MAML algorithm (Finn et al., 2017) and use its lightweight first-order approximation to train the attention attention-mapper and soft prompts. We focus on visual question answering (VQA; Antol et al. 2015; see example in Figure 1), a general-purpose
2

---

Published as a conference paper at ICLR 2026
task often used to evaluate the image understanding capabilities of LMMs, and demonstrate the benefits of MAML training applied to LMM architectures. Our contributions are as follows:
We introduce MAPD (Meta-Adaptive Adaptive Prompt Distillation), an alternative to in-context learning that meta-learns fixed set of soft prompts within large multimodal models (LMMs) via distillation. MAPD enables adaptation to new tasks with a few examples using a few gradient updates at test time, and consistently improves performance as the number of shots increases, To our knowledge, this is the first exploration of meta-learned prompt distillation for cross-task generalization in LMMs under low-data settings.
We propose flexible attention -eaeeeee--ppperr module, inspired by Najdenkoska et al. (2023). that exploits all patch features from the vision encoder and can be readily incorporated into the projection layer of any LMM architecture. It is trained jointly with seft prompts and can be efficiently adapted at test-time to distill task-specific visual information.
Extensive evaluation VL-ICL Bench.² 2 (Zong al., 2025), diverse benchmark for image perception and mathematical reasoning, demonstrates that our approach outperforms ICL and several other prompt distillation and parameter--fffcient finetuning methods.
2 RELATED WORK
Our approach, Meta-Adaptive Prompt Distillation (MAPD), builds upon several existing research areas including few-shot learning, prompt tuning and test-time adaptation.
Multimodal Few-shot Learning Learning from few examples has been a long-standing goal in machine learning. Early work by Vinyals et al. (2016) introduced Matching Networks for one-shot maage-to-text classification. This approach leverages a support set of labeled images to classify an unlabeled query image, laying the foundation for few-shot learning in vision tasks. With the advent of large language models (LLMs) and large multimodal models (LMMs: Alayrac et al. 2022; Zhao et al. 2024), in-context learning (ICI.; Zhao et al. 2024; Lester et al. 2021) has emerged as a popular method for few-shot adaptation. ICL involves providing a few input-output examples directly in the model's prompt without updating its parameters. While this is a computationally inexpensive method, its performance for LMMs can be inconsistent (Zong al., 2025) and may even degrade as more examples are added, particularly in smaller models.
Learning with Prompts Another widely adopted approach for adapting models to task-specine data is prompt optmization. Wang et al. (2022) explored this idea with small language models (~ 0. 0.IM) such as BERT for text classification. Subsequent work (Hou et al., 2022) introduced soft prompts to overcome the limitations of optimizing over discrete vocabulary tokens. Khattak et al. (2023) further proposed PromptSRC for CLIP-based vision- laiguale encoders, mitigating overfitting of soft prompts. While these methods perform well on classification tasks, their extension to LLM-based architectures and problems such as anestion-answeriin and mathematical reasoning remains limited.
Test-Time Adaptation These methods aim to dynamically adapt models during inference on test examples, that may have distributional differences from the training data. This adaptation can either involve training of model parameters (Hardt & Sun, 2024) or can be entirely training free (Karmanov et al., 2024). Additionally, previous work (Hu et al., 2025) has taken advantage of prompt tuning and other PEFT methods such LoRA (Hu et al., 2022) resolve catastrophic forgetting issues during test time training and achieve state-of-the-ar performance. Shu et al. (2022) propose Test-time Prompt Tuning (TPT), method that adapts vision-language models for zero-shot classification by tuning soft prompts on image augmentations. Previous work (Najdenkoska et al., 2023; Li et al., 2023b) has also explored meta-learning of soft prompts for small models and limited range of vision-languee tasks such as fast-concept binding.
We extend upon this idea to provide an alternative for few-shot adaptation in LMMs. Specifically, we design meta-learning procedure, namely MAPD, to learn soft prompts that can distill task-relevant visual features from image embeddings and can be rapidly adapted at test time for variety of new tasks using few examples. Najdenkoska et al. (2023) rely on single [CLS] token from CLIP's s vision encoder, which limits the attention-mapper's is capacity. We instead use the complete set of hidden patch features, enabling the attention attention-mapper to encode fine- ameiinadeed visual information for
We only focus on single-image few-shot VQA tasks and leave the multi-image scenario for future work.
3

---

Published as a conference paper at ICLR 2026
Support Response
X supp
Query
X
Response
Language Model
fo
Hp aape
H supp
H query
H query
upu Support Instruction
Xq
Xuueery
Query Instruction
Inner-loop step
ATT-mapper Mo
ATT-mapper M
Voleng
Compp
Cquery
pr pLanup
Z
P'
Z
Trained
Vision Encoder 9+
X PPP
X Query Image
Support Image
Figure 3: Our proposed MAPD framework based LLaVA v1.5-7B (Liu et al., 2024): im- age embeddings are distilled into soft prompts /' during instruction finetuning. The support set (X-""P .gg _ xP= is processed initially the obtain loss value Lupp which is used in the inner- loop obtain task-specific parameters {0', P'). Next, the query set (Xyueryy Xquery X ) used to calculate the query loss for the outer-loop meta mneta-parameter optimization {0, PJ.
distillation into soft prompts. We show that MAPD can be applied any LMM architecture and achieves sfate-of-the-ar performance on visual question answering tasks (Antol et al., 2015).
3 PROBLEM FORMULATION
3.1 FEW-SHOT VISUAL QUESTION ANSWERING
Visual Question Answering (VQA; Antol et al. 2015) key task for evaluating the ability of vision-language models understand images by accurately responding to questions about various aspects of visual content. These questions can vary widely, ranging from descriptions of objects inside bounding boxes (Krishna et al., 2017) to solving high-school geometry problems (Gao et al., 2025), but are mostly grounded in the visual information present the image.
In VQA, we typically have a dataset D = {(X;, X 12) where X, E II, X, E Q and X, A. and Z
set all images, Q the set of all questions, and A the set of all answers. Our goal is to learn a function fa parametrized by 0, that maximizes the likelihood of the answer given the image and the question, 11 (X X;). Following the standard train-test paradigm in deep learning, we evaluate whether fe generalizes well by dividing dataset D into (Dimin Diest) such that maximizing the above likelihood
Drain also maximizes the likelihood of answer Diest, common assumption is that the size of Damin large enough so that function fo does not overfit on Daain In the context of feo- VQA, we treat the in-context examples (or shots) given LMM during ICL as Dtain Since the examples in Dirain are limited (as few as 1-shot), it becomes harder avoid overfitting while training and still perform well on Diest We conceptualize this problem as one of learning about an underlying task represented by Dirain and adopt meta-learning (Finn et al., 2017) which exploits the shared structure across a distribution of tasks to learn a prior over model parameters, thereby enabling stable transfer to new tasks with limited data. In the following, we describe how we enforce this prior over parameters through the curation meta-tasks containing few-shots. A sketch of our model architecture and training procedure is shown in Figure 3.
3.2 IMPROVING TASK UNDERSTANDING WITH META-TASKE
The core idea of optimization- based meta-learning is to learn a good initialization of parameters, which when finetuned specific task, enables stable transfer for that task with few gradient
4

---

Published as a conference paper at ICLR 2026
steps (Finn al., 2017). To promote this capabiliiy, training involves processing batches of few-shot datasets that represent an underlying task. We refer to these few-shot datasets as meta-tasks and propose to create them from our finetuning data mixture based on the original LLaVA datasets. We provide details of our specific data mixture in Appendix A.1.1.
More formally, let p(D) denote our data mixture. We create meta-task T by randomly sampling a fixed subset of VQA examples (image, question, answer triplets) from dataset D ~ p(D) and partitioning the examples further into support and query sets 7 = (Duup, Dauar j. To be consistent with the notation introduced in Section 3.1, we treat the support set as D = Ditain and the query set as Dquery = Diest We continue this process until all samples from D' have been assigned to at least one meta-task. This meta-task construction is performed for each dataset in p(D), resulting in meta-task distribution p p(T We now describe our model architecture designed to process these meta-tasks. Further details on the number and composition of meta-tasks for training and evaluation are provided in Appendix 1.1.2.
3.3 MODEL ARCHITECTURE
We design our LMM architecture (Figure 3) based on the visual instruction tuning framework of LLaVA v1.5* (Liu et al., 2024) and further describe our modifications for incorporating the attention- mapper. For clarity, we omit the distinction between support and query sets in this section both are processed in the same manner. As shown in Figure 3, the model consists of pretrained CLIP ViT-L/14 visual encoder (g+) with an aspect ratio of 336px; for an input image Xu. the encoder gives us hidden visual features Zü which are then passed to the projection layer that consists of an attention-mapper Mj responsible for extracting useful features from Z
Attention Mapper We re-design the projection layer of LLaVA v1.5 to include soft prompts P by introducing an attention citention--aaper M3 for improved task-specific feature extraction. Specifically, we prepend Z, with set of m learnable prompt tokens P to obtain a sequence C' = (P, Z.) which is then passed to the attention-mapper (see Figure 3). Both prompt tokens P and weights are initialized with Xavier Uniform initialization (Glorot & Bengio, 2010). We define the mapper
H Hp+u M,
(QQK)) *
where the query is Q = -M C, the key is K = Mb C the value is V = VM C. and their corresponding matrices are { AMg.MM, My). The mapper computes the dot product of the query and key vectors which are then passed to a softmax function compute activation scores for every feature in vector V. Finally, we extract the first m embeddings corresponding to the learnable prompt tokens from the set Ho+e that correspond to the specific image embeddings Hp. These are now passed to the LLM (f&) as prompts for further processing. We denote the trainable parameters for the attention-mapper with 0p- = (0, P).
Language Model The quality of the learned prompts highly depends on the underlying language model. We update the L.M of LLaVA v1.5 with the state-of-the-arr Qwen2.5-7B-insttrue LLM, which has demonstrated strong performance on complex tasks such as mathematical reasoning and coding and supports the generation of up to 8K tokens. The LLM (fa) receives the concatenated sequence of image and text tokens to generate the answer X, = ((H p. Hq)). In this pipeline, only the attention mapper parameters tp are trained, making our approach parameter-efficiee generalization. The number of trainable parameters is approximately 24M (see Appendix A.1.3 for hyperparameters). The training objective maximizes the likelihood function, (X, (X,|X, Xg), parametrized by 0y. where X, is the answer, X, is the image, and X, is the question. For clarity, refer to this model, namely LLaVA-ATT-QQwen2.5 7B, as our base LMM in the following sections.
for cross-task
3.4 MODEL TRAINING
We train the attention mapper parameters to learn image-conditioned soft prompts in two stages following curriculum learning procedure similar to LLaVA v1.5 (Liu et al., 2023). In the first-stage, which is aimed at feature alignment, the attention-mapper is pretrained on the LCS-558K subset
We adopt LLaVA v1.5 due to its simplicity and publicly available training code and datasets (Section A.1. This prevents mixing between training and test datasets and enables evaluation over unseen tasks. We demonstrate in Section& 4.3 that our method can be easily applied other LMM architectures.
5

---

Published as a conference paper at ICLR 2026
of the LAION/CC/SBU dataset filtered with more balanced concept coverage (Liu al., 2023). Further details on pretraining are mentioned in Appendix A.1.4. In the second stage, which aims to distill task-specific image features into prompts Hy. the attention attention-mapper parameters Op are finetuned on diverse task-specific instructions. We describe our MAMI --Mseb finetuning procedure below and also introduce alternative methods which we compare against in our experiments.
3.4.1 LEARNING TO DISTILL PROMPTS WITH FIRST-ORDER META LEARNING
Our prompt distillation procedure, MAPD, uses the -adel-agiostic first-order approximation of MAML (Finn et al., 2017) which aims to learn robust initialization of meta-pant peta-parsmeters that enable efficient adaptation to new tasks with just few gradient updates. We borrow the implementation of Antoniou et al. (2019) and use their first-order version and (learnable) per-step learning rates (a) to further optimize the training process. We sample batch B of meta-tasks from p(T and use the support set of each task to convert fp into task specific parameters 6 with a few gradient steps. Equations (2) and (3) show a single step of this inner loop:
D
supp 0g
log(po, (X 61 x)) (2)
0/ = 0p
Ct VVV 0p 10p
supp
33)
Dsupp
The outer loop involves optimizing the meta- meta-parameters which in our case are the original attention- mapper parameters Op on the query set using the task-specific parameters Oz:
| Σ
query '6
log(no, (X:l x;)) (4)
0p:=0p Op -
query 8
(5)
Dqquery
Equation (5) the first-order approximation of the meta-update in MAML (Finn et al., 2017) that treats the gradient of 6% w.r.t. O, for meta task as a constant. This approximation avoids backpropagating through the entire computation graph of the inner loop and avoids the Hessian-vector product estimation of the query loss. This saves huge GPU memory while still approximating gradient in the same direction as the true MAML gradient (Weng, 2018). We provide a sketch of MAPD training in Figure 3 and more detailed algorithm in Appendix A.1.5 as Algorithm 1
3.4.2 ALTERNATIVE METHODS FOR PROMPT DISTILLATION
We also implement other prompt distillation methods based on our model architecture to compare their performance with MAPD on few-shot VQA tasks. We provide more formal description of these methods below, highlighting important differences from our framework.
Multi-Task Prompt Distillation We define a multi-task baseline where we eliminate the bi-level optimization of MAPD. Specifically, at each iteration, we sample a batch of meta-tasks from p(T metn) and optimize the following loss per task:
-1 N N Σ (X x)))
Lop
such that N = + [[query] This loss accumulated across the entire batch of meta-tasks used to update 6p. We refer to this baseline as Multi-Task
In-Context Prompt Distillation Previous work (Chen et al., 2022; Min et al., 2022) suggests is possible to meta-learn task information by reducing the bi-level optimization of MAML to sequence prediction problem over in-context examples with the help of pretrained LLMs. We develop method called In-Context where we concatenate the support set with each query example in a meta-task, and optimize the following loss function to distill this task information from LLMs into soft prompts:
Lo
log(po, IX X D)WPP)
Dquery
Methods without Meta-tasks To further understand the benefit of curating meta-tasks (see Sec- tion 3.2), compare with the original finetuning procedure of L.L.VA-v1.5 7B but only train O,

---

Published as a conference paper at ICLR 2026
without any meta-tasks for fair comparison. We refer to this method NoMeta-taske in subsequent sections. We also compare with model averaging, which is computationally efficient and has been shown to increase performance 2022). We separately finetune the attention-mapper on each dataset D ~ p(D), and take an average of all dataset-specific parameters lb weighted by their corresponding dataset size ratios:
out-of-distribttio datasets (Choshen al., 2022; Wortsman et al.,
w
where w = |D [D]. We refer to this baseline as Model-Avg"DD in subsequent sections.
3.5 TEST-TIME ADAPTATION
After learning optimal parameters with MAPD (and alternative distillation strategies), adapt the ineetiio-mmpper to a new test task by fine-tuning for K gradient steps. We empirically find that K I 30 is sufficient for all prompt distillation methods to converge, which we attribute to our adaptation procedure training only 24M parameters over a few examples. We further explain how this value is selected in Appendix A.2.3. Concretely, given K s 30 steps, we perform task- specific finetuning of the parameters 0, on the support set Dist of test task Ties using the inner-loop optimizer mentioned in equation 3. We then evaluate model performance the query set Dquery for that task.
4 EXPERIMENTAL RESULTS
4.1 EVALUATION DATASETS
For evaluation purposes, our datasets follow the same structure the meta-tasks introduced in Section 3.2, with support and query examples. We use the recently introduced VL-ICL benchmark (Zong et al., 2025), designed test the ICL capabilities of LMMs on various tasks like fast concept binding, multimodal reasoning, and fine-grained perception. Meta-tasks for testing are created by randomly sampling a support set from the training split of the VL-ICL datasets and test/query set from their respective testing splits. In line with our training pipeline, which exclusively utilizes datasets containing a single image per example (see Section A. I.I. we focus solely on single image- to-text scenarios, leaving multi-image cases for future work. We report results on four tasks from VL-ICL: a) Fast Open MinilmageNet ( (Open-MI), where the model must name new objects based on few examples; b) Operator Induction, where the model must solve image tasks of the type 2?7 ? given training examples like 1 ? = 4; e) CLEVR Count Induction, where the model must count objects that satisfy given attributes like "shape: sphere"; and d) TexOORR where the model must transcribe highlighted text contained in an image. We provide more details on these tasks in Appendix A.2.2. The final model performance is calculated as the average across all meta-tasks.
4.2 MODEL COMPARISONS
Our results are summarized in Table 1, which compares MAPD against alternative prompt distillation methods (see Section 3.4.2) and reports the mean accuracy of up to eight shots. We compare two types of test-time adaptation methods, namely in-context learning (ICL) which prompts the underlying LLM with no distillation of image embeddings and finetuning (FT) with K < 30 gradient steps, which further distinguished based on whether they meta-tasks during training. Results for individual shots are in Appendix A.2.4; additional results for more shots are in Appendix A.2.
Prompt distillation improves task induction in LMMs at test-time. Our results in Table show that FT adaptation with few-shots (support examples) largely outperforms ICL at test time evaluated over query examples, with average increase of 21.2% over all datasets. These results highly support our hypothesis that distilling task-specific information from image embeddings to create targeted prompts improves the few-shot capabilities of the underlying LLM (in our case Dwen-2.5-7BB-nnstteee Instruct).
*We also keep separate validation set for each VI.IICI. dataset (sampled from the training split) to select the best model which we then evaluate on the test (query) set. More details can be found in Scction A.2.3 We also provide ICI. performance of publicly available models in Appendix A.2.5 for reference.
7

---

Published as a conference paper at ICLR 2026
Methods
MT
Open-MI
OP IND
CLEVR
TextOCR
TTA with ICL
No.Meta-task
X
43.8 + 0.9
12.1 + 0.6
18.0 0.2
6.8 0.4
Model-Avg
X
26.6 0.7
0.5
7.1
2.8 0.3
In- In-Context
51.1 + 0.9
1.6
0.2
23.8
Multi-TaskPP
48.6 0.9
10.0 0.6
1.1
6.9 = 0.4
MAPD
53.3 0.9
9.60 + 0.5
12.3
7.30 1 0.4
TTA with FT < 30
NoMeta-taskk"D
x
68.0 0.8
3
0.2
Model-Avg
X
6.8
40.0
29.1
In-Context
30.9 0.8
3 0.2
18.9 0.3
Multi-Task
74.6 = 0.7
4
29.9 + 0.2
22.9 + 0.4
/
MAPD
77.9 + 0.7
47.7 + 0.5
31.4 + 0.2
26.4 + 0.5
Table 1: Evaluation tasks from the VL-ICL Bench (Zong al., 2025) with LLa 7B as the base LMM. Each method trains 24M attention attention-mapper mapper parameters. We report the mean accuracy across shots {1, 2, 4, 5, 8} with 95% binomial confidence intervals and compare different prompt distillation approaches. TTA:Test-Timee Adaptation, FT: Finetuning with K 30 gradient steps, ICL: In-Context Learning, MT: Meta-Tasks used (V) not (X) during training. Qualitative results are in Appendix A.2.6 and A.2.9.
Additionally, our results show that finetuning just the attention-mapper param- eters only requires few gradient steps (K < 30) test-time to generalize to unseen tasks and does not lead overfitting over the support examples (Ap- pendix A.2.3). For one- to-one comparison, we look into In-context which performs better with on 3 out of tasks compared to its ICL adaptation and enables prompting the underlying LLM with fixed set of learned task-specific embeddings.
LoRA
Open-MI
OP.IND
CLEVR
Tex1OCR
TTA with FT < 30
All
LLM layers
55.1 + 0.7
13.3 + 0.6
15.1 + 0.2
10.4 + 0.4
[0-15] LLM layers
67.3 + 0.8
25.5 - 0.6
30.0 = 0.2
23.8 + 0.3
[0-15] LLM layers + ATT
69.1 11 0.8
30.5 1 0.5
28.7 1 0.2
24.5 1 0.3
Table 2: LoRA configurations for base LMM and evaluated on the VL-
ICL bench. We report the mean accuracy across shots {1, 4,5, 8} with 95% binomial confidence intervals. ATT: Attention-Mapper; TTA: Test-Time Adaptation; FT: Finetuning with K < 30 gradient steps.
Meta-learning and meta-tasks improve few-shot learning. Table shows that methods using meta-tasks indeed superior. For ICL-based adaptation, In-Context performs best, while for FT-based adaptation, our proposed approach, MAPD. achieves the best overall performance across all four datasets test time. This further suggests that first-order MAML. learns the best initialization of attention-mappe parameters 9p- These parameters are subsequently adapted for a test task with a few gradient steps and few-shot examples to produce precise set of soft prompts that improves LMM predictions on that task. Our detailed results in Table 11 in Appendix A.2.4 further show that for FT-based adaptation, MAPD most effective in the 2-shot case for Operator Induction, surpassing Multi-Task PD by 10%. Finally, MAPD with FT is the only approach that exhibits strictly monotonic improvements as the number of shots increases, showing better scaling behavior.
MAPD surpasses other efficient finetuning approaches for few-shot adaptation. We compare MAPD with LoRA (Hu et al., 2022), ofate-of-the-art parameter-efficient finetuning (PEFT) approach. In Table 2, we integrate LoRA in the base LMM in three configurations and evaluate on VL-ICL:
(1) naively applying LoRA to underlying LLM layers (as done in LLa v v1.;; 5; Liu et al. 2023) increases the number of trainable parameters (~ 300M) and the model finds it difficult to converge within 30 gradient steps at test-time; (2) restricting LoRA to the first LLM layers (~ 24M parameters) offers better test-time performance; and (3) adding LoRA to the attention-mappe layers further boosts performance as it provides some distillation over the image embeddings before prompting the underlying L.M.M Ultimately, MAPD still outperforms the best LoRA configuration
8

---

Published as a conference paper at ICLR 2026
25hmn(M)
Iiscamce [iMtsds-Aag" DiMuls
MAMA
Indrat))
2-hott]) - 4mboiti)
55
30
Moan Save (52)
$3.3
Mar Accuracy
45
60
55
401
30
20
Oriiiaal (FM)
Took Indoction
Porcrption (FMI
Math Reswrring
(FM)
(Qwes-Vi Soor))
Soft Prompts P (ohooaii in (PD)
Figure 4: (a) Left: Performance comparison between MAPD+FT (M) and In Io- (I). Mean Accuracy is computed across all VL-ICL datasets. We consider different prompt token lengths P [4, 16, 64, 256) which are shown in loga (P) scale for different shots. (b) Right: Performance of different prompt distillation methods on three Operator Induction subtasks: Task Induction, Perception, and Math Reasoning. We report mean exact-match (EM; %) for 1,2 and 8-shots as defined in the VL-ICL Bench (Zong et al., 2025) except for Mathematical Reasoning, which uses mean ratings generated by Qwwn-225-VVL332B-Inmmee More details can be found in Appendix A.2.9
by an average 7.7% across all VL-ICL datasets. This demonstrates that MAPD is the best choice for achieving fast test-time adaptation in low-data scenarios. We provide additional LoRA training details in Appendix A.1.4 and further detailed results can be found in Appendix A.2.4.
4.3 ABLATION STUDIES AND ANALYSIS
In this section, we present ablation studies across various model architectures and sizes, along with a more in-depth analysis of the benefits of test-time fine-tuning using MAPD. Please refer to appendix for further ablations on testing 1) robustness image perturbations (Appendix A.2.7) and 2) different few-shot selection strategies (Appendix A.2.8).
100
1MP SPAML ATT AITT
Mcan Score (9)
80
77.8
60
-40
20-2
20
18.5
29.6
211
16.1
What are the benefits of the attention map- per and soft prompts? In Figure 5, we com- pare different designs for the projection layer the base LMM for rapid few-shot learning. We clearly that MAPD benefits most by incorporating the attention attention-mapper mapper and soft prompts (SP+ATT). We draw two key conclusions from this experiment: (1) distilling task-relevant information from CLIP embeddings with soft prompts yields substantial improvements, with an average gain of 16.3% across architectures. (2) replacing the 2-layer MLP used in LLaVA v1.5 with attention mapper leads an additional average gain of 13.1% %6 thanks to its inherent weighting mechanism of pairwise similarities over CLIP embeddings.
TexOUR
CLEVR
Operator
OPEN MM
Induction
Figure 5: Projection layer architectures in the base
LMM. SP: Soft Prompts, ATT: Attention- -Mapper, MIP: 2-layer MIP (originally used in L.LVVA v1.5).
How does the number of soft prompts affect performance? We examine how MAPD's performance changes with the number of soft prompts across varying shot settings for VL-ICL datasets in Figure 4(a). Additionally, we show results of our best ICL approach, In-Context as a baseline for this comparison. We see that MAPD scales favorably and learns more consistent task information from marginal
gradient updates at test time as the number of soft prompts is increased. Furthermore, improvement per added prompt token is substantially greater when more shots are provided. In contrast, the performance of In-Context generally deteriorates with more prompts and struggles to jointly attend to more examples and longer prompts. We also investigate this further by analysing the softmax attention values of the underlying LLM for soft prompt embeddings containing image information in Appendix A.2.10. We note that the attention entropy decreases as the context length grows for PD- This highlights its inherent limitation in being unable to attend to all the soft prompts when answering query, while MAPD consistently attends to all the soft prompts.

---

Published as a conference paper at ICLR 2026
Vision Encoder
LLM
TTA
NoMeta-tusk
Model-Avgg
In- In-Contexd
Multi-Task
MAPD
CLIP VIT1/14
Qwen2,5-7B
ICL
43.8 + 0.9
26.6 0.7
51.1 0.9
48.6 0.9
53.3 +: 0.9
Instruct
FT=30
68.0 = 0.8
63.1 0.8
64.5 0.8
74.6 0.7
77.9 = 0.7
Qwen2.5-3B
ICL
24.3 0.7
30.5 0.7
48.3 0.9
39.1 0.7
32.9 0.7
CLIP VIT-L/14
Instruct
3T
56.5 - 0.9
66.0
4.9
61.1 1.1::00
67.3 -: 0.6
ICL
20.0 = 0.7
26.2 0.7
46.3 0.9
29.1 0.7
49.9 = 0.9
CLIP VIT-L/14
Vieumu v1.5-7B
30
69.1 = 0.8
74.9 - 0.4
66.7 0.8
70.3 0.7
75.8 = 0.4
Qwen2.5-7B
ICL
42.6 +
40.7 0.9
47.3 0.9
50.0 0.9
43.6 0.9
SigLIP-SO400M
Instruct
FT:30
52.0 0.9
56.5
56.0
59.3 + 0.5
60.5
CLIP VIT-L/14
Qwen3-8B
ICL
55.0 0.9
48.5
63.5
57.6
60.3 0.5
FT
72.3 0.9
69.1 0.7 0.7
71.4 0.9
80.4 0.6
83,5 - 0.6
Table 3: Comparison of prompt distillation approaches under different LMM settings while keeping the attention irrention-mapper and soft prompts fixed. We report mean accuracy across to shots with 95% binomial confidence intervals for the OPEN_MI benchmark. The original LLa LLaVA-ATT-Qwen! 7B architecture is highlighted in gray. FT: Finetuning with K < 30 gradient steps, ICL: In-Context Learning, TTA: Test-Tine Time Adaptation. NoMeta-task" PD and Model-Avg do not use meta-tasks.
use of fixed number of soft prompts is superior to In-Context where the number of soft prompts vary according the number of shots,
To what extent does MAPD facilitate task understanding at test time? We take a closer look at how effectively MAPD captures task understanding at test time, using the Operator Induction task (See Figure 8) as a case study. To solve this task, the model should correctly (a) identify the operands in the query example (Perception); (b) identify the operation from few-shot examples (Task Induction); and (c) use its own mathematical knowledge over the identified elements to reason towards the answer (Mathematical Reasoning). To test whether the model understands these subtasks, we design specific prompts and modify query examples listed in Appendix A.2.9. In Figure 4(b), observe that MAPD outperforms other prompt distillation approaches on all three subtasks, leading to better performance on the original task. MAPD shows a major improvement for task induction with an increase of 11.7% compared to MultiTask"" We also observe that solving each subtask individually easier than tackling the original task, as the latter requires integrating knowledge from all subtasks, which is challenging when only a few shots are available at test time. Finally, MAPD excels mathematical reasoning, effectively utilizing the underlying LLM's reasoning capabilities.
Does MAPD generalize across different LMM architectures? We next examine different LMM architectures that affect MAPD's performance. Specifically, we report results in four settings that vary the underlying LLM and vision encoder: a) using smaller LLM (Qwen2.5-3BB -nnssttt--B- b) using a different and relatively weaker L.M (Vicuna v1.5 15-57B); e) using a different vision encoder, SigL.P (Zhai et al., 2023); and d) and using relatively powerful L.M (Qwen3-8B) (Yang et al., 2025). In Table 3, MAPD outperforms other baselines with FT adaptation across different model configurations on the Open MI task, demonstrating its robustness and generalizability. Fine-tuning based test-time adaptation for prompt distillation substantially outperforms ICL, with average improvements of +24.6, +37.06, + +2.02,, and +18.04 across the four settings, respectively. This highlights the significant benefits of test-time prompt distillation.
5 CONCLUSION
This work introduced Meta-Adaptive Prompt Distillation (MAPD), novel meta-learning approach that endows LMMs with few-shot capabilities. MAPD employs a fixed set of soft prompts, distilled from task-relevant image features, which can be adapted at test time using only a few examples. A key component of our method is an attention- Sammee module, which can be flexibly integrated with any LMM architectures and is jointly learned with soft prompts to facilitate distillation. Extensive evaluation on the VL-ICL benchmark shows that MAPD consistently outperforms traditional ICL and other efficient finetuning approaches across a diverse range of VQA tasks. Additional analysis (Appendix A.3) shows that MAPD incurs higher per-query latency than ICL due to gradient- based adaptation, but scales more favorably with increased test-time compute budget and is more data- efficient. Future work could focus on improving MAPD's computational efficiency for resource- constrained scenarios and extending it multi-image tasks and complex reasoning problems.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDEGMENTS
We thank the anonymous reviewers for their feedback. We gratefully acknowledge the support of the UK Engineering and Physical Sciences Research Council (grant EP/W002876/1).
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances Neural Information Processing Systems, 2022. URL. https: /ppnreview net forum? i d intMiiiii..ss..
Stanislaw Antol. Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433 2015.
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How train your MAML. In International Conference on Learning Representations, 2019. URL https: /pwp//epeeree..ee////.e net/ forum?
id=HJGven05577.
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. Meta-learning via language model in-context tuning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 719-730, Dublin, Ireland, May 2022. Association for Computational Lin- guistics. doi: 10. 1..118651112222.acclle lonno. URL httpst /aaet///llllliaae httaa/aaannooooooooooooooooooooa www.holl....oooooooo.ooooooooooo .rgg..rgg//o org/2022. acl-long 53/.
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining, 2022. URL
abs 2tt//2224...3 23440. 03044.
Julian Coda-Forno, Marcel Binz, Zeynep Akata, Matthew Botvinick, Jane X Wang, and Eric Schulz. Meta-in-context learning in large language models. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL https: http///ppeeeeeeeeee/ee/eeee net forum??rm cerummide 2id= sxCxpa00za.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning. pp. 1126-113, PMLR, 2017.
Chelsea B Finn. Learning to learn with gradients. University of California, Berkeley, 2018
Jiahui Gao, Renjie Pi. Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing HONG, Jianhua Han, Hang Xu, Zhenguo Li, and Lingpeng Kong. G-LLaVA: Solving geometric problem with multi-modal large language model. In The Thirteenth International Conference Learning Representations, 2025. URL https: ht/p//peeevveeeeeeeeeeeeeeeeeeee n forum ? d=px1 1 67 4wp3c.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Thomas L Griffiths, Frederick Callaway, Michael B Chang. Erin Grant, Paul M Krueger, and Falk Lieder. Doing more with less: meta-reasoning and meta-learning in humans and machines. Current Opinion in Behavioral Sciences, 29:24-30, 2019. ISSN 2352- 1546. doi: https:/ddoi.or//10.000 2019.00 005. URL. https: /ww.. sci enced rect. wwm www.///ciine//aatiii...... pii/S235211 618302222.22.. Artificial Intelligence.
Moritz Hardt and Yu Sun. Test-time training nearest neighbors for large language models.
In The Twelfth International Conference on Learning Representations, 2024. URL https:
//oppreeviieeeee eew net www.://oromm..dcon.naeeae..

---

Published as a conference paper at ICLR 2026
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Compu- Singapore, December 2023. Association
national Linguistics: EMNLP 2023, pp. for Computational Linguistics. doi: 20.18633/v1/0223 121.1655212222....mn.gmmmmmmmmm findings--emllp.22.. URL https: //aclanthology www.oolooyyooggg2222 wwwp/ogg....... f rindi Findings-enne nas-emns-mmn 1p 624/.
Yutai Hou, Hongyuan Dong. Xinghao Wang, Bohan Li, and Wanxiang Che. MetaPromptiin: Learning to learn better prompts. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Scung-Hoon Na (eds.), Proceedings of the 29th International Conference on Computational Linguistics, pp. 3251-3262 Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. URL https: / aaaalcaann awwlnnhhlloggtoo org/ 2022. coling-1.287.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL ht https: http:////peeeeeeee/eeee www.eiiwwwweeeeeeeeeeee.ee.eeeee forum? id=nzevKeeFYES.
Jinwu Hu, Zitian Zhang, Guohao Chen, Xutao Wen, Chao Shuai, Wei Luo, Bin Xiao, Yuanqing Li, and Mingkui Tan. Test-time learning for large language models. In Forty-seconl International Conference on Machine Learning, 2025. URL https: /ttps/////ppeeeeeeeee net /tff/ooormmii iCYbIaGKSR.
Brandon Huang, Chancharik Mitra, Leonid Karlinsky, Assaf Arbelle, Trevor Darrell, and Roei Herzig. Multimodal task vectors enable many-shot multimodal in-context learning. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024. http:: //openreview net wwwprummid-ooooooo
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and patter recognition, pp. 2901-2910, 2017.
Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. The IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024.
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming- Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15190-15200, October 2023.
Louis Kirsch and Jürgen Schmidhuber. Self-referential meta learning. In First Conference Automated Machine Learning (Late-Breaking Workshop). 2022. URL n forum? d CixP7..
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-75, May 2017. ISSN 1573-1405. doi: 10. URL http:: //op::///do... org/10 1007 00000//s126630000 01900111--.1111111
1007/s11263-016-
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?, 2024, URL
org/ abs/2405. 02246.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih (eds.). Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v 100653//1/2221.... 1/2021.emmlp.pp......... wwtillpmaii..2.. main.243, URL https: / //cl///caaaaaaa// antho 1 ogy. wrg//02/002 emnle n. 243/.
12

---

Published as a conference paper at ICLR 2026
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning, 2023a. URL abs/ abs/2305. 03726.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-onevision Easy visual task trans- fer. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https: / //openreview.net/ net forum?2dd=22&&@
Juncheng Li, Minghe Gao, Longhui Wei, Siliang Tang, Wenqiao Zhang, Mengze Li, Wei Ji, Qi Tian, Tat-Seng Chua, and Yueting Zhuang. Gradiiet-reguulaee meta-prompl learning for generalizable vision-language models. In 2023 IEEE/CVF International Conference Computer Vision (ICCV), pp. 2551-2566 2023b. doi: 10. 00110000CCV5100002223...2222
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Teirty-seventh Conference Neural Information Processing Systems, 2023. URL https: // //openreview. net/ forum? i 1==0##2
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26296-26306, June 2024.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations, 2024. URL https:::////. http:///oppeeeeeeeeeeeeeee net/ forum KiN-m
Sewon Min, Mike Lewis. Luke Zettlemoyer, and Hannaneh Hajishirzi. MetalCL: in context. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2791-2809, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 1t.11863111/2222 nwacl-main.nnnna.mm- moalmmmi..111111 201. URL https::p//// / aclanthology org/ 20220022222222 maaaci-mainn 201/.
Learning to learn
Ivona Najdenkoska, Xiantong Zhen, and Marcel Worring. Meta learning to bridge vision and language models for multimodal few-shot learning. In The Eleventh International Conference Learning Representations, 2023. URL https: / http///peeeeeeeeeeeeeeee f Forum? ? d= 92cQxxI.
Chengwei Qin, Shafiq Joty, Qian Li, and Ruochen Zhao. Learning to initialize: Can meta learning improve cross-task generalization in prompt tuning? In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11802-11832. Toronto, Canada, July URL
2023. Association for Computational Linguistics. doi: 10. 1ttp:///11/23../lll/// ht https: /tt///acla /aa/ccaannti aaclaanaaanooyyyccoaooaa org/ 2222/222222 aaaclcllonn 659/
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su. Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: aaa/axivvv org/ alos 2412..111. 15115.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference Learning Representattions, 2017. URL https: //ppnreeeee//eeeeeeeeee new..eee//ee /frrum?
1d=rJYO-Kc11..
ShareGPT. Sharegpt. https: httess//shaeeppt.e com/ 2023.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization vision-language models. In Al- ice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https:/ openreviee net forummtrmmmmmmm did= @8PVEk5a4Fg
13

---

Published as a conference paper at ICLR 2026
Amanpreet Singh, Guan Pang. Mandy Toh, Jing Huang. Wojeiech Galuba, and Tal Hassner. Textoer: Towards large-scale end-to-end reasoning for urbitrary-shaped scene text. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8802-8812, 2021.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 200-212. Curran Associates, Inc., 2021. URL https: h http::::s/prrocedinngss nww.ips.. lareer__ileeee wwwessipapeeeeee / pdf.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 3637-3645, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819
Jianing Wang, Chengyu Wang, Fuli Luo, Chuanqi Tan, Minghui Qiu, Fei Yang. Qiuhui Shi, Songfang Huang, and Ming Gao. Towards unified prompt tuning for few-shot text classification. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 524-536, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 22.1863/11/2222.fmm finddings-eennp.3... URL https://aclanthology org.orgg/////
wiindinngseenn.. 37/.
Lilian Weng. Meta-lcarning: Learning to learn fast. attps://Lie http::iiiiaaegggtttiti.. github............... 1o/ posts/2018-11 10011-130-mttaatm -ew.ringriinnn..... 2018.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, 2022. URL https: / /ttxxaxxvvv////////////////aaav/ org/ abs/ 2203. 05482.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang. Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: //ax/aiivii../////v///aa////aavv org/ abs / /505 09388.
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Scong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10.1 11.109///CV.22299000122
2019. doi:
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. 2023. URL
/2003 15343.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. URL https: //oppnreview... net. forum? daarmm@iddr
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. MMICL: Empowering vision-language model with multi- modal in-context learning. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttp://////////i////ee/e net forum? (
14

---

Published as a conference paper at ICLR 2026
Yongshuo Zong, Ondrej Bohdal, and Timothy Hospedales. VL-ICL bench: The devil in the details of multimodal in-context learning. In The Thirteenth International Conference on Learning Representations, 2025, URL https: /ttpp///rreeeeeeeeeeorruummmeeee neer//oorumm..dccppPPLXY
15

---

2. Section A.2 Evaluation Published as a conference paper at ICLR 2026
A APPENDIX
Section A.I: Implementation Details
(a) Section A.I.I Finetuning Data Mixture
(b) Section A.1.2 Details on Meta-Task Creation
(c) Section A.1.3 Model Configurations
(d) Section A.1.4 Training Details
(e) Section A.1.5 Psuedo Algorithm of MAPD
(a) Section A.2.1 Detailed Task Instructions for LMM Evaluation.
(b) Section A.2.2 Evaluation Datasets from VL-ICL Bench
(c) Section A.2.3 Test-Time Adaptation Details
(d) Section A.2.4 Detailed Results VL-ICL Bench
(e) Section A.2.5 Performance of Publicly Available LMMs on VL-ICL Bench
(f) Section A.2.6 Qualitative Results on VL-ICL Bench
(g) Section A.2.7 Robustness Against Image Perturbations
(h) Section A.2.8 How to Select Few-Shot Examples for Better Performance?
(i) Section A.2.9 Details Ablation Study for Operator Induction
Section A.2.10 Attention Entropy Analysis
(k) Section A.2.11 Scaling to More Shots
3. Section A.3 Test-time Compute Analysis for ICL FT
16

---

Published as a conference paper at ICLR 2026
Table Finetuning Data Mixture Statistics
Dataset
No. of examples
Question Types
Conversations (57.669)
LLsVA-Instruct
157,712
Detailed Image Description (23,240)
Complex Reasoning (76,803)
GQA
72,140
Visual Reasoning
Image Question Answering with Reading Comprehension
OCR-VQA
80,000
Image Question Answering with Reading Comprehension
TextVQA
21,953
Image Question Answering and Bounding Box Prediction
Visual Genome
86,417
Visual Math
MAVIS-Math-Metagen
87,348
Question Answering
TabMWP-Cauldron
22,717
Tabular Math Reasoning
Image Question Answering and Bounding Box Prediction
RefCOCo
48,447
Knowledge Grounded
OKVQA
8,998
Image Question Answering
VQAv2
82,783
Image Question Answering
MultipleChhoie Question Answering
A-OKVQA
66,160
Math Question Answering and Reasoning
Geo-170k (QA)
67,823
Total
802,498
A.1 IMPLEMENTATION DETAILS
A.1.1 FINETUNING DATA MIXTURE
For model finetuning, we create our multi-task data mixture for single image per example using the visual instruction tuning data of LLaVA v1.5 (Liu al., 2023) which contains a mixture of 12 different datasets 6 ranging from long conversations to academic multiple-choice questions. Since we are only training image-based prompts, we remove the language-only ShareGPT-40K dataset (ShareGPT, 2023). Additionally,
include 3 different math reasoning/QA datasets from the LLaVA One Vision data mixture (Li et al., 2025) which are known to improve LMM performance on difficult reasoning and logical QA tasks (Lu et al., 2024). We further get rid of the extra answer formatting instructions to test the true few-shot transfer learning ability of our approach without the need of external task induction. Table 4 shows the list of all the datasets along with their size and question types.
A.1.2 DETAILS META-TASK CREATION
As mentioned in Section 3.2, meta-tasks are small subsets of examples randomly sampled from a single VQA dataset (D') within the training data mixture (p(D)). Each meta-task consists of support and query sets, each containing a fixed number of VQA examples (image, question, answer triplets). The support provides few-shot demonstrations the model, either in-context examples for gradient- adaptation, depending on the prompt distillation method. The query set is used to
'We use this dataset only for academic research purposes as mentioned by the original authors and follow the Open AI Usage Policy for GPT-4 generated datasets. Additionally, we conform to the license (CC-BY-4.0) for Cauldron datasets.
17

---

Published as a conference paper at ICLR 2026
Table 5: Meta-Task composition during test-time adaptation
Open_MI
Operator Induction
CLEVR
TextOCR
No. of Meta-tasks
5000
4000
6000
5000
Support examples per meta-task
[1,2,4,5]
[1,2,4,8]
[1,2,4,8]
[1.2.4,8]
Query examples per meta-task
1
Table 6: Meta-Task composition during the finetuning stage
MAPD
Multi-task"
In Pa-Coneext
No. of Meta-tasks
39,650
79,300 8000
72,100 8000
(train/val)
8000
Support examples per meta-task (train/val)
10/
5/
10/
[1,2,4.5,8]
[1,2,4,5,8]
[1,2,4,5,8]
Query examples
per meta-task (train/val)
10/1
5
1/1
Total no. of examples (train/val)
793,000
793,000 16,000
793,000 /
16,000
16,000
optimize the LMM (specifically, the attention- mapiie parameters (0p)) in our case) during fine-tuning, and to evaluate performance during inference. This meta-task construction protocol remains consistent across both the fine-tuning stage and test-time fine-tuning, following the framework established by (Zong et al., 2025).
During test-time adaptation, meta-tasks of fixed sizes. VQA examples are randomly sampled from the predefined training and test splits of each dataset. Table 5 specifies the number of meta-tasks per test set, which remains constant throughout our evaluation. All results reported in the paper represent average accuracy computed the query examples of these meta-tasks, ensuring fair comparison across all prompt distillation methods and shot configurations.
the publicly available VL-ICL benchmark code? to construct
During the attention-mapper fine-tuning stage, in order keep a balanced ratio of train-validation splits across multiple datasets in Section A. used in this stage, we divide each dataset into 98% for training and 2% for validation and then combine them separately create the final train and validation splits. We then construct meta-tasks by randomly sampling VQA examples. We treat the support-query composition as tunable hyperparameter alongside those listed in Table, performing grid search to identify the configuration that minimizes validation loss for each prompt distillation method. Table 6 details the optimal support-query compositions, number of meta-tasks, and total number of training and validation examples used for each method.
Additionally, for In-Context we follow the in-context tuning algorithm of (Chen et al., 2022), which uses only 1 query example per meta-task during training and yields optimal performance for this prompt distillation method. Note that the validation done across a different number of support examples for robustness and the total number of training and validation examples remains constant across all methods to ensure fair comparison, regardless of meta-task composition.
VVI.ICL:
18

---

Published as a conference paper at ICLR 2026
Table 7: Grid search values for meta-task methods
No. of support/query meta task
Inner-loop learning rate
Batch Size
Learning Rate
(# of meta-tasks
Search Values
[1 10, 15]
[1e-3, 5e-4, 2e-5]
[1e-1, 5e-2, 5e-1]
[1. 5]
Table 8: Grid search values for non meta-task methods
Learning Rate
Batch Size
Search Values
[le-3, 5e-4, 2e-5]
[16, 32, 64, 80)
A.1.3 MODEL. CONFIGURATIONS
Models We use the publicly available implementation of LLaVA v1.5 and first-order MAML to implement our baselines. Additionally, we use the pretrained model weights from Huggingface for Qwen2.5-7B 2---n.5----nntrrcc LLM and the CLIP ViT-L/14-336px visual encoder"! The output embedding dimension size of CLIP is 1 1,024 and the input word embedding size of the Qwen LLM is 3,584. We set the training context length 4096 for all baselines except for in-context baseline where 8, 192 as it requires training with longer sequences. The attention-mapper is a single multi-head attention block with 8 heads. The token length of the soft prompt P described in Section 3.3 for the attention mapper is set to m 256. The total number of trainable parameters for our model is approximately 24M making our approach significantly parameter-efficient for finetuning.
A.1.4 TRAINING DETAILS
Pretraining Stage During the pretraining stage, we only train the attention and soft prompts for 4 epochs and use a learning rate of 2e-3 with a batch size of 64 per GPU. We perform a train- validation split on the LCS-558K dataset (Liu et al., 2023) by keeping 98% of the examples for training and 2% for validation and take the checkpoint with the lowest validation loss. We use this checkpoint as our base for further task-specific finetuning.
Finetuning Stage For finetuning, we perform grid search across fixed set of values we are constrained by our GPU resources (4 H200 GPUs). For each prompt distillation method, we select the configuration that achieves the lowest validation loss following standard train-val-test procedures. Table (for meta-task methods) and Table (for non non-meta-task methods) provide details of all hyperparameters for which performed grid search. We also provide additional training details below, separately for each method, along with their corresponding best sets hyperparameters after grid search. All approaches were finetuned for 1 epoch to ensure complete pass over the entire finetuning data mixture.
1. MAPD: We use 5 inner-loop steps and initialize the inner-loop learning rate a=1e-1 The outer-loop learning rate set as 1e-3 with a per GPU batch size of meta-task with a gradient accumulation of 2 steps. Each meta-task for training contains 10 support and 10 query examples. Training time
hours.
2. Multi-Taste TDs Similar to MAPD, we use a learning rate of le-3 with a per GPU batch size of meta-task with a gradient accumulation of 4 steps. Each meta-task for training contains 5 support and 5 query examples. Training time ~ 4.5 hours
3. In-Context We use a learning rate of 1e-3 with a gradient accumulation of 4 steps and 5 meta tasks per GPU. Each meta task for training contains 10 support examples and 1 query example. The support examples were concatenated with the strategy that ensured all image tokens of meta-task present in the sequence and we truncate the text tokens
'LLaVA htt.Ah/..wts...c.....ooaaaaaaaaa
"CL (TLP--WT1/1443330x: htt.a////W////a////////////ooooa
19

---

Published as a conference paper at ICLR 2026
if the sequence exceeded the context length of 8192. Further, the few-shot question and answers were concatenated by inserting "Question:" " and "Answer:" strings in between them, inspired from (Alayrac al., 2022). Training time ~ 4.5 hours
4. ModelAvg PD:
We first finetune individual models on each dataset in the finetuning data mixture (Section A. 1.) with a learning rate of 5e-4, For all the datasets, we choose a per GPU batch size of with gradient accumulation of 2 steps. Average time per dataset hours
5. NaMskt ;D: Here, we finetune on the complete data mixture in one training run by sampling batches randomly and again use per GPU batch size of with a gradient accumulation of steps. We also use learning rate of Se-4. Training time ~ 4 hours.
6. LoRA: We only apply LoRA to the attention matrices (Q, K, V) of each layer. For training, we use learning rate of 5e-4 and a per GPU batch size of with gradient accumulation of 2 steps. Further, we performed hyperparameter search for choosing LoRA parameters rank
(I) and scaling factor (a) for the three settings shown in Table 2. Training time ~ 4 hours.
(a) All LLM layers (r = 128, a = 256)
(b) [0-15] LLM layers (r 16, a 64)
(c) [0-15] LLM layers ATT: (r 16, 64)
Computational Requirements We find that the GPU requirement for training the attention attention-mapper mapper mostly depends on the size of the underlying LLM used. For the 7B model training, we use 4 H200 GPUs with VRAM of 143GB per GPU and for 3B models only 2 H200 GPUs were needed. For both the stages, the hyperparameters were tuned using their corresponding validation sets and we choose the checkpoints at the end of first epoch report our results.
A.1.5 PSEUDO ALGORITHM FOR MAPD
We highlight full MAPD algorithm based FoMAML in detail with inner and outer loop that is used to train the attention-mapper parameters Op in Algorithm 1.
Algorithm 1: Meta-Adaphin Prompt Distillation (MAPD)
Input: Meta-Task distribution p(T inner-loop learning rate a, meta learning rate 3
Output: Meta-parameters 8, {0, P}
Initialize 6p with Xavier Uniform Initialization; while not converged do Sample batch of meta-tasks (Tj))= p(Tmee);
foreach task T (Dr Dquiery } in batch do
D
Evaluate 1
log(po.. (Xax X());
[D
supp
Adapt parameters with K gradient steps: for l 1,..., ......K do
0% 0 aV o L supp
Evaluate query 98
log(Pax (Xix X;)):
D query
23
First-Order Meta- Update:
8p x 0p B
V
Bk
L n
2-
20

---

Published as a conference paper at ICLR 2026
Operator Induction "The image contains two digit numbers and representing the mathematical operator. Induce the mathematical operator (addition, multiplication, minus) according the results of the in-context examples and calculate the result."
CLEVR Count Induction "The image contains objects of different shapes, colors, sizes and materials. The question describes the attribute and its value. You need to find all objects within the image that satisfy the condition. You should induce what operation according the results of the in-context examples and then calculate the result."
Figure 6: Detailed task instruction for LLaVA-OneVision-77 LMM evaluation VL-ICL tasks.
Table Evaluation Dataset Statistics
Train Set (Support)
Test Set (Query)
Dataset
Task Category
Size (GB)
Fast Open-MinilmageNet OPPENMMI)
Fast-Concept Binding
5,000
200
0.18
Fine-Grained Perception,
CLEVR Count Induction
800
200
0.18
Task Induction
Perception, Task Induction
Operator Induction
80
60
0.01
Mathematical Reasoning
Tex1OC
Perception, Task Induction
800
200
0.01
A.2
EVALUATION DETAILS
A.2.1 DETAILED TASK INSTRUCTIONS FOR LMM EVALUATION.
Figure shows the detailed task instructions used evaluate LLaVA-OL Vision-BB the Image-to- Text (12T) ICL setting.
A.2.2 EVALUATION DATASETS FROM VL-ICL BENCH
The VL-ICL Bench Zong et al. (2025) includes a diverse variety tasks to test different capabilities of models like Fast-Concept binding, Mathematical Induction, and Fine-grained perception. Given the nature of our model architecture and training (Section 3), we only focus on the single-image Image-to-text (I2T) tasks. Table 9 shows the dataset statistics. We also give brief descriptions of these tasks below along with some examples for better understanding.
1. Fast Open-Ended MinilmageNNet (OPEN_MI) This a variant of the MinilmageNeet few-shot object recognition task (Vinyals et al., 2016), which was repurposed for few-shot prompting (Tsimpoukelli et al., 2021). It is essentially an open-ended image classfication problem, but contains nonsense categorical names like dax or blicker making the test performance not influenced by the prior knowledge an LMM but only dependent on the support examples. This design ensures to test the few-shot abilities of LMMs and how quickly they can learn about new concepts. For the results shown Table 11, we use the 2-way version of this task involving classification between two nonsense categories. An example of a 2-way 1-shot task is shown in Figure
2. Operator Induction Initially proposed by (Zong et al., 2025), this dataset tests various capabilties of LMMs like Task Induction, Perception and Mathematical Reasoning. The support examples involve two operands with a missing mathematical operation and an answer. When testing, the task is to identify the hidden operation from the support example and use it to calculate the result over the operands in the query. An example of a 2-shot task is shown in Figure 8.
21

---

Published as a conference paper at ICLR 2026
Support set for 2-way 1-shot classification
Query example
SEL
Question: This a? Answer: Dax
Question: This is a? Answer: Blicket
Question: This is a? Answer: Blicket
Figure 7: 2-way Fast Open-Ended MinilmageNet
Support set with 2-shots for a multiplication operation induction
Query example
7?0
5?3
4 2
Question: What is the result of the mathematical expression? Answer:
Question: What is the result of the mathematical expression? Answer:
Question: What result the mathematical expression?
Answer:
Figure 8:
Operator Induction
CLEVR Count Induction This dataset contains images from the widely used CLEVR dataset (Johnson et al., 2017) where each image contains set of objects that have certain characteristics based on attributes like shape, size, color and material. The task is to learn to count the objects of the given attribute in the support example and transfer that knowledge to count the objects of any attribute the query example. An example of 2-shot task is shown in Figure 9.
4. Tex1OCR This dataset has been repurposed by (Zong et al., 2025) from the Tex1OCR dataset (Singh et al., 2021) to create a task where the LMM should learn to output the text within red bounding box from the support examples. Even though this task could be solved in a zero-shot setting as we see in the O-shot case with a detailed prompt, we still only focus inducing task knowledge from the few-shot examples. An example of s-shot task is shown in Figure 10.
22

---

Published as a conference paper at ICLR 2026
Support with 2-shots for CLEVR with random attributes
Query example
Question: Size: Large Answer:
Question: Color: Red Answer: 2
Question: Shape: Sphere Answer:
Figure 9: CLEVR Count Induction
Support set with 2-shots wirh red bounding boxes
Query example
KARAOKE
Questions
100
Answers
Question: What text is shown in the red box?
Question: What text shown in the red box? Answer: Answers
Question: What text is shown the red box?
Answer: KARAOKE
Answer: 100
Figure 10: TextOCR
A.2.3 TEST-TIME TIME ADAPTATION DETAILS
follow a similar test-time adaptation procedure to (Qin et al., 2023) to find the best hyperparameter settings for each prompt distillation method, ensuring fair comparison. We first sample 10% of the examples from the training split of each test task and combine them to form a validation set. After meta-task creation
the VL-ICL datasets (Zong et al., 2025) using the remaining training and test splits, we perform a maximum of K = 30 inner-loop steps over each support set and select the Kith-tep model that achieves the lowest validation loss. This model is then used to compute results on the query set. To further validate whether K 30 is sufficient number of fine-tuning steps, we plot average test accuracy curves (up to 40 gradient steps) across VL-ICL datasets, methods, and shot settings in Figure 1.. Accuracies converge within 30 steps, confirming our choice of K. We also provide examples of how predictions evolve during test-time time adaptation in Figures 13, 14, and 15. For reproducibility, Table 10 reports the best learning rates for each method, selected via grid search over (0.1, 1.0] on the validation set with a batch size of one meta-task.
23

---

Published as a conference paper at ICLR 2026
Accuracy
20
Gradient Steps
Gradient steps
(a) OPEN_MI test performance
(b) Operator Induction test performance
MAFO
Nall Tapa
Contor
Aug
Accuracy 15 12
10
30
Gradient steps
Gradient steps
(c) CLEVR performance
(d) TEXTOCR test performance
Figure 11: Average test performances of MAPD with finetuning on different datasets
Table 10: Learning rates for Ametuning-based (FT) test-time adaptation for results shown in Table 1,
Table 2, Table 11 and Table 12.
Training Methods
Learning Rate (LR)
MAPD
1.0
Multi-Taskk?
0.8
PD
0.8
AvgelAvgPl PD
0.6
NoMeta-taske PD
1.0
LoRA
0.2
24

---

Published as a conference paper at ICLR 2026
A.2.4 DETAILED RESULTS
Table 11: Comparison of different prompt distillation approaches single-image tasks from VI.-ICL Bench (Zong et al., 2025). We report accuracy for different numbers of shots (-S). "Avg" is only calculated for shot(s). FT Finetuning with - 30 gradient steps, ICL In-Context Learning. TTA= Test-Time Adaptation. More details are mentioned in Appendix A.2.3. We do not compare on O-shot results. The model used for this evaluation is LLa -LaVA-ATT-Qwen2. 7B which is described in Section 3.3. Meta- Tasks used (V) or not (X) during training. We also provide results for higher number of shots in Appendix A.2. 11 and qualitative results in Appendix A.2.6 and A.2.9.
Meta Task
Open-MI (2-way)
Operator Induction
Methods
0-S
1-S
2-S
4-S
5-S
Avg
0-S
1-S
2-S
4-S
8-$
Avg
TTA ICL
NoMeta-task
X
0.0
35.0
47.0
48.0
45.0
43.8
11.7
13.3
13.3
10.0
11.7
12.1
Model-Avg"D
X
0.0
20.0
22.0
30.0
34.5
26.6
8.3
11.7
6.7
8.3
10.0
9.2
In T
0.0
30.0
56.0
55.0
63.5
51.1
10.0
20.0
18.5
18.0
26.0
20.6
Multi Taski-TaskPD
0.0
43.0
50.0
51.0
50.5
48.6
8.3
13.3
11.7
3.3
11.7
10.0
MAPD
0.0
42.5
53.0
57.0
60.5
53.3
15.0
13.3
13.3
1.7
10.0
9.6
TTA with FT <:30
NoMeta-task
0.0
21.5
67.5
89.0
68.0
26.7
23.3
46.7
58.3
38.8
Model-Avg"M
X
0.0
8.3
31.5
28.0
45.0
55.5
40.0
In m--Cotextt"
0.0
35.5
54.5
79.5
88.5
64.5
10.0
21.7
18.3
41.7
41.7
30.9
Multi- Tosk
0.0
37.0
73.5
93.5
94.5
74.6
8.3
31.0
28.3
61.0
60.0
45.1
MAPD
0.0
43.5
78.0
94.5
95.5
77.9
15.0
32.0
38.3
58.3
62.0
47.7
Meta Task
CLEVR
Induction
Tex1OCR
Methods
0-S
1-S
2-S
4-S
8-$
Avg
0-S
1-S
2-S
4-S
8-S
Avg
TTA with ICI.
x
0.0
8.0
10.5
23.0
30.5
18.0
20.0
4.5
9.5
8.5
4.5
6.8
Model-Avg
1.5
17.0
8.5
4.0
1.0
7.6
12.0
3.0
2.5
3.0
1.0
2.8
RD
0.0
13.5
23.0
28.5
24.1
16.0
22.5
21.0
23.5
28.0
23.8
Multi-Task
1.0
9.0
12.5
18.0
4.0
10.5
MAPD
2.0
11.0
7.0
15.5
15.5
12.3
21.5
7.0
TTA with FT <30
NoMeta-taskk
X
0.0
18.5
21.5
26.0
37.0
25.8
20.0
20.5
23.0
24.0
22.5
22.5
Model-Avg
X
1.5
26.5
25.0
29.5
35.5
29.1
12.0
17.5
20.0
23.0
25.5
21.5
In DD
0.5
24.5
30
34.5
34.5
30.9
16.0
16.0
18.0
19.5
22.0
18.9
Multi-Task
0.0
25.0
25.5
31.0
38.0
29.9
18.0
21.0
20.5
24.5
25.5
22.9
MAPD
0.0
26.5
27.5
31.0
40.5
31.4
21.5
23.5
26.5
27.0
28.5
26.4
Table 12: Comparison of the LoRA baselines on VL-ICL Bench (Zong et al., 2025). "Avg" is only
calculated for 2 1 shot(s) (-S). TTA= Test-Time Adaptation. FT=Finetuning with S 30 gradient steps. ATT=AAttenion-Maappaatta The model used for this evaluation is LLaVA-ATT-Qwent. 7B.
Open-MI (2-way)
Operator Induction
LoRA
0-S
1-S
2-S
4
5-S
Avg
0-S
1-S
2-S
4
8-S
Avg
TTA with FT FT'S 30
All LLM layers
0.0
24.5
45.7
68.3
81.9
55.1
8.1
11.7
10.0
13.3
18.2
13.3
[0-15] LLM layers
30.9
65.3
81.1
91.9
67.3
8.3
18.3
26.3
23.1
34.3
25.5
[0-15] LLM layers + ATT
0.0
37.3
64.1
83.5
91.5
10.0
28.3
35.5
36.7
30.5
CLEVR Count Induction
Tex1OR
LoRA
0-$
1-S
2-S
4-S
S-S
Avg
0-S
1-S
2-S
4-S
8-S
Avg
TTA with FT 30
All LLM layers
0.0
9.3
11.7
15.5
23.9
15.1
15.0
6.7
9.1
13.3
12.5
10.4
[0-15) LLM layers
0.0
21.5
28.3
32.5
37.7
30.0
18.3
20.3
24.5
25.5
24.9
23.8
[0-15] L.LM layers ATT
0.0
26.0
23.1
30.0
35.7
28.7
18.3
20.6
23.4
26.5
27.5
24.5
25

---

Published as a conference paper at ICLR 2026
A.2.5 PERFORMANCE PUBLICLY AVAILABLE LMMs ON VL-ICL BENCH
Table 13: Performance of different LMMs single-image tasks from VI.-CCL Bench. We report the "Agg" accuracy for different numbers shots - {1.2, 2, 4, 5, 8) with 95% binomial confidence intervals. FT Finetuning with < 30 gradient steps, ICL In- Context Learning, TTA= Test-Time Adaptation, CLIP-CLP-ViT-LLL--3-pp MLP=2-layer MLP, ATT=AAtention-Mappss Bold shows best performance and Underline MAPD's
VL-Duta=Vision-Languee
Data, --Q-TB-LLaVV--AT---weel22.
performance with LAQ-7B LMM.
Params trained
Methods
VL-Data
TTA
Open-MI
OP_IND
CLEVR
TextOC
LLaVA v1.5-7B
1.2M
7B
ICL.
12.4 + 0.4
5.4 + 0.5
10.9
1 0.1
4.4 0.3
LLaVA v1.5-7B
1.2M
7B
FT
38.4 + 0.7
11.4 + 0.6
16.9 +.2
15.6
0.6
LLaVA-Next-78
1.3M
7.06B
ICI.
34.4 + 0.7
5.4 0.5
21.1 0.2
0.4
LLaVA-Next-7B
1.3M
7.06B
FT
55.1
+ 0.9
13.4
0.2
7.8 0.4
10.4M
8B
ICL
42.1 0.9
41.7
0.2
42.3 0.5
LLaVA-One OneVnninn-
10.4M
8B
FT
1.7
46.1
38.9 0.2
45,5 0.5
LLaVA- One
10.4M
73.2B
ICL
75.1
0.6
69.1 0.9
37.2
52.2 111
Qwen2-VL-7B..n -L-T--Instuuttt
-NA-
8B
ICL
73.5
0.6
69.6 0.9
27.9
50.5 0.9
Qwen2.5- Dwen2.5-VL-7BB-iig -L-7-Intttt
-NA-
8B
ICL
44.0 0.9
8.2
0.2
36.9 0.7
Owen2.5-VL-7B-Immm
-NA-
8B
FT
85.6 0.7
89
0.2
41.1 0.5
LAQ-7B In DD-ConeexeDD
1.3M
24M
ICL
5.9
2.8
0.2
23.8 0.8
LAQ-7B + In "D
1.3M
FT
0.8
05
18.9 0.2
LAQ-7B MAPD LAQ-7B MAPD)
1.3M
24M
ICI.
53.3 0.9
9.5
0.1
7.3 0.4
1.3M
24M
FT:30
77.9 0.7
47.7
0.2
26.4 + 0.8
We report the performance of publicly available LMMs alongside best-performing architecture on the single-image tasks from VL-ICL Bench in Table 13. We provide this reference and note that direct comparison across LMMs is not straightforward, given their fundamental differences in architecture, scale, and training data.
Test-time fine-tuning of the MLP connector consistently improves over ICL for all public LMMs, supporting hypothesis that these models overwhelmed by image embeddings during ICI. Fine-tuning enables the connector to distil task-specific information into image embeddings before prompting the I.M,, thereby improving few-shot performance.
2. Our model with MAPD-based meta-learning and fine-tuning adaptation performs compa- rably to other publicly available LMMs and, notably, surpasses LLa LLaVA-One AiA-oneisioon-- ICL the Fast -nded MinilmageNet (Open-MI) task, as well ICL with its 7B counterpart (trained on substantially more data) and the stronger Qwen- VL models on other tasks.
3. Unlike other LMMs, LLaVA-ATT-Qwenn.... during training and uses significantly less vision-language data (1.3M examples) and fewer trainable parameters (24M), compared to LLaVA-One Vision which trains the full model on 10.4M examples. This highlights the data and parameter efficiency of MAPD, which achieves state-of-the-art performance Open-MI by fine-tuning only the attention-mapper for up to 30 gradient steps on the few-shot examples.
(LAQ-7B) does not fine-tune the LLM
4. For LMMs such as LLaVA-One Vision,, fine-tuning the attention attention-mappe mapper requires substantial compute (2 H200 GPUs) due their sarge-scale training mixture (10.4M vision-lanuuag examples) and high-dimensional vision encoder embeddings, exceeding our available re- sources. Similarly, fine-tuning data for the Qwen- VL models is not publicly available. Given these constraints, we are unable to conduct attention-mapper fine-tuning experiments on these architectures.
26

---

Published as a conference paper at ICLR 2026
A.2.6
QUALITATIVE RESULTS
predictions diflerent
2-way 1-shot Set
Cuery Example
gradien: sleps al lest-lime
Step This is photo yallow
Step 10: School Bus
Question: This a?
Question: This is a?
Question: This is a?
Step 23: Dax
Answer: Dax
Answer: Shously
Answer: Dax
Figure 12: OPEN MI predictions test-time
predictions difierent gradien: sleps test-lime
2-shol Suppoit Sel
Cuery Example
Step The material of objects image is rubber
Step 6: Gray sphere is rubber
Question: Shape: Cylinder
Question: Shape: Sphere
Question: Material: Rubber
Step 16:
Answer: 5
Answer: 5
Answer:
Figure 13: CLEVR predictions at test-time
LMM predictions for difierent gradien: steps test test-lime
2-shot Support Set
Cuery Example
Cheelss
Step 0: This is hanging
sign restaurant Willie Mae's 's Restaurant.
Segeee Game Box
Willie Mae's RESTRURANT
Step Restaurant
Question: What text
Question: What text shown in red box?
Question: What text
Step 25: Restaurant
Answer: Game
Answer: Cheetos
box?
Answer: Restaurant
Figure 14: TEXTOCR predictions at test-time
UMM predictions for different
shat
Query Example
Step S: The expression 8
will
??7
?
8 ?
two answer
Question: the the mathomatical expreesion?
Question:
Stap 3: The answer
the mathoraticaal expression?
Question: result the mathomatical expression? ? Answer: 32
Answer:
Answer:
Step 7: 82
Figure 15: Operator Induction predictions at test-time
27

---

Published as a conference paper at ICLR 2026
Table 14: Robustness of prompt distillation methods against image perturbations on the Fast Open- Ended MinilmageNet dataset (2-way classification) for LLaVA-ATI-Qwen2.5 7B LMM. We report accuracy scores as defined in VL-ICL Bench (Zong al., 2025) across 2, and shots. Test-Time Adaptation Finetuning with v 30 gradient steps.
NoMeta tD
Mvdel-Avge
In In-Contex
Multi-task
MAPD
2-8
5-$
2-S
5-S
2-S
5-S
2-S
5-S
2-S
5-S
Original
67.5
94.0
|
53.5
87.5
54.5
88.5
73.5
94.5
78.0
95.5
Cropping
65.0
94.0
51.5
87.5
51.5
83.0
72.0
91.5
76.5
95.0
Rotation
67.0
91.0
50.5
81.5
50.5
83.5
72.5
93.5
78.0
95.5
Gaussian Blur
67.5
92.5
51.5
84.5
49.5
78.0
71.5
92.5
77.5
96.0
Color Jitter
66.5
92.5
50.5
89.0
49.5
81.5
71.5
94.0
77.0
94.0
CutMix
58.5
86.0
45.5
70.5
49.0
75.0
72.0
92.0
75.5
92.5
Mi
58.0
84.0
46.0
70.5
48.0
75.5
69.0
89.0
76.5
91.0
Mcan Drop in Accuracy
-3.8
4.0
4.3
6.99
4.8
9.11
2.1
2244
1.2
-1.4
Net Mean Drop across Shots
3.9
5.6
7.0
2..
-1.3
Origical L Atrilanc ||
Oripinal
Altribt:
Mosin Accunccy (9)
-00
Mean Accuracy
24.
2132033
180
15-4
0
NoM: PD Mudk--Aug
In-Cuntca:
Muli-TTck
MAPD
McMosTTaa
Madd--aa
InCCoted
MAPD
Figure 16: (a) Performance comparison of different prompt distillation approaches on the CLEVR Count Induction (details in Appendix A.2.2). Few-shot examples for Same Attribute and Same Pair are selected based on their attribute-value similarity with the query (test) example. Mean Accuracy computed for 1,2,4 and 8 shots. Left: Finetuning (FT) based Test-time Adaptation. (b) Right: In-Context Learning (ICL) based Test-time Adaptation.
A.2.7 ROBUSTNESS AGAINST IMAGE PERTURBATIONS
We assess if our prompt distillation methods are robust enough to handle perturbations applied to the images in the support set as shown in Table 14. We see that our method, MAPD, is most robust even in the presence of noise in the support examples as compared to other distillation methods that suffer huge drop in performance. Advanced techniques like CutMix (Yun et al., 2019) and MixUp (Zhang et al., 2018) 8) change the original image distribution substantially. affecting all methods to a greater degree but MAPD is still close to its original performance for both and shots. This robustness likely stems from MAPD's 's meta-learned initialization, which learns the underlying task structure from meta-tasks without over- fittingt to any other spurious visual patterns and this allows to adapt quickly to newer tasks without being influenced by noisy visual artifacts in the examples.
A.2.8 How TO SELECT FEW-SHOT EXAMPLES FOR BETTER PERFORMANCE?
We further assess how performance varies for different prompt distillation approaches based on the selection of fhw-shot examples on the CLEVR Count Induction task (details in Appendix A.2.2) as an example. We propose two selection methods based on similarity of attributes and their corresponding values for every query (test) example. the query has attribute and value as shape: sphere, we select the few-shot examples based on a) Same Attribute shape, (b) Same Pair shape: sphere and compare both of them with the original setup as proposed in the VL-ICL benchmark (Zong et al., 2025) which retrieves the few-shot examples randomly. In Figure 16(a), we first see that for finetuning- based (FT) adaptation, the performance of all the baselines increases by 4.8% for Same Attribute and 5.3% for Same Pair on an average. MAPD performs best in the Same Attribute
28

---

Published as a conference paper at ICLR 2026
setting (Mean Acc 35,4%) and Multi-Task" performing best in the Same Pair setting (Mean Acc 35.8%). In Figure 16(b), we see that for In-Context Learning (ICL) adaptation, the similarity- -ased few-shot selection methods have greater impact in performance and improve the mean accuracy of all the baselines by 7.7% for Same Autribute and 8.6% for Same Pair on an average. In-Context performs the best in both Same Attribute and Same Pair settings with mean accuracies of 28.8% and 30.5% respectively for ICL adaptation. We also notice that the Same Pair setup is generally the best few-shot selection method giving best performance for all the approaches. This indicates that choosing few-shot examples that are similar to query example induces better task understanding during test-time adaptation. We also see that the selection of few-shot examples shows less variance with FT adaptation compared to ICL adaptation, thereby showing higher robustness of FT adaptation.
2.2.9 DETAILS ON ABLATION STUDY FOR OPERATOR INDUCTION
We break down the ablation study on operator induction tasks (Section 4.3; Figure 4(b)) into components: 1) Task Induction, 2) Perception, and 3) Mathematical Reasoning. We test these components separately with the help of suitable prompts for our LMM to answer questions in specific formats. Figure 17 shows our prompts used for different components.
Task Induction - "What mathematical operation should be used in this example? Strictly answer in word."
"What are the numbers in this example? Do not calculate the answer after applying mathematical operation. Only give the numbers shown in the example. Stricly give numbers in numeric digits and your result should be in the format Number A: xxx Number B: xxx."
Perception
Mathematical Reasoning- "Think step-by-step and give proper reasoning steps first and then given your final answer. The format should be > Reasoning: xxx Answer: xxx. The Reasoning part should contain reasons to derive the answer and the Answer part should only contain the answer. Your response should strictly follow this format and not just give the answer of the mathematical operation. It's important that you give reasoning before you answer."
Figure 17: (Operator Induction Task) Prompts to the LMM for generating answers in specific formats suited for evaluation.
We list out few examples which we curate for the Operator Induction task to enhance mathematical reasoning, Each image in the dataset contains a set of 2 numbers or operands and hidden mathemat- ical operation. The result of the correct mathematical operation is also provided for the support set examples. The task to induce the mathematical operation used in the support set to calculate the answer of the query image containing two new operands. As finetuning on single answer token limits the token generation capacity the LMM, we further modify the support set examples list out detailed mathematical steps before calculating the answer. Finetuning on this reasoning data improves both the generation capacity and reasoning ability of the LMM. We further provide a few examples of this hand-curated data in Figure 18. We used Swwen2.5VL-222 ----t----nttttt (Qwen et al., 2025) as a judge for evaluating the Mathematical Reasoning component of the problem where LMMs responded with detailed reasoning steps before the answer. Evaluation of responses was done by prompting the judge to score a response between 0-3 based if it thinks the reasoning and answer are correct. We then calculated mean score as the percentage of total score assigned by the Qwen-2.5-VL (Judge) to the responses relative to the maximum possible score.
ΣN S
Mean Percent Score =
X 100
(9)
3 x N
where St is the score assigned by Qwen2.5-VL. for the ith response and is the total number of responses. We provide the prompt the judge for this evaluation in Figure 19.
We also provide a few examples of LMM predictions for task induction (Figure 20) and perception (Figure 21) and mathematical reasoning (Example 1: Figure 22, 23 and Example 2: Figure 24, 25)
29

---

Published as a conference paper at ICLR 2026
Original Answer:
Detailed Answer: There numbers, 5 and 1. Performing some mathematical operation gives the answer 4. So if we think about adding the numbers, + subtracting them, 5- 1
5?1
4, multiplying them, 5 x 11=5 5. This implies that the hidden operation must be subtraction (-) and the result 4.
Original Answer: 21
Detailed Answer: There two numbers, 3 and 7. Performing some mathematical operation gives the answer 21. So if we think about multiplying the numbers, 3 x 7 21, adding the numbers, 3 + 10, subtracting the numbers, 3 -4. This implies that the hidden operation must be multiplication or and the result is 21.
3?7 7
Figure 18: (Operator Induction Math Reasoning) Few examples of our hand-curated data with mathematical reasoning steps.
Judge Prompt- "You are given a few in-context examples of a mathematical induction problem. The in-context examples each have an image with two numbers and '?' which is supposed to be some mathematical operation. You are given solution that gives the answer and the reasoning on how to calculate that answer using some mathematical operation applied on those two numbers in the image. The task is to induce the correct mathematical operation from the given examples, and use that operation to calculate the result of a query image with different numbers.
After this, you then given reference response. The candidate response is in format Reasoning: xxx Answer: xxx The reasoning part contains reasoning about how the candidate arrived the solution, and the Answer part contains their final answer. Your task is to judge if the reasoning and the answer of the candidate response are correct or after considering the in-context examples, query image, question, reference answer, and your own reasoning of the mathematical induction problem.
written by experts and candidate
The rating should be done on scale of 0-3, where O indicates when the response is ambiguous or does not follow the format, 1 is for when both the reasoning and answer are wrong, 2 for when either only reasoning answer is correct, is for when both the reasoning and answer are correct.
Be strict in your judgement and do not give a higher rating unless the candidate re- sponse contains accurate reasoning and exact answer. Thorougly check each and every part of the candidate response and make sure it does not contain extra irrelevant operations or answers. If does then give lower rating accordingly. The candidate response should follow the format and conclude with the correct answer. If it does not, that means their answer is wrong. Also give your rationale before rating. Give the final rating > Rating: xxx"
Figure 19: (Operator Induction Math Reasoning) Prompts for the S0-.2.5 to evaluate LMM responses scale of 0-3. is given 4 in-context examples for understanding the mathematical induction task before the LMM (candidate) response for better evaluation.
30

---

Published as a conference paper at ICLR 2026
Example (Task Induction): Support shot
7?6
Question: What is the result of the following mathematical expression? Answer: 42
Example (Task Induction): Support shot 2
Question: What the result of the following mathematical expression? Answer: 0
0?8
Example (Task Induction): Query
Question:
What mathematical operation should be used in this example? Strictly answer in word.
3?6
LMM prediction: Multiplication
Figure 20: (Operator Induction Task Induction) An example of 2-shot task induction for multiplica- tion operation
31

---

Published as a conference paper at ICLR 2026
Example (Perception): Support shot 1
2?2
Question: What the result of the following mathematical expression? Answer: 0
Example (Perception): Support shot 2
Question:
What is the result of the following mathematical expression?
2?4
Answer:
Example (Perception): Query
Question: What are the numbers in this example? Do not calculate the answer after applying mathematical operation. Only give the numbers shown in the example. Stricly give numbers in numeric digits and your result should be in the format Number A: xxx || Number B: xxx.
5?4
LMM prediction: Number A: Number B:
Figure 21: (Operator Induction Perception) An example of 2-shot perception task detect operands
32

---

Published as a conference paper at ICLR 2026
Example 1: Support
What the result of the following mathematical expression? There are two numbers, and 8. Performing some mathemati- cal operation gives the answer 8, So we think about subtracting the numbers, - 8 = -7, multiplying the numbers, 1 x 8 = 8, adding the This implies that the hidden operation must be multiplication or x and the result is 8
Question:
Answer:
1?8
numbers,
Example1: Query
Question: What is the result of the following mathematical expression? Think step-by-step and give proper reasoning steps first and then given your final answer. The format should be Reasoning: xxx || Answer: xxx The Reasoning part should contain reasons derive the answer and the Answer part should only contain the answer. Your response should stricly follow this format and not just give the answer of the mathematical operation. It's important that you give reasoning before you answer.
6?2 2
LMM prediction:
There are two numbers. 6 and 2. We need to find the result of an operation between them. The question asking for the result of the operation, which is 12. So the answer is Reasoning: The two numbers are and 2. The operation between them is multiplication. 6 2 = 12 Answer: 12
Figure 22: (Operator Induction Math Reasoning) An example of 1-shot mathematical reasoning task with Judge Rating: (shown in Figure 23)
33

---

Published as a conference paper at ICLR 2026
Example 1: Judge Response
Judge Response ### Analysis of the Problem
#### In- Context Example:
The in-context example provides the expression 'I 8 and states that the result is **g**, The solution explains that among the possible operations (subtraction, multiplication, addition), the operation that yields the result **g** is **mulliplicatiion ** x This establishes the pattern that the hidden operation is **muliplicaation**
#### Query Image:
The query image presents the expression ? 2'. Based the in-context example, we infer that the hidden operation is **multiplication*
#### Candidate Response:
The candidate response states:
**Rasooning **: The two numbers are 6 and 2. The operation between them is
multiplication. 6 x 12.
**Answer*s; 12
#### Reference Answer:
The reference answer is also #*1
### Evaluation the Candidate Response
1. Correct Identification of the Operation**; The candidate correctly identifies the operation as **multiplication ** based on the in-context example. This is consistent with the pattern established in the in-context example.
2. ** *Crrrett Application of the Operation **; The candidate applies the multiplication operation correctly: 6
3. ** Final Answer**; The candidate's 's final answer is ##12**, which matches the reference answer.
4. ** *Claritt of Reasoning**; The reasoning provided is clear and logically follows the pattern established in the in-context example. The candidate explicitly states the operation and shows the calculation step.
5. ** **Format Compliance* TTe candidate response follows the required format: "Rea- soning: xxx
Answer: xxx".
### Final Verdict
The candidate's response is both logically sound and mathematically correct. Both the reasoning and the final answer are accurate, and the response adheres to the required format.
* **atingg 3*
Figure 23: (Operator Induction Math Reasoning) The Judge (Qwen2.5-VL-32B) evaluates the response of the LMM in Figure 22 to provide correct rating.
34

---

numbers, 4 and 1. Per- Published as a conference paper at ICLR 2026
Example 2: Support
Question: What is the result the following mathematical expression? Answer: There are two numbers, 1 and 6. Performing some mathemati- cal operation gives the answer 6. So if we think about subtracting the
1?6
numbers, - 6 -5, multiplying the numbers, x 6 = 6, adding the numbers, 1 + 7. This implies that the hidden operation must be multiplication or and the result is 6
Example 2: Query
Question: What is the result of the following mathematical expression? Think step-by-step and give proper reasoning steps first and then given your final answer. The format should be > Reasoning: xxx || Answer: xxx The Reasoning part should contain reasons derive the answer and the Answer part should only contain the answer. Your response should stricly follow this format and not just give the answer of the mathematical operation. Its important that you give reasoning before you answer.
4?0
LMM prediction: Reasoning: There
forming some mathematical operation gives the answer 5. So if we think about adding the numbers,
5, subtracting the numbers, 4 3, multiplying the numbers, x 1 = 4. This implies that the hidden operation must be addition + and the result is 5. Answer:
Figure 24: (Operator Induction Math Reasoning) An example of 1-shot mathematical reasoning task with Judge Rating: (shown in Figure 25)
35

---

Published as a conference paper at ICLR 2026
Judge Response- #### Analysis of the Problem
#### In-Context Example:
The in-context example provides the expression ? 6' and the solution states that the result is **6**, The reasoning provided suggests testing different operations:
Subtraction: 1 66 -5;
Multiplication: x 6 = 6;
Addition: 1 + -
From these tests, the solution concludes that the hidden operation **multiplication **, as 6
#### Query Image:
The query image is '4 ? 0'. We need to determine the result of this expression based on the same hidden operation identified the in-context example.
#### Reference Answer:
The reference answer provided is **0**
#### Candidate Response:
The candidate's 's reasoning and answer are as follows:
** **Rasoning *s; The candidate incorrectly identifies the numbers as *44 and I **
instead of * and () They test different operations:
Addition: 4 5; Subtraction: 4 1 3; Multiplication: 4 X =
Conclusion: The hidden operation is **addition **, and the result is $25** **Answer**; The candidate concludes that the result is #w #n5w*.
### Evaluation of the Candidate Response
** **Correct Identification of the Hidden Operation **;
The in-context example clearly indicates that the hidden operation is **multi- plication ** (1 x 6 6).
However, the candidate incorrectly assumes the hidden operation is **adii- tion ** based on their misinterpretation of the numbers.
2. **Correct Interpretation of the Numbers** **: The query image shows the numbers #*4 and 0**, but the candidate mistakenly uses #*4 and ** their reasoning.
This error leads to incorrect calculations and conclusions.
3. ** **Correct Application of the Operation **;
Even if we overlook the misidentification of the numbers, the candidate' 's rea- soning flawed because they do not apply the correct hidden operation (multi- plication) as indicated by the in-context example.
4. 4. ** **Final *nswer**:
The candidate" final answer is **5**, which incorrect. The correct answer, based on the reference and the hidden operation of multiplication, should be **0 x 00)
### Rating Justification
The candidate's 's reasoning is **incorrect** because:
They misidentify the numbers in the query image.
2. They incorrectly deduce the hidden operation as addition instead of multiplication. The candidate's ''s final answer (**5**) is also **ong
Given these errors, the candidate's 's response does not align with the in-context example or the reference answer. Both the reasoning and the answer are incorrect.
### Final Rating: Rating:
Figure 25: (Operator Induction Math Reasoning) The Judge (Qwen2.5-VL-32B) evaluates the response of the LMM in Figure 24 to provide correct rating.
36

---

Published as a conference paper at ICLR 2026
A.2.10 ATTENTION ENTROPY ANALYSIS
We analyze attention patterns by extracting post-softmax attention scores of the underlying LLM for the LLaVA-ATT-Qwen2.55 7B model on the VL-ICL bench for different shot scenarios. For each generation step, we first re-normalize the extracted attention scores over soft prompt tokens (containing image information)
where P refers to soft prompts. We then calculate
Σ 2==1% Dj
the normalized Shannon entropy as,
H(Q) -i= qi log log(n)
to account for the difference in the number of soft prompts. We average this across all generation steps, LLM attention heads, layers, and examples across different VL-ICL datasets for MAPD with FT 30 and best In-Context baseline that uses ICL and report results in Table 15. We a more uniform distribution of attention scores for MAPD over the fixed set of soft prompts, i.c. 256 and hence the entropy remains consistent even on increasing the number of shots. On the other hand, for In-Context as we increase the number of shots, the number of soft prompts increase. For example, 8-shot + query, ICL requires 256 (8 + 1) = 2304 soft prompts. Given this relatively larger context, we see many soft prompt tokens containing very low attention scores leading to lower value of normalized entropy. This highlights an inherent limitation of the LMM that is unable to attend to all the soft prompts over longer context lengths.
Table 15: Normalized Attention Entropy VL-ICL Bench. TTA = Test-Time Adaptation. FT S 30 Finetuning with 30 gradient steps. ICI. In-Context Learning.
Training Methods
TTA
0-S
1-S
2-S
4-$
8-S
In-Context
ICL
0.84
0.75
0.63
0.51
0.45
MAPD
FT 30
0.84
0.81
0.84
0.80
0.81
A.2.11 SCALING TO MORE SHOTS
Here, look into the performance of different prompt distillation methods with finetuning based test time adaptation for larger number of shots and for 3 tasks from the VL-ICL datasets for LLaVA- ATT-Qwen2.5 7B LMM. We see similar performance gains with the introduction of more shots as shown in Table 11. Both the meta-task learning methods, Multi-Task P and MAPD perform quite well with MAPD showing outstanding performance for Operator Induction.
Table 16: Operator Induction Results.
Training Methods
Meta-task
16-$
32-S
64-S
Po
X
73.3
73.3
80.0
Model-Avg.PD
X
71.7
78.3
80.5
In-Context
V
58.3
53.3
76.7
Multi-TaskPP
73.3
67.7
80.0
MAPD
80.0
81.0
83.3
Table 17: CLEVR Count Induction Results.
Training Methods
Meta-task
16-S
32-S
64-S
PoMeta-task
35.5
30.0
36.5
Model-AvgPD
X
30.0
34.5
37.0
In- Context PD
/
25.5
34.5
32.5
Multi-TaakkPD
38.0
41.5
38.5
MAPD
40.0
40.5
41.0
37

---

Published as a conference paper at ICLR 2026
Table 18: TextOCR Results.
Training Methods
Meta-task
16-S
32-S
64-S
NoMeta-task
X
29.0
26.5
30.5
Model-Avg
X
29.0
29.5
31.5
In In-ContexttD
V
26.5
26.0
28.5
Multi-Tanne Task
27.0
32.5
33.5
MAPD
30.5
31.5
31.5
A.3 TEST- TIME COMPUTE ANALYSIS FOR ICL VS FT
D D DEPT
40
15
Time talcn secondo)
32.2
53
Moo Aposmny
30
00
1-Shot
2-Shot
4-Shux
5-Shot
10
2000
400
(10
800
TFLOPs
Figure 26: (a) Left: Computational time taken per test example (query) by ICL (blue) and FT (orange) for different number of shots. (b) Right: FLOOPs matched evaluation across all the VI.ICL test sets with mean accuracy for ICL (blue) and FT (orange).
Test-time finetuning (FT) for 30 gradient steps takes about twice as much inference time per test example (query) compared to in-context learning (ICL) under different few-shot scenarios as shown in Figure 26(a). This is not surprising as fine-tuning involves gradient computation, which is more expensive to run than single forward pass in ICL.
For more fair comparison, we examine the amount of computation required between these different test-time adaptation methods. Figure 26(b), shows FLOPs-matched evaluation curves for ICL and FT, using In- mn-Context and MAPD representative training methods, respectively. We report mean accuracy across (single-image) VL-ICL datasets. Test-time computation (TFLOPs) scales with the number of shots for ICL, while for FT, it can be scaled by increasing either number of shots or gradient steps. We note that given low test-time computational budget, ICL performs better than FT, but the amount of computation is increased FT outperforms ICL. This indicates that FT adaptation resource- resource-intensive but scales better than ICL as the amount of computation is increased at test time.
After 400 TFLOPs, In-Context performance begins to decline because the large number of shots used (232) exceeds its trained context length of 8,192 tokens. Training In-Context"D with longer context would require >4 H200 GPUs, which exceeds other hand, MAPD by design does not require training on long context lengths due to the use of fixed set of distilled soft prompts for all shots. Additionally, find that MAPD much more data-efficient: at 400 TFLOPs, it achieves comparable performance with only 8 shots and 20 gradient steps, indicating better fho- test-time adaptation.
available compute resources. On the
38


Reviews:
[{"review_id": "B1JqAKnZwF", "reviewer": "Reviewer_WEYe", "summary": "This paper proposes a meta learning based approach for adding few shot capabilities to LLMs. The paper gives a convincing argument why this is needed and also provide a comprehensive set of baseline methods to show how the new method compares to those. Claim is that through meta learning the method is not hampered by the problem of others methods that they cannot leverage the longer context with more images. There is an extensive set of experiments both in the paper and in the appendix. State-of-the-art results are obtained.", "questions": "Please see the two elements indicating for weaknesses and reflect on those.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is very clear and comprehensive. Experiments are well defined and serve a clear purpose.\n- New approach to the problem which gives clear improvement in the results. \n- Good ablation study to analyze specific aspects of the methods.", "weaknesses": "- The paper indicates that existing methods cannot leverage the benefits of having longer contexts. This is clearly demonstrated through experimentation. But some more theoretical and intuition basis for this could be given. \n- The method can be applied to any existing method but only shown for a limited number of models. It is also not made explicit what is required for a model to be able to apply your method. You are using prompts and not every model is using exactly the same type of prompts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iRr1l33DOX", "reviewer": "Reviewer_6iP1", "summary": "This paper proposes a test-time fine-tuning method to enhance the accuracy of MLLMs in visual question answering (VQA) compared to in-context learning. The proposed method introduces an attention-based mapping network on top of visual tokens that utilizes soft prompt tokens to distill visual information into soft prompts that are then provided to the LLM. The mapping network is pretrained, then fine-tuned on meta-tasks using meta-learning techniques, and finally is fine-tuned for each task during test-time for few-shot VQA. The paper also constructs several alternative methods for fine-tuning the proposed mapping network for comparison with the meta-learning technique. The results show that the proposed method can outperform in-context learning in few-shot VQA.", "questions": "1. Can you provide more details about the number of meta-tasks, their composition, and why they differ between different methods?\n\n2. Can you provide a baseline of fine-tuning the mapping network of LLaVA-OV, Qwen-VL?\n\n3. Can you provide confidence intervals for the results?\n\n4. Can you report time overhead of your method compared to ICL?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed method is novel, and shows promising improvements on few-shot VQA. The experiments also provide several interesting insights about the role of attention mapper, soft prompts, and various fine-tuning strategies.", "weaknesses": "1. The description of how meta tasks are constructed lacks clarity. The paper must provide a clear table describing the number and composition of meta tasks used for fine-tuning and test-time fine-tuning. It is therefore hard to understand how the test accuracies are computed and can be compared.\n\n2. Looking at the Appendix, it seems that different baselines are fine-tuned with different composition of meta-tasks (Appendix A.1.3): e.g., MAPD has 10-10 (support-query), whereas Multi-Task has 5-5, and In-Context has 10-1. No explanation is provided for these choices, and how they might affect the reported results.\n\n3. The choice of bounding the number of test-time fine-tuning iterations (K<=30) seems arbitrary and makes the results of comparisons unreliable. For example, without any other information, it is possible that at K=40, which may take only a few more minutes, another method outperforms MAPD or the ranking of the methods changes completely. Comparing how long it takes each method to reach a certain accuracy level would be a more clear comparison.\n\n4. The paper seems to miss the simple baseline of test-time fine-tuning just the connector (mapper MLP) of other open-source SOTA MLLMs (LLaVA-OV and Qwen-VL). This is critical to show whether the proposed adaptor matters in practice.\n\n5. The results in Table 9 of Appendix show that ICL is still the SOTA on 3 out of the 4 tasks in VL-ICL when used with other MLLMs (LLaVA-OV and Qwen-VL). These results show that the best current method for few-shot VQA is still performing ICL with SOTA MLLMs. This limits the practical relevance of the proposed method.\n\n6. All the reported results lack confidence intervals, so it is unclear how statistically significant the differences are. I suggest reporting binomial confidence intervals for accuracy to clarify this.\n\n7. The paper seem to not report the time and computation over-head of its method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zznXWyJbIF", "reviewer": "Reviewer_pV3j", "summary": "The paper presents MAPD, a meta-learning framework that enables few-shot adaptation in large multimodal models (LMMs) for visual question answering (VQA) by distilling task-relevant visual features into a fixed set of soft prompts via an attention-mapper integrated into the projection layer. Unlike in-context learning (ICL), which can degrade in performance with more examples in smaller LMMs due to irrelevant image token noise, MAPD leverages first-order MAML to train the attention-mapper and prompts on meta-tasks constructed from support/query splits. At test time, it adapts to new VQA tasks with a few gradient steps on the support set. Evaluated on the VL-ICL Bench, MAPD consistently outperforms ICL and parameter-efficient fine-tuning methods (e.g., LoRA, TPT), with performance improving monotonically as the number of shots increases.", "questions": "Please refer to weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Novel and effective integration of meta-learning with multimodal prompt distillation: MAPD is the first approach to apply MAML-style bi-level optimization to distill visual features into soft prompts for LMMs.  Its lightweight attention-mapper (~24M trainable parameters) enables efficient task adaptation, achieving state-of-the-art few-shot VQA performance across diverse VL-ICL tasks, while scaling reliably with support set size.\n2. Rigorous experimental design with strong baselines and reproducibility: The authors provide comprehensive ablations (Multi-TaskPD, In-ContextPD, NoMeta-taskPD, Model-AvgPD) and leverage public code, datasets, and a standardized benchmark.  The observed monotonic improvement with increasing shots, coupled with substantial gains over ICL and PEFT methods, offers compelling evidence of MAPD’s practical superiority in low-data regimes.", "weaknesses": "1. The paper attributes the non-monotonic improvement in ICL performance of small-parameter LMMs to \"irrelevant information interference in image embedding,\" but it does not rule out other contributing factors, such as the inherent limitations of small models or deviations in instruction understanding.\n2. While much prior work has focused on optimizing the projection layer of LMMs, the advantages of this paper relative to existing methods appear limited. his study focuses on scenarios involving small models, single images and limited samples. Does this narrow scope limit its applicability? Additionally, the paper claims that the approach can be \"easily incorporated into the projection layer of any LMM architecture,\" but it remains unclear whether it is effective for other architectures, such as variants of models like Qwen3-VL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p8grb7AAyb", "reviewer": "Reviewer_XpJA", "summary": "The authors propose a meta-learning–based approach for few-shot adaptation in Large Multimodal Models (LMMs). They observe that training-free in-context learning often yields inconsistent performance in smaller LMMs and fails to scale reliably with more shots. To address this, they introduce learnable soft prompts distilled from image features using a first-order meta-learning framework. Additionally, they replace the standard shallow visual–language projector with a multi-layer self-attention “attention-mapper” module, enabling richer interactions between visual embeddings and prompt tokens before projection into the language space.", "questions": "I am looking forward to the authors addressing the weaknesses. In particular, I would appreciate them answering the following questions:\n\n1. How does the proposed attention-mapper differ from the adapter module introduced by Najdenkoska et al. (2023)?\n2. How does the meta-learning–based adaptation affect latency or response time of the LMMs compared to standard training-free in-context learning?\n3. Does the proposed approach scale to larger LMMs (e.g., 13B or 34B), and do the performance gains persist at that scale?\n4. How does adding more soft embeddings (via learnable prompts) help reduce the issue of overwhelming image information in smaller LMMs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and clearly explained. It proposes a parameter-efficient approach to fine-tuning Large Multimodal Models (LMMs) using simple yet effective ideas such as learnable soft prompts and meta-learning. The work is timely and highlights an important limitation of standard in-context learning, namely its inconsistent scaling with increasing shots. The proposed method shows strong empirical performance, achieving substantial improvements over standard in-context learning and other baselines on few-shot VQA tasks with LMMs of up to 7B parameters.", "weaknesses": "1. Limited technical novelty: The paper borrows heavily from existing approaches. The use of soft or learnable prompts is already well established in few-shot learning and LLM adaptation [1, 2, 3]. Moreover, the meta-learning framework and the attention-mapper module are directly borrowed from Antoniou et al. (2019) and Najdenkoska et al. (2023), respectively.\n\n- [1] Hou et al., MetaPrompting: Learning to Learn Better Prompts, COLING 2022. \n\n- [2] Wang et al., Towards Unified Prompt Tuning for Few-shot Text Classification, EMNLP Findings 2022.\n\n- [3] Khattak et al., Self-Regulating Prompts: Foundational Model Adaptation Without Forgetting, ICCV 2023.\n\n2. Latency and efficiency: Although the method demonstrates strong few-shot performance, one of the main advantages of training-free in-context learning is its low latency. The paper does not provide any analysis or discussion of the computational cost or response-time impact of meta-learning-based adaptation.\n\n3. Insufficient explanation of motivation: The authors claim that additional soft prompts mitigate the issue of overwhelming image embeddings in smaller LMMs, but this is not analyzed beyond aggregate performance results. A visualization of attention patterns could have helped verify whether the prompts actually improve focus. Moreover, it seems counterintuitive that adding more embeddings (via soft learnable prompts) and information would resolve a capacity limitation.\n\n4. Limited scalability evaluation: While the method is tested on multiple LMMs, all models are limited to 7B parameters. It remains unclear how well the approach scales to larger backbones (e.g., 13B or 34B) or whether the benefits persist at that scale.\n\n5. Potential over-claiming in contributions: The contributions section slightly overstates novelty. MAPD largely reuses the first-order meta-learning framework of Antoniou et al. (2019), applied to prompt adapters. Similarly, the authors did not clearly distinguish their flexible adapter design from that of Najdenkoska et al. (2023). Although the authors claim to have proposed the mapper (inspired by Najdenkoska et al.), it appears to be essentially the same module.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)