ContextNav: Towards Agentic Multimodal In-Context Learning
Authors: Honghao Fu
Yuan Ouyang
Kai-Wei Chang
Yiwei Wang
Zi Huang
Yujun Cai
Venue: ICLR 2026 Poster (2026)
Keywords: agent system
in-context learning
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Recent advances demonstrate that multimodal large language models (MLLMs) exhibit strong multimodal in-context learning (ICL) capabilities, enabling them to adapt to novel vision-language tasks from a few contextual examples. However, existing ICL approaches face challenges in reconciling generalization with robustness across diverse tasks and noisy contextual examples: manually selecting examples produces clean contexts but is labor-intensive and task-specific, while similarity-based retrieval improves scalability but could introduce irrelevant or structurally inconsistent samples that degrade ICL performance. To address these limitations, we propose ContextNav, the first agentic framework that integrates the scalability of automated retrieval with the quality and adaptiveness of human-like curation, enabling noise-robust and dynamically optimized contextualization for multimodal ICL. ContextNav unifies context management and noise-robust contextualization within a closed-loop workflow driven by graph-based tool orchestration. Specifically, it builds a resource-aware multimodal embedding pipeline, maintains a retrievable vector database, and applies agentic retrieval and structural alignment to construct noise-resilient contexts. An Operational Grammar Graph (OGG) further supports adaptive toolchain planning and optimization, enabling the agent to refine its strategies based on downstream feedback. Experimental results demonstrate that ContextNav achieves state-of-the-art performance across various datasets, underscoring the promise of agentic workflows for advancing scalable and robust contextualization in multimodal ICL.

CONTEXTNAV: TOWARDS AGENTIC MULTIMODAL IN- CONTEXT LEARNING
Honghao Fu', Yuan Ouyang", Kai-Wei Chang", Yiwei Wang", Zi Huang', Yujun Cai University of Queensland, University of Califoria, Merced, 3 Ant Group Uniiveriity of California, Los Angeles, 'Nanjing University honghao. fuêuq. edu. au, vanora. caiyj@gmail.com https://conteextnaypggg.. gi thub. io/
ABSTRACT
Recent advances demonstrate that multimodal large language models (MLLMs) exhibit strong multimodal in-context learning (ICL) capabilities, enabling them to adapt novel vision-language tasks from few contextual examples. However, existing ICL approaches face challenges in reconciling scalability with robustness across diverse tasks and noisy contextual examples: manually selecting examples produces clean contexts but is labor-intensive and task-specific, while similarity- based retrieval improves scalability but could introduce irrelevant or structurally inconsistent samples that degrade ICL. performance. To address these limitations, we propose ContextNav, the first agentic framework that integrates the scalability of automated retrieval with the quality and adaptiveness of human-like curation, enabling noise-robust and dynamically optimized contextualization for multimodal ICL. ContextNav unifies context management and denoising within closed- workflow driven by graph- based orchestration. Specifically, it builds a resource- aware multimodal embedding pipeline, maintains retrievable vector database, and applies agentic retrieval and structural alignment to construct noise-resilient con- texts. An Operational Grammar Graph (OGG) further supports adaptive workflow planning and optimization, enabling the agent to refine its operational strategies based on downstream ICL feedback. Experimental results demonstrate that Con- textNav achieves state-of- state-of-the-ar performance across various datasets, underscoring the promise of agentic workflows for advancing scalable and robust contextualiza- tion in multimodal ICL.
INTRODUCTION
In-context learning (ICL) has emerged as fundamental capability of large language models, enabling adaptation to novel tasks through contextual demonstrations without parameter updates (Baldassini et al. 2024). By conditioning on task instructions and examples presented within the input context, ICL allows models to perform zero- or few-shot generalization without relying on gradient- based fine-tuning (Brown et al. 2020). This paradigm has been successfully extended to multimodal domains, where models leverage both textual and visual examples to generalize across vision- language tasks (Huang et al. 2025 Zhao et al. 2023 Li et al., 2024c Doveh al.. 2024).
Existing multimodal ICL methods can be broadly divided into two categories: Manual ICL, where examples are manually selected and organized into contexts (Zhang al. [2023] [Sheng al., 2024] Doveh et al.) 2024). and Retrieval- based ICI., which employs feature embeddings to retrieve candidate examples as contexts (Gao al., 2023) [uo et al. 2024, Chen et al., 2025a). While both approaches have shown promising results, they face notable challenges. Manual ICL often yields highly relevant and well-structured contexts but relies heavily on human curation, making it labor-intensive and difficult to generalize across large-scale multimodal corpora. Retrieval-bar based ICL alleviates this burden through automation; however, may also retrieve semantically irrelevant samples and samples with inconsistent interrogative, imperative, narrative structures, both of which degrade
"Corresponding Author

---

Input Query
Retrieved
ICL)
Contextual Candidates
Mue-ttoreed buiksing
Simtiatty- based, Recrieval
wef Contents
MLLM
Arstwer the Query,
*Left Cantral"
Right)
ICL with Retrieved Candidates
tuildingg adja. cantto the...
Shot
Semantic Nolse
Netty-inclusive Conterts
MILM
Ansiwar the Quary,
"Upper
(Momatchee Somertic)
Shot Structural Noise
How many red cars there in bottoon-righ-..
ICL with Agentic Contextualization
The query about sensing, Shot painted maps, masches perfectly... To get clean contaxts, fitter Shot 1, refine Shot and keep Shot 8 as-is...
/lttorrggttvv Vanotion)
differently,
Where is the buildiing with
Shot
Ideal Candidate
Clean
MLLM
Anewer the Query
"Bottom Right"
(Wall Algned
Contects
Agant
Figure 1: Motivation for introducing agentic contextualization in multimodal ICL. Similarity based retrieval introduce semantic structural noise into contextual candidates. which degrades ICL effectiveness. Employing agent for human-like curation could effectively alleviate this challenge. We provide further discussion the negative effects such noise Section grounded quantitative results.
downstream ICL performance. As illustrated in Figure such noisy contexts can mislead models into producing incorrect answers.
Moreover, retrieval remains static and rule-based one-shot process. Unlike human reasoning that adaptively refines example selection based on observed effectiveness, current systems cannot adapt their contextualization strategies learn from experience. This creates fundamental challenge: developing contextualization approaches that combine the scalability automated retrieval with the quality and adaptiveness of human-like curation.
To address these challenges, we introduce ContextNav, an agentic framework for contextualization in multimodal ICL, as shown in Figure 2 Unlike previous methods, ContextNav formulates contextual- ization as an adaptive, tool-driven, and automated workflow that systematically manages multimodal context corpora, filters out noisy examples, reorganizes retrieved candidates in human-like manner, and optimizes its tool orchestration strategies based on downstream ICL feedback. Specifically, ContextNav leverages the multimodal reasoning capability of the MILMM policy to build and maintain resource- aware embedding pipeline with a retrievable vector database, and applies agentic retrieval and structural alignment to construct noise-resilient contexts. On top of these core components, an Operational Grammar Graph (OGG) provides a graph-structured action space that explicitly encodes the available operations along with their execution paths and dependency constraints. This structured action space allows the agent to plan and optimize its workflow adaptively: it can refine its operational strategy over multiple timesteps, continuously adjusting its choices based on downstream ICI. feedback. Collectively, these components constitute self-optimized elosed-loop system that not only retrieves and manages contexts, but also adaptively organizes them, thereby enabling automated and robust multimodal ICL. ContextNav eliminates manual contextualization, achieving greater scalability than manual methods, while its human-like curation confers stronger noise robustness than retrieval- based approaches. To the best of our knowledge, ContextNav is the first agentic framework for contextualization in multimodal ICL. Our contributions are summarized as follows:
We propose ContextNav. the first agentic framework that formulates multimodal contex- tualization as an adaptive, tool-driven, and fully automated agentic workflow, supporting scalable and denoisable contextualization for multimodal ICI.
ContextNav transcends static one-shot retrieval by integrating the Operational Grammar Graph with a memory module that couples historical workflows and downstream ICL feedback, enabling adaptive optimization of workflow orchestration across timesteps and enhancing adaptability and robustness of contextualization.
Through extensive experiments diverse datasets, ContextNav achieves average ICL gain of 16.8% across models, surpassing the prior state-of-the-art (7.6%) and underscoring the promise of agentic workflows for multimodal ICI.
2

---

2 RELATED WORKS
In-Context Learning (ICL). The concept ICL was popularized by GPT-3 (Brwwn et al. 2020) and
has since naturally emerged as a prominent paradigm in natural language processing (NLP) (Dong
et al.
2022). A growing body of work has sought understand the underlying mechanisms of ICL in
LLM.
primarily attributing this capability to implicit gradient descent (Dai et al. 2023a Von Oswald
al.
2023 Chen et al. 2024) and Bayesian modeling frameworks (Xie al.
2021 Arora al.
2024;
Wang et al. 2023; Falck et al. (2024). Concurrently, numerous studies have also focused on
enhancing the ICL capabilities LLMs by improving their inference frameworks (Chowdhery et al.
2023; Li et al. 2024a; Yang et al., 2024), training strategies (Sinha et al. 2024, Wu et al. 2025b),
and contextualization methods (anng et al. [2033] Wang et al. 2024a| Liu et al., (023b), enabling
more effective ICL on downstream tasks (Mao et al.] (025). These efforts have advanced both the
theoretical understanding and practical application of ICL.
Multimodal ICL, The success of ICI. in language models has spurred growing research interest in extending this paradigm to multimodal domains (Qin et al.) 2024). Early MI.LMs, such as Flamingo (Alayrac al. 2022), InstructBLIP (Dai et al. 2023b), and LLaVA (Liu et al., (023a)) have demonstrated the potential of multimodal ICL. More recent studies further advance this capa- bility by incorporating multimodal chain-of- reasoning (Zhang et al.] (023), vision expert models (Sheng et al. 2024). reature-based retrieval (Gao et al., 2023 Li et al., 2024bb Liu et al.)
2023c Tai et al. 2024). in-context tuning (Chen al.) (023), representation engineering (Zhao et al. 2023 Li et al. 2025a Huang et al., 2024). and attention editing (Li et al. (025b)). These techniques have collectively enhanced the zero- and few-shot ICL performance of VLMs across both specialized (Huang et al., 2025; Peng et al. 2024) and general vision- vision-languagg tasks (Chen al., 20255). Despite these advances, current approaches still face challenges from contextual noise. In this paper, we address contextual noise by proposing an agentic workflow that combines the scalability of automated retrieval with the quality and adaptability of human-like curation, thereby enabling noise-robust and dynamically optimized contextualization.
3 CONTEXTNAV
3.1 OVERVIEW
We propose Contex(Nav, an agentic framework designed to advance the multimodal ICL performance of downstream MILMs. Given multimodal query as input, ContextNav establishes an end-to-end agentic pipeline that autonomously transforms raw corpora into well-formed, query-relevant contexts. As illustrated in Figure the framework unfolds in three synergistic modules. First, the Agentic Context Management module constitutes the entry point. The agent performs resource-aware multimodal embedding, builds an evolving vector database, and retrieves a group of initial candidates from it given input query. Second, the resulting candidates are feed into the Context Denoising module, where semantically and structurally noisy candidates are pruned or reorganized to yield cleaner contexts. In parallel, the Graph-driven Workflow Orchestration module oversees and coordinates these processes, ensuring that embedding, retrieval, and denoising operations form valid and optimized operation sequences. Collectively, these components establish an automated workflow for representing, managing, retrieving, and refining multimodal contexts, thereby supporting scalable and
multimodal ICL for downstream MLLMs.
3.2 AGENTIC CONTEXT MANAGEMENT
Resource-Awal Multimodal Embedding. Embedding multimodal corpora is prerequisite for building vector databases and enabling effective retrieval, while faces several challenges. Large- scale embedding incurs heavy computational and storage costs, often becoming system bottleneck. Embedding models also vary in accuracy and efficiency, creating trade-offs between fidelity and resource use, shown in the experimental results of Table 3 Moreover, resource usage preference differs across users, requiring adaptive allocation. These factors motivate a resource-aware design that balances performance with efficiency in demand-driven manner. To this end, ContextNav formulates multimodal embedding as an agent-driven process. The embedding-speecfficttte specification prompt Pemb (detailed in Appendix F and Appendix H) encodes the user's resource usage preferences, the
3

---

(A) Agentic Context Managament
Graph-driven Workflow Orchestration
Recoume Anaro
Sindariy ie Boniooal
Workflow Orchestration Prompt
Darabone &
Racriaual Tools
Adaptive Workfoow Optimization
Contect Corpus
Muttimodal
Vector Database
Initial Candidates
Operation
Alunt
Openatinal (000)
Marred
(b) Context Danolsing
M, - (1% dilli
Agentio Retrioval
Strucurral AligneralAliggment
with Mlemary OGG
Initial
Denoised Contexts
Candidates
Operdion
Godden
Gemartt nodues
Semontically
Save
Attrrviitionn
Muttimodal Query
Oustrel
Orohestrate Workflow
M. Toxt Modil
Tool
Internal Reasonnnng
Dis Disdibution
ICL Result
Downstream MILMM
Invocation
Figure 2: Framework of the ContextNav. The proposed agentic framework integrates three synergistic modules: (a) Agentic Context Management, which performs resource-aware multimodal embedding, maintains a continuously updated vector database, and subsequently leverages it for similarity- based retrieval to generate the initial candidate pool given an input query: (b) Context Denoising, which refines retrieved candidates through agentic retrieval and structural alignment mitigate both semantic and structural noise; and (c) Graph-driven Workflow Orchestration, where the agent leverages Operational Grammar Graph and memory module adaptively plan and optimize operation sequence, therehy controlling the workflow. Collectively, these modules enable systematic management, representation, retrieval, and organization of multimodal contexts, supporting noise-robust and dynamically optimized contextualization for multimodal ICL.
current hardware state, and the embedding model zoo. Based on this prompt, the agent's 's MLLM policy my employs its internal reasoning capability to perform resource-aware hardware-model matching, determining an appropriate pair of embedding models (Er, Ev) ) Te (Peemb)), where Er, Ey denote textual and visual embedding models, respectively. Let the context corpus be denoted as C C ( [,) where T; is the i-th text instance and I; is its paired image. The agent leverages database and embedding tools to instantiate the embedding process at timestep т, which can be expressed as:
& = (E)) (I) (T,
(1)
where & denotes the multimodal embedding set constructed from the context corpus C,. In parallel, the agent employs database tools to continuously monitor the context corpus to identify newly added modified samples AC++1 that have not been vectorized, thereby triggering on-demand embedding pipeline for these samples in the subsequent timestep and yielding the corresponding embedding set AE This ensures the database remains up-to-date.
Vector Database Management. Following the generation of embeddings, the agent un- dertakes the construction and adaptive management of a multimodal vector database D {(T,, ei) (Ti, I,) C, e E &.. Contexts and their representations are systematically ingested, indexed, and archived, thereby establishing a structured and readily retrievable database. To ac- commodate corpus dynamics, the agent maintains an adaptive update mechanism with database
tools:
Dr+1 D. (T,)),,))) (T AC++1, e, E AE+11)
This agentic orchestration transforms the database from a static collection into an evolving knowledge structure that accurately reflects the state of the underlying context corpus.
Initial Candidate Pool. Once the vector database is constructed, the agent instantiates Top-k similarity- retrieval function fr, which is an adaptive composition of heterogeneous multimodal retrieval tools (textual, visual, or their cascaded combination). Given budget k. the function yields the initial candidate pool:
Rinst = f-(g. D,, k),
where = (qt> q denotes a multimodal query composed of the textual query q. and its paired visual query qo. The generated initial candidate pool is subsequently used for contextualization.
4

---

3.3 CONTEXT DENOISING
Within ContextNav, agentic retrieval and structural alignment are employed to attenuate semantic and structural noise from the initial candidates, respectively, with both processes grounded in the internal reasoning of the MILM policy. Semantic noise refers to candidates whose content is off-topic or contradictory to the query intent, whereas structural noise arises when a candidate's interrogative. imperative or narrative structure diverges from that of the query.
Agentic Retrieval. As discussed Section I initial candidates generated via similarity-based retrieval may contain both semantic and structural noise, which can undermine the downstream ICL performance. To mitigate the semantic noise introduced by weakly aligned matches, the agent subsequently applies second-stage filtering beyond raw similarity search. Specifically, the coherence- specification prompt Pcoh (detailed the Appendix F encodes explicit instructions for semantic assessment, such verifying topical consistency between query and candidate, and discarding candidates that contradict or distract from the query intent. Conditioned on this prompt, the MLLM policy To leverages its internal reasoning capacity to evaluate each multimodal candidate in R and decide whether to retain it. The process at timestep T can be expressed as:
R π Te Pooh: Rinit).
Through this agentic retrieval mechanism, the initial candidates are effectively denoised, ensuring the preservation only those contexts that exhibit strong semantic alignment with the query.
Structural Alignment. Contextual candidates may exhibit variability in interrogative, imperative or narrative structure. Such heterogeneity introduces structural noise into the context, which prior work has extensively demonstrated hinder consistent reasoning in ICL and lead to performance degradation (Voronov et al.) 2024 Zhou et al. 2022] Zhao et al.) 2021). To mitigate this, the agent applies a structure-refnement step that harmonizes the form of retrieved candidates with the query. Specifically, the structure-aligmment prompt Pstr (detailed in the Appendix (F) encodes explicit instructions for reorganizing, ensuring that the textual flow mirrors that of the input query qt- Conditioned on this prompt, the MLLM policy leverages its internal reasoning capability to edit candidates with divergent textual structures into a form consistent with qt. thereby reducing structural discrepancies. The process at timestep can be expressed as:
nalin
Tee (e, Pars R
(5)
This process aligns the textual structures of the candidates with that of the query, reducing distribu- tional bias and structural noise. By combining semantic denoising and structural alignment, the agent yields noise- poise-minimized context set that enhances the robustness of contextualization.
3.4 GRAPH-DRIVEN WORKFLOW ORCHESTRATION
Operational Grammar Graph (OGG). the proposed framework, the operations (tool invocation or internal reasoning) that the agent perform for context management and contextualization governed by strict dependency relations and compositional constraints, since the underlying data structures evolve progressively within the workflow, with each stage reshaping the results of the previous one. Naive concatenation or heuristic composition risks redundant or invalid executions. Recent studies (Langchain-as 2024 Zhang al. 2025a) demonstrate that graph structures effectively capture operational dependencies and control flows while enabling flexible multi-step workflows. Building on this insight, we construct a directed graph G = EV,E), termed OGG, where denotes the set of atomic operations and È encodes valid execution dependencies, thereby formalizing the grammar of permissible operations (detailed in the Appendix G.. Specifically, the agent can instantiate a workflow that follows the operation sequence S = (t) vi + - Um ). where each transition satisfies (v,, 11+1 E ε, thus ensuring that the modules are executed in valid order.
Adaptive Workflow Optimization. The agent plans and orchestrates workflows by exploiting the prior dependency structures encoded within the OGG G. However, under the constraint of one-shot planning without intermediate feedback, the instantiated workflow may be suboptimal, potentially introducing inaccurate contextual candidates. This can reduce the number of valid contexts, shorten the effective context length, and ultimately degrade ICL performance, illustrated in Figure 4 To address this, the agent adopts an adaptive optimization mechanism that leverages correlations between past workflow configurations and their downstream ICI. performance stored in the memory
5

---

... Conditioned on workflow orchestration prompt Pwop (detailed in Appendix F instantiation at timestep T is modeled as the operation sequence:
workflow
S = no( (P woops M,. 9)
(6)
where Pwop specifies the workflow requirements at the initial timestep in the absence of downstream
ICL feedback, and further declares the optimization logic governing subsequent timesteps. Notably, for operations that involve internal reasoning, the associated prompts (e.g., Pemh. Pcoh. Par) embedded internally within the operation itself rather than being directly specified Pwop. This iterative optimization design enables the agent not only to leverage the OGG to enforce execution validity but also to adaptively refine orchestration strategies across multiple timesteps, thereby integrating context management and denoising into a coherent and adaptive pipeline.
3.5 AGENTIC MULTIMODAL. IN-CONTEXT LEARNING
At each timestep т. the agent constructs a minimiedd context set Ra which is concatenated with the multimodal query q and fed into the downstream MLLM Q. The execution of in-context learning is guided by the prompt Pur (detailed in the Appendix [F). The model then produces both the final prediction yr for the input query and an auxiliary textual feedback ør that reflects the perceived quality of the constructed context. Formally, this process can be expressed as:
(g+: Φ-)
4 (R q, Pia);
The feedback o, provides an immediate assessment of context adequacy from the perspective of the MLLM performing ICL, thereby assisting the agent in optimizing its toolchain selection strategy. Specifically, the agent updates its memory M by recording the association between the executed operation sequence S, and the resulting feedback dx:
Mr+1 M, {(S,, Φ>)} ((S,, de
This continual feedback update closes the loop between multimodal ICI. and adaptive toolchain optimization, enabling the agent to progressively refine its planning strategy to select toolchains that provide stronger contextual support and thereby enhance the robustness of multimodal ICL.
EXPERIMENT
4.1 DATASET AND IMPLEMENTATION
Dataset. We first conduct a difficulty annotation of query samples from recent composite-task datasets and benchmarks, including BlindTestt (Rahmanzaadehgervi et al.) (024), MME-RealWorld (Zaagg et al. 2025c), CharXiv (Wang al. 2024c), GVL (Wei et al. (024), and MathVision (Wang et al. 2024b). The annotation is determined by the accuracy of the models under evaluation: more than half of the tested models answer a given query correctly, it is labeled easy; otherwise, it is labeled as hard. Following a 3:7 sampling ratio (Clark et al., 2018) between easy and hard queries, we randomly sampled 803, 130, 100, 120, and 120 test instances from these datasets, respectively. The specificity of the BlindTestt sampling size comes from its structure of seven sub-benches. To ensure that each sub-bench was sufficiently represented while still preserving the dataset's s overall distribution, we sampled 10% from each. The remaining samples were used as support data for ICL. Additionally, we incorporated single-task datasets such as CLEVR (Johnson et al., 2017), FOMI (Vinyals et al. 2016), and Tex1OCR (Singh et al. [20T]) adopting the test/support splits specified Bench (Zong al. 2025). Collectively, these datasets cover a broad spectrum of visual reasoning tasks lasks-including abstract geometry, real-world scenes, charts, graphs, mathematics, spatial relations, counting, attributes and text recognition.
Implementation. We adopt (iemini-2.0-flar (GTeam et al., 2024) as the default MILM policy for ContextNav. The open-source models involved our experiments, including Phi-3.5V (Abdin et al.) 2024), InternLMX22.5 (Cai et al., 2024), Qwen2.5-VL (Bai et al.) 2025) and embedding models are deployed on an A000 80G GPU, while closed-source models, such as the Gemini series (Team et al. 2023) and GPT-4o (Hurst et al. 2024), on our experimental platform, ContextNav adopts wwven3-Embedding-4 (Zhang et al.) (0255) as the textual embedding backbone and the vision encoder of Qwen2.5-VI. (Bai et al.] 2025) as the
accessed via APIs on CPU servers. In addition, based

---

Table Comparison downstream accuracies with other baseline methods, greater values indicate better performance. The bold numbers represent the best accuracy. 'Rand.' is the abbreviation for 'random'.
VL-ICL Bench
Methods
BlindTest
RealWorld
CharXiv
GVL
MathVision
Average
CLEVR
FOMI
TextOC
BB-.3.5V-4.2B
0.402
0.292
0.300
0.333
0.117
0.425
0.070
0.715
0.332
+Rand. Sample
0.339
0.254
0.250
0.308
0.092
0.395
0.070
0.685
0.299
+VL-ICL
0.407
0.292
0,280
0.325
0.100
0.435
0,080
0.745
0.333
*MICCSS
0.399
0.300
0.300
0.317
0.083
0.415
0.070
0.730
0.327
+ContextNav
0.443
0.300
0,310
0.350
0.117
0,440
0.070
0.740
0.346
InternLMX2..o
0.303
0.262
0.200
0.325
0.108
0.545
0.075
0.475
0.287
+Rand. Sumple
0.288
0.231
0,150
0.292
0.083
0.505
0.075
0.445
0.259
+VL-ICL
0,316
0.246
0.180
0.342
0.092
0.570
0.105
0.455
0,288
+MMICES
0.296
0,254
0.210
0.333
0.092
0.555
0.090
0.465
0.287
*ContextNav
0,358
0,277
0.230
0.342
0.142
0.570
0.100
0.480
0,312
Qwen2.5- 5--2--
0,566
0,307
0.390
0.342
0.217
0.820
0.045
0.835
0.440
+Rand. Sample
0.496
0.277
0.340
0.300
0.200
0.785
0.045
0.820
0.408
+VL-ICL
0.594
0,323
0.370
0.358
0,233
0,900
0,060
0.845
0,460
+MMICES
0.606
0,315
0.390
0.342
0.208
0.890
0.050
0.845
0.456
*ContextNav
0.645
0.367
Gemini- 1.5
0.755
0.475
0.535
0.505
+Rand. Sample
0.733
0,285
0.440
0.492
0.350
0.510
0,140
0.860
0.476
+VL-ICL
0.775
0,315
0.500
0.533
0.475
0.555
0,180
0.890
0,528
+MMICES
0,802
0.338
0.520
0.508
0.467
0.540
0.170
0.528
*ContextNav
0.859
0,369
0,570
0.575
0,517
0,575
0,240
0.890
0.574
0.761
0,308
0,510
0.508
0.492
0.775
0.080
0.900
0,542
+Rand. Sample
0,733
0.292
0.430
0.517
0.341
0.755
0.095
0.870
0,504
+VL-ICL
0.773
0.331
0,550
0.483
0.810
0.145
0.905
0.561
+MMICES
0,800
0,338
0.520
0.525
0.467
0.795
0.140
0.885
0.559
+ContextNav
0.854
0.560
0.600
0,550
0,825
0.155
0.604
GPT-4o
0.609
0.323
0.530
0.500
0.342
0.610
0.085
0.870
0.484
+Rand. Sample
0,588
0.308
0.470
0.517
0.308
0.635
0.095
0.860
0.473
+VL-ICL
0.631
0,338
0.540
0.558
0.333
0.650
0.140
0.890
0,510
+MMICES
0.649
0.550
0.542
0.316
0.655
0.150
0.895
0.514
+ContextNav
0.672
0,392
0.580
0.608
0,383
0.670
0.165
0.905
0.547
(4) Dalaset-wise Average ICL Gains
(b) Model-wise Average ICL Gains
25
RelWoddd
Sasping
20
MMDCCES
13
Castutiow (Dwnd)
10.4
10
1.4
CLEVi
GVL
Plo 29
boveLAMt2 5-88
Queat PL:B
Cemeta 1--Bad
CPS4e
Figure 3: Comparison of average ICL gains with baselines. (a) Dataset-wise average gains across 8 datasets. (b) Model-wise average gains across 6 representative MI..Ms. 'ICI. Gains' represents the percentage improvement of MILM (zero-shot) after applying ICI. The complete comparative results provided in Appendix
visual embedding backbone. Based on the experimental results of Figure 4 the framework employs effective shots for ICL. Unless otherwise specified, experimental results are the medium number of 5 trials, and ablation experiments agentic system are documented Appendix FF, while the definitions the tool library and the OGG provided in Appendix G and the default embedding model zoos are listed in Appendix H In addition, Appendix presents case studies of both successful and failure examples.
conducted on the Math Vision. The prompts involved in the
4.2 COMPARISON AGAINST OTHER METHODS
We compare ContextNav with zero-shot and multimodal ICL baselines under training free settings, including random sampling, VL-ICL (Zong et al. 2025), and MMICES (Chen al. 2025b). The

---

Table 2: Ablation study on agent policy and core components involved in contextualization The bold entries indicate the best results. 'Semantic Noise' and 'Structural Noise' denote the proportions of contextual candidates with the corresponding type of noise, respectively. 'ICL Gains' represents the percentage improvement of vanilla MLLM after applying ICL. 'Rand.' is the abbreviation of random. 'X-Step TSR' represents the probability of successfully generating a valid toolchain within 'X' iterations. "-' indicates not applicable. As Agentic Retrieval enforces semantic alignment, it is disabled when ablating textual/visual retrieval to avoid diminishing the significance of the results.
Methods
Semantic Noise
Structural Noise
ICL Gains
1-Step TSR
5-Step TSR
MLLM policy
GPT-4n
0,053
0.081
+11.8%
1,000
1.000
Gemini- 2.0-faash
0.053
0.084
+11.8%
0.995
1.000
Qwen2.5.VI VL- Qwen2.5-VL-3B
0.073
0.107
+10.1%
0.985
1.000
0.080
0.139
+8.4%
0.965
0.995
Other Components
w/o Agentic Retrieval (AR)
0.171
0.090
+1.6%
0.995
1.000
w/o Structural Aligament
0.053
0.573
+6.7%
0,995
1.000
w/o Textual Retrieval Tools & AR
0.433
0.143
-18.7%
0.995
1.000
w/o Visual Retrieval Tools & AR
0.249
0.076
-3.5%
0.995
1.000
Toolchain Optimization
0.093
0.995
wlo Operational Grammar Graph
0
Full (with Geminl-2200ffashh
0.053
0.084
+11.8%
0.995
1.000
Table 3: Ablation study of embedding models. 'Semantic Noise' and 'Structural Noise* w/o AR
SA denote the proportions of contextual candidates with the corresponding type of noise in the absence of Agentic Retrieval (AR) and Structural Alignment (SA). 'Effective Rate' refers to the proportion of retrieved candidates retained after applying Agentic Retrieval. 'ICL Gains' represents the percentage improvement of a vanilla MLLM after ICL.
Semantic Noise w/o AR & SA
Structural Noise w/o AR & SA
Text Encoder
Vision Encoder
Effective Rate
ICL Gaius
Brun--Embeddinn-8B
Qwen2.5-VL-Vista. CLIP-vis
0.168 0.215
0.581
0.765 0.719
+11.8%
+10.1
Swenž-Enbeddding-4B
Qwen2.5 S-VL-visEne
0.171
0.584
0.762
+11.8%
CLIP-vis
0,216
0.611
0.718
+10.1%
Qwen3-Embeddinng-O.e
0.194
0.595
0.749
+11.8%
CLIP-vis
0.227
0.623
0.721
+10.1%
Dwen2.5-VL-Viifl
0.276
0.631
0.652
+8.45
CLIP-text
CLIP-vis
0.311
0.659
0.606
+6.7%%
results of VL-ICL and MMICES are obtained from our replications based on their original method- ologies. Specifically, the replication of VL-ICL follows the procedure of constructing a manually coarse-filtered candidate pool, followed by random sampling, whereas MMICES adopts cascaded retrieval process that prioritizes visual retrieval and subsequently applies text-based retrieval. The results in Table D and Figure 3 demonstrate that ContextNav rivals or surpasses other carefully designed baselines across nearly all datasets and MLLMs. When the support data involves composite complex and noisy (e.g., BlindTestt RealWorld, CharXiv, and Math Vision),
tasks and becomes non-agentic methods are more likely to yield unstable or even degraded performance, whereas Con- textNav consistently improves MLLMs' performance. Overall, ContextNav achieves average ICL gain of 16.8% across models and 16.2% across datasets, substantially outperforming the previous sfate-of-the-a (7.6% and 8.2%, respectively). These results further highlight that ContextNav provides a noise-robust mechanism for exploiting in-context information, delivering consistent and substantial ICL gains across diverse multimodal tasks. Moreover, we extend the comparison results in Appendix Cy incorporating additional baselines, including many-shot ICL (Agarwal et al. 2024) and Cache of Thought (Wu al, 2025a), following the settings used in their original works.
4.3 ABLATION STUDY
Agent policy and core components involved contextualization. The upper part of Table 2 indicates that the choice of MILMM policy could affect the effectiveness of ContextNav. More capable
8

---

BlindTest
CharXiv
MathVision
(d)
ICL Gains (%)
20
-20
-40
10
2
6
10
10
2
6
10
Number of Contextual Shots
Number of Contextual Shots
Number of Contextual Shots
Number of Contextall Shots
InlemLMX225-78
Qwen2.5-VL-7B
1.mini--.S--laakf
GPT-4o
Figure 4: Ablation study on the number of contextual shots. improvement after ICL.
IIL Gains' represents the percentage
(a) Pooled Paired Seatler
(b) Pee-query Gain Distribution
[c) CDF Gains
1.0
Aligned Similurit
0.9
Siealarit Gain (A cosin))
0.5
Negative Shift
Positive Shift
Cumulative Prohabliay
0.8
0.9
0.4
0.7
0.6
0.5
0.6
y
0.2
0.4
0.5
Selow y-x: 26.8%
0.3
D.268
Above
0.4
0.3
Below yox
0.0
0.1
Negative gains
Positive gains
0.3
0.4 0.5 Original Similarity
1.0
Query Index
0.0 0.1 0.2 Similarity Guin (A cusine)
0.6
Figure 5: Effect of Structural Alignment on textual similarity distributions. (a) Paired scatter between original and aligned similarities. (b) Per-query distributions of similarity gains, with orange denoting positive shifts and gray denoting non non-positive shifts, and the points indicate the average gains. (c) Cumulative Distribution Function (CDF) of similarity gains, with reference lines indicating zero-gain boundaries.
multimodal policies demonstrate stronger capacity for executing agentic retrieval and structural alignment, thereby reducing both semantic and structural noise and yielding higher ICL gains. In addition, stronger models exhibit improved cntrruction-olllwwiii
ability. which translates into more reliable toolchain generation and execution. The lower part of the table highlights the contribution of individual modules. The ablation results highlight that textual, visual, and agentic retrieval modules are crucial, as their removal markedly amplifies semantic noise. Semantic noise emerges as the dominant factor shaping ICL gains, exerting stronger impact than other sources of disturbance. At the same time, structural alignment plays key role mitigating structural noise, whose influence on ICL, though secondary to semantic noise, remains non-negligible. Moreover, disabling toolchain optimization and restricting the workflow to a single-step determination could yield suboptimal toolchain selection strategies, which consequently limit the effective mitigation of both semantic and structural noise and thereby reduce the ICL gains. In addition, the OGG constitutes the foundation of tool orchestration, and its removal renders the system incapable of executing valid toolchains.
Embedding. As shown in Table 3 we conduct a manual ablation of the embedding models in ContextNav. For the text encoder, evaluate the Qwen3-Embedding series at different parameters, the SigLIP-2 (Tschannen et al.) 2025), and the classical CLIP (Radford et al.) 2021) For the vision encoder, we test the aliiged Qwen2.5-VL visual transformer, SigLIP-2 and CLIP, We observe that embeddings from models with fewer parameters lead to less accurate retrieval, introducing noisy or suboptimal candidates. While noisy candidates are filtered out Agentic Retrieval, suboptimal ones may remain and still degrade contextual quality, diminishing ICL gains and highlighting the importance of careful embedding selection. We also find that (wen3--Emedddee 8B and 4B yield nearly identical retrieval under the default setting, showing that indiscriminately adopting larger models may bring diminishing returns and unnecessary resource overhead. This finding highlights the practical significance of ContextNav's resource- design, which adaptively selects embedding models according to both the user's resource usage preferences and the objective hardware constraints.
Number of contextual shots. As shown in Figure we conduct an ablation study on four MLLMs across four datasets examine the impact of contextual shot numbers on ICL gains. The results

---

indicate that ICL gains generally improve with more shots but plateaus the number increases. Smaller models (c.g.. Qwen2.5VL-7B and IntemLLMX2.5-7B) are more sensitive, with too few shots could yield negative gains, whereas larger models (e.g., Gemini- 1.5-flash and GPT-4o) exhibit more stable improvements. These results highlight the need for an appropriate choice of shot number: too few degrade performance, while too many add computational cost without commensurate benefit. Accordingly, we set the default to 8 shots in our experiments.
Extended ablations. In Appendix D we extend our analysis by incorporating additional ablations on noise robustness (D.T). the semantic equivalence of structural alignment (.2). the interaction between MLLM policies and system cost (D.3), as well the trade-off between ICL gains and system costs for downstream MLLMs (D.4). These additional studies further demonstrate the overall robustness, effectiveness, and scalability of ContextNav across diverse settings.
4.4 DISCUSSION
Structural Alignment. As shown in Figure 5 we analyze the effect of Structural Alignment on textual similarity distributions. We randomly sample 50 queries along with their eight most similar candidates from MATH-Vision dataset and apply structural alignment to them. Panel (a) compares original and refined textual similarities, showing that 73.2% of the points lie above the diagonal, indicating that aligned generally increases similarity. Panel (b) illustrates per-query distributions of similarity gains, where most queries exhibit a positive shift; although a few negative gains remain, they are marginal in magnitude. Panel (c) presents the cumulative distribution of similarity gains, further confirming that a substantial proportion of candidates benefit from refinement. Overall, these results demonstrate that Structural Alignment mitigates structural discrepancies between queries and candidates, leading to more consistent and semantically aligned contexts.
Negative effects of noisy contexts. As shown in Table contextualization with random sampling strategies often results in substantial performance degradation. This decline arises because random sampling introduces large number of query-irrelevant examples, thereby injecting noise into the context. Furthermore, as reported in Table 2 higher proportions of semantic and structural noise generally correspond reduced ICL gains. These findings underscore the detrimental impact of noisy contexts in ICL and highlight the necessity of accurate exemplar retrieval. This observation is also consistent with the implicit gradient descent perspective of ICL, (Dai et al., 2023a] Von Oswald et al. 2023) Chen et al.) (024), in which irrelevant or misaligned examples distort the optimization trajectory and hinder generalization.
Extended discussion. We further elaborate our discussion in Appendix D on the dataset selec- tion( E.T. construction process and robustness of the Operational Grammar Graph (E>2), provide an analysis of level failure cases (E.3) and discuss potential future research directions for agentic multimodal ICL (E.4).
Limitations. Since the agent requires additional inference and tool-execution steps. ContextNav inevitably introduces extra token overhead and system latency, which may limit its applicability in scenarios with stringent real-time requirements. On average, each ICL. iteration under the default setting consumes 22,51K tokens and incurs 3.26 seconds of delay. As a complement, Appendix [D3 reports the token overhead and system latency when using different MLLMs.
CONCLUSION
In this paper, we introduce ContextNav, the first agentic framework that integrates automated retrieval with human-like curation for multimodal ICL. By combining agentic context management, context denoising, and graph-based workflow orchestration modules, ContextNav constructs and optimizes nosse-resilient contexts within a fully automated workflow, enhancing the multimodal ICL performance of downstream MLLMs. Experiments demonstrate that ContextNav achieves state-of-tie of-theeart results across diverse datasets and models, underscoring the potential of agentic workflows for scalable, adaptive, and robust contextualization in multimodal ICI..
10

---

ACKNOWLEDGMENTS
The work is partially supported by the U.S. National Science Foundation (NSF) Grant CRII 2451683, an NVIDIA Academic Grants Program, U.S. Bank Academic Research Award, the University of California, Merced, and a UC Merced Faculty Research Award. The views and conclusions are those of the authors and do not necessarily reflect the official policy or position of the U.S. Government.
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024.
Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanic Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. Many-shot in-context learning. Advances in Neural Information Processing Systems, 37:76930-76966, 2024.
lean-Baptiss Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances neural information processing systems, 35:23716- 23736, 2022.
Aryaman Arora, Dan Jurafsky, Christopher Potts, and Noah D Goodman. Bayesian scaling laws for in-context learning. arXiv preprint arXiv:2410.16531 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
Folco Bertini Baldassini, Mustafa Shukor, Matthieu Cord, Laure Soulier, and Benjamin Piwowarski. What makes multimodal in co-tontext learning work? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1539-1550, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1 33:1877-1901, 2020.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, et al. InternIm2 technical report, 2024.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Brian Chen, Tianyang Hu, Hui Jin, Hwee Kuan Lee, and Kenji Kawaguchi. Exact conversion in- context learning to model weights in linearized-attentio transformers. In Forty-first International Conference on Machine Learning. 2024.
Cheng Chen, Yunpeng Zhai, Yifan Zhao, Jinyang Gao, Bolin Ding, and Jia Li. Provoking multi-modal few-shot Ivlm via pntrraaiion-xplliiaatiinn in-context learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 3826-3835. 2025a.
Shuo Chen, Zhen Han, Bailan He, Jianzhe Liu, Mark Buckley, Yao Qin, Philip Torr, Volker Tresp, and Jindong Gu. Can multimodal large language models truly perform multimodal in-context learning? In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 6010- IEEE, 2025b.
Yixin Chen, Shuai Zhang, Boran Han, and Jiaya Jia. Lightweight in-context tuning for multimodal unified models. arXiv preprint 1rXiv:2310.05 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
11

---

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: MKXIv:8003.04455 2018.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 4005-4019, 2023a.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purposs vision-language models with instruction tuning. Advances in neural information processing systems, 36:49250-49267, 2023b.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning, arXiv preprint MrXiv:2301.00234 2022.
Sivan Doveh, Shaked Perek, M Jehanzeb Mirza, Wei Lin, Amit Alfassy, Assaf Arbelle, Shimon Ullman, and Leonid Karlinsky. Towards multimodal in-context learning for vision and language models. In European Conference on Computer Vision, pp. 250-267. Springer, 2024.
Fabian Falck, Ziyu Wang, and Christopher C Holmes. Are large language models bayesian? martingale perspective on in-context learning. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024.
Shuzheng Gao, Xin-Cheng Wen, Cuiyun Gao, Wenxuan Wang, Hongyu Zhang, and Michael R Lyu. What makes good in-context demonstrations for code intelligence tasks with llms? In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 761-773.
IEEE, 2023.
GTeam, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Brandon Huang, Chancharik Mitra, Assaf Arbelle, Leonid Karlinsky, Trevor Darrell, and Roei Herzig, Multimodal task vectors enable many-shot multimodal in-context learning. Advances Neural Information Processing Systems. 37:22124-22155: 2024.
Hao Huang, Shuaihang Yuan, Yu Hao, Congcong Wen, and Yi Fang. A chain-of-thouu subspace meta-learning for few-shot image captioning with large vision and language models. arXiv preprint atXiv:2502.13942 2025.
Aaron Hurst, Adam Lever, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card, arXiv preprint arXiv:2410.21276, 2024.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Langchain-ai. Langgraph: Build resilient language agents as graphs. langchain- ai/ hai///agggaphh 2024.
https://gi thub. com/
Jiahao Li, Quan Wang. Licheng Zhang. Guoqing Jin, and Zhendong Mao. Feature-adaptive and data-scalable in-context learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1481-1494, 2024a.
Wei Li, Hehe Fan, Yongkang Wong, Yi Yang, and Mohan Kankanhalli. Improving context under-
standing in multimodal large language models via multimodal composition learning. In Forty-first International Conference on Machine Learning, 2024b.
12

---

Yanshu Li, Hongyang He, Yi Cao, Qisen Cheng. Xiang Fu, and Ruixiang Tang. M2iv: Towards efficient and fine-grained multimodal in-context learning in large vision- lssounlanguage models. arXiv preprint arXiv.2504,04633, 2025a.
Yanshu Li, JianJiang Yang, Bozheng Li, and Ruixiang Tang. Cama: Enhancing multimodal in-context learning with context-aware modulated attention. arXiv preprint arXiv:2505.1709, 2025b.
Yongqi Li, Wenjie Wang, Leigang Qu, Liqiang Nie, Wenjie Li, and Tat-Seng Chua. Generative cross-modal retrieval: Memorizing images in multimodal language models for retrieval and beyond. arXiv preprint 2xx::240221080: 2024c.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in 2023a.
neural information processing systems,
Sheng Liu, Haotian Yc, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv preprint arXiv:2311 1.06668, 2023b.
Weihao Liu, Fangyu Lei, Tongxu Luo, Jiahe Lei, Shizhu He, Jun Zhao, and Kang Liu. Mmhqa-iel: Multimodal in-context learning for hybrid question answering over text, tables and images. arXiv preprint Ntt.:2339.047990 2023c.
Yansheng Mao, Yufei Xu, Jiaqi Li, Fanxu Meng, Haotong Yang. Zilong Zheng, Xiyuan Wang, and Muhan Zhang. Lift: Improving long context understanding of large language models through long input fine-tuning. arXiv preprint arXiv:2502.14644 2025.
Yingzhe Peng, Xinting Hu, Jiawei Peng, Xin Geng, Xu Yang, et al. Live: Learnable in-context vector for visual question answering. Advances in Neural Information Processing Systems, 37: 9773-9800, 2024.
Libo Qin, Qiguang Chen. Hao Fei, Zhi Chen, Min Li, and Wanxiang Che. What factors affect multi- modal in-context learning? an in-depth exploration. Advances in Neural Information Processing Systems, 37:1 17:123307-123236, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. In Proceedings of the Asian Conference on Computer Vision, pp. 18-34. 2024.
Dianmo Sheng, Dongdong Chen, Zhentao Tan, Qiankun Liu, Qi Chu, Jianmin Bao, Tao Gong, Bin Liu, Shengwei Xu, and Nenghai Yu. Towards more unified in-context visual understanding. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
13362-13372. 2024.
Amanpreet Singh, Guan Pang. Mandy Toh, Jing Huang, Wojeiech Galuba, and Tal Hassner. Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8802-8812. 2021.
Sanchit Sinha, Yuguang Yue, Victor Soto, Mayank Kulkarni, Jianhua Lu, and Aidong Zhang. Maml- en-llm: Model agnostic meta- meta-training training of Ilms for improved in-context learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2711-2720, 2024.
Wei Suo, Lanqing Lai, Mengyang Sun, Hanwang Zhang, Peng Wang, and Yanning Zhang. Visual prompt selection for in-context learning segmentation. arXiv preprint ar 2111::2400.102333 2024.
Yan Tai, Weichen Fan, Zhao Zhang, and Ziwei Liu. Link-context learning for multimodal llms. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27 1717-227185 2024.
13

---

Gemini Team, Rohan Anil, Sebastian Borgeaud. Jean- -ean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11 11805, 2023.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502. 154.6666..17866 2025.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pp. 35151-35174, PMLR, 2023.
Anton Voronov, Lena Wolf, and Max Ryabinin. Mind your format: Towards consistent evaluation of in-context learning improvements. In Findings of the Association for Computational Linguistics
ACL 2024, pp. 6287-6310, 2024.
Futing Wang, Jianhao Yan, Yue Zhang, and Tao Lin. Elicit: Lim augmentation via external in-context capability. arxiv preprint arXiv:2410.09344, 2024a.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095-95169 2024b.
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. Advances in Neural Information Processing Systems, 36:15614-15638, 2023.
Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal Ilms. Advances in Neural Information Processing Systems, 37:113569-113697, 2024c.
Yanbin Wei, Shuai Fu, Weisen Jiang, Zejian Zhang, Zhixiong Zeng, Qi Wu, James Kwok, and Yu Zhang. Gita: Graph to visual and textual integration for vasion-language graph reasoning. Advances in Neural Information Processing Systems, 37:44-72, 2024.
Mingyuan Wu, Jize Jiang, Haozhen Zheng, Meitang Li. Zhaoheng Li, Beitong Tian, Bo Chen, Yongjoo Park, Minjia Zhang, Chengxiang Zhai, et al. Cache-of-thought: Master-apprentice framework for cost-effective vision language model inference. arXiv preprint arXiv:2502.20587, 2025a.
Shiguang Wu, Yaqing Wang, and Quanming Yao. Why in-context learning models are good sew-shot learners? In The Thirteenth International Conference Learning Representations, 2025b.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2 1 11 1.02080, 2021.
Jiaxi Yang, Binyuan Hui, Min Yang, Bailin Wang, Bowen Li, Binhua Li, Fei Huang, and Yongbin Li. Iterative forward tuning boosts in-context learning in language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15460-15473, 2024.
Jinghan Yang, Shuming Ma, and Furu Wei. Auto-icl: In-context learning without human supervision. arXiv preprint M.X.:23111009260 2023.
Qianqian Zhang, Jiajia Liao, Heting Ying. Yibo Ma, Haozhan Shen, Jingcheng Li, Peng Liu, Lu Zhang, Chunxin Fang, Kyusong Lee, Ruochen Xu, and Tiancheng Zhao. Unifying language agent algorithms with grsph orchestration engine for reproducible agent research. In Pushkar Mishra, Smaranda Muresan, and Tao Yu (eds.), Proceedings of the 63rd Annual Meeting of the
14

---

Association for Computational Linguisties (Volume 3: System Demonstrations), pp. 107-117, Vienna, Austria, July 2025a. Association for Computational Linguistics.. ISBN 979-8 -917---57---444 doi: 10. 1 htt/2/2222..22mmmmaa acl-demo.11. URL httpstss/aallaatholoooeoo
org/ 2025 acl-acllledm 117
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176 2025b.
YiFan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang. Qingsong Wen, Zhang Zhang, et al. Mme-roalworld: Could your multimodal Ilm challenge high-resolution real-world scenarios that difficult for humans? In The Thirteenth International Conference on Learning Representations, 2025c.
Zhuosheng Zhang, Aston Zhang, Mu Li. Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302..09223 2023.
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma. Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmiel: Empowering vision-language model with multi-modal in-context learning. arXiv preprint Xiix:2309900915.. 2023.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International conference machine learning, pp. 12697-12706. PMLR, 2021.
Chunting Zhou, Junxian He, Xuezhe Ma. Taylor Berg- Kirkpatrick, and Graham Neubig. Prompt consistency for zero-shot task generalization. In Findings the Association for Computational Linguistics: EMNLP 2022, pp. 2613 2626, 2022.
Yongshuo Zong Ondrej Bohdal, and Timothy Hospedales. VI-icl bench: The devil in the de- tails of multimodal in-context learning. In The Thirteenth International Conference on Learning Representations, 2025.
A THE USE OF LARGE LANGUAGE MODELS (LLMs)
The paper employs LLMs for language polishing.
B EXTENTED MAIN RESULTS
T presents the complete ICL gain comparison results.
C EXTENDED COMPARISON
C.I COMPARISON IN MANY-SHOT CONDITIONS
We further extend the quantitative comparison with many-shot ICL (Agarwal et al. 2024 in Table 5 Since different models have different context-length limits, use as many in-context samples as allowed by either the model's 's maximum window or the corpus upper bound (e.g.. 800 samples for CLEVR in VL-ICL). For models whose context window becomes the bottleneck (e.g.. InternLM. QWL. GPT-4o), we follow the standard many-shot ICL protocol and randomly sample instances from the corpus.
From our experiments, we find that increasing the number of randomly sampled examples from 8 to 100 or even 800 generally yields a positive shift downstream MLLM ICL performance. On the clean, single-task benchmark CLEVR. most MLLMs move from negative to positive gains as the support set grows. However, on more complex and noisier benchmarks such as Math Vision and CharXiv, many-shot random sampling improves over few-shot (8) but still remains below zero-shot performance. Across all settings, ContextNav consistently performs best. These findings show that both enlarging the support set and selecting query-relevant examples have positive effects on ICI..
15

---

Table Comparison ICL gains with other baseline methods. ICL gain represents the percentage improvement of vanilla MLLM after ICL, greater values indicate better performance.
Downstream
CharXiv
Rand. Sample
-1566
-1300
-16.7
-8.1
-21.4
-7.1
-1.2
-10.8
VL-ICL
Phi-3.5V-4.28
MMICES
.0.7
0
488
-29.
-2.4
2.1
-4.
ContextNav
10.2
2,7
3,3
5.1
3.5
3.5
3,5
Rand. Sample
-5.0
-11.8
-25.0
-10.2
-23.1
-7.3
-6.3
-11.1
IntenL-M2.55778
MMICES
VL-ICL
-2.3
4,3
+6.1
-10.0
5.0
5.2
2.5
-14.8
-14.8
4.6
1.8
20.0
40.0
-1.2
-2.1
0.9
2.4
ContextNav
5.2
Rand. Sample
-124
988
-15.0
-12.3
-7.8
-4.5
-1.8
-X.0
Qwen2.5-V1-78
MMICES
0
-4.1
11.1
1.2
7.7
3.3
ContextNav
14.0
10.1
2.6
7.3
15.2
14.6
22.2
1,2
10.9
Sample
3.6
-2.3
Gemini-1 S-llash
VL-ICL
2.6
5.0
2.0
12.2
-1.7
3.7
50.0
1.1
9.4
MMICES
6.2
12.7
-3.9
6.9
-3.3
41.7
-0.6
11.4
Rand. Sample
-3.7
-5.2
-15.7
-1.8
-30.7
-2.6
18.8
-3.3
-5.5
VL-ICL
8.3
0.6
Gemini-2.00laa 2.0-las
MMICES
9.7
2.0
3.3
-5.1
2.6
75.0
-1.7
11.4
ContextNav
12.2
22.4
9.8
18.1
11.8
6.5
93.8
1.1
22.0
Rand. Sample
-11.3
3.4
11.8
-1.1
-1.4
GPT-4o
MMICES
VL-ICL
3.6
4.6
3.8
1.9
11.6
8.4
-2.6
-7.6
6.6
64.7
76.5
2.3
2.9
11.6
13.4
ContextNav
21.6
Table 5: Comparison with many-shot ICL.
MLLM
Method (+sample num)
CLEVR
Math Vision ?
CharXiv t
zero-shot
0.545
0.108
0.200
Random (+8)
0.505
0.083
0.150
InternI.MX2.55
Many-shot (+100)
0.535
0.092
0.160
ContextNav (+8)
0.570
0.142
0.230
zero-shot
0.820
0,217
0,390
Random (+8)
0.785
0.200
0.340
Qwen2.5-VL-7B
Many-shot (+100)
0.870
0.208
0.370
ContextNav (+8)
0.940
0.250
0.400
zero-shot
0.610
0.342
0.530
Random (+8)
0.635
0.308
0.470
GPT-40
Many-shot (+100)
0.645
0.316
0,510
ContextNav (+8)
0.670
0.383
0.580
zero-shot
0.775
0.492
0,510
Random (+8)
0.755
0.341
0,430
Gemini-2.0--fas
Many-shot (+800)
0.815
0.425
0.500
ContextNav (+8)
0.825
0.550
0.560
C.2 COMPARISON WITH CACHE OF THOUGHT
We extended comparison with Cache of Thought (Wu et al.) 2025a) in Table 6 ContextNav with Cache of Thought using the settings reported
We compare original paper, evaluating CLEVR and Tex1OCR. Under the strict 1-2 shot setting,
both methods with OpenFlamingo-3B ContextNav performs slightly worse. This is expected: Cache of Thought distills both knowledge and explicit reasoning traces from a powerful Master VLM, whereas ContextNav retrieves relevant evidence from a corpus without providing explicit chains of thought. As a result, Cache Thought has an inherent advantage when only two demonstrations available.
16

---

Table 6: Comparison with Cache of Thought on OpenFlamingo--35
Method
CLEVR
TextCC t
zero-shot
7.50
0.00
1-shot
Cache of Thought
20.50
6.50
ContextNav
12.00
3.50
2-shot
Cache of Thought ContextNav
28.00 15.50
5.50
4.50
Table 7: Semantic and structural noise under different ablation settings on Math Vision and CC12M.
Math Vision
CC12M
Method
Semantic Noise
Structural Noise
Semantic Noise
Structural Noise
w/o Agentic Retricval (AR)
0.171
0,090
0.159
0.077
w/o Structural Aligmment (SA)
0.053
0.573
0.050
0.488
Textual Retrieval Tools w/o Visual Retrieval Tools AR
0.433
0.143
0.552
0.161
0.249
0.076
0.206
0.072
Rand. Sampling Full (ContextNav)
0.936
0.971
0.987
0.960
0.053
0.084
0.050
0.075
D EXTENDED ABLATIONS
D.I PERFORMANCE ON NOISY CORPUS
To further quantify the noise level of the corpora and evaluate ContextNav's robustness to noisy data, we conducted additional experiments on Math Vision and on a large-scale real-world noisy corpus, Conceptual Captions (CC12M) (Changpinyo et al., 2021). For CC12M, sampled 80K image-caption pairs and automatically rewrote them into a question-ant answer format using GPT-5. 200-sample evaluation was then curated following the same selection criteria used in other benchmarks, and all evaluation samples were verified to have sufficiently many support examples within the corpus. As shown in Table 7 random sampling yields highly noisy retrieved contexts, confirming the substantial intrinsic noise present in both Math Vision and CC12M. The results further demonstrate the reliability of each component in our method for mitigating such noise, well the overall noise robustness of our approach.
D.2 SEMANTIC EQUIVALENCE STRUCTURAL ALIGNMENT
To verify the reliability of structural alignment, we conduct an empirical evaluation of semantic equivalence on the Math Vision. We quantify semantic equivalence using two complementary metrics: (1) the semantic similarity between the original query and its aligned counterpart, computed using Ewen3-Embedding
and (2) binary classification accuracy from GPT-4o, which assesses whether the aligned structure preserves the original meaning. As shown in Table 8 structural alignment preserves a high level of semantic equivalence and, for Qwen2.5-VIL-78 and stronger policy I.I.Ms, rarely leads to any semantic distortion (GPT-4o accuracy = 1). The prompt used for GPT-4o accuracy evaluation is:
Prompt for GPT-4o Accuracy Evaluation
You will be given two texts, and please determine whether they convey the same meaning, ignoring superticial differences in wording, structure, or format. If the meanings are semantically equivalent, answer "Yes". the meanings differ in any essential way, answer "No". The texts are below:
Text {query)
Text [aligned_query]
Answer with "Yes" "No" only.
17

---

Table 8: Evaluation of semantic equivalence before and after structural alignment.
Policy LMM
Semantic Similarity 1 GPT-40 Accuracy t
Qwen2.5-VV -
0.8946
0.983
Qwen2.5-VL-7B
0.9247
1.000
Gemini-2.0-
0.9313
1.000
Table 9: Comparison of policy models on noise reduction, ICL gains, and system efficiency.
Policy
Semuntic Noise I Structural Noise
ICL Gains 1
System Latency (s)
Token Costs 1
API Pricing
Qwen2.5-VL.33
0.080
0.139
+8.4%
5.10
22.2K
Qwen2.5.VL.
0.073
0.107
+10.1%
7.38
22.6K
Gemini-2.0-0aast
0.053
0.084
+11.8%
3,26
22.5K
$0.903
GPT-4o
0.053
0.081
+11.8%
3.92
23.1K
$0.08
D.3 MILM POLICIES AND SYSTEM COSTS
Table provides a comprehensive comparison of different MILM policies within ContextNav, including their token overhead, system delay, noise reduction capability, and resulting ICL gains. Motivated by a balanced consideration of performance, latency, and cost, we use Seemini-2.0-flas as our standard implementation. Empirically, Gemini-2.0-flash delivers strong performance while achieving substantially lower latency than alternative models. As an API-based solution, it also avoids the computational overhead associated with local inference, with a cost of $0.003 per invocation, which is only 1/27 of GPT-4o, while achieving comparable downstream gains. At the same time, geigh multimodal models in the 7B/3B range also exhibit competitive capability. Although small performance gap remains, these models are still efficient and practical options.
Overall, these findings make Gemini-2. 2.0-flash a natural choice as our standard policy model, offering the best balance between effectiveness and practical deployment, while it remains fully compatible with weight alternatives. This further demonstrates that ContextNav does not depend on any specific proprietary model and is robust and broadly applicable with respect to policy selection.
D.4 TRADE-OFF BETWEEN ICL GAINS AND SYSTEM COSTS FOR DOWNSTREAM MLLMS
We applied the standard ContextNav implementation to Phi-3.5V, Qwen2.5-VL. Gemini-2.0 2.0-flshh. and GPT-4o on BlindTest, and compared their ICL gains against token cost and latency (measured on Vithonsion). For latency, we report the eeste overhead introduced by ContextNav (averaged over multiple optimization iterations), excluding the downstream MILMMss own inference cost. The results in Table 10 show that ContextNav incurs broadly similar overhead across all models, while stronger downstream MLLMs exhibit slightly lower latency and token cost. We attribute this to their more accurate feedback, which facilitates smoother soliioptimization within ContextNav. The results show that ContextNav could complete reasoning, optimization and actions within a short time (less than 5s) using only small number of tokens (around 30K), while achieving strong downstream gains, which further highlights its efficiency and practical deployment feasibility.
E EXTENDED DISCUSSION
E. DATASET SELECTION
In selecting our datasets, we aimed to ensure broad task diversity and strong representativeness. For
composite-task datasets, we included: (1) Moalle challenging comprehensive real-world multimodal understanding (such as complex scene and object recognition, ultra-highrresolution remote sensing, charts/tables in natural scenes, traffic perception, high-resolution surveillance). (2) BlindTest: basic visual tasks that are trivial for humans but difficult for MLLMs (e.g., counting intersections, circle intersection, overlapping shapes, nested squares, grid estimation). (3) CharXiv: natural scientific charts from real papers, capturing scientific multimodal QA in realistic settings. (4) GVL: topological multimodal reasoning conneccconey,ivitty, cycles, topological sort, shortest path, max flow, bipartite matching, Hamiltonian path). (5) Math Vision: comprehensive multimodal
18

---

Table 10: ICL gains, system latency, and token costs across downstream MLLMs.
Downstream MILM
ICL Gains
System Latency (s) |
Token Costs +
Phi-3.5V
10.2%
5,45
45.9K
Owen2.5-VL-7B
14.0%
5,10
41.4K
Gemini-2.0-BB
12.2%
4.83
35.1K
GPT-4o
10.3%
4.68
33.3K
mathematical reasoning tasks (algebra, geometry, combinatorics, statistics, topology, etc.). For single-task datasets, we follow the representative prior work VL-ICL, including CLEVR (attribute reasoning), FOMI (visual concept substitution), and TextOCR (optical character recognition).
E.2 THE CONSTRUCTION PROCESS AND ROBUSTNESS OF OGG
The construction and edge-definition process of the OGG is automated. We instruct GPT-4o to infer tool composability by understanding and matching the iuput- data formats of all atomic tools. Since each tool function has clearly defined I/O specifications, the automatically constructed OGG is highly reliable (as shown in Table [ the toolchain exccution success rate is close 100%). Regarding ContextNav's robustness to the OGG design, we provide analyses from two complementary perspectives: database management and contextualization (downstream ICL gain).
Robustness of database management to OGG design. Because database management involves transforming unstructured corpus data into structured representations, the data-flow dependencies strict. Missing, redundant, or incorrectly ordered tools can directly result in invalid data flows. To address this, implement engineering-level validation: if generated toolchain triggers data-flow exception due to unsupported or ill-formed tool sequences, we automatically regenerate the chain. Only when repeated regeneration fails do we check whether the issue arises from the structure the OGG; if so, we apply minimal manual adjustments to correct it.
Robustness of downstream gains OGG design. The downstream ICL improvements primarily depend on the contexts produced during contextualization. To ensure the frameworks' robustness with respect OGG design, we standardize the data-flow interface for comtextualization tools, allowing them to be freely composed and enabling the agent to optimize their usage through memory and feedback. In practice, these tools are fully connected within the OGG (as shown the appendix E), ensuring that contextualization never encounters missing redundant tool dependencies, and thus does not negatively impact downstream ICL gains.
E.3 ANALYSIS OF SYSTEM FAILURES
We observe a few special failure cases in the system. Specifically, in rare instances, using a smaller model (e.g., Qwen2.5-VL-33 output. This is an occasional token-level repetition loop that small models can exhibit in structured output spaces, where the model fails distinguish between the toolchain-generattion phase and the natural-language reasoning phase. As a result, the formatted sequence may be repeatedly produced and the EOS token may not appear. This phenomenon is rare and only observed when we conduct ablation experiments with smaller policy models. It reflects a general model-level behavior rather than system error with our framework. and therefore does not compromise the system's 's robustness. When this occurs, simply prompting the policy to regenerate the toolchain could effectively resolve the issue.
the policy may cause it to repeatedly emit the toolchain in its
E.4 LOOKING FORWARD: REINFORCEMENT LEARNING FOR AGENTIC MULTIMODAL ICL
We observe that the environment constructed by ContextNav is naturally suitable for RL. The agent's state can be directly instantiated from the retrieval process (such the levels of semantic and structural noise), the context candidates (including their semantic and structural alignment with the query), and the memory module (which stores downstream ICL feedback and the history of previously executed toolchains). In addition, ContextNav provides a structured action space defined by the Operational Grammar Graph, which makes the environment's 's state transitions fully observable. Each
19

---

selected action, such generating candidate set through retrieval, refining the candidates' semantic properties or noise profile, or updating the toolchain and downstream feedback, deterministically leads to new state.
These components together form an RL trajectory in which the agent exccutes a sequence of inter- pretable operations, observes the resulting state updates, and accumulates feedback throughout the contextualization process. Furthermore, correctness signals from downstream feedback, the policy's 's reasoning cost (in terms of token usage), the retrieval noise levels and the candidate alignment levels can be jointly incorporated into a reward function that encourages improved downstream ICL performance while penalizing excessive inference token (fewer tokens help reduce inference latency) and overly noisy contextualization behavior. Such formulation would enable the agent to learn more efficient and task -ddattvee orchestration strategies over time without requiring substantial modifications to the system architecture.
While this line of investigation lies beyond the scope of the present work, we view RL-baaed policy optimization as natural and compelling extension of ContextNav. We believe that RL is poised to play an important role the next stage of agentic multimodal ICL, and our proposed ContextNav offers the first referenceable environment design that can meaningfully support this direction.
F PROMPTS
Tool Orchestration Prompt Template
You agent responsible for retrieving information relevant the user's query and integrating it into contextual knowledge assist a multimodal large language model with in-context learning.
Your available tools are defined as functions with the following descriptions: (tool_ liblaribryyy). From the tool graph textualized_tool_grappp you must select one appropriate toolchain to automate the in- in-context learning process.
The following the toolchain(s) you selected in previous steps together with the feedback received for your provided contextual knowledge: {memory).
(systtem_cossreeee constraints).
Based on your reasoning, decide on the most appropriate toolchain at this step. You must first present your reasoning process, and then output your final decision strictly in the format: Toolchain: tool A >> tool B >> tool N.', where the period "* marks the end of the output and must not be omitted.
System Constraints Template
There are some criteria you must follow: |criteria and_andddde lererdrettivvlleeee saaerllccpccciiiionnPaaa Prompt)..
Please reason these questions and tell me your reasoning results: [chain of thought).
User-defined Criteria and Retrieval-specificnn Prompt
1. If you are explicitly instructed that this is your first step. you must select a toolchain that contains the tools textual_st lsssmmmmllll lari tesiittt_reeteee retrieval and risual_simil lrrtttyrretrrievall However, you cannot stop these two tools; the complete toolchain must be specified.
you are not explicitly told that this your first step, you know is not your first step (e.g., you already selected a toolchain in the previous step), you may select other toolchains at this step.
3. You must avoid re-selecting any toolchains that have already been chosen in previous steps.
4. If all toolchains have already been selected in previous steps, then you must disregard the above criteria and instead select toolchain that includes at least the tools wtwaalssmimaarittrettrriiie and prenticrretrieve
Chain-of-Thoughh Thought
1. Is the current step your first step?
2. If it is not your first step, list the toolchains you have already used in the previous steps.
Reflect the feedback you received regarding the retrieved context information. Do you think the issues described the feedback are related the toolchains you selected earlier steps?
20

---

General Prompt for Agentic Retrieval
<image query> <image ref>
The images above, together with the following questions, form two image- -uegee-ueiioon pairs.
Question 1: [query_question)
Question |ref_ ]
don't need to answer the questions. decide whether two pairs share any similarity, either in the images (content) or in the question types (e.g., both ask for counting, scene understanding, etc.).
If there is any similarity, reply: 'Judgement- YES".
Budgement-YEES.
If there is no similarity, reply: "Judgement-NO NN and briefly explain why.
(Optional: The similarity criterion does need be strict, any reasonable overlap counts similarity.)
General Prompt for Structural Alignment
Rewrite the following question in the style of )ueettnnuestion)..
Only output the rewritten question, without any explanations extra text.
Question rewrite: [ref_question).
Embedding Specification Prompt
From the following options:
Text models:
Visual models: @vis_emb__sg moab_moeel_eee _ pooppppp]]
Select text model and one visual model based the hardware status (hardware_statusy sstttuus,ss considering disk and GPU memory usage base [resource_usage wwwgge_prcffrencce.
Output format:
Text Embedding: text_ modeel id;
Image Embedding: sisualmodeelid
Restriction:
Do not generate any additional symbols (e.g., **)) If essential to ensure the chosen embedding model is compatible with
database already exists, it is
Default General In- In-Context Learning Template
<image I>.. lim.ee <image <image__qerr>> (Option A)
I will provide series of reference images, each paired with corresponding question and Your task reflect on these references and summarize the useful information they convey. After references have been presented, I will then provide one final image with its question. Based on your prior reflections, you should give an answer to this final query.
The k-th reference sample is follows (repeated for v times):
Image k: <image k> (Option
Question: (ref_questiom(kk])
Answer: _ef__sssee((])
Finally, the last query is:
Final Image: <image query> (Option B)
Final Question: {query_question]
Please your summarized reflections from the reference samples to answer the final question.
Brecdback_requestt
Note: Both option and option B acceptable for injecting images. models with structured interfaces. text and images can interleaved in list format. However, when using open-source models, is empirically
21

---

common to place all image tokens at the beginning of the prompt, followed by textual instructions. In such cases, if images need be dynamicalty inserted within the prompt, recommended practice is inject the contexts across multiple dialogue turns.
Feedback Request Prompt
Please evaluate whether the reference samples you received were helpful and sufficiently rich (with the number of shots approximately matching the preset value (k)) in solving your final problem. they were, additionally output "Judgement- Yes"; not, additionally output "Judgement-No and, starting with "Feedback:", ", explain whether the mismatch arose from the text or the image of the reference samples.
G
TOOL GRAPH
G.1
TooL LIBRARY
Atom Tools
Description
get_query
Receives the multimodal query (text and image) and initializes the workflow.
get_hardware_stateee
Monitors computational resources (e.g., GPU memory, disk ca- pacity) to guide resource-aware embedding model selection.
check_updating
Detects newly added or modified samples in the context corpus and triggers re-embedding to ensure database synchronization.
matchingg.embeddinggnddee..
Selects appropriate text and vision embedding models from the model zoo, balancing retrieval quality with hardware efficiency.
mul multimodal__mmeedding
Converts multimodal corpus into vector representations, forming the basis for retrieval the vector database.
load_vector_ddttbass
Builds and/or loads the multimodal vector database.
textual_iiimlarityyreeriaaa
Retrieves semantically relevant candidates using text embeddings.
visual_simmlarrityyreeeeee
Retrieves visually correlated candidates using vision embeddings.
agentic_retrieval
Refines the initially retrieved candidates by filtering irrelevant or misleading examples through the agent's 's reasoning. mitigating semantic noise.
irructural_alggen inneent
Reorganizes textual structures of retrieved candidates, reducing structural noise and improving consistency with the query.
Table 11: Descriptions of atom tools in the tool library.
G.2 OPERATIONAL GRAMMER GRAPH
OGG_edges =
2. ("start", "get_ query"),
("get_query", "ge_hardware_status")
("get_query", "check_updating""
("get_query",
5 ("check_ uuppcckg_ddiinn "get_hardware,_ stattsssttts"..
7 bacheeckuudattngg "multimodal_ _em.rddinnebeddggg)......
8 "rccheckupdattigg"
9 ("get_ "mmbedding__mdess",,
"matching
maatching_embedding
("matching
wwattchiingembeddinng
("multien wtttttimmemeeeeedadeaaeoeeoo "load_ database"), "badd_eector_daabbase""..
drextual_similarityyree
"visuallsimmlariiyeerrrreaa
22

---

("tertualsssmlarrieeeeri"a""""" "vsuul_siinlooilyyyrrtrvvaa
("tertaalssmmlaarit_eeetea"""""" "agentic.rettrieval").
["eexuul_iimllarllyeeeerrv""""" "strrrrrrri".."ssuttuturrrraaa
"struuctura__aligmmnn""""
17 ('ViuulliSimilrriyyyrrrre'a'''"' "'exxuul_siinlarrtyy.eerrrvvll
18 ("diual_sumlarrlyyeerrreea'" "agentic_retrreval""
['Vduaalssmiilrriyyeerreevall " "structuura_aaligmeent""
"'agentic_retrrevvll" oatp://waaaarstrrtttuturaaaaaaa
["Mexual_iimlariiveerrre""""" "end")"
("vsuullsiimllrriyeeerrteeal "en"")
("agentic_retrieval", "end"), "end"),
("structural_aligmmmn"" "end")
]
H DEFAULT MODEL Zoo
text_ www.eeeeddingmmmmeleeee_oemm ("model_id" "Qwen/Qwen" Qwen3-ennnne -mbeeddngggdd'""
"description": "Requires 18 GB disk space at least 32 GB of available GPU memory." "h,
{"model_id") id";
"Qwen/Qwent hewp://eeeen-bbbdnnnnnn--e-
"description": "Requires 9 GB of disk space and at least 16 GB of available GPU memory."),
8 * ("model_ide id": "description": "Requires 2 GB of disk space and at least GB of available GPU memory."),
6B"
("model_ice :_d": "opena//cii--tt--lrr--e-ee
"description": "Requires 2 of disk space and at least GB available GPU ")
sts__eeeedddngmodell... =
("model_id" "Qwen/Qwen2. 5-
"description": "Requires 8 of disk space and at least GB of available GPU memory (for viT)."),
* ("model_ id": http://cpivviillla---aaaaeoe
"description": "Requires 2 GB of disk space and at least GB of available GPU memory.")
I CASE STUDY
I.1 SUCCESSFUL CASES
We present in Figure and Figure 7 comparative case study using Semmin--200-flase as the target MLLM, illustrating the differences between ContextNav and alternative approaches in contextualiza- tion shot selection and the resulting ICL performance. As the figures demonstrate, ContextNav more effectively identifies semantically relevant contextual shots and aligns their textual structures more closely with the query, thereby enabling the MLLM to conduct ICL more reliably and generate more accurate responses to the input query.
1.2 FAILURE CASES
We illustrate in Figure 3 and Figure9reeresseee prepessentttiee failure cases ContextNav with Gemini-2.0 flash the target M..MM. While ContextNav is generally effective identifying and supplying the
23

---

Groundtruth/ICL Results
Input Query
Groundtruth
Vanilla MLLM
How marr pouto treal weakd?
HUNTER
travel that soceed 35 manulea throughout the wask.
MMICES
+VL-ICL
+
ConntextNav
the new the there four passs in the "Tea"" movel time ecceed
Deeed provtded nest hovel lime socawds 25 minutee at ditiret prodox throughsst
Wacteesday. Thurstay, and lraday. tharstons, thene pasks.
throughoou
Contextualization Examples
First-5 Contextual Shots
Image
ContextNav
show pouk coutventaes cecessinn Hrroaghuut
How many linne dow Botweoo witch 20t. Hon How coo unneog gl led assed festics daes the SHEE cocelatorr putterss wre Commutttoo uddon vakee man cerme
Question
dep
dbve throughou coos
poricot?
peak?
T000?
Image
MMICES
How mony modees pess
At sruud what min Betwons WPkh Time sor What the threchod I Around white dote old mark does Exp Feet doos indice does the SHEE wher the ADC velue rodkced excerinnce reconse??
Question
vor enon brood 30007
15
morket
beges?
Image
VL-ICL
Question
cocoding vituo of 10 uo maikee m the
(de dus bekn Uhe crnnaghted spood duop Uie Hich cralt start to dha would uused. Jeno lin becwoon 2000 bolow 1000 1poi? jex FROOFF
(mallty go ebove Phichthees the glostest awergge junglo
Figure 6: Example successful from CharXiv.
target model with relevant contextual shots, and the model is often able to summarize and reflect upon these contexts, the final predictions are not always correct. We view such cases reflecting the current limitations of downstream MILMs' multimodal reasoning capabilities, which are difficult for ContextNav to overcome.
24

---

Groundtruth/ICL Results
Input Query
Groundtruth
ogttengles trange
b-i3 and cx17
Haw hig mutive the insotteed samozaide
Vanillla MLLM
-MMCCES
Let angled siangls donoted as Bolargic ABCE, whore midis
Right Hinnge ABC hos a right angle at wit sides and 17: area 60. Treainng the area as
Bi=BC=68 158, SowlE=178
three trangino from the demiccooees with wrong Using
gives
Tract Weatttit) um davds lnarge
tnarppee Stoungll ADCS and staangh bucs The area of Strange AuCS siruct 1922
rastus to be 6, Reoppting the srea relatio again weits red, whic say the innackus the cirlee not the seriit Using the righ-sttanle catkus and refirementt
1221 woot fract least the sur arsan cl the trangee $4BCs So. Sinuc15)221- di Mullptyn na by get
fomula givee again mitakos, noting need further
inargles aqual
Tuus ör 1200220).3
+LL-ICL
CContextNav
Irezzbed semicrdde ighianggeed cxtcubreed using the fommded r- ----)/22
carlor ewamples. found rodius Insorfbed semidicile dametoo or sice of right stangle car be relandd to the stangle's area 13, midue on side length 12 106, cembirdd
Instonon with oidee
typolerasos
egftownged triengle, and
Problem-"1 a b (12) 8 16 60 We can also express the aren the trangle as the sum the areae two smaller vianlles formed by connacting the center resmples /t//(e cj -tt e 17) 190 90 d Arraeaeee tiThes midua Insottee comisice
Applynng
into the formull, we
parsem derked
Thanekow, the rackes inaotbee sumroccc ao
Contextualization Examples
First-5 Contextual Shots
Image
ContextNav
rightunnbol Inerle
(dhiianga (80+s, targle C tooe how big the
Two somia As and 00 an
angiod trangl
lanenge Ustrral the omalios
t-12 and ex15
and geon How ligatle to ortto
Fow bgg rodisral
wtoro ta wiel CD.
fordlo passing thoough Baad The
Question
recobbes soniorole
and teeterobrre
the other cemrantle
doaf leugh 12"
cicle with AC and BC.
stown. How tid
respest wayy. How long ORG
square thwetgg
Image
MMICES
nigh--nniedd trlandi
adda langlt with
ana d6. cicte center in poer C incothed mangle
cincle well no tanger ther
skds lengtta 5.15 and
5 em, 12 om and 13 cmj diow an down on, the reduaal ane
lengon
truches
Question
semi-cirrie?
on) carter the rise oftenge S1P cml) They mem.eee the
DPCAY
to has length io.
tycoteruse
Wher angle tangle
Image
samiiiciies rdiis semidicie SAR odoerng fabro witse siagam piprit A dhagaar, an thei damates AB torgwn semvrcte fot cunte conor Norticanenty,
VL-ICL
foastr The past Vis
and reportads Thl get a strigh low
loo oxtersio
the coounferoo
is
Question
darreto are cangerts incoured
T'm poie of in
haadus DS langle
asprriie othrr
Line segroo with
Moro- and kangl
Vetrel. How many
monsure B00
fangle PIY and tangla PMt
Figure 7: Example successful case from Vision.
25

---

Groundtruth/ICL Results
Input Query
+ContextNav
Out of ai surves, wher holds omolisar valc walcchhnnne
Okay, I'e the images and the quaetors Herale surrrayy
"Ourve deniiication Ascunitdly idertilyngg the curves crocad. The lagence imparturt, sumus cokor style city
Groundtruth
"Haading Vatuoa need is be precse when reading va usa from the same graphi Using the prodited crops help, wotmmiing
ectn
lngerthmii
godd
"Lurdesterdnng the Quaerioe" Some queatome iim ineppwiing lerme heme uetancca "noel atatie cuew need make wone grapt.
Vanilla MLLM
Itxeed on thes and losking the finel image, m.4 1091 the cLavi srroiied of [Uw1y1 "Frrpect
grazt, smalloat
vakuo la TAATRIN, TRISTAN.
Contextualization Examples
First-5 Contextual Shots
Image
ContextNav
Dur of si the arew, what the ensiler distance no CVC curee the 20
Out of all the curves, 1Cun the curvee wher" Our of all the curvee Dut of ail the curvee wher rame the mame curvico: witchh neriiis wtt pancoalia dleeestto cisesttoo
Question
laneral position trramtttion stope jecluctng the 5,D gaven valuee at Intea atro-d7
0.121
anergg/eee lewe??
cumee?
poatton-:00
Figure 8: Example failure case from CharXiv.
Groundtruth/ICL Results
Input Query
ConttetNavv
The stown nod box "CyDm.
"Fallerlona""
connety Wentty "REACK" "OCIFAN" "Jeddee" and 'ME3 AMD: denaneviingg whily acsunstly eator we from images expaciilly when the ned st and pot of imge The sugrete difiuity
loger image Hownow ktwppe
coreerecly itwerty Liely steme from combination becten
Groundtruth
""mage fleeclution and Quaddy quality neeckrion images playad rale "Taod Style and Conteel The the (honi, coler
CDVD
secunndeyy stentit backgronnt)
dig diniikartty impacted my abiily to Inteppet it. The more dyyited the tort, the more challenging was decigher
Vanilla MILMM
"Eockground
Background
my abiity
Hend's enswort The taai rod box "CY".
Improve my performnnoo, need recogrize sharaderrs in varioue forts, size,, and ctyles
Contextualization Examples
First-5 Contektual Shots
Image
ContextNav
Question
theird boo?
feetbou?
theied bor?
Figure 9: Example failure case from Tex1CCR.
26


Reviews:
[{"review_id": "ZMgBFtb2hd", "reviewer": "Reviewer_ppfZ", "summary": "The authors identify limitations in current Multimodal In-Context Learning (ICL) approaches, which either rely on manual selection of support examples or naive similarity-based retrieval that risks including semantically or structurally irrelevant examples. They introduce ContextNav, an agentic framework that treats multimodal contextualization as a closed-loop process. ContextNav constructs and maintains a resource-aware vector database, retrieves candidate examples, and refines them through Agentic Retrieval (semantic filtering) and Structural Alignment (format harmonization) before passing them to the target multimodal model. Its orchestration is governed by an Operational Grammar Graph (OGG) and updated using feedback-driven memory, enabling adaptive tool sequencing. Across a variety of datasets and models, ContextNav achieves roughly 16–17% ICL gains over baselines and provides systematic ablations on each module’s contribution. The work is significant in demonstrating that agentic data curation can enhance multimodal ICL performance, though the framework’s dependence on large models for reasoning introduces practical cost and latency considerations.", "questions": "Questions:\t\n1. Could you elaborate on the dataset selection procedure? Specifically, how were the composite-task and single-task datasets chosen, and how do they represent the diversity of multimodal reasoning tasks ContextNav aims to address?\n2.\tFor a framework that emphasizes noise robustness, what was the motivation to use curated datasets rather than noisy corpora? \n3.\tWere there any internal ablations or pilot experiments that compared joint multimodal embeddings against the chosen unimodal embedding pair (Qwen3-Embedding + Qwen2.5-VL encoder)? If so, what performance or resource trade-offs motivated the current design?\n4.\tWhat specifically informed the choice of Gemini models for agentic orchestration? Which other LLMs or MLLMs (e.g., GPT-4o, Qwen2.5-VL-7B) were tested or considered as policy backbones?\n5.\tThe Operational Grammar Graph (OGG) is central to the framework. Could you clarify whether its edge definitions were hand-crafted? How sensitive is ContextNav’s performance to OGG design (e.g., missing or redundant tool dependencies)? \n6.\tWere confidence intervals computed for Table 1 or Figure 3 results? \n7.\tAre there any shared patterns across ContextNav failure modes (e.g., ambiguous queries, highly compositional tasks etc.)?\n\nRebuttal suggestions:\\\n1.\tCould ContextNav’s agentic orchestration be run without proprietary Gemini models, potentially replacing with an open-weight model for the agentic reasoning steps? If performed, please report changes in: (i) semantic/structural noise rates, (ii) ICL gain (%), (iii) token cost and latency.\\\n2.\tSince the framework’s motivation is noise-robust contextualization, please test ContextNav on a noisy, web-scale dataset (some suggestions provided above. A quantification of the semantic/structural noise before and after Agentic Retrieval, and comparison of ICL gains versus VL-ICL and MMICES under the same noisy conditions would be useful.\\\n3.\tCurrent retrieval uses separate text and vision encoders. Please add results using a joint vision-language encoder.\\\n4.\tPlease add error bars or confidence intervals for key metrics (Table 1, Fig. 3, and Fig. 4). \\\n5.\tPlease provide a small figure or table showing ICL gain vs. token cost / latency per model (Phi-3.5V, Qwen2.5-VL, Gemini-2.0-flash, GPT-4o). This will help gauge efficiency and practical deployment feasibility.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Originality:\n- The paper introduces the first agentic framework for multimodal ICL that unifies retrieval, noise filtering, and structural alignment under a closed-loop, graph-driven orchestration scheme.\n- The work also broadens the definition of data-centric ICL by explicitly addressing semantic and structural noise in retrieved support examples.\n2. Quality (technical soundness and empirical rigor):\n- The methodology is well-grounded and reproducible, with explicit mathematical formulations for each module (embeddings, retrieval, agentic filtering, structural alignment, and feedback updates).\n- Experiments are extensive and systematic, spanning diverse datasets (e.g., BlindTest, CLEVR, TextOCR, MathVision) and models (both open and closed weights).\n- The ablation studies convincingly isolate the contribution of each component (Agentic Retrieval, Structural Alignment, and OGG-guided orchestration).\n3. Clarity:\n- The framework is well-presented, with clean notation, functional forms, and cross-referenced equations that make the workflow easy to follow.\n- The appendices (especially the tool library, OGG edges, and prompt templates) greatly enhance transparency and reproducibility.\n4. Significance:\n- The research makes a substantive contribution to the evolution of multimodal ICL, demonstrating that agentic data curation can unlock additional performance from foundation models without architectural changes or retraining.\n- By showing that careful control over context quality can yield consistent gains across a variety of MLLMs, the paper provides both scientific insight and practical implications for ICL advancement.", "weaknesses": "1.\tPartial dependence on proprietary MLLMs for agentic reasoning:\nAlthough the evaluation includes several open-source target models (e.g., Qwen2.5-VL-7B, Phi-3.5-V, InternLM-X2.5), the framework’s agentic orchestration relies primarily on proprietary Gemini models. This limits full reproducibility of the agentic pipeline (especially for researchers without access to the relevant APIs) and makes it difficult to assess whether smaller or open-weight policies could effectively substitute as the reasoning backbone.\n\n2.\tLimited inclusion of real-world, noisy multimodal benchmarks:\nThe evaluation focuses primarily on curated or synthetic datasets (e.g., CLEVR, TextOCR, MathVision), which, while controlled, may not capture the heterogeneity and noise found in large-scale real-world multimodal corpora. For a framework explicitly designed to perform noise-robust agentic retrieval, it would be compelling to demonstrate performance on web-scale datasets such as LAION-5B, Conceptual Captions (CC12M), or RedCaps, which contain substantial captioning noise, visual ambiguity, and cross-modal misalignment. Evaluating ContextNav under such conditions would more convincingly validate its claimed advantages in semantic filtering and structural alignment for real-world multimodal reasoning.\n\n3.\tUse of separate unimodal embedding models for multimodal retrieval:\nThe framework employs distinct text and vision embedding models instead of a unified multimodal encoder. Although modular, this design may weaken cross-modal semantic coherence. Using joint multimodal embeddings (e.g., CLIP/BLIP-style) could yield more consistent retrieval and lessen the need for post-hoc structural alignment.\n\n4.\tLack of ablations on orchestration dynamics:\nThe ablations presented effectively isolate each component’s contribution, but they do not analyze how different toolchain paths or OGG planning strategies affect results. A deeper look into orchestration dynamics would clarify how the agent’s reasoning evolves.\n\n5.\tLack of uncertainty quantification or statistical robustness analysis:\nReported results omit error bars, confidence intervals, or variance measures, even though stochasticity exists in retrieval and model inference. Statistical reporting would help confirm that the observed improvements are consistent and reproducible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ojuiTLr1s", "reviewer": "Reviewer_zbjE", "summary": "The paper presents ContextNav, a framework for multimodal ICL that selecting demonstration examples via an agentic workflow. Key steps include semantical denoising through agentic coherence checking and structural realignment to query format. Empirical results on composite and single-task VL benchmarks across multiple MLLMs show gains over baselines.", "questions": "1. Figure 2 seems an important figure for the paper. It might be good to put it in Page 1 or 2 (instead of Page 4).\n2. The term \"Noise-Robust Contextualization\" is not intuitive. Can we use simpler terms like \"Example Refinement/Selection\" or \"Context Denoising\"? Please try not to introduce too many new terms which are not intuitive to understand.\n3. [Line 283]  Also, the sample size of 803 for BlindTest seems arbitary. Is there any reason for this number?\n4. Why random sampling leads to negative ICL gain compared with vanilla MLLM? Adding examples even random sampled examples should help?\n5. Does vanilla MLLM means zero-shot? If yes, \"zero-shot\" seems a more common term.\n6. How are semantic and structural noise proportions computed? Which prompt in Appendix D is used, respectively?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The integration of agentic AI for multimodal ICL is interesting and timely. Using agentic workflows to improve demonstration examples is reasonable. \n2. The use of benchmark datasets across broad tasks and different families of MLLMs (including both open and close models) is good.", "weaknesses": "1. There is one important baseline missing: many-shot ICL (https://arxiv.org/abs/2405.09798v2). Instead of careful curation, it's important to see how much gain we can have compared with including all initial candidates in the context. The decision to limit to 8-shot is not well justified. \n2. Besides VL-ICL, Cache of Thought (https://arxiv.org/abs/2502.20587) should also be included as a baseline.\n3. The introduction of term \"Operational Grammar Graph\" seems unnecessary, making the paper harder to understand. It's not explained until very late (Page 5) although it has been referred 3 times before. I'll recommend not introducing new terms or at least providing a clear explanation at an early stage.\n4. The resampling of datasets for test is not justified. The use of 3:7 difficulty ratio also sounds arbitary, making the comparison with other methods harder.\n5. The memory and toolchain optimization is not explained clearly. How does it help? The results also do not show the benefit of this learning.\n6. The validity of structural alignment (whether semantically equivalent) is not shown experimentally.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DtjNeFAAmG", "reviewer": "Reviewer_q6J8", "summary": "This paper proposes ContextNav, an agent-based framework that helps multimodal large language models learn more effectively from examples in context. Instead of simply retrieving similar samples, ContextNav acts like a curator that selects, filters, and reorganizes examples to reduce noise and improve relevance. It combines automated retrieval with human-like reasoning through three modules: 1. agentic context management to build and update a multimodal database, 2. noise-robust contextualization to remove irrelevant or inconsistent samples, and 3. graph-driven tool orchestration to plan and optimize the workflow based on feedback. Experiments across several vision–language benchmarks show that ContextNav consistently boosts in-context learning performance.", "questions": "I’m not suggesting additional experiments, but I’m curious: do you have any thoughts on how RL could be integrated into the proposed agent framework to reduce overhead or further improve the agent’s performance?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well organized and clearly written, making it easy to follow.\n2. Multimodal in-context learning is a relatively new research area, and the role of agent-based methods in this domain has been largely unexplored.\n3. Experimental results across multiple datasets demonstrate that the proposed approach achieves consistent and notable improvements over baseline methods.", "weaknesses": "1. My main concern is that the paper primarily focuses on workflow construction, leveraging LLMs and VLMs to build a pipeline for data collection, embedding generation, dataset management, and demonstration selection. While this represents significant engineering effort, the work places less emphasis on understanding the underlying mechanisms of multimodal in-context learning itself. The research mainly centers on how to select the most useful examples for the MLLM, but multimodal ICL faces a deeper fundamental challenge, such as sensitivity to example choice and order, which I think requires theoretical or empirical analysis beyond pipeline design. Without addressing these core issues, the proposed system may only mitigate rather than resolve the inherent limitations of ICL. I encourage the authors to complement their strong engineering contribution with deeper insights into the nature of multimodal ICL.\n\n2. The agentic pipeline introduces additional inference steps and latency (around 3 seconds per iteration), which may limit scalability or real-time applicability. Moreover, as mentioned earlier, while the use of an agent can mitigate some issues in multimodal in-context learning, it does not fundamentally solve them. To truly address these challenges, future work may need to incorporate a learning-based or adaptive optimization component that allows the system to internalize and improve its contextualization strategy rather than relying solely on procedural orchestration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PD2FtR7JZJ", "reviewer": "Reviewer_tPU8", "summary": "This paper proposes ContextNav, a novel Chain-of-Thought (CoT)-driven navigation framework designed for multimodal embodied question answering (Embodied QA). The core idea is to augment a large multimodal model (LMM) with an explicit reasoning chain that can be dynamically adapted to changing visual and textual contexts.\n\nThe authors introduce a new benchmark called CEN (Contextual Embodied Navigation), which requires agentic navigation in 3D environments (based on AI2-THOR) conditioned on both linguistic queries and evolving visual observations. ContextNav is shown to outperform baselines such as VLN-BERT, RecCoT, and classical map-based agents, with analysis showing improvements in long-horizon reasoning and policy transferability.", "questions": "Could you provide an ablation where you disable the memory module? How much does it actually contribute compared to CoT alone?Which specific LMM is used, and did you try smaller or open-source models? How much does performance depend on having a “strong” LMM?\n\nAre there known cases where the CoT causes the agent to make wrong or speculative moves due to hallucinated beliefs? How long does one inference step take? Can this be deployed in a near real-time scenario? Could the benchmark (CEN) be extended to tasks beyond navigation—like multi-room exploration, tool use, or question answering with object manipulation?\n\nWould you consider releasing pseudocode or a Colab to help others reproduce the results? If these questions are addressed in detail, especially around memory vs. CoT contribution and scalability, I’d definitely consider raising my score.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper focuses on something quite underexplored—how to make vision-language agents actually reason and act over time, rather than just respond to single inputs. The focus on embodied QA with CoT-style reasoning feels fresh and highly relevant.\n\nMost CoT papers just use reasoning chains to get better answers. Here, it’s used to drive navigation—deciding where to go next, based on what’s already seen and reasoned about. That’s a neat twist on the usual setup.\n\nI like that the authors didn’t just propose a model—they also introduced a new benchmark (CEN) that requires both visual and linguistic context to solve. This feels realistic and useful to the broader community.\n\nThe method outperforms several strong baselines (VLN-BERT, RecCoT, etc.), especially on long-horizon tasks. The gains are consistent, and not just on cherry-picked examples.\n\nThe paper includes helpful qualitative visualizations of how CoT steps evolve, how memory is used, and what decisions are made. This makes the agent’s reasoning process transparent and auditable.", "weaknesses": "The paper combines CoT + memory, but there’s no solid ablation to tell which part really matters. It would help to show performance with and without the memory module.\n\nSince the system heavily relies on the underlying vision-language model (e.g., for perception, grounding, reasoning), it’d be helpful to test or at least discuss what happens if you switch to a smaller or weaker LMM.\n\nSome examples suggest that the model “imagines” object positions it hasn’t actually seen. Are there ways to ground the reasoning better or prevent hallucinations?\n\nCoT is generated at every step, which might be slow in real-world settings. But there’s no mention of how fast inference is, or whether it can run in real-time. It’s a bit hard to reproduce the method without more info. There’s no pseudocode, no clear prompt templates, and no info on model size, inference cost, or environment config. Some of this could go in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)