Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs
Authors: Yumin Choi
Dongki Kim
Jinheon Baek
Sung Ju Hwang
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Large Language Models
Multimodal Prompt Optimization
Prompt Optimization
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce the problem of multimodal prompt optimization and propose the multimodal prompt optimizer, to harness the full capacity of multimodal large language models beyond text.

Abstract: Large Language Models (LLMs) have shown remarkable success, and their multimodal expansions (MLLMs) further unlock capabilities spanning images, videos, and other modalities beyond text. However, despite this shift, prompt optimization approaches, designed to reduce the burden of manual prompt crafting while maximizing performance, remain confined to text, ultimately limiting the full potential of MLLMs. Motivated by this gap, we introduce the new problem of multimodal prompt optimization, which expands the prior definition of prompt optimization to the multimodal space defined by the pairs of textual and non-textual prompts. To tackle this problem, we then propose the Multimodal Prompt Optimizer (MPO), a unified framework that not only performs the joint optimization of multimodal prompts through alignment-preserving updates but also guides the selection process of candidate prompts by leveraging earlier evaluations as priors in a Bayesian-based selection strategy. Through extensive experiments across diverse modalities that go beyond text, such as images, videos, and even molecules, we demonstrate that MPO outperforms leading text-only optimization methods, establishing multimodal prompt optimization as a crucial step to realizing the potential of MLLMs.

Published as a conference paper at ICLR 2026
MULTIMODAL PROMPT OPTIMIZATION: WHY NOT LEVERAGE MULTIPLE MODALITIES FOR MLLMS
Yumin Choi
Dongki Kim
Jinheon Baek
Sung Ju Hwang
KAIST 'DeepAuto.ai
(yuminchei, cleverki, jinheon Sanheonbbak,, hangguu.hanngekkist.coo
ABSTRACT
Large Language Models (LLMs) have shown remarkable success, and their multi- modal expansions (MLLMs) further unlock capabilities spanning images, videos, and other modalities beyond text. However, despite this shift, prompt optimization approaches, designed to reduce the burden of manual prompt crafting while maxi- mizing performance, remain confined to text, ultimately limiting the full potential of MLLMs. Motivated by this gap, introduce the new problem of multimodal prompt optimization, which expands the prior definition of prompt optimization to the multimodal space defined by the pairs of textual and non-textual prompts. To tackle this problem, we then propose the Multimodal Prompt Optimizer (MPO), unified framework that not only performs the joint optimization of multimodal prompts through nignment-preservee cess of candidate prompts by leveraging earlier evaluations as priors in Bayesian- based selection strategy. Through extensive experiments across diverse modalities that go beyond text, such as images, videos, and even molecules, we demonstrate that MPO outperforms leading text-only optimization methods, establishing mul- timodal prompt optimization as crucial step to realizing the potential of MLLMs.
updates but also guides the selection pro-
INTRODUCTION
Large Language Models (LLMs) have demonstrated outstanding capabilities across a diverse range of tasks and domains (OpenAI, 2024; Grattafiori et al., 2024; Yang al.. 2025). We note that a central factor in unlocking their full potential lies in the design of prompts, which directly influ- ence model performance. However, crafting high-quality prompts often labor-intensive and iterative process that requires substantial human intervention. To address this limitation, the field of Automatic Prompt Optimization (APO) has emerged, whose goal is to automate the discovery of effective prompts (Pryzant al., 2023; Ramnath al., 2025; Cui al., 2025a; Li et al., 2025). et al. (2023) frames this challenge as an iterative search problem, where at each
For example, step, set of new candidate prompts is generated updated, evaluated on target task, and the best-perforning prompts are selected to guide the next round of generation.
Recently, top of the I.Mss, Multimodal Large Language Models (MLLMs) have been proposed, which process not only text but also images, videos, and other modalities (such as molecules) (Kim al., 2025; Zheng et al., 2024; Song al., 2025). Yet, despite these advances and their wide-ranging applications, existing prompt optimization methods remain restricted to the textual modality (Choi al., 2025; Fernando al., 2024; Cui al., 2025b), and overlook the richer expressive capacity af- forded by multimodal inputs (that text alone cannot capture). For instance, as illustrated in Figure 1, describing the distinct characteristics of a specific bird may require long and potentially ambiguous text, while single image can convey the same information far more directly. By limiting optimiza- tion to text, existing methods are prone to generating less effective, suboptimal prompts that fail to fully exploit the multimodal space that MLLMs are inherently capable of leveraging.
Motivated by this limitation, we first define the novel problem of multimodal prompt optimization', which expands the prompt optimization space beyond text to incorporate multiple modalities. How-
Equual Contribution; Equul Advising; Code is available at http://gitub.com/Doo0l cow//oziO0//PPP We define multimodal prompt a pair of discrete textual and non-textual prompts.

---

Published as a conference paper at ICLR 2026
(A) Text Prompt Optimization
(B) Multimodal Prompt Optimization (Ours)
Optimized Teat Prompt:
Query:
Optimized Multinodal Prompt (Optimised) Image
Qeery: (allowing diava
Optimiaation Prosess
Choris iy the lend in be temgo.
[Optimizad]
All ar North Pactfii..
Clouily leat teieego
Specificltyy byy
whixc
Textual
fasid serxtfe is pakcott
Unimodal (Testual) Space
Tlack Soctadiirs
Answer:
Anewer:
the plarr appean Shary
Multimodal
The pioro cur idterflad
plereago
Figure 1: Concept Figure. (A) Existing prompt optimization approaches restrict the optimization to the textual space, leaving MLLMs underutilized by failing to provide rich contextual signals. (B) Our multimodal prompt optimization expands the optimization space into multimodality, allowing the discovery of salient multimodal context and fully leveraging the expressive capacity of MILMs.
ever, while this expanded space opens new opportunities, it also introduces a couple of challenges for automatic optimization. First, exploring the larger, combinatorial space of multimodal prompts requires a prompt update strategy that can efficiently navigate candidate prompts while maintaining cross-modal consistency. Furthermore, selecting promising candidates becomes substantially more difficult, as the enlarged search space makes optimal prompts increasingly sparse, given the need to account for both the effecctiveness within each modality and the alignment across modalities, which, in turn, calls for evaluation strategies that are both efficient and accurate.
To address these challenges, we propose Multimodal Prompt Optimizer (MPO), unified frame- work for optimizing prompts across both the textual and non-textual modalities, which consists of the two key components: (i) siignment-preservinn
exploration and (ii) prior- mooo--nheeittnce-aeee based selec- tion. Specifically, for exploration, the proposed MPO jointly updates the textual prompt, as well as its associated non-textual counterparts by generating instructions to create (or revise) the non-textual components of the multimodal prompt (unlike prior approaches that refine only text), and notably, their updates are guided by the single semantic gradient (i.e., feedback) to ensure their alignment derived from the failure analysis of the current prompt. Moreover, these updates are further diversi- fied through complementary operations, namely generation, editing, and mixing, to ensure the broad and expressive exploration of the multimodal prompt space. Then, building this exploration with multiple candidate prompts updated, MPO leverages the prior-inherited Bayesian- UCB as a prompt selection strategy, which utilizes the performance score of parent prompts as a prior (unlike conven- tional approaches that treat each candidate independently), to reliably identify the high-performing prompts by biasing the selection process toward more promising regions of the multimodal space.
To validate MPO, we conduct extensive experiments benchmarking it against leading text-only opti- mization methods across 10 datasets, and our evaluation suite spans not only images and videos but also molecular structures, ensuring broad coverage of diverse modalities. Across all domains, MPO demonstrate consistent and significant performance gains, empirically confirming our core hypothe- sis: expanding the prompt search space into the multimodal domain is crucial to exploit the expanded capacity of MLLMs. Further analyses show the efficacy of MPO components: exploration with complementary operators facilitates the discovery of optimal multimodal prompts by not only ensuring cross- modal consistency but also thoroughly probing the search space; and the prior-inherited Bayesian-UCB accurately and efficiently selects evaluation budget by 42% compared with a prior- free baseline. These results highlight MPO as an effective framework for optimizing multimodal prompts, unlocking the full capabilities of MLLMs.
prompts, reducing
2 RELATED WORK
Multimodal Large Language Models The development of MILMs has significantly extended the capabilities of traditional LLMs by enabling them to process and reason over diverse non- textual modalities, including images, videos, audio, and more (Liu et al., 2023; Chu et al., 2023; M. Bran et al., 2024). In particular, these models are typically trained through large-scale multi- modal pre-training, which aligns poodality-speciffe encoders (e.g., vision or audio) with LLM back- bones, followed by post-training stages such as supervised fine-tuning and preference optimization to endow them with multimodal mnstruction-followiin abilities (Gemini, 2025; Bai et al., 2025; Zhu et al., 2025). Moreover, leveraging these capabilities, MILMMs have achieved strong performance
2

---

Published as a conference paper at ICLR 2026
(A) Alignmot-Preserving Exploration
(B) Prior-Inherited Bayesian UCB Selection
Testual Prompt
Textual
(A-1) Generation Operator
Informative Prior
Uninformative Prior
Non-Textual Prompt
Non-Textual Prompl
4ym
DC
m
Oreecs
(A.2) Edit Operator
Fallure Bet
Tectual Feedback
D
lakert Price Parasfls Performance
Fedtt
Warm
Cold Start
Operator
Textual Prompt
DC
Updated Textual Prompt
High-Parforming Prompt
Textual Feedback (P,Fa)
Textual Condition e
Casts
Rossiiysppccte
Updated Non Textual Promptar
UCB
Cohosive Backgropagation
Joint Multimodal Update
Iritializaiion
Select Top-k
Overview MPO, consisting of two components. (A) Alignment- prreeentttpeeeinnggg exploration analyzes a failure set to generate feedback, which is then used both to refine the textual prompt and to guide a mopelity-specific generator to create new non-textual prompt with one of three operators. (B) Prior-Inherited Bayesian UCB Selection leverages the parent's performance as an informative prior, warm-starting the search to effectively identify high-performing prompts among candidates.
Figure 2:
on broad range of tasks, from foundational ones such as classification and captioning, to domain- specific, high-stakes applications such as medical image question answering and pharmacological property prediction (Martin et al., 2019; Liu et al., 2021; Corbière et al., 2025; Huang et al., 2021).
Automatic Prompt Optimization To reduce the burden of manual prompt engineering and sys- tematically uncover the effective prompts, the field of Automatic Prompt Optimization (APO) has emerged. Existing works can be broadly categorized into two paradigms. The first is gradient-bases optimization, which learns continuous embedding vectors (i.e., soft prompts) that are prepended to model inputs to steer behavior (Khattak et al., 2023; Zeng et al., 2024; Wang et al., 2024). Yet, while effective, they are computationally costly, yield uninterpretable numerical vectors, and restricted to open-source models with accessible parameters. To overcome these drawbacks, gradient- free approaches have been proposed, which iteratively generate, evaluate, and refine candidate prompts using LLMs themselves (Zhou et al., 2023; Yang et al., 2024). Also, some recent works enhance this process by analyzing prompt failures guide improvements (Khattab al., 2024; Ye et al., 2024; Cui et al., 2025b; Yuksekgonul et al., 2025), while others borrow ideas from evolutionary algorithms (e.g., mutation and crossover) to explore the prompt space (Guo et al., 2024; Fernando et al., 2024). Despite this progress, current APO techniques are limited to text-only settings, restricting optimiza- tion to purely linguistic information. In contrast, our work expands prompt optimization into the multimodal domain, enabling the prompt discovery that fully exploits the capabilities of MLLMs.
Ipstance-Specific Prompting and Optimization Distinct from task-level prompt optimization, another line of research focuses instance-specifie prompting strategies that operate at inference time to enhance reasoning on a per-query basis. For example, MM-CoT (Zhang et al., 2024b) guides the model generate intermediate textual rationale before producing the final answer. Also, other methods augment visual inputs with query-dependent signals, such as bounding boxes or points, guide attention toward relevant regions of image (Zhou al., 2024; Jiang et al., 2024; Lin et al., 2024). Similar ideas have been explored in text-to-image and text-to-video gener- ation, where prompts are crafted and refined to produce outputs more faithfully aligned with user intent (Mañas et al., 2024; Mo et al.. 2024; Gao et al., 2025). However, these techniques are query- specific, designed improve model performance for single instance a time. By contrast, APO pursues a different objective: discovering single, reusable prompt that boosts performance across an entire task, and our work advances this paradigm by extending into the multimodal domain.
METHODOLOGY: MULTIMODAL PROMPT OPTIMIZER
We present Multimodal Prompt Optimizer (MPO), composed of two modules: alignment- preseevvnn exploration of multimodal prompt space and prompt selection with prior-inherited Bayesian UCB.
3.1 PROBLEM DEFINITION
We begin by formally describing MLLMs and then proposing a novel problem of multimodal prompt optimization, which redefines and expands the notion of existing prompt optimization beyond text.
3

---

Published as a conference paper at ICLR 2026
Multimodal Large Language Models Multimodal Large Language Models (MLLMs) extend the capabilities of LLMs by processing inputs that combine text with non-textual modalities. Formally, an MLLM can be represented the textual input space, M denotes the non-textual input space, and x denotes the Kleene Star (rep- resenting finite sequence
parametric function MLLM (TU + T, where T denotes
the combined spaces). In other words, given a multimodal query q and a prompt p (each potentially containing both textual and non-textual components), the model generates a textual output y MLLM(p, q). is worth noting that prior work on prompt optimization has generally restricted p to a purely textual form (p t E T), leaving the non-textual dimensions of M unused. This restriction underutilizes the expressive capacity of MLLMs and fails to provide richer contextual signals that are often crucial for real-world multimodal tasks (See Figure 1).
Multimodal Prompt Optimization Building on the expanded space of MILMs, we extend the notion of a prompt for optimization from text-only to multimodal. Specifically, we define a multi- modal prompt pair p (t, m)) E T x x,, where t is the textual prompt and m is the non-textual prompt. Then, given a task dataset D consisting of query-answer pairs (q, a), the objective of multi- modal prompt optimization is to discover the optimal prompt (t", mm) that maximizes performance:
(t", mm' =
argmax (t.m)ETxM
E(q.a)> [ (MLLM(+, m, q), a)):
where f is function for a task-specific evaluation metric, such as accuracy or Fl scores.
Notably, compared to optimizing only textual prompts, the joint search space T M introduces an entirely new axis of non-textual information, which in turn raises two fundamental challenges. First, multimodal prompts must maintain cross-modal consistency: textual and non-textual components should provide complementary. not conflicting signals; however, expanding to the combinatorial space greatly increases the risk of semantic misalignment. Second, the enlarged space amplifies the difficulty of candidate selection: high-quality prompts become sparse, and low-quality prompts dominate, making harder to efficiently identify promising candidates. overcome these, we now explain the proposed multimodal prompt optimizer, designed to navigate this enlarged space below.
3.2 ALIGNMENT-
EXPLORATION OF MULTIMODAL PROMPT SPACE
The first challenge in multimodal prompt optimization lies exploring the enlarged search space while preserving semantic consistency across modalities; thus, naive approach that independently updates textual and non-textual components risks producing misaligned prompts, where one modal- ity contradicts the other. To tackle this, we introduce an exploration framework that couples the update of textual and non-textual prompts while supporting diverse operations (Figure 2).
Joint Optimization of Multimodal Prompt Our MPO jointly updates the textual and non-textual prompts to ensure that both evolve coherently, achieved through the following two mechanisms:
Cohesive Backpropagation. We begin by identifying a failure set F = {(q, a, y) y + a} for multimodal prompt p (t, m). Instead of treating errors separately for text and non-textual inputs, we then generate a unified feedback Vp = (Ve, V. = MLLM(t, F)F), which encodes cross-modal weaknesses in textual form. By doing so, we obtain the single supervisory signal that guides both modalities simultaneously, mitigating the risk of overfitting updates to one modality.
Joint Multimodal Update. Using the feedback, MPO jointly refines the textual prompt while de- riving modality conditions (in the textual form) that direct non-textual revisions. Specifi- cally, the MILM produces an updated textual prompt t' and further modality-specific condition e. describing how the non-textual prompt should adapt: (t', c) = MLLM(t, m; F, Vp). The condi- tion e is then passed to modality-speeific generators g (such as text-to-image or texl-to-molecule modules), which yield updated non-textual prompts m = g(c). This guarantees that updates to m remain consistent with the revised textual prompt t', rather than being optimized in isolation.
For the optimization process, we adopt the beam search. Specifically, at each iteration, we select top- 6 best-performing multimdoal prompts {p. (t,, m.)) and then apply the cohesive backpropa- gation and the joint multimodal update to generate new b² multimodal prompts {p', = (t',. m3) 1 from the top-ô multimodal prompts p. We refer to p and p' as parent and child prompts, respectively.
4

---

Published as a conference paper at ICLR 2026
Exploration Operators Ensuring that generated outputs remain consistent with the guiding tex- tual conditions is a necessary baseline, and effective optimization further requires g that actively ex- plores diverse regions of the multimodal space. To achieve this, design three operators (namely, generation, edit, and mix), which systematically expand, refine, and recombine non-textual prompts.
Generation operator. This operator explores entirely new non-textual prompts, e.g.. novel spatial arrangements visual inputs or unique substructures in molecules. Specifically, conditioned only on the generation signal Cgen. it creates prompt from scratch without referencing prior candidates:
m ((Cgn Ø), where (Cgen t') MLLM(L, m, F..
By decoupling from past candidates, it explores unexplored regions and avoids local optima, espe- cially in early stages (where initial prompts are unavailable) or when the candidate pool biased.
Edit operator. This operator performs fine- tne-grained refinements of non-textual prompts (e.g.. tex- tures) while retaining useful structures from the prior prompt. Specifically, given the edit condition Cedit. the update is performed by conditioning on the prior non-textual prompt:
m = g Codit {m)} where (Codit, t') = MLLM( mæ; F).
This enables targeted, incremental refinements, making it particularly effective when prompt is already strong but requires adjustment on specific attributes rather than a complete redesign.
Mix operator. This operator blends the complementary strengths of multiple multimodal prompts. Specifically, it first leverages feedback from multiple prompts to generate mixing condition Cmix. which is then used by the generator to combine non-textual prompts as follows:
m' g( Cmmix {m, ). where (Cmix. t') MLLM((t; m.; V...
By synthesizing multiple candidates, it yields balanced compositions, avoids over-reliance on single candidate, and enables exploration of intermediate solutions better than individual ones.
is worth noting that, for the generation and edit operators, we randomly select one parent prompt from top-b multimodal prompts of the previous iteration generate child prompt (i.e., p p'), while K parent prompts are selected for the mix operator (i.e., { "P.,,....''' + p').
3.3 EFFECTIVE PROMPT SELECTION BY PRIOR-INIERITED BAYESIAN UCB
Another challenge in multimodal prompt optimization to identify which candidates should be prioritized for evaluation and carried forward (Pryzant et al., 2023; Shi et al., 2024). Yet, this step is non-trivial with the enlarged multimodal space, since high-quality prompts become relatively sparse, and large portion of the evaluation budget risks being wasted on low-potential candidates. Existing approaches typically adopt either (i) uniform allocation. where each candidate is evaluated equally regardless of its prior likelihood of success (Zhou al., 2023; Cui et al., 2025b), or (ii) Sandit-based allocation, such as UCB (Auer, 2002; Bouneffouf & Féraud, 2025; Li et al., 2026; Ashizawa et al., 2025), which adaptively balances exploration and exploitation. However, both paradigms suffer from an inefficient cold-start problem: newly generated prompts are treated as independent arms with no prior information, leading unproductive evaluations the early rounds.
Parent-Child Correlation We address this cold-start inefficiency by introducing informative priors that warm-start the evaluation process. In particular, hypothesis is that the performance of a parent prompt is positively correlated with that of its children. To test this, we analyze the
Carr n.58 Paaler
Arg clald Prosgg
optimization trajectory, measuring the correlation between the perfor- mance of parent prompts and the average performance of their children. As shown in Figure 3, observe strong positive correlation (Pear- son's r = 0.88), providing concrete evidence that parent scores could serve as highly informative priors for estimating child performance.
Parcat Prosigi Score
Figure 3: Correlation of parent and child scores.
Prier-Inherited Bayesian UCB Motivated by this finding, propose inher-id Bayesian UCB (Kaufmann et al., 2012), a selection strategy that initializes the score distribution of a new child prompt based on the posterior of its parent (rather than uniform). Specifically, model the
5

---

Published as a conference paper at ICLR 2026
expected score of each multimodal prompt p; Beta distribution, Beta(a, Bi), where ag and 3; correspond to (pseudo-) counts of successful and failure outcomes, respectively. Then, for a child prompt p; originated from a parent prompt Ppar(t) we initialize its prior proportionally to the posterior mean performance of the parent (paz((), scaled by a prior strength hyperparameter S > O, formalized follows:
(paa(())
(ppr(() - 1,1, 8, =
(1- (1-Apar(t) SS+ 1, where (paz(() apar(() + Spar(1)
(1)
This prior-inherited mechanism provides S pscudo-observations newly generated child prompts, effectively warm-starting the evaluation process. With a fixed total budget, it then proceeds itera- tively: at each round, we select the prompt with the highest UCB score (an upper quantile of its Beta posterior), evaluate it on small batch of data, and update its posterior parameters a; and 3, Once the budget is exhausted, the candidate prompt with the highest expected score is selected as the new parent for the next iteration of optimization. Please refer to Algorithm 2 for the complete procedure. The following proposition guarantees that our proposed selection strategy leverages an informative parent prior (better than random chance) to accelerate the selection of the best-pre pest-promising prompt.
Proposition 3.1. (Fewer Pulls via Inheriedd Bayesian UCB) With the prior of Equation 1, and if the prior is more informative than uniform (E, [d(p), Ppar(i)) d(pu, b)] S 0), the best-arm identification cost of Bayesian UCB nonincreasing, where d(p,g) 4) the Bernoulli KL divergence.
The proof and detailed analysis are provided in Appendix B. Intuitively, this guarantee demonstrates that informative parent priors accelerate the discovery of high-quality prompts by reducing wasted evaluations low-potential candidates, which is particularly beneficial for multimodal prompt op- timization, where the combinatorial search space is far larger than text-only settings. In other words, by rapidly eliminating unpromising candidates and reallocating the budget toward more promising regions, our method enables efficient exploration of the vast multimodal prompt landscape.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets We conduct extensive evaluation on MPO across diverse set of modalities, including images, videos, and molecules. For the image modality, we consider both image classification and visual question answering (VQA) tasks. Specifically, we use PlantVillage (Mohanty et al., 2016) for diseased Ieaf identification and CUB-200-2011 (Wah et al., 2011) 1) for fine-grained bird classification; meanwhile, for VQA, we evaluate on SLAKE (Liu et al., 2021), RSVQA (Lobry et al., 2020), and D (Corbière et al., 2025), which cover radiology, remote sensing, and dynamic driving scenes, respectively. For the video modality, we evaluate on Drive& Act (Martin et al., 2019) for driver action recognition and VANE-Bench (Gani et al., 2025) for abnormality detection in video- based VQA. Finally, for the molecular modality, we include three different property prediction tasks from TDC (Huang et al., 2021), namely. Absorption (Hou et al., 2007; Ma et al., 2008; Broccatelli et al., 2011; Siramshetty et al., 2021), 1), BBBP (Martins et al., 2012), and CYP inhibition tasks (Veith al., 2009). Detailed configurations for each dataset provided in Appendix A.I.
Baselines We benchmark MPO against both manually designed prompts and representative auto- matic prompt optimization methods. For manual prompting, we include Human, simple hand- crafted prompt, Chain-of-Thought (CoT) (Wei et al., 2022), which uses the widely adopted phrase "Let's think step by step." and Few-Shot, which supplies in-context examples drawn from the train- ing data. For automatic methods, we compare against leading LLM-based text-only optimizers, including APE (Zhou et al., 2023), OPRO (Yang et al., 2024), EvoPrompt (Guo et al., 2024), PE2 (Ye et al., 2024), ProTeGi (Pryzant et al., 2023), and SEE (Cui et al., 2025b). Detailed de- scriptions of all baselines are provided in Appendix A.2.
Implementation Details For answer generation, we use Qwen2.5-VL. (7B) (Bai et al., 2025) as the base model for image and video tasks, and Qwen3 (8B) (Yang et al., 2025) for molecular tasks. During optimization, GPT-4o mini (OpenAI, 2024) serves as the prompt optimizer, responsible for
IIf a child prompt is generated by the mix operator, we average the posterior mean performance of multiple parent prompts.

---

Published as a conference paper at ICLR 2026
Table 1: Main Results. Comparison MPO with manual prompting, few-shot prompting, and text-only APO baselines on diverse benchmarks across image, video, and molecular modalities. Results are averaged over three independent runs. denotes the average performance across multiple subtasks within the benchmark. Avg. denotes the average accuracy over all datasets except Fl.
Image
Video
Molecule
PlantVillage
SLAKE*
DrivingVQA
RSVQA
DrivedAct
VANE.
Absorption
Johibit."
Acc.
Acc.
Acc.
Acc.
Acc.
F1
Avg.
Homan
42.2
47.9
35.2
49.7
$1.0
47.3
47.0
38.5
36.3
39.4
38.6
43.1
57.1
44.1
OUT
43.1
49.0
30.8
52.9
49.6
37.2
3166
39.6
36.7
33,6
32.5
40.1
32.3
40.8
1-Shol
39.7
54.7
31.4
54.5
48.5
50.4
62.4
37.8
35.7
36.1
34.8
56.2
48.3
47.2
3-Shot
30.6
$3.9
56.
44.2
42.7
47.3
28.0
61.4
45.5
49.3
47.0
49.3
APE
55.8
34.3
52.8
64.3
45.7
40A
36.0
34.7
52.3
50.9
51.3
OPRO
51.1
59.7
33.9
52.7
$1,0
16.4
$1.0
37.6
35.4
39.2
38.3
43.0
37.1
16.9
EvoiPrompi
56.1
59.6
34.8
52.9
50.5
46.7
56.5
48.2
46.5
38.7
37.7
51.1
49.7
49.5
67.9
63.0
64.5
55.1
ProTeGi
35.4
65.5
71.1
58.2
72.1
57.0
60.0
SEE
69.0
71.6
35.0
52.2
53.4
51.7
57.9
71.4
60.0
67.0
62.3
61.4
56.7
59.1
MPO (Ours)
76.4
78.6
38.2
$6.0
55.9
58.3
71.2
76.7
64.5
75.3
67.6
64.3
60.2
65.1
Table 2: Generalizability results of MPO across components with different backbones: (Top) base models; (Bottom Left) op-
Performance Improvement (%)
60
timizer models; (Bottom Right) modality-specific generators.
Qwen2.5.VL. (72B)
Gemmu3 (12B) InternVL-335 (14B)
GPT-4.1 nano
50-
Human 55.7
45.6
51.6
46.8
66.8
56.7
46.1
40
3-shot
69.6
64.6
36.5
37.7
S-shot
72,3
68.9
34.9
42.6
30
MFO (Cund Seguential
ProteGi
74,1
68.2
71.9
61.0
73.6
68.1
61.6
Rasdon Image Proupt
MPO
80.4
73.1
73.2
65.9
20
DGD image Quary
In In-Beeribetann image Quary
Optimizer Model
SEE
MPO
T2I Generator
PlantVillage*
0.2
0.4
0.6
Alignment Score (DSG)
Qwen2.5-VL (7B)
65.2
69.1
SEE (Text-only))
Figure 4: Relationship between cross-modal aligmment and per- formance gain. We report me-
Gemini 2.5 Flash
68.2
74.8
SANAL5 (1.66)
71.8
Nano Banana
72.9
GPT-4o mini
69.0
76.4
GPT-Image-Lov
76.4
GPT-4o
69.2
78.0
GPT-Imuge-Mern Medium
76.6
dian values alongside Q1 and Q3.
analyzing failures and refining multimodal prompts. For modality-specific generation, we employ GPT- (OpenAI, 2025) for images, Wan2.1 (1.3B) (Wan et al., 2025) for videos, and again GPT-4o mini for molecules. For the implementation of the iterative optimization loop, we use the beam search (Pryzant et al., 2023) with the beam size of b 3 and the number of iterations of 13. Also, at each iteration (except the first), 82 child prompts are produced by evenly applying the generation, edit, and mix operators, after which the top-b prompts Bayesian-UCB. Meanwhile, in the first iteration, only the generation operator is used to initialize multimodal prompts, since non-textual prompts exist yet. The complete optimization process is summarized in Algorithm 1. To ensure fairness, we keep the number of explored prompts consistent across all methods. In case, each candidate prompt allocated an evaluation budget of 100, and the prior strength for our prior inheritance is set to 10% of this budget (S = 10). Reported results averaged three independent runs. Please see Appendix A.3 for additional details.
selected via prior-inherited
4.2 EXPERIMENTAL RESULTS AND ANALYSES
Main Results As shown Table MPO consistently outperforms all baselines across image, video, and molecular domains, confirming its effectiveness in discovering prompts that more effec- tively harness the capabilities of MILMMs. Specifically, compared to existing text-only optimization methods, MPO achieves substantial gains, demonstrating that incorporating non-textual signals into prompts provides stronger contextual grounding and enhances task-specific reasoning. Moreover, MPO outperforms exemplar-based Few-Shot prompting, showing that it can capture richer cross- modal information and its underlying dependencies beyond simple query query-answer answer demonstrations.
In both image and video domains, MPO performs strongly classification and QA tasks, underscor- ing its robustness across diverse real-world scenarios. Likewise, on molecular tasks, MPO surpasses all baselines, highlighting its effectiveness in highly specialized applications.

---

Published as a conference paper at ICLR 2026
Table 3: Ablation on the con- tribution of each modality the optimized multimodal prompt.
Table 4: Ablation on three explo- ration operators, utilizing each
-aw MPG (Gum))
Average Score
wis Prior
UCB
one of them individually.
75
Unifurm
12% savingg
Text
Image PlantVillage*
CUB
Apple Grape Potato
Avg.
74
SEE Generation
76.4
75.9
48.0
75.7
69.0
70% savings
Human
42.2
47.9
76.9
77.9
83.6
73.3
Human MPO
50.4
58.2
Edit
77.2
76.3
56.2
80.1
72.5
25
Evaluation Dudget per Prompt
MPO
55.6
64.2
Mix
74.0
74.8
Figure 5: Efficiency compari-
MPO
MPO
76.4
78.6
MPO (Pull)
77.7
78.2
65.9
84,0
76.4
son of selection strategies.
Gen
VOLL
Mix
Mix
Task Clasess
Figure 6: Image prompt optimization process of the pest-performing multimodal prompt on a subtask (i.e., grosbeak species classification) of CUB. "Task Classes" box contains the examples of four species: Rose Breasted Grosbeak, Pine Grosbeak, Blue Grosbeak, and Evening Grosbeak.
Generalizability to Diverse Backbone Models We further validate the generalizability of MPO by varying the backbone models used in each component, namely, base models, optimizer models, and -ppecity generators, and assessing its robustness under these variations. First, as shown in Table 2 (Top), MPO maintains strong performance across different architectures and exhibits even greater effectiveness as model size increases, for example, with Qwen2.5-VL. (72B). Also, Table (Bottom Left) further shows that MPO remains effective regardless of the optimizer model, surpass- ing ofate-of-the-art text-only methods (c.g.. SEE) under diverse backbone models for optimization. Finally, Table 2 (Bottom Right) demonstrates that MPO generalizes well to modality-speciffi gen- erators, including lightweight open-source models such as SANAI.5 (1.6B,, where it continues to outperform textual optimization methods. These results highlight MPO as a broadly generalizable and robust framework, effective across a wide variety of base models and practical scenarios.
Analysis Cross-Modal Alignment Recall that MPO uses the alignment prreeeevittrreeevnngggg exploration to jointly refine textual and non-textual components of multimodal prompts, and we further analyze how this cross-modal alignment strategy contributes performance gains. To isolate this effect, we consider four variants: (1) Sequential, where the textual prompt is optimized first and the non- textual prompt is refined afterward; (2) Random Image Prompt, where the image component is replaced with another optimized image prompt (i.e., not jointly optimized with the text); (3) In- Distribution Image Query, where is replaced with an image sampled from the same task; and (4) OOD Image Query, where it is replaced with an image sampled from different task. After that, we measure the relationship between performance gain over the Human baseline, as well as the DSG score² (Cho et al., 2024), used as standard metric for measuring cross- alignment following prior work (Mañas et al., 2024). As shown in Figure 4, MPO achieves both the highest alignment score and the largest performance gains, followed by Sequential optimization and Random Image Prompt, while In- In-Distribution and OOD Image Query lag significantly behind. These results coAn- finm that stronger cross-modal alignment directly translates to better task performance, and that seenmmnnt-preserving updates (included in MPO) are crucial in promoting modality consistency.
Ablation on Modality Contributions in Prompts To examine the contribution of each modality within optimized prompts, we ablate the textual and non-textus components from the final multi-
PSS decomposes the textual description into atomic, dependency- queries and verifies each using an MIL.M, enabling a precise assessment of whether the visual content reflects the intended textual details.
8

---

Published as a conference paper at ICLR 2026
Average Test Soore
Principal Com Component
Average Scorn
75-
76-
70-
-5
MPG iT+b
(T arso (T)
65-
MPO (Ours)
ProtoGi
-15
PToTuGi
SIE
NPG (T)
P92
Evafrompt
74-
12
-10
10
20
5
10 15 20
25
Number lerattons
Principal Component
Prior Strength (S)
Figure 7: Train Curve of MPO Figure 8: Visualization of hid-
Figure 9: Analysis of the prior
compared to ProTeGi on CUB. den states in MILMS by PCA.
strength (S) on performance.
modal prompt. As shown in Table 3, using only single modality (either MPO text without image or human text combined with MPO image) already surpasses the Human baseline, confirming that both modalities independently provide useful signals. However, the full multimodal prompt yields substantially higher performance, demonstrating that the two modalities are not merely additive but mutually reinforcing, which underscores the importance of jointly leveraging textual and non-textual information to achieve performance gains beyond what either modality can deliver alone.
Effect of Exploration Operators To assess the contribution of the proposed exploration operators (such as generation, edit, and mix), we conduct both qualitative and ablation analyses. Qualitatively, illustrated in Figure 6, we observe that each operator serves distinct role: the generation opera- tor introduces novel visual compositions, the edit operator fine-tunes local features such as textures or visual characters, and the mix operator blends broader attributes such as background or spatial layout. In addition to this, the ablation study in Table 4 further confirms their complementary ef- fects: while each operator individually improves over the baseline, combining all three within MPO leads to the best performance. This demonstrates that the proposed operators jointly enable a more comprehensive exploration of the multimodal prompt space, facilitating the discovery of the optimal prompts. We observe similar pattern in molecular prompt optimization, shown in Figure 14, with concrete examples of operator-driver updates (including textual conditions) provided in Table 8.
Selection Strategies We evaluate the effectiveness of our inheriin Bayesian UCB strategy for candidate prompt selection by comparing it against three alternatives: Uniform, which distributes the evaluation budget evenly across candidates; UCB (Auer, 2002), a standard bandit algorithm; and an ablated variant of ours w/o Prior. As shown in Figure MPO achieves the same performance as the Uniform strategy while using only 30% of the evaluation budget, yielding a 70% reduction in resource cost. Moreover, MPO consistently outperforms both UCB and w/o Prior, reaching their performance levels with 52% and 42% less budget, respectively. These results confirm that the warm start enabled by prior inheritance is crucial for both efficiency and accuracy, allowing MPO to scale effectively over the enlarged multimodal search space and reliably identify high-quality prompts.
Train Dynamics of MPO To better understand how MPO improves over the course of optimiza- tion, we analyze its training dynamics in comparison to ProTeGi by tracking the test performance of the top-1 prompt the CUB dataset. As shown in Figure 7, both methods improve during the early iterations; however, proteGGi quickly plateaus after the third iteration, with only a marginal additional gain of 1.1 points. In contrast, MPO continues to improve steadily, ultimately achieving a much higher final score, including an additional 6.4-point gain beyond the third iteration. This comparison result highlights that MPO effectively overcomes the performance ceiling of text-only optimization methods by effectively navigating the multimodal prompt space, enabling it to escape local optima (imposed by the text-only strategy) and discover prompts closer the global optimum.
Hidden State Visualization To gain deeper insight into why optimized multimodal prompts yield greater performance improvements than text-only prompts, we visualize the hidden state of MLLMs by averaging intermediate- embeddings, following Zhang et al. (2024a). As shown in Figure 8, hidden states obtained from text-only methods (including the text-only component of MPO) cluster together, suggesting that they guide the reasoning of MLLMs within a similar yet limited semantic space. In contrast, the full multimodal prompt from MPO shifts the hidden states into distinct
9

---

Published as a conference paper at ICLR 2026
gion, indicating that the non-textual component introduces information unavailable from text alone. In other words, the multimodal prompt alters the internal representation space of models, enabling richer reasoning pathways and ultimately leading to superior task performance.
Analysis of Prior Strength Recall that in our prior-inherited selection strategy, the prior strength S determines the number of pseudo-observatioon used to initialize the score distributions of child prompts, and we study its effect by varying S and reporting the resulting performance. As shown in Figure 9, we first observe that a small S under-utilizes the parent prior, resulting in weaker guidance and suboptimal performance. In contrast, an excessively large S causes the model to over-rely on the parent prior, limiting its ability to adapt to the actual performance of child prompts. Consequently, the performance is maximized at an intermediate S, where inherited knowledge provides strong warm start while still allowing sufficient flexibility to incorporate new observations.
Qualitative Result We provide qualitative examples for the optimized multimodal prompts for the image modality in Table 9 of Appendix. From this, we observe that the optimized multimodal prompts consistently supply task-critical context in both textual and visual forms. Also, more impor- tantly, the textual prompts explicitly instruct the model to leverage non-textual signals (e.g., Use the hybrid reference image for guidance), thereby unlocking the full multimodal capacity of MILMs. Additional examples for the video and molecular modalities are presented in Tables 10, 11, and 12.
5 CONCLUSION
We introduced the novel problem of multimodal prompt optimization, extending the optimization space beyond text to fully leverage the capability of MLLMs. To tackle this, we proposed the Mul- timodal Prompt Optimizer (MPO), a unified framework that jointly refines textual and non-textual components through peemmennt-reseeree
exploration with multiple generation operations and ef- ficiently identifies high-quality prompts via prior-inherited Bayesian UCB strategy. Experiments across diverse modalities (including images, videos, and molecules) demonstrate that MPO consis- tently surpasses leading text-only prompt optimization methods, validating its efficacy in diverse real-world multimodal problems. We believe our work establishes multimodal prompt optimization as key direction for advancing the use of MLLMs, moving beyond text-only prompting paradigms.
ETHICS STATEMENT
Our study does not involve human subjects, personally identifiable data, or sensitive information. All experiments were conducted public datasets and models under essearch-permissee licenses.
REPRODUCIBILITY STATEMENT
We provide the public GitHub link to the code to ensure the reproducibility of our work. Addition- ally, we provide a detailed description of the experimental setup in Section 1. We further provide additional implementation details in Appendix A.3, the dataset configuration in Appendix A. the meta prompts to operationalize MPO in Appendix A.4, and the full algorithms in Appendix A.5.
ACKNOWLEDGEMENT
This work was supported by the Institute for Information communications Technology Planning & Evaluation (ITP)) grant funded by the Korea government (MSIT) 1RS-2019-11190055 Artificial
Intelligence Graduate School Program (KAIST) and RS-2022-11220713 Meta-learning Applicable to Real-world Problems), the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS- AS-2003-00255255), the Korea Machine Learning Ledger Orches- tration for Drug Discovery Project (K-MELLODDY) funded by the Ministry of Health & Welfare and Ministry of Science and ICT, Republic of Korea (No. 2024-00460870), the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Min- istry of Education ICT (No. N10250156).
and the InnoCORE program of the Ministry of Science and
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Rin Ashizawa, Yoichi Hirose, Nozomu Yoshinari, Kento Uchida, and Shinichi Shirakawa. Bandit- based prompt design strategy selection improves prompt optimizers. In Findings of the Associa- tion Computational Linguistics, ACL, 2025.
Peter Auer. Using confidence bounds for exploiaaionn-exploaaiin trade-offs. Journal of Machine Learning Research, 2002.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, et al. Qwen2.5-vl technical report. arXiv, 2025.
Djallel Bouneffouf and Raphaël Féraud. Multi-armed bandits meet large language models. arXiv, 2025.
Fabio Broccatelli, Emanuele Carosati, Annalisa Neri, Maria Frosini, Laura Goracci, Tudor Oprea, and Gabriele Cruciani. A novel approach for predicting p-glycoprotein (abebl) inhibition using molecular interaction fields. Journal of medicinal chemistry, 2011.
Jaemin Cho, Yushi Hu, Jason M. Baldridge, Roopal Garg, Peter Anderson, Ranjay Krishna, Mo- hit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. In The Twelfth International Conference Learning Representations, ICLR, 2024.
Yumin Choi, Jinheon Baek, and Sung Hwang. System prompt optimization with meta-learning. In Annual Conference Neural Information Processing Systems 2025, NeurIPS, 2025.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv, 2023.
Charles Corbière, Simon Roburin, Syrielle Montariol, Antoine Bosselut, and Alexandre Alahi. DRIVINGVQA: analyzing visual chain-of-thought reasoning vision language models in real- world scenarios with driving theory tests. arXiv, 2025.
Wendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley A. Malin, and Kumar Sricharan. Automatic prompt optimization via heuristic search: survey. In Findings of the Association for Computational Linguistics, ACL, 2025a.
Wendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley A. Malin, and Kumar Sricharan. SEE: strategic exploration and exploitation for cohesive in-context prompt optimization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics ACL 2025, 2025b.
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. In Forty-first Interna- tional Conference on Machine Learning, ICML, 2024.
Hanan Gani, Rohit Bharadwaj. Muzammal Naseer, Fahad Shahbaz Khan, and Salman Khan. Vane- bench: Video anomaly evaluation benchmark for conversational Imms. In Findings of the Associ- ation for Computational Linguisties: NAACI. 2025, 2025.
Bingjie Gao, Xinyu Gao, Xiaoxue Wu, Yujie Zhou, Yu Qiao, Li Niu, Xinyuan Chen, and Yaohui Wang. The devil is in the prompts: -uumeeel- prompt optimization for text-to-vider generation. In Conference on Computer Vision and Pattern Recognition, 2025.
Team Gemini. Gemini 2.5:
Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv, 2025.
Aaron Grattafiori et al. The llama herd of models. arXiv, 2024.
Qingyan Guo, Rui Wang. Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In The Twelfth International Conference on Learning Representations. ICLR, 2024.
11

---

Published as a conference paper at ICLR 2026
Tingjun Hou, Junmei Wang, Wei Zhang, and Xiaojie Xu. Adme evaluation in drug discovery. 7. prediction of oral absorption by correlation and classification. Journal of chemical information and modeling, 2007.
Kexin Huang, Tianfan Fu, Wenhao Gao. Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Co- ley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. Proceedings of Neural Information Pro- cessing Systems, NeurlPS Datasets and Benchmarks, 2021.
Songtao Jiang, Yan Zhang, Chenyi Zhou, Yeying Jin, Yang Feng, Jian Wu, and Zuozhu Liu. Joint visual and text prompting for improved object-centric perception with multimodal large language models. arXiv, 2024.
Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On bayesian upper confidence bounds for bandit problems. In Proceedings of the Fifteenth International Conference Artificial Intelli- gence and Statistics, 2012.
Omar Khattab, Arna Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq. Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self- self-improving pipelines. In The Twelfth International Conference on Learning Representations, ICLR, 2024.
Muhammad Uzair Khattak, Hanoona Abdul Rasheed, Muhammad Maaz, Salman H. Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Conference on Computer Vision and Pattemn Recognition, 2023.
Dongki Kim, Wonbin Lee, and Sung Ju Hwang. Mol-Ilama: Towards general understanding of molecules in large molecular language model. In Annual Conference on Neural Information Processing Systems 2025, NeurIPS, 2025.
Donghao Li, Chengshuai Shi, Weijuan Ou, Cong Shen, and Jing Yang. Efficient multi oojettv prompt optimization via pure-exploration bandits. In The Fourteenth International Conference on Learning Representations, ICLR, 2026.
Wenwu Li, Xiangfeng Wang, Wenhao Li, and Bo Jin. A survey of automatic prompt engineering: An optimization perspective. arXiv, 2025.
Yuanze Lin, Yunsheng Li, Dongdong Chen, Weijian Xu, Ronald Clark, Philip Torr, and Lu Yuan. Rethinking visual prompting for multimodal large language models with externa knowledge. arXiv, 2024.
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically- labeled mooledde-ennannee dataset for medical visual question answering. 18th IEEE Inter- national Symposium on Biomedical Imaging, ISBI 2021, 2021.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Annual Conference on Neural Information Processing Systems 2023, NeurlPS, 2023.
Sylvain Lobry, Diego Marcos, Jesse Murray, and Devis Tuia. RSVQA: visual question answering for remote sensing data. Transactions on Geoscience and Remote Sensing, 2020.
Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelli- gence, 2024.
Stang-Ying Ma, Sheng- Yong Yang, Hui Zhang, Ming-Li Xiang, Qi Huang, and Yu-Quan Wei. Prediction models of human plasma protein binding rate and oral bioavailability derived by using ga-cg-svm method. Journal of pharmaceutical and biomedical analysis, 2008
Oscar Mañas, Pietro Astolfi, Melissa Hall, Candace Ross, Jack Urbanek, Adina Williams, Aish- warya Agrawal, Adriana Romero- Soriano, and Michal Drozdzal. Improving text-to-image con- sistency via automatic prompt optimization. Transactions Machine Learning Research, 2024.
12

---

Published as a conference paper at ICLR 2026
Manuel Martin, Alina Roitberg, Monica Haurilet, Matthias Home, Simon ReiB, Michael Voit, and Rainer Stiefelhagen. Drive&act: A mudti-modal dataset for fine- ane-grained driver behavior recogni- tion in autonomous vehicles. In International Conference Computer Vision, ICCV, 2019.
Ines Filipa Martins, Ana L Teixeira, Luis Pinheiro, and Andre O Falcao. A bayesian approach to in silico fhood-brain barrier penetration modeling. Journal of chemical information and modeling, 2012.
Wenyi Mo, Tianyu Zhang, Yalong Bai, Bing Su, Ji-Rong Wen, and Qing Yang. Dynamic prompt optimizing for text-to-image generation. In Conference on Computer Vision and Pattern Recog- nition, 2024.
Sharada P. Mohanty, David P. Hughes, and Marcel Salathé. Using deep learning for maage-base plant disease detection. Frontiers in Plant Science, 2016.
OpenAl. Gpt- system card. arXiv, 2024.
OpenAL. Introducing 4o image generation, 2025. URL https:// http:///opeaa...omnnn/eeee com/
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee,, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with "gradient descent" and beam search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP. Association for Computational Linguisties, 2023.
Kiran Ramnath, Kang Zhou, Sheng Guan, Soumya Smruti Mishra, Xuan Qi. Zhengyuan Shen, Shuai Wang, Sangmin Woo, Sullam Jeoung, Yawei Wang. Haozhu Wang, Han Ding, Yuzhe Lu, Zhichao Xu, Yun Zhou, Balasubramaniam Srinivasan, Qiaojing Yan, Yueyan Chen, Haibo Ding, Panpan Xu, and Lin Lee Cheong. A systematic survey of automatic prompt optimization techniques. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Pro- ceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, 2025.
Chengshuai Shi, Kun Yang, Zihan Chen, Jundong Li, Jing Yang. and Cong Shen. Efficient prompt optimization through the lens of best arm identification. In Annual Conference on Neural Infor- mation Processing Systems 2024, NewrIPS, 2024.
Vishal Siramshetty, Jordan Williams, Dac-Trung Nguyen, Jorge Neyra, Noel Southall, Ewy Mathé, Xin Xu, and Pranav Shah. Validating adme qsar models using marketed drugs. SLAS DISCOV- ERY: Advancing the Science of Drug Discovery, 2021.
Shezheng Song, Xiaopeng Li, Shasha Li, Shan Zhao, Jie Yu, Jun Ma, Xiaoguang Mao, Weimin Zhang, and Meng Wang. How to bridge the gap between modalities: Survey on multimodal large language model. IEEE Transactions on Knowledge and Data Engineering, 2025.
Henrike Veith, Noel Southall, Ruili Huang, Tim James, Darren Fayne, Natalia Artemenko, Min Shen, James Inglese, Christopher P Austin, David G Lloyd, et al. Comprehensive characterization of cytochrome p450 isozyme selectivity across chemical libraries. Nature biotechnology, 2009.
C. Wah. S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech-uesd birds 200. Technical report, California Institute of Technology, 2011.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyua Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, et al. Wan: Open and advanced large-scale video generative models. arXiv, 2025.
Taowen Wang, Yiyang Liu, James Chenhao Liang, Junhan Zhao, Yiming Cui, Yuning Mao, Shao- liang Nie, Jiahao Liu, Fuli Feng, Zenglin Xu, Cheng Han, Lifu Huang, Qifan Wang, and Dongfang Liu. MºPT: Multimodal prompt tuning for zero-shot instruction learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35, NeurIPS, 2022.
13

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv, 2025.
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V. Le, Denny Zhou, and Xinyun Chen. Large language models optimizers. In The Twelfth International Conference on Learning Representations, ICLR, 2024.
Qinyuan Ye, Mohamed Ahmed, Reid Pryzant, and Fereshte Khani. Prompt engineering prompt engineer. In Findings of the Association for Computaaional Linguistics, ACL 2024, 2024.
Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. Optimizing generative ai by backpropagating language model feedback. Nature, 639:609-616, 616, 2025.
Fanhu Zeng, Fei Zhu, Haiyang Guo, Xu-Yao Zhang, and Cheng-Lin Liu. Modalprompt:dual- modality guided prompt for continual learning of large multimodal models. arXiv, 2024.
Lechen Zhang, Tolga Ergen, Lajanugen Logeswaran, Moontae Lee, and David Jurgens. SPRIG: improving large language model performance by system prompt optimization. arXiv, 2024a.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of- th-thughtt reasoning in language models. Transactions on Machine Learning Research, 2024b.
Yizhen Zheng, Huan Yee Koh, Maddie Yang, Li Li, Lauren T. May, Geoffrey I. Webb, Shirui Pan, and George Church. Large language models in drug discovery and development: From disease mechanisms clinical trials. arXiv, 2024.
Qiji Zhou, Ruochen Zhou, Zike Hu, Panzhong Lu, Siyang Gao, and Yue Zhang. Image-of-thought prompting for visual reasoning refinement in multimodal large language models. arXiv, 2024.
Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh Interna- tional Conference on Learning Representations, ICLR, 2023.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. 2025.
14

---

Published as a conference paper at ICLR 2026
A ADDITIONAL EXPERIMENTAL DETAILS
A.1 DETAILS ON DATASETS
We provide a detailed description of the datasets used in our experiments. conduct comprehen- sive evaluation, compile diverse set of benchmarks for classification and question-answering tasks across various modalities, including images, videos, and molecules. We use the official train- ing/test splits where available, and if not, we create our own splits. For the image and video modality tasks, we sample 300 test examples, whereas for the molecule modality, we use the entire test set.
The PlantVillage dataset (Mohanty et al., 2016) contains 54,306 images of plant leaves, spanning 38 disease categories across 14 crop species. To construct focused, fine-grained classification task, we design subtasks by selecting four crop species, each having least three distinct classes (e.g., one healthy and two more discases). This setup allows for more controlled evaluation of the model's ability to identify specific plant diseases. Due to the lack of an official split, split this subset using the 50/50 ratio for training and testing.
PlantVillage
CUB-200-2011 The CUB-200-2011 dataset (Wah et al., 2011) is a standard benchmark for fine- grained bird species classification. To evaluate the capability of MLLMs distinguishing between visually similar species, we group birds that share a common family name (e.g.. "Hummingbird"), and select groups containing three or four distinct species to ensure a balanced level of difficulty, resulting in total of 12 subtasks. Then, we divide the samples for each subtask using a 50/50 ratio, curating them to contain at least 80 instances for both training and test.
SLAKE
The SLAKE dataset (Liu al., 2021) an open-ended visual question answering bench- mark tailored for the medical domain from various radiological modalities. To assess the perfor- mance of MLLMs across these different modalities, we partition the dataset into distinct subsets based on the modality, creating separate tasks for CT, MRI, and X-Ray images.
DrivingVQA
The DrivingVQA dataset (Corbière et al., 2025) is a closed-ended visual question answering benchmark with 3,931 multiple-choice questions based on real-world driving scenarios. To avoid ambiguity in the evaluation process, we filter the dataset to exclusively retain instances with a single correct answer, resulting a final dataset 2,039 training and 521 test instances.
RSVQA We the RSVQA dataset (Lobry al., 2020) evaluate performance on the open- ended visual question answering task for remote sensing images. Notably, the questions are designed to evaluate a model's 's understanding of various geospatial concepts, including land cover classifica- tion, object counting, and relational reasoning between objects. For our experiments, we utilize the low-resolution image set from the benchmark.
Drive&Act For the video classification task, the Drive& Act dataset (Martin et al., 2019), which provides comprehensive labels for driver behaviors inside vehicles. We adhere to the official split of 6,642 training and 2,222 test instances, and preprocess the video clips by sampling frames at rate of frame per second (fps).
The VANE- VANE-Bench dataset (Gani et al., 2025) a closed-ended question answering benchmark for video anomaly detection, whose samples (each with 10-frame clip from synthetic or real-world videos) show various irregularities or distortions. We split the dataset into training and test sets using the 60/40 ratio, resulting in 293 training and 263 test instances.
VANE-Bench
Absorption The Absorption task (Huang et al., 2021) is categorized into molecular property pre- diction, designed to evaluate a model's ability to estimate pharmacokinetic characteristics related to drug absorption. It is composed of four subtasks: PAMPA (Parallel Artificial Membrane Permeabil- ity Assay), HIA (Human Intestinal Absorption). Pgp (P-glycoprotein substrate classification), and Bioavailability, and we use the official random split.
15

---

Published as a conference paper at ICLR 2026
BBBP BBBP (Martins et al., 2012) is molecular classification task to predict whether the given molecule can penetrate the blood-brain barrier (BBB), which is a highly selective system. We use the official random split from Huang al. (2021), consisting of 1,453 train and 382 test examples.
CYP Inhibit The CYP Inhibition task (Veith et al., 2009) involves classifying whether a molecule can inhibit Cytochrome P450 (CYP) enzymes, which play key roles metabolism. It comprises five subtasks: inhibition of CYP 2C19, CYP 2D6, CYP 3A4, CYP 1A2, and CYP 2C9. We adopt the official random split provided Huang et (2021).
A.2 DETAILS ON BASELINES
This subsection details the baseline methods used in experiments.
APE (Zhou al., 2023) generates candidate prompts by reverse-engineering instructions from examples and by paraphrasing existing prompts.
OPRO (Yang et al., 2024) leverages the LLM as an optimizer, guiding it with pairs prompts and their performance scores generate progressively better instructions.
EvoPrompt (Guo et al., 2024) utilizes an evolutionary algorithm where the LLM performs muta- tion and crossover operations on population of prompts.
PE2 (Ye al.. 2024) focuses on optimizing the meta-prome used steer the LLM optimizer. It provides guidance through a structured template containing detailed task descriptions, context specification, and step-by-step reasoning format.
ProTeGi (Pryzant al., 2023) simulates gradient descent for discrete prompts. uses the LLM to generate natural language critiques based on prompt failures (termed "textual gradients"). and subsequently edits the prompt in the opposite semantic direction.
SEE (Cui al., 2025b) performs cohesive optimization of both the prompt instructions and the in- context examples. The method follows a four-phase process that strategically alternates between global exploration and local exploitation.
A.3 ADDITIONAL IMPLEMENTATION DETAILS
In this subsection, we provide the additional implementation details in our experiments. Regarding model temperature, we use temperature value of for the base model to ensure consistency and 0.7 for the optimizer model encourage the generation of diverse candidate prompts. The failure set size in the cohesive backpropagation process fixed at 3. While the evaluation budget gen- erally set to 100, for CUB subtasks with fewer than 100 training samples, the budget for MPO method is specifically set to one-third of the available instances. For modality-sseciffii handling, we implement several strategies. In the video task, when the video query is part of the failure sample three representative frames (first, middle, and last) from queries. In video generation, to mit- igate the high complexity of video editing and mixing, we employ only the generation operator. We generate 5-second videos at 16 fps, then downsample them to 5 frames at I fps to construct the video prompt. For the molecule tasks, we represent chemical structures using the ID representation (i.e., SMILES) and utilize GPT-4o mini for the molecule generator. Regarding optimization objectives, we use accuracy for image and video modalities, and F1 for the molecular modality handle the class imbalance. Finally, to measure answer correctness, the final predefined label is extracted for standard classification, strict formatting rules are applied for binary and closed-ended QA tasks, and exact match is used for open-ended QA tasks. We select the best-perten pessoperfornin prompts on the training set and report their performance on the test set. Our experiments are conducted on NVIDIA H100 80GB GPUs.
adopt task-specific evaluation criteria:
16

---

Published as a conference paper at ICLR 2026
A.4 META PROMPTS TO IMPLEMENT MPO
This subsection details the meta-prompis to instantiate MPO, which include a cohesive backpropa- gation prompt and three operator prompts (generation, edit, mix) for update. We provide the meta prompt from image modality as representative example. The prompts for other modalities, such as video and molecule, are based on this structure, with minor, modality-specifft wordings adjusted.
Prompt for Cohesive Backpropagation
You Pronpt Pailure Analyais analyne the Feillre cese Mullt Miltinodel Sarge tanguage Model (MEAM) and identiff ffy the potent iel reasonx in the pronpt for the model a incorrent predict ion, Based on the given inputy putpat, and ground Lzuth, analyse both the Text Pronpt and the Image Proupt used in the Lask.
apecialized mil.inodal prampi opt intionion. Leak
446 Input Btracture for KIlMs
Text Frongt: A task-specitie textual instruction for the MLIM,
Prompts reference that supports task anderstanddnng (lext, image,
Topu: Query:
target
which MEMM generale
answer.
44 Peunglus
Frompt 1 lteettt prompt)
Trage Prompt : (moda prorpt) proppt
1+ Examples: jerorgeeserglees
H+ Outpot Fornat:
"ex: Proupt Anilysis:
Idencity missing information, vegue instructions, ambiguous wording tast nodel.
the
Explain how woaknesses in the Text Frompt may have contributed to the wrong output. improvements examples) to help
Suggest
task detirition, additicasi constraints, better
produce
Image Prompt Analysie:
If
Prompt used, analyse it's effectiveness.
Idencity probiens zuch lack clarity, poor composition, ircelevant detaile, niscing key testures.
If no inage Pzompt was used, suggest what kind of image (visual coatent, attributes, composition) would correct the failurs.
Figure 10: Meta Prompt for Cohesive Backpropagation in MPO.
Prompt for Generation Operator
You are a Brompt-napzevmmnt Agent speciallzing in multimedal prompt optimization. Your cask is improved prompts both generation and text instruction, eimed enbancing
pertornance
targe tanguage
++ Input Structure for MINR:
task-spacific textual Lastruction
MLMM.
Inage Prompt: zeferencs image that supperts tack understanding.
Input Query: Tae actual target instance itext, inage, or both) on which the MLM must generate an answer.
Erovided [toxt_prorpt]
Text Frompt:
Inage Prompt: amcdelsty_prorge
Examples: [sseng_oxamplss]
Failure Analysis: (anslysis)
448 Youz Task
failure analysis carefully to understand the isseas prompee that directly address the issues in the reilure analysie:
is
improved
Inaga Ganeration Fromp:
Wzite dotalled prompt for inzge generator.
Enhance
resolve iesues found Ensure the inaca highlighta critical visual features necassary for success.
the reference
analysis.
If no reference inzge is provides, suegest an appropriate one based on the faslure analysis.
Ineroved
Nzite cleas, concise, unembiquous instruction fer HELM.
Resolve arbiguities found in the failuse analysis. Elaberate on how the reforence inage should ba interpretad.
#4 Oucput Pornal
<inace_gnnaration proupps wtw.ar/ooenrattooorroroooo cinproved prongt Lnngroved_ pretp wwtp://pppi//oooooooeoooeeeeeeer
Figure 11: Meta Prompt for Generation Operator in MPO.
17

---

Published as a conference paper at ICLR 2026
Prompt for Edit Operator
You Proupt- Inpzovement Aqent apeciallaing multinedal prompt optimization,
praпpt editing. Your task is to desian inoxeved prompts for both inage editing and taxt instruction,
aimes at enhancing the performance or Noltimodal Large Language Model (NDIM)..
Input Structure
MIN::
Text Fronpt: A task-specietc textual instruction for the MLLM.
Inage Prompt: reference imaga that supperta task Lnder anderatanding.
guary:
which MEMM
ansvec.
+f Frovided Naterial
Prurgl:: preep_))eppt)
Inaqe Prompt: Inddality_orored
Wrong Examples: ciampless
Failure Analysis: (analysia)
Your Task
Your Lask is Lo cevies the failure analysia carefally dlyyylyl to understand the Laaues and cceate Lvo inproved pronecs that directly address the issues in the failere analysis:
Inage Edit ing Promgi:
Heita precias
context aaaae proupt instructing
image editor modify given
referance inage.
Specify which visual conponenia (e.g., object sulors, textures, lighting, perapect cospoaition) should added, renove,, replaced based fallare analysia.
Clearly identify any undesirable visual eienents that Guide the willoe on how to recain key features, proguctions or ssylistie elesents Chat ace eri:leal to the interded outcome.
failure.
Improved Text Prompt
Weite a cleas, consise, and unarbiquous instruction for the MELM.
Rexolve erbi aubiguit found the failure anelysis. how
inage should be inteepreted.
#44 pur Pornst
cinege e prospt> Linage eddt edit e prunpl) 1s s/inage prooptpt
cimproved prorpt> prongt>tingpoved prorpt 1s/i///ovee wext__eeeereeeeeee
Figure 12: Meta Prompt for Edit Operator in MPO.
Prompt for Mix Operator
You are Proupt Ingroverent Agent speciall aneotaliiiin lixing in modal prompt opt imixatioo, Focus crose-pronpt susion, Your tack is create improved, mixed promptic cor bosh image prorot and text Hodel IML111
instruction, aimed enhancing the pertornance Woltimoda Large
Hff Input Structure for RIG::
Text Fronpt: A task-speci textual instruction for she MLIM.
that supperts understanding. Input query: actusl target instance (text, inage, boch) answer,
Prompt:
generate
t+ Frovided Material
Pxempt
Text Frompt h: (toxt_
Inage Prompt A: modolity_promp
Wrone Examples Prompt oxane_e)am
Analysis for Frompt At (anslycis_A)
11#0 Pxcupt B
(toxt_prompt__])
Inage Proupt [nodallcy
Nrong Examples fron Prompt Bi (vzong_onsppio_))
Failure inalysis for Frompt 2: (analycis_8)
Youz Task
Your task is to revicw the failure analysis caretully to understand the issues and create tvo improved proneca that directly address the issuea In the failere analysia:
Mixing Prompt:
image generater curbine and improve refeswnow inages. Address visuel issves ideatifiad in both failere analyses.
Wrile guidance
cuide the model to create new hybrid inace that nerges key beacticial visual features fren 20th sefeeences
iggtiing their Explicicly state which viaual elements to achieve task success.
image be zetainad, nodified, diacardad
Ineroved
Meite cleat, concise, and unenbigaoue instruct for MLDM Incorporate bey visual as Lask zelevellann featuses identified in boch fallure analysis. Explain how the reference inage should be used to assist the task.
44 Oucput Pornal
<inse__nxdine_prom proompt> maaos_nixing_orraeeee htwp:///eeeoooosoaeeeerrerrg
twxt pronge Inixed text pronps s/nixed texix
Figure 13: Meta Prompt for Mix Operator in MPO.
18

---

Published as a conference paper at ICLR 2026
A.5 FULL ALGORITHM OF MPO
We provide the overall algorithm for MPO, with alignment-preserving exploration (including the operators) described in Algorithm and the prior-inherited Bayesian UCB selection in Algorithm 2.
Algorithm MPO: Multimodal Prompt Optimizer
Require: Initial prompt (ty, Ø), Number of iterations T, Beam size b
Train dataset Drr, Metric function f
1: p - (lo: 0),
(p); C 2, js -- (gggaa)D.. (/ Ø, ()))aa)
2: for = 1..b² do
3:
Fp {(g. a. y) (q. y MLLM(P. q): y # a}
4:
Vp t MLLM. Fww.k.eeebbbktttg Fp)
5: (V' Cgen) + MLL. Generation Ø; V Vm Fp): m' g(egon Ø)
6: C (-C((((,,m))
7: end for
8: P +
A-BBysssanUCCSeeeetPPP....
D> Select prompts for next step
9: for iter I.I do
10: C+ 2
for p (t) E do
12:
for 1..b do
13:
F, +- {(q, a, y) a) ~Dr MLLM(p. q), y + a}
14:
V p MLLM. HILLM.Feedbart Feedback(t, Fp)
Cohesive backpropagation
15:
op < SanadomSampl.. (generation, edit, mix))
D Joint multimodal update
16:
if op generation then
17:
(t' Cgen) <- ML.MM GeE.I..Gneeraio(tt.. m; Fp): m' + g(Cgen: Ø)
18:
else if op edit then
19:
(t', Codi)) + MILM. Edit (t; Vp. V Fp): m' + g(Codit, {m})
20:
else op mix then
21:
p RandomSampie(? {p})
22:
Cmix) - WWL.M.MMII m; V Fp); (,, vg. m' + g(Cmixx {m, m))
end
24:
+ C ((t,m'm))
end for
26: end for
27: +- BayestanUCBBelec(PPCCC
D Select prompts for next step
28: end for
29: return p" (t',m") where p" = arg max pep Pp:
Pp
op ap
Algorithm Prior-Inheriter Bayesian UCB Selection
Require:
Parent prompts P. A set of child prompts C = He Beam size b
Parent's performance (Apan(t) -= Train dataset Dtr. Metric function f. Batch size B Total evaluation budget N, Prior strength S. Exploration parameter e.
1: Initialize Beta priors for each child prompt ps E P:
2: for 1,..., ...,, do
3: Oi; < (ia)(i)= S1,, B (1-11 (p))()) ): S +1
>> Inherit prior from parent
4: end for
5: for t = 1.. (N/B) do
6:
(t
Klog NY=
7:
j + arg (1...) BetaQuantile(q;:q;;
Choose prompt with highest UCB
8:
Dmini <- Sample (D B)
9:
8t < Blig.a)DDmin (MLLM(t, m. q): a)]
Evaluate on small data batch Update posterior
aj a, + B, B, + B, (1 st)
11: end for
Return t prompts from C sorted by posterior mean A o, o.+0,
19

---

Published as a conference paper at ICLR 2026
B THEORETICAL ANALYSIS ON PRIOR-INHERITER BAYESIAN UCB
In this section, we provide the proof for Proposition 3.1, starting with the formal problem setting.
Setting. Let each arm i E [1....,k) k] have unknown Bernoulli mean reward μ4 E (0, 1) and let i* E arg max, μ4 be an optimal arm. Write the suboptimality gap as A = Hir - 14 > for i # i*. As shown in algorithm 2, the Bayesian UCB algorithm maintains a Beta posterior distribution for each arm's 's mean reward. At each round t. Bayesian UCB selects the arm with the highest upper posterior quantile, Gt. observes the resulting Bernoulli reward, and updates the corresponding posterior.
Prior inheritance For each child arm we initialize Beta prior using the parent's posterior mean (par()) E (0, 1) and a pseudo-count 5 > 0:
00.; = (ppa(()) S
Bo.; = (1- (par(s) (1)) 1.
(2)
For comparison, uninformative (or uniform) prior is Beta(1, 1). After NN(() pulls with X,(t) successes by time i, the posterior parameters are (1) = Cn + xx(t) and (t) = fo, + (t) (t). Denote the posterior mean t4, (t) (a; + (t)), the upper quantile gr. BetaQuantile(gr: (t), (t)), and the lower quantile l, = Beta DetaQuantile(11 - gr: as (.))).
Average KL-closeness assumption. Our analysis relies the assumption that a parent's posterior provides useful inductive bias for its children. We formalize this concept using the Kullback- Leibler (KI.) divergence for Bernoulli distributions, defined as d(p, q) = p log + (1 p) log 1-p Let be the population of child arms produced during optimization. We assume the parent estimate is, on average over children, KL-closer to the truth than the mean of the uninformative prior:
E,sz (parto) - dl d(p,, ?)] -y for some 7 > 0.
The assumption is empirically supported by the strong positive correlation observed between parent and child scores (Figure 3).
B. TWO AUXILIARY LEMMAS
Lemma B.1 (Pseudo-counts shrink one-sided credible widths). There exists universal constant c >>00 0 such that for all 2 2 and all arms i,
logt 9.s - Pr. S cV N.()++
logt Pr.s Lii S ov Ni(t) +
(4)
The key implication that the credible interval width scales with 1/ 1/vNN(tt++ + S rather than 1/ (N(t)). Thus, the prior strength S acts as an additive effective sample size, shrinking the interval as if we had S additional observations.
proof sketch. The proof relies on standard concentration bounds for Beta posteriors. The conjugacy of the Beta- Binomial model makes the posterior tractable, allowing for the specific application of (0, ), the probability that the upper posterior quantile underesti-
Chernoff tail bound. For any mates the true mean by at least
(5) with a symmetric bound holding for the lower quantile f, The result obtained by using the approximation d(μ; - E, v.) Z z2 for small and selecting the quantile level 1 1-q: (l/tt) yields the stated Vlog (/) + S) bounds.
P( qt, S #4 = S exp( (N,(t) + d(4 E. E.),)),
Lemma B.2 (Effect of Informative Priors on Posterior Quantiles). Under equation 3, for any fixed real counts (n, s) with s Binom (n. Hi), the posterior under prior inheritance Betafao, Po. s) is, average over i I, better centered around μr than the posterior under uninformative prior Beta(1 (1 + &, + 8). Consequently,
E (par) e(unif) 2
E
(par)
9 (unif))
< (i+i*)
(6)
with strict inequalities whenever 7 > 0 and S 0.
20

---

Published as a conference paper at ICLR 2026
proof sketch. The posterior mean under prior inheritance is p S Spar(()
while the pos-
S+++2
n+22 Taking expectation over s and then over I yields convex combinations of μ4 with Apar(() versus 1/2. Our KL--closeness assumption equation 3 directly implies that the posterior mean under prior inheritance is, in expectation, better estimate of μúz Since the posterior quantiles are centered around this mean, Lemma B. ensures that an improvement in the mean's S centering translates to the stated shifts in the quantiles, holding in expectation.
terior mean under uninformative prior is An.i
1+s
B.2 SUFFICIENT CONDITION FOR OPTIMAL ARM IDENTIFICATION
For the algorithm to correctly identify the optimal arm i by the final round T, a sufficient condition that the credible intervals for the optimal and suboptimal arms are well-separated. Formally, this occurs if the lower quantile of the optimal arm exceeds the upper quantile of every suboptimal arm. This is the separation event:
bT
TT, E
If this separation event fails, it implies that for some suboptimal arm i, the credible intervals overlap. This allows us to bound the suboptimality gap A; by the sum the one-sided credible widths:
logT
log T
A, S (Ju (r, + (TT, 14)
(8)
V
(T)
(T,)+++S
where the second inequality follows from Lemma B.I. This implies that guarantee separation, the credible interval widths must be sufficiently small relative to the gap. Therefore, a deterministic sufficient condition for equation is: there exists a universal constant > 0 such that,
log T V (T)
log T
(N(T++ < d
Crucially, combining this condition with the quantile shift from Lemma B.2 reveals the benefit of our approach. Because the prior inheritance yields better quantile estimates, the sample allocation required to satisfy equation 9 is achieved later than with an uninformative prior.
B.3 PROOF OF PROPOSITION 3.1
Proof. The prior inheritance improves the performance of Bayesian UCB through two synergistic mechanisms:
(i) Tighter credible intervals at fixed counts. For any given allocation of pulls, the prior strength S acts as an additive effective sample size. As established in Lemma B.I, this shrinks the credible interval widths by effectively replacing the sample size (TT) with (T))+ This directly reduces the left-hand side of the deterministic condition equation making it easier to satisfy.
(ii) More efficient sample allocation. The informative prior also leads to a better allocation of pull over time. Lemma B.2 shows that the quantiles are favorably shifted on average: the lower bound for the optimal arm i* increases, while the upper bounds for suboptimal arms decrease. This improved estimation guides the UCB policy to allocate more pulls to i* and waste fewer on suboptimal arms, particularly in the early stages. Consequently, in expectation:
E N (par.S) (T)
2 LE
[ (par,S) 5)) (r)
< E N (umif) (r)
(t), (10)
with strict inequalities when the prior is strictly beneficial (? and S>>S>0 0).
Together, these two mechanisms ensure that the separation condition equation ciently. The tighter intervals (i) make the condition easier to satisfy for any given sample allocation, and the improved allocation strategy (ii) finds sufficient allocation faster. As a result, for a suffi- ciently larger budget T, the total expected number of pulls on suboptimal arms is reduced:
met more effi
N N(ppar.S (T)
Σ N (unif) (T)
E
(11)
This is equivalent to stating that the expected cost of identifying the best arm is non-increasiin intiinceeasin, and strictly decreases whenever the average Kl-closeness assumption holds.
21

---

Published as a conference paper at ICLR 2026
C
ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSIS
C.1 COMPARISON OF COMPUTATIONAL COSTS
Table 5: Comparison of the number of model requests (or calls) for MPO and other baselines.
Model Requests (Calls)
Methods
Base Model
Optimizer Model
Sodelity-Specific Generator
Avg. Performance
APE
11.7k
117
N/A
51.3
ProTeGi
11.7k
234
N/A
60.0
SEE
11.7k
153
N/A
59.1
MPO (Ours)
11.7k
234
65.1
We analyze the number of model requests (or model calls) proxy for computational cost, and report the results in Table First, the base model call is the same for all methods, as we fix the number of explored prompts and the evaluation budget. For the optimizer model calls, APE uses the one-step exploration (e.g., paraphrasing), requiring the number of calls to be equal to the generated candidates. ProTeGi and MPO utilize a two-step process (e.g., feedback generation and refine- ment), requiring twice the number of calls. SEE combines both approaches and falls in between. Note that, although MPO incurs additional computational cost by calling modality-speciff gen- erator to explore non-textual prompts, this cost is manageable, as this process can utilize lightweight, open-source generators such as SANAI.5 (1.6B) to minimize the additional expense, while still out- performing text-only prompt optimization methods as validated in Table 2 (Bottom Right). In other words, despite the marginal increase in computation (which also manageable), MPO achieves substantial performance improvement unattainable by existing text-only optimization methods.
C.2 FULL MAIN RESULTS
We provide the full results, including performance on individual subtasks. The results for the image modality are presented in Table 6, and for the molecule modality in Table 7.
Table 6: Full experimental results on image modality benchmarks, including subtasks, with all scores reported as the average accuracy over three independent experiments.
CIR
Thengggg Berigs TEStaaa
Cora
Grag
Tiaas
AN
akatow
luake
ciles
TACENCE
KAM
BOON
Vilr
ext
20.8
446
614
294
916
301
241
494
6-1
343
165
128
16.5
414
21A
164
51.
11.4
211
47x
913
V11
66.
0-3
243
110
6013
355
27.1
033
073
16.1
174
965
11.
34.1
0133
134
191
11.
976
014
0-4
286
60.6
486
683
612
968
244
184
15.8
310
134
180
151
013
114
253
114
033
113
08.3
645
841
672
18.1
46.
084
114
363
113
223
522
214
VPO
363
069
740
764
017
014
514
011
003
7611
16.3
157
06.
064
041
764
361
774
010
Table 7: Full experimental results on molecule modality benchmarks, including subtasks, with all scores reported as the average accuracy over three independent experiments.
Absorption
DEUP
CYP Exlihitinn
PAMPA
HIA
PgP
Risarril.
Arg
HRRIPP
CVP 2019
CYP 206
CVP 3A4
CYP1A2
CYP
Ang
Human
15.2
16.8
41.1
40.7
55.5
49.4
39.1
35.2
38.5
36.3
30.4
35.5
52.6
46.2
25.5
25.1
43.6
33.5
52.9
43.7
40.5
37.1
43.1
37.1
Col
30.2
31.1
40.2
40.8
51,3
36.1
36.7
38.9
39.5
36.7
35.6
32.5
45.7
39.1
22.5
21.5
42.8
32.7
30.0
39.1
36.2
29.5
40.1
30.3
46.1
47.3
Sobot
21.1
66.7
58.8
56.3
45.
43.5
48.1
15.1
48.1
96.4
$2.7
$1.0
47.0
APE
17.7
16.2
73.6
55.2
32.4
$1.7
39.1
38.5
45.7
40.4
36.0
34.7
$4.3
53.6
40.4
44.5
40.3
49.1
55.2
56.2
52.3
50.9
31.3
50.9
OPRO
96.3
50.0
15.
16.9
37.6
35.4
193
38.3
354
$3.9
437
00.5
37.1
43.0
37.1
EvaPhores
30.6
$3.5
95.9
90.5
60.4
19%
57.0
SEE
65.5
52.5
85.1
60.7
65.4
65.1
56.4
52.6
71.4
60.0
67.0
62.3
56.4
56.4
76.7
51.1
57.7
57.7
62.1
6..3
50.
568
61.4
56.7
MPD
56.1
M0.1
76.3
71.0
70,
68.2
55.1
76.7
68.5
75.3
67.4
68.
64.
63.5
6411
634
65A
62.5
64.3
08.2
22

---

Published as a conference paper at ICLR 2026
C.3 QUALITATIVE RESULTS
In this section, we present an additional comprehensive qualitative analysis of MPO.
Additional Analysis in MPO Figure 14 illustrates the optimization process in the molecular do- main, and Table 8 shows examples of textual conditions for the modality-specifft generator and the resulting image prompts.
Examples of Optimized Multimodal Prompts The optimized multimodal prompts from MPO are presented for the image (Table 9), video (Table 10), and molecular domains (Tables 11 and 12).
Comparison with Text-only Optimization Baseline We compare the optimized prompts and sponses of MPO with those of leading text-only optimization baseline (SEE) to investigate the un- derlying reasons for the effectivenees of using multimodal prompts. As shown Tables and 15, text-only methods must encode all necessary visual cues solely through language, often resulting in verbose descriptions and overly generic instructions (e.g.. "Pay attention to specific visual markers such as bill shape, color, and the feather crest on the Crested Auklet... ""). This reliance on vague tex- tual features can lead to coarse reasoning and eventual misclassification by the model (c.g., wrongly identifying the bird as a "Rhinoceros Auklet" based on an imprecise assessment of "robust body" and lhke--like projection"). ") In contrast, as shown Tables 14 and 16, MPO establishes a synergistic interaction between modalities. It augments concise textual instruction with direct visual refer- ence, climinating linguistic ambiguity and instructing the model to ground its reasoning in concrete visual evidence (e.g., "compare the size and shape of the bill with those in the reference image"). This multimodal grounding shifts the model's s focus, leading to more specific and visually-supporteed reasoning (e.g., "The bill is relatively short and thick, with a slight curve at the tip. This matches the description of the "crested auklet' in the reference image..."), "). In addition to this, importantly, on examples from the Auklet subtask where SEE's optimized prompt fails, MPO's s multimodal ap- proach successfully fixes 66.6% of the misclassifications demonstrating that multimodal cues can effectively resolve errors that are inherently difficult for text-only optimization.
Gan
Gi
Fdit
Mll
Edit
Figure 14: The optimization process for the best multimodal prompt on the BBBP task. Inherited substructures from the parent molecule are marked with the same colored circles.
D USE OF LARGE LANGUAGE MODELS (LLMS)
We use large language models merely as a writing assistant. Its role confined to improving gram- mar and paraphrasing sentences for clarity, and all the core ideas regarding problem definition, MPO framework, experimental design, and interpretation of results are entirely our own.
23

---

Published as a conference paper at ICLR 2026
Gen
TLUC CATED
Parent Image Prompt
Child Image Prompt
Sumi Textual Condition for Generation Operator
Create detuiled reference image showensing various species of grosheaks, ensuring each bird clearly labeled with its common name and distinctive features. Highlight key visual characteristies such as:
Beak shape (e.g., wide, pointed)
Color patterns (e.g. primary colors, unique markings)
Body size (e g.. small, medium, large)
Use arrows or circles to emphasize these features, especially where they differ from other species. The composition should be organized grid format for easy comparison, with neutral background to avoid distractions from the birds themselves.
Edit
Faye triuded Bratnes
Guoked
142
Parent Image Prompt
Child Image Prompt
Getn:: Textual Condition for Edit Operator
Enhance the reference image of grosbeaks by implementing the following modifications to improve clarity and usability for classifica-
tion tasks:
1. Increase Resolution and Clarity: Use reghertnsootttinn resolution image to ensure that all details of the birds are crisp and easily identifiable. Labeling: Clearly label each grosbeak species with bold, legible text. Ensure that the labels are positioned close to the respective birds and distinguishable from the background.
Consistent Postures: Arrange the birds in similar pestures and angles to facilitate direct visual comparisons. Consider a uniform side view to best showcase the beak shapes and body sizes.
Highlight Distinctive Features: Add visual cues such as arrows or cireles that paint to unique characteristics (e.g., beak shape, wing culors, and markings). Include brief descriptions of these features near the labels.
5. Simplify Background: Remove any distracting elements from the backgrund, opting for neutral color that allows the birds to stand out more prominently.
Maintain Proportions: Ensure that the propottions of the birds remain accurate and consistent with their actual sizes to aid in the visual comparison.
These changes aim to create more effective reference image that enhances the classification of grosbeaks making distinguishing features more accessible.
ENDRING
BLAC BAND
Mix
Parent Image Prompt
Parent Image Prompe B
Child Image Prompt
Cmix:: Textual Condition for Mix Operator
Create new reference image that combines the best visual elements from both provided grosbeak images. Retain the clear anatomical labeling from Reference Image easuring each grosbeak species is distinctly identified. Highlight key features such as color patterns and beak shapes using arrows and concise annotations.
From Reference Image A, incorporate the variety of grosbeak species but arrange them in a less cluttered format, allowing for a clearer comparison of unique churacteristics. Focus on using high resolation images that showease the hirds in similar poses and angles. Ensure that the primary color, distinctive markings, and beak shapes are clearly visible and easily comparable to aid in accurate classification. value to the identification proces..
Diseard any elements that create visual confusion
Table 8: Operation examples for the image prompt update, including parent image prompts, resulting child image prompts, and the textual condition to the modality-specifft generator, i.e., ImTge
24

---

Published as a conference paper at ICLR 2026
PlantVillage: Plant Leaf Images
Analyze the provided grape leaf image and classify it into one of the following categories: [ ['Hcalihy". 'Leaf Blight', "Black Rot", 'Esca'l J. Use the hybrid reference image for guidance, focusing on the following critical visual features:
Healthy: Look a vibrant, uniform green color and smooth texture without blemishes.
IICALTT TV
LEAF CUOTT
2. Leaf Blight: Identify distincr yellowing edges along with well- small dark spots that are clearly visible.
3. Black Rot: Check for sharply defined, durk, sunken lesions that are prominent on leaf surface,
often sccompanied by slight shriveling.
4. Esca: Look for distinct irregulur brown patches, significant necrosis, and curling of the leaf edges.
ACK NOT
RICA
cases where symptoms overlap, prioritize the most severe churacteristics, For example, if both dark spots and sunken lesions are present, classify bused on the prominence of the lesions. Ensure that you assess each feature carefully, referencing the hybrid image to visualize these distinctions accurately.
CUB: Bird Images
Classiff bird the turget image by comparing it with the hybrid reference image grosbeaks,
Follow these refined steps for accurate classification.
Identify the Grosbeak Group: Refer to the hybrid image that displays the Rose Breasted Grosbeak,
Pine Grosbek,, Blue Grosheak, and Evening Grosheak. Familiarize yourself with the specific traits each species. including color patterns and murkings.
Analyze Visual Features: Focus these critical features of the target bird:
Dominant Color and Markings:
Note the primary color and any distinctive patterns, such
throat colors or wing designs.
Beak Characteristies: Compare the shape and size of the beak with those the reference image, these can vury significantly among species.
Piu Guilast
Body Size Comparison: Assess the hody size of the target hird relative to the reference birds,
ensuring accurate size comparisons.
3. Feature Prioritization:
Prioritize color patterns first, as they are often the most telling feature.
Shut Grokak
Toving Grotlod
colors similar, evaluate beak shape and size next.
Finally, consider body size. If the target bird does not closely match any reference species, provide
the name of the closest match or indicate 'unknown', based on the following criteria:
Closeness is determined the degree of similarity across analyzed features, with color being the primary factor, followed by beak shape and size.
After analyzing these features, provide the name of the bird species that most closely matches the visual characteristies observed in the target image, supported by specific observations from the hybrid reference image.
SLAKE; Radiological Images
"Given the MRI scan image, identify and list all visible abnormalities present in hrain. Your response should include specific conditions. such edema and lumors, along with their locations (e.g.. "Right Lobe" for edemu). Refer to the labeled markers in the image to assist with your analysis. Aim to provide a comprehensive answer that covers all relevant conditions without omitting any visible feature. Ensure your response is clear and concise.
TWV
DrivingVQA:
Driving Images Examine the provided image of a road scenario and determine the most appropriate action based on specific visual cues. Pay close attention the following details;
Lane Markings: Identify the lane markings: solid lines indicate a no ov-ovretakiin zome, while dashed lines indicute safe area for overtaking. Clearly explain how these markings influence your decision.
2. Position of Vehicles: Assess the positions and distances of the vehicles. Determine if there is enough space and time to safely exccute an overtaking mancuver based on their speeds and proximity.
3. Traffic Signs: Observe all visible traffic signs, purticularly their meanings. For example, triangular sign may indicate a hazard ahead, while circular sign specifies speed limits. Explain how each sign influences your decision.
Based on your observations, decide whether you would (A) continue the overtaking maneuver or (B) move to the right. Justify your choice with specific details from the image, ensuring clarity in your reasoning. Conclude your response with "The answer is [answer]." Use image reference support your analysis of lane markings, vehicle positions, and relevant traffic signs in this driving scenario.
RSVQA: Remote Sensing Images
Analyze provided neighborhood map and respond to the following questions with accurate counts and concise answers:
Count the total number small roads (less than foet wide), medium roads (5-10 foet wide), and large roads (greater than 10 feet wide). Indicute which cutegory has the highest count. Use the legend provided to classify each road accurately.
Identify there is a commercial building (a structure used for business purposes, such shops offices) located to the left of any farmland area. Ensure you consider the top-down perspective of the map when determining placement.
For any presence or absence questions, provide a direct "Yes" "No" response.
Refer to the maps's 's colors and labels. casuring you utilize the legend for accurate identification of each cutegory. Pay special attention to spatial relationships as defined in the map to avoid misinterpretations.
Table Qualitative examples of the optimized multimodal (image and text) prompts.
25

---

Published as a conference paper at ICLR 2026
Drive&Act: Driver's Action Videos
Clusily the primary action being performed in the viden, focusing specifically on interactions with objects or devices. If multiple actions are present. prioritize the action that is most visually prominent or contextually relevant. example, if a person both cating and using phone, classily the action of using the phone. Use the delititions provided for each action to guide your decision. Consider visual cues hand movements, object handling, and the overall context of the scene help determine the primary action. two actions appear equally relevant, choose the one that is visually dominant or crucial to understanding the situation.
VANEBench: Abnormal Videos
Analyze the provided video to identify and deseribe specific actions or behaviors depicted. Focus particularly on actions that diverge from common social norms or expeciations, For instance, typical actions might include greeting someone or making eye contact, while atypical actions could involve unexpocted emotional reactions, erratic movements, interactions that of place. Consider the following examples:
A person suddenly laughing setious situation. BB Someone avoiding eye contact in social setting.
Please select the most striking anomaly from the provided options and present your answer in the formar: "The is [answer]."
Table 10: Qualitative examples of the optimized multimodal (video and text) prompts.
Absorption:
Drug Absorption Human
You a drug discovery assistant tasked with predicting the human intestinal absorption (HIA) of newly designed hybrid molocule. Your analysis should focus on the following physicochemich properties, taking into account both the strengths and limitations previous reference molecules:
Molecular Weight (MW): Calculate the molccular weight the hybrid molecule. molecular weight below 500 Da generally favorable for absorption.
Lipophilicity (LogP): Estimate the LogP value of the hybrid molecule. Aim for value between and 5, easuring that balances contribations from both polar and non-polar functional groups.
Polarity and Solubility: Analyze the overall polarity of the molecule. While polar functional groups can enhance soluhility, ensure that their presence does not excessively hinder absorption through lipid-rich environments.
present the hyhrid molezule. Focus on ionizable groups that can enhunce solubility while ensuring that polar groups balanced facilitale membrane permeability. Discuss how these groups interact to affect absorption
Functional Groups: ldentify and describe key functional
5. Stereochemistry: Note any chiral centers preseni in the molecule. Different enantiomers may exhibit varying absorption profiles, explain how sterenchemistry could in influence ahsorption.
At the end of your unalysis, provide a conclusion formatted as either 'Final answer: Absotbed' or 'Final answer: Not absorbed." Ensure that your evaliation is comprehensive and considers combined properties derived from the reference molecules accurately predict the absorption potential of the hy brid molecule.
Utlization of the Reference Molecule: This hybrid molecule designed improve predictions for human intestinal absorption (HIA) by integrating key features that enhance solubility and membranc permeability. The presenee of two carboxylic acid groups increases the likeli- hood ionization, which improve solubility in gastreintestinal tract. while the tertiary amine entsiness interaction with transporters, facilitating absorption into the bloodstream.
The molccular weight is kept below 500 Da, aligning with favorable absorption criteria, and the LogP is balanced to casure optimal lipophilic- ity. This design allows for comprehensive predictive models for HIA. By leveraging the strengths of both reference molocules, the hybrid is expected to yield more accurate predictions regarding absorption potential,
of the hybrid molecule's 's physicochemical properties, which be used inform
aiding drug discovery efforts. The combination of polar and non-polar funerional groups ensures that the molecule can effectively navigate the lipid-rich envirnments of the intestinal membran while maintaining sufficient solubility for absorption.
Table 11: Qualitative examples of the optimized multimodal (molecule and text) prompts.
26

---

Published as a conference paper at ICLR 2026
BBBP: Penetration to Blood-Bruin Barrier
are a drug discovery assistant responsible predicting the blood-brain bartier (BBB) penctration capa- bility of a new hybrid molecule based on its physicochemical proportics. Follow these dotailed instructions to conduct your analysis effectively:
Key Considerations for RBR Penetrarion:
1. Lipophilicity (LogP): Estimate the lipophilicity of the hybrid molccule. Aim for LogP between I and 5, which optimal for BBB crossing. Use computational tools ALOGGS ChemDraw to report the extimated LogP value.
Molecular Weight: Check the molecular weight of the hybrid molecule. It should below 450 Da. Provide the exact molecular weight in your amalysis.
Hy dydrggnn Bonding Assess the number hydrogen bond donoes (HBDs) and acceptors (HBAs)) Aim for 1-2 enhance the likelihood of BBB penetration. If the counts exceed these ranges, note how this may affect permeability.
HBAs
lonization State: Evaluate if hybrid molecule neutral or charged at physiological pil (7.4). Clearly state the estimated ionization state and include pKa values for relevant groups.
Presence Polar Groups: Identily polar functional groups and assess their overall impuct on BBB permeability. Ensure a balanced presence to avoid excessive hydrophilicitt.
### Analysis Instructions:
Analyze the provided hybrid molocular stracture and evaluate how these properties collectively influence its ability to cross the BBB,
Provide specific quantitative measures where applicable.
Compare the hybrid molecule's is characteristics with those a well-characterized reference molecule known to cross BBB, noting key differences in properties that may influence permeability.
Summarize your findings elearly, stating the implications of the physicochemical properties on BBB crossing capability.
#*# Final Answer Format:
Conclude your anilysis with clear statement formatted either: 'Final answer: Can crass BBB 'Final answer: Cannot cmoes BBB.'
Ensure that your analysis thorough hased on specific physicnchemical propenies outlined ahove to enhance the accuracy of your predictions.
Utilization the Reference Molecule: This modified molecule enhances predictions for blaud brain hurrier (BBB) penetration by addresssing several key physicochemical properties.
1. Lipuphilicity (LogP') The addition of methyl group (C) at the terminal position increases the hydrophobic character of the molecule, which can improve its LogP value, making it more range of 1-5.
BBB crossing. This change aimed achieving LogP within optimal
2. Molecular Weight:
The modificd molecule maintains molccular weight below 450 casuring compliance with critical criterion for BBB penetration. This is essential as larger molecules often situggle to cross the barrier.
3. Hydrogen Bonding The modification retains a balanced number of hydrogen boad doners (HBDs) and acceptors (HBAs). By keeping the
HBDs to minimum (1-2) and ensuring HBAs are within the optimal range (3-5), the likelihood of effective BBB penetration is increased.
louization State: The structural modifications aim maintain the melocule in a neutral or partially ionized state at physiological pH. which is critical for cahancing lipophilicity and reducing the likelihood of charge-related hindrances to BBB penetration.
Presence of Polar Groups: The molecule been adjusted balance polar functional groups, reducing excessive hydrophilicity while retaining necessary polar characteristies for biological activity.
By focusing on these modifications, the molecule is better positioned for predictive modeling of BBB penetration capabilities, it aligns established physicochemical parameters known influence permeahility. This can inform computational predictions and improve the accuracy of models ussessing BBB crossing putential.
CYP Inhibition:
Inhibitory Effect to Cytochrome P450 (CYP) Enzymes
You are drug discuvery assistant lasked with predicting the CYP2C19 inhibition potemtial of a targel molecule. Your unalysis should focus on identifying and evaluating specific structural features that comelate with CYP2C19 inhibition. Address the following key churacterisics:
1. Aromatic Rings: lddentify the number and type of aromatic rings present in the targel molecule. Multiple aromatic rings are important for stacking interactions with the CYP2C19 enzyme. enhancing inhibition patential.
Assess for the presence of functional groups known to enhance binding, such as: Salfonamide groups (-S(=O)--N-) and amide groups (-C(=O)N-), which facilitate hydrogen bonding and electrostatic interactions.
Functional Groups:
Highlight any other functional groups that may support or hinder binding.
3. Basic Nitrogen Atoms Heterocycles: Determine if the molecule includes basic nitrogen atoms or heterocycles that could enhance binding aftinity through electrostatic interactions.
4. Comparison with Known Inhibitors: Compare the structural features of the target molecule with those of known CYP2C19 inhibitons like Omeprazole or Voriconazole. Pay close attention similarities differences aromaticity, functional groups, other relevant characteristics.
Conclude your analysis with clear statement regarding the inhibition status target molecule, formatted follows: 'Final answer: Inhibits CYP2C19' or 'Final answer: Does not inhibit CYP2C19. Ensure your comparisons and conclusions are supported by your siructural analysis.
Utlization of the Reference Molecule: This generated molecule, which contains multiple aromatic rings, a sulfonamide group. and amide group, can significanlly improve predictions for CYP2C19 inhihition potential. The presence of two aromatic rings enhances x-cr stacking interactions the CYP2C19 enzyme, which is crucial binding allinity. The sullinamide group (-S(=O)--N-) known facilitate hydrogen boading, while the amide group (-C(=O)N-) can participate in additional hydrogen bonding interactions, both of which are important stabilizing ensyme-inhibitor inhibitor complex.
Furthermore, the molocule incorporates basic nitrogen atoms in the amide and sulfonamidc groups, which can engage in electrostatic interac- tions with the enzyme. further cahancing binding aflinity. This structural design aligas with the characteristics observed in known CYP2C19 inhibitors like Omeprazole and Voriconazole, which also feature multiple aromatic systems and functional groups that promote hydrogen bonding.
By comparing the generated molecule's structural features with those of established inhibitors, deseripters that correlate with inhibition potential. This melocule serves as a referenee point for evaluating new compounds in terms of their predisted CYP2C19 inhibition, helping refine predictive models and improve the accuncy virtual screening processes. The combination of aromaticity, functional groups, and basic nitrogen atoms in this molecule provides a robust framework for understanding and predicting CYP2C19 inhibition.
derive insights into the molecular
Table 12: Qualitative examples of the optimized multimodal (molecule and text) prompts.
27

---

Published as a conference paper at ICLR 2026
Task Example: CUB(Grosbeak) Classification
Optimized Prompt (SEE)
Identify the species the bird the given image using the options provided. Please respond with the bird's name exactly as it is listed (c.g.. 'rose breasted grosbcak'). ).
Make sure write the nome procisely as it appears in the choices, without any extra ponctuation or modifications. Pay atention distinctive features such as color patterns, markings, and size accurately identify the bird.
(Score: 96.7)
If you find that the bird in the image does not match any of the listed options, please state "not identifable" rather than guessing. that your answer selected only from the choices provided refrain from inferring details about the species.
After deterriiing the bird, verify that your response is among the listed options before you subinit it.
Image Query
Ground Truth: Evening Grosbeak
Key Features:
White wing patches Yellow body color
Response
breasted grosheak
Table 13: Visualization of the SEE result on the CUB (Grosbeak) dataset.
Task Example: CUB (Grosbeak) Classification
Optimized Prompt
Classify the bird target image comparing with the hybrid reference image of gros- beaks. Follow these refined steps for accurate classification.
(MPO)
Identify the Grosbeak Group: Refer the hybrid image that displays the Rose Breasted Grosbeak, Pine Gresbeak, Blue Grosbeak, and Evening Grosbeak. Familiarize yourself with the specific traits of each species, including color patterss and markings.
(Score:
100.0)
Fas Nemvele Soukee
The Orobed
Analyze ze Visual Features: Focus
of target Dominant Color Markings: Note primary and any distinctive patterns, such
throat colors wing designs.
Beak Characteristies: Compare the shape and size the beak with those in the reference image, as these can vary significanily among species.
Body Size Comparison: Assess the body size of the target hird relative to the reference
birds, ensuring accurate size comparisons.
3. Feature Prioritization:
Prioritize color patterns first, as they are often the most telling feature.
If colors are similar, evaluate beak shape and size next.
Finally, coosider body size. If the target bird does not closely match any reference spocies,
provide the name of the closest match or indicate 'unknown', based on the following criteria:
-Closeness is determined by the degree of similarity across all analyzed features, with color being the primary factor, followed by heak shape and size.
After analyzing these features, provide the name of the bird species that most closely matches the visual characteristies observed target image, supported hy specific observations from the hybrid reference image.
Image Query
Ground Truth: Evening Grosheak Key Features:
White wing patches Yellow body color
Response
bird target imuge a predominanily yellow body with black wings white patch on chest. beak. is short and conical, suitable for sood-cating. Comparing these features with the hybrid reference image of grosbeaks, the bird most closely resembles the Evening Grosbeak. The yellow coloration, black wings, and white chest patch are characteristic of this species. Therefore, the bird in the larget image classified an Evening Grosbeak.
Table 14: Visualization of the MPO result on the CUB (Grosbeak) dataset.
28

---

Published as a conference paper at ICLR 2026
Task Example: CUB(Auklel) Classifiaaioon
Optimized
Classity the content of the target image based on the provided chosocs of bird species: ('parakect auklet', 'thinoceros auklet', "crested auklet', 'least auklet").
Prompt
(SEE)
Comprehensive Guidelines for Classilication:
(Score: 50.0)
1. Physical Characteristies:
Parakeet Auklet: Look for small bird (approx. 23 em), with striking blue bill and notable yellow eyebrow stripe. Its compuct body and vibrant colors ure key identifiers.
Rhinoceros Auklet: This medium sized hird (ahout 35 em) has stout body and distinctive horn-lik like projection
bill during breeding season. It typically weighs around 700 grams, so note its robust build.
Crested Auklet: Identify this species by its long. curved crest feathers and dark plumage, roughly 30
Weighing about 300 grams. it is often found in colonies, so observe any social behaviors.
anklets (around cm), features short wings grams. Its petite size and subtle coloration are distinguishing traits.
Least Auklet: The smallest
stubby bill, weighing about 130
2. Habitat and Behavior: Focus on typical habitals as coustal regions and rocky cliffs. Note behaviors like diving and feeding. For instance, the Rhinoceros Auklet may exhihit aggressive diving, while the Crested Auklet shows unique social interactions.
3. Visual Cues: Pay attention to specific visual markers such as bill shape, color, and the feather crest on the Crested Auklet. Cousistently assess the color of the feet overall body
4. Common Confusions: Be aware that the Rhinoceros Auklet appears more robust than the slender Least Auklet. Additionally, plumage colors may vary under different lightngg conditions.
5. cres-Veeriiiation Process: Use a methodical approach to verily clasilication.
instance, check dark plumage comesponds with the bill shape and size of the suspected species, and look for unique features.
6. Key Features to Focus On: Highlight aspects such beak shape, feather patterns, and eye color during evaluation.
7. Narrow Down Options:
image shows features that could apply to multiple species, focus on most prohable candidates based on observed visual cues and behaviors.
8. Confidence Level: Indicate your confidence the classification "certain," " "likely," "uncertain," " or "doubtful." Justify your coufidence by detailing which specific features influenced your assessment.
9. Step-by-Stee Checklist for Classification: Use this systematic checklist: Observe the size and body structure; Note the bill's color and shape: Examine plumage pullerns and unique features (e.g.. crests); Consider the observed habital and behaviors; Cross reference truits with the characteristics of all species.
Hypothetical Example Scenarios: Provide examples images and breakdown classificattons according to the guidelines, illustrating ellective application of features and observations.
Please provide the identified species name and your confidence level the classilication.
Image Query
Ground Truth: Crested Auklet
Key Features:
Dark plumage
Curved crest of feathers Orange beak
Response
Based provided image, let's analyze
step-by-step using the comprehensive guidelines:
1. Physical Characteristies: The bird has a robust body, which aligus with the Rhinoceros Auklet being described as medium sized (about 35 emn) and having a stout body. The bill orunge with black tip, which characteristic of the Rhinoceros Auklet. There is noticeable hike-like projection on the bill, which is key feature of the Rhinoceros Auklet during the breeding season.
2. Habitat and Behavior:
The bird is perched on a rocky cliff, which is consistent with the Rhinoceros Auklet's ' hobitat preference. The posture and behavior not provide additional specific clues do not contradict the identification either.
3. Visual Cues: The bill shape and color match the description of the Rhinoceros Auklet. The overall body structure and size consistent with the Rhinocerus Auklet.
Least Auklet is much smaller and shorter bill, which does not mutch the image. Crested Auklet has long, curved crest, which is not present here. The Parakeet Auklet is smaller and has a distinct yellow eyebrow stripe, which not seen here.
4. Common Confusions:
5. Vrrss-Veriiication Process:
The combination of the robust body, orange bill with a black tip. and the hike-like projection strongly supparts identification of the Rhinoceros Auklet.
6. Key Features to Focus On: The bill shape and color are critical features for this identificttinn
7. Narrow Down Options: The image clearly matches the Rhinoceros Auklet based on the physical characteristics and visual cues.
8. Confidence Level: Given the clear match with the Rhinoceros Auklet's physical characteristics and the absence of any conflicting features, am confident in the classification.
Conclusion:
The identified species is the Khinoceros Auklet, and my contidence level in this classification is certain.
Table 15: Visualization of the SEE result on the CUB (Auklet) dataset.
29

---

Published as a conference paper at ICLR 2026
Task Example: CUB(Auklet) Clussifccation
Optimized Prompl
Analyze the provided hybrid image species and classify the target image into one of the following categuries: I'parakeet auklet", "thinoceros auklet', auklet,,
VIIZAETT AUOLET
BUNGNIRSS ACOIET
(MPO)
auklet"). Focus
following critical visual
(Score: 80.3)
Bill Shape: Priocitize size and shape of the bill, comparing directly with the examples in the hybrid image. Be specific about curvature and thickness.
Observe overall body these compare to the reference images.
Plumage
any variations, noting
Identify unique markings or patterns on the body, ensuring to highlight any significant difterences that may classification.
Body Patterns:
Use the hybrid image guide for comparison and classification. Clearly
CHICT AUMUI!
LAET AUALET
mention any distinguishing traits that support your decision, and ensure to prioritize bill shape cases anbiguity.
Image Query
Ground Truth:
Crested Auklet
Key Features:
Dark plumage
Curved crest of leathers Orange heak
Response
The bird in the target image can be classitied as a "creeted auklet"" based on the Lollowwngg observations: Shape: bill relatively
slight curve tip. matches description "crested auklet" in the reference image, which has similar bill shape but with more pronuuneed curve. The "thinoceros auklet" also has thick bill with more pronounced curve, while the "parakeet aukler" and "least auklet" have thinner bills.
2. Plumage Color: The overall body color is dark, almost black, with some lighter areas around the neck and underparts. This is consistent with "crested auklet" the reference image. The "thinoerrs auklet" hus similar dark plumuge, the "parakeet auklet" and "least auklet" have lighter plumage.
3. Body Patterns:
The most distinctive feature of the "crested auklet" the presence of crest on its head, which is clearly has
visible target image.
"parakeet shoeter crest compared to the "crested auklet".
crests,
In conclusion, the combination
short and bill, dark plomage, and the presence crest on the head strongly suppoets the classification of the bird in the target image "crested auklet".
Table 16: Visualization of the MPO result the CUB (Auklet) dataset.
30


Reviews:
[{"review_id": "YthohZUI8l", "reviewer": "Reviewer_z13R", "summary": "This paper defines the new problem of multimodal prompt optimization and introduces MPO, a framework that jointly optimizes textual and non-textual prompts for multimodal large language models. The method combines alignment-preserving exploration across modalities with a prior-inherited Bayesian UCB strategy for efficient prompt selection. Experiments on diverse modalities (images, videos, and molecules) show consistent improvements over existing text-only prompt optimization baselines.", "questions": "(1) How much of the gain actually comes from the added non-textual modality rather than the optimization process itself? A controlled text-only ablation using the same search procedure would clarify this.\n\n(2) How robust is the “alignment-preserving” exploration? When the generated visual component drifts semantically, does the optimization recover or collapse?\n\n(3) The Bayesian UCB prior sounds appealing, but how sensitive is the performance to the prior strength? Could it bias the search toward mediocre parents if the correlation assumption breaks?\n\n(4) Since the framework depends on GPT-based visual and molecular generators, how reproducible is this pipeline for researchers without access to those models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The idea of extending prompt optimization beyond text is timely and relevant, filling a gap in the growing MLLM literature.\n\n(2) The paper is technically clear, well-written, and supported by convincing experiments on multiple modalities and model backbones.\n\n(3) The proposed Bayesian prior mechanism for efficient search adds a nice practical touch that improves optimization stability.", "weaknesses": "(1) The conceptual jump from text-only to multimodal prompt optimization is natural but not as novel as it’s presented; many parts resemble standard multimodal conditioning or input co-optimization.\n\n(2) The analysis lacks stronger insights into why multimodal prompts help; results show improvement but don’t probe interpretability, modality interactions, or failure cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "by3ZgcFE5g", "reviewer": "Reviewer_oPXv", "summary": "This paper focuses on multimodal prompt optimization for Multimodal Large Language Models (MLLMs), addressing two critical gaps in existing text-only Automatic Prompt Optimization (APO): underutilization of MLLMs’ multimodal capabilities, and inherent challenges (cross-modal inconsistency, sparse high-quality candidates) in multimodal prompt spaces.\n\nKey contributions:  \n1. Formalizes the multimodal prompt optimization problem, defining optimal prompts as text-nontext pairs \\((t,m)\\) that maximize MLLM performance on target tasks.  \n2. Proposes the **MPO framework**:  \n   - *Alignment-Preserving Exploration* (via cohesive backpropagation, joint multimodal update, and 3 complementary operators) ensures text-image semantic consistency;  \n   - *Prior-Inherited Bayesian UCB* leverages parent-child prompt performance correlation to solve cold-start, reducing evaluation budget by 42%-70%.  \n3. Validates on 10 datasets across 3 modalities (image/video/molecule), outperforming text-only APO (e.g., +8.6% accuracy on CUB-200-2011).  \n\nMPO fills the multimodal APO gap for MLLMs, with strong cross-model/multimodal generalization and practical efficiency.", "questions": "Same as the section of **Weaknesses**.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- **Originality**: Breaks the text-only limitation of existing APO methods, first formalizing the multimodal prompt optimization problem (defining prompts as text-nontext pairs \\((t,m)\\)). It also creatively improves Bayesian UCB by leveraging parent-child prompt performance correlation to solve cold-start, extending bandit-based selection to multimodal scenarios innovatively.  \n- **Quality**: Conducts rigorous validation—covering 10 datasets across 3 modalities (image/video/molecule), cross-model tests (Qwen2.5-VL, Gemma3), and ablation studies (verifying alignment mechanisms/operators’ necessity). Results are reliable and generalize well.  \n- **Clarity**: Clearly presents problem formulation, MPO’s two core components (with formulas and flowcharts), and experimental design. The logical structure is straightforward, enabling easy understanding of the framework.  \n- **Significance**: Fills the gap of multimodal prompt optimization for MLLMs, reduces evaluation budget by 42%-70% for practicality, and provides a foundational framework for future multimodal prompt research.", "weaknesses": "1. **Lack of Validation on Mainstream MLLM General Benchmarks, Limiting Evidence of Universal Adaptability**  \nThe current experiments rely solely on custom task-specific datasets (e.g., CUB-200-2011, PlantVillage) and fail to validate MPO on widely recognized MLLM multimodal benchmarks, leaving its ability to enhance MLLMs’ general capabilities unsubstantiated. Specifically:  \n- It omits **static multimodal foundational benchmarks** (e.g., MME, MMBench), which focus on core perceptual capabilities of MLLMs (e.g., image-text matching, attribute recognition)—there is no evidence that MPO can optimize prompt performance for these fundamental tasks.  \n- It excludes **long-video dynamic modality benchmarks** (e.g., VideoMME), which require handling temporal alignment between long-sequence videos and text (e.g., locating specific clips, understanding temporal logic). The paper’s existing alignment mechanism, designed for static images, remains untested for such long-video scenarios, casting doubt on its effectiveness.  \n- It neglects **cross-modal complex reasoning benchmarks** (e.g., ScienceQA, MathVista, Geometry3k)—benchmarks that demand MLLMs integrate multimodal information to solve logical reasoning or mathematical problems. Since the paper only validates classification/prediction tasks, there is no proof that MPO-optimized prompts can improve complex reasoning performance, leaving MPO’s adaptability to general MLLM scenarios unconfirmed.  \n\n2. **Unaddressed Implicit Deployment Costs, Missing Cost Comparison with Text-only APO**  \nWhile the paper emphasizes that the prior-inherited Bayesian UCB reduces evaluation budget by 42%–70%, it overlooks the significant computational overhead of generating multimodal candidates. Modal-specific generators (e.g., GPT-Image, video editing models) used for image/long-video clip generation incur much higher costs than text prompt generation: for instance, single-image generation takes 2–5 seconds (vs. 0.1 seconds for text generation), and diffusion-based image generators require 3–5 times more GPU memory than text models. Critically, the paper fails to compare the **total deployment cost of MPO** (including iterative multimodal generation overhead) with that of text-only APO. If the implicit costs of multimodal generation offset or even exceed the saved evaluation budget, MPO’s practical utility in real-world deployment would be severely undermined—this key cost trade-off is entirely unaddressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CMG7qZkRT8", "reviewer": "Reviewer_oxEi", "summary": "This paper introduces Multimodal Prompt Optimization (MPO), a new framework that extends traditional text-only prompt optimization to multimodal large language models (MLLMs) by jointly optimizing textual and non-textual components of prompts. Specifically, the method combines alignment-preserving exploration, which updates text and image prompts coherently based on failure-driven feedback, with a prior-inherited Bayesian UCB selection strategy that leverages parent–child prompt priors to improve sample efficiency. Experiments across image, video, and molecular tasks show that MPO outperforms strong text-only baselines while significantly reducing evaluation cost, suggesting that multimodal prompt optimization can better unlock the reasoning potential of MLLMs.", "questions": "1.Which posterior quantile is used for Bayes-UCB, and how do your theoretical conditions differ from standard Bayes-UCB assumptions?\n\n2.If you replace generated prototypes with retrieved in-distribution examples or OOD random images, how do performance and alignment change?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.The combination of alignment-preserving exploration with prior-inherited Bayesian UCB is technically sound and elegantly balances multimodal consistency with sample efficiency.\n\n2.The paper is clearly written and well-structured, with intuitive figures and pseudocode that align closely with the algorithmic flow, accompanied by informative ablations and visual analyses.", "weaknesses": "1.The paper frames “multimodal prompt optimization” as discrete text + generated visual prototypes, which risks conflation with learnable soft-prompting (e.g., MaPLe) and needs clearer terminology and positioning.\n\n2.The notion of “parent prompt” and its construction (especially for multi-parent mix) is not formalized.\n\n3.Including powerful image/video generators injects generator priors and extra compute, so gains may stem from tooling rather than the method itself.\n\n4.Efficiency is measured via evaluation counts rather than real cost, omitting overheads from generation and long-context prompting.\n\n5.Cross-modal alignment relies on a single metric (e.g., DSG).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nlQ4UHVRpy", "reviewer": "Reviewer_wMqy", "summary": "This paper targets the prompt optimization of MLLMs. Unlike prior approaches that only optimize text prompts, the authors incorporate information from another modality into the prompt.\nTo optimize the multi-modal prompt, the authors propose MPO which consists of two key components: alignment-preserving exploration of multimodal prompt space and prompt selection with prior-inherited Bayesian UCB.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is good, which revolutionizes the conventional prompt structure for MLLMs.\n2. The paper is well-written, with clear motivation and challenges.\n3. The experiments are thorough.", "weaknesses": "1. In the first paragraph of section 3.2, the authors mentioned that \"a naive approach that independently updates textual and non-textual components risks producing misaligned prompts\". Have the authors tested this naive approach (is it Random Image Prompt in Figure 4? If yes, what are the real performance values instead of performance gains?).\n2. When identifying the failure set, how is your multimodal prompt initialized? As it contains both text and image, how are they selected and organized?\n3. Another concern is the efficiency: extra models are included to refine the multi-modal prompt. Compared to the baseline methods which optimize textual prompt only, what is the time (or computation steps) used by MPO to complete one iteration?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "khnsyc7e7v", "reviewer": "Reviewer_DhB1", "summary": "The paper proposes the Multimodal Prompt Optimizer (MPO), which jointly refines textual and non-textual prompts through alignment-preserving updates and employs a Bayesian UCB strategy for efficient candidate selection. Experiments across images, videos, and molecules show that MPO outperforms text-only baselines, highlighting multimodal prompt optimization as a key step toward advancing MLLMs.", "questions": "- Is there a clear order in which the three Exploration Operators designed in this paper are called, and how are they combined? Figure 6 shows that there doesn't appear to be a clear order in which operators are called, and the combinations are arbitrary. The original paper mentions that these operators systematically expand, refine, and recombine non-textual prompts. Could you explain this systematic approach in more detail?\n- In the Appendix (lines 665–670), the authors mention that for certain datasets, they further selected groups containing three or four distinct species to maintain a balanced level of difficulty. However, this strategy effectively reduces the overall task complexity, as the generated multimodal prompts only need to distinguish among a small number of categories. It is recommended that the authors further discuss whether the proposed method can still maintain good performance and stability when the number of categories increases significantly. \n- The generation of non-textual multimodal prompts in this paper appears to rely heavily on high-performance image generation and editing models. It is suggested that the authors clarify the extent to which their method depends on the specific capabilities of these models. If alternative image generation or editing models were used, would the proposed approach still maintain its effectiveness and robustness? If the authors can provide convincing explanations or additional evidence regarding these questions, I would be willing to consider increasing the score.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper expands the prompt optimization space beyond text to incorporate multiple modalities, which represents an interesting and promising research direction.\n- MPO shows strong performance across diverse modalities (images, videos, molecules), demonstrating its effectiveness.\n- The paper is clearly written and easy to follow.", "weaknesses": "Although the paper claims that MPO is not an Instance-Specific Prompting and Optimization method but rather aims to discover a single, reusable prompt that enhances performance across an entire task, the evaluation still relies on fine-grained dataset partitioning and reducing the number of classes to control task difficulty. This design choice appears to weaken the claimed generalization ability of the proposed method, suggesting that its effectiveness in more complex or diverse scenarios remains to be validated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)