Multimodal Policy Internalization for Conversational Agents
Authors: Zhenhailong Wang
Jiateng Liu
Amin Fazel
Ritesh Sarkhel
Xing Fan
Xiang Li
Chenlei Guo
Heng Ji
Ruhi Sarikaya
Venue: ICLR 2026 Poster (2026)
Keywords: Conversational AI
Multimodal models
Policy internalization
Reinforcement learning with verifiable rewards
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce a new task, multimodal policy internalization, along with a robust three-stage training algorithm that embeds complex policies into model parameters, enabling efficient and reliable policy-following without lengthy prompts.

Abstract: Modern conversational agents such as ChatGPT and Alexa+ have become indispensable in everyday life. To handle diverse business requirements and enable agentic capabilities, these LLM-based systems often rely on predefined policies, which specify instructions such as model metadata, response styles, and tool-using rules. These policies, typically implemented as in-context prompts, are becoming increasingly complex and lengthy, posing challenges for models in faithfully following them. Moreover, they impose a large fixed computational cost regardless of the input query. As multimodal conversational agents emerge, complex policies that govern multimodal tasks and even involve visual instructions are becoming increasingly necessary, yet they have been rarely studied in previous work. In particular, prior work on prompt compression has focused solely on reducing the length of task templates and demonstrations, which require limited reasoning compared to policies. Meanwhile, related work on policy alignment has been limited to internalizing text-only safety instructions. To bridge this gap, we introduce Multimodal Policy Internalization (MPI), a new task that aims to internalize reasoning-intensive multimodal policies into the parameters of a large multimodal model, enabling stronger policy-following behavior without requiring the policy to be included in-context during inference. MPI presents unique challenges from both data and algorithmic perspectives. We construct two new datasets that cover complex decision-making and tool-using tasks across both synthetic and real-world visual inputs. We investigate diverse internalization strategies and propose a novel three-stage training framework, TriMPI, which enables stronger guidance from the original policy during internalization. Specifically, we first introduce a continual pretraining stage before supervised finetuning, which directly injects policy knowledge into the model. We then propose PolicyRollout, a simple yet effective extension to GRPO-style RL algorithms, which enables more grounded exploration by augmenting the rollout space with policy-aware responses. We show significant improvements of TriMPI over strong baselines in end-to-end performance, generalization capability, and robustness to catastrophic forgetting. As the first work on multimodal policy internalization, we aim to build a strong foundation for future research by providing datasets, training recipes, and comprehensive evaluations.

Published as a conference paper at ICLR 2026
MULTIMODAL POLICY INTERNALIZATION FOR CON- VERSATIONAL AGENTS
Zhenhailong Wang', Jiateng Liu', Amin Fazel", Ritesh Sarkhel', Xing Fan", Xiang Li², Chenlei Guo", Heng Ji Ruhi Sarikaya² 'University of Illinois Urbana- Shaanaa-happainn, 2Amazon wangz381 wwiioii..eeei........ .ddu, jihjeamazon. com
ABSTRACT
Modern conversational agents such as ChatGPT and Alexa+ have become indis- pensable in everyday life. To handle diverse business requirements and enable agentic capabilities, these LLM-based bbseeddd systems often rely on predefined policies, which specify instructions such as model metadata, response styles, and tool- usinu rules. These policies, typically implemented as in-context prompts, are becoming increasingly complex and lengthy, posing challenges for models in faithfully fol- lowing them. Moreover, they impose large fixed computational cost regardless of the input query. As multimodal conversational agents emerge, complex policies that govern multimodal tasks and even involve visual instructions are becoming increasingly necessary, yet they have been rarely studied in previous work. In particular, prior work on prompt compression has focused solely on reducing the length of task templates and demonstrations, which require limited reasoning com- pared to policies. Meanwhile, related work on policy alignment has been limited to internalizing text-only safety instructions. To bridge this gap, we introduce Multimodal Policy Internalization (MPI), new task that aims to internalize reasoning- cessoning-intesive multimodal policies into the parameters of large multimodal model, enabling stronger policy-following behavior without requiring the policy to be included in-context during inference. MPI presents unique challenges from both data and algorithmic perspectives. We construct two new datasets that cover complex decision-making and tool-using tasks across both synthetic and real-world visual inputs. We investigate diverse internalization strategies and propose novel three-stage training framework, TriMPI, which enables stronger guidance from the original policy during internalization. Specifically, we first introduce a contin- ual pretraining stage before supervised finetuning, which directly injects policy knowledge into the model. We then propose Policy Rollout, a simple yet effective extension to GRPO-style RL algorithms, which enables more grounded exploration by augmenting the rollout space with policy-aware responses. We show significant improvements of TriMPI over strong baselines in end-to-end performance, general- ization capability, and robustness to catastrophic forgetting. As the first work on multimodal policy internalization, we aim to build a strong foundation for future research by providing datasets, training recipes, and comprehensive evaluations.
INTRODUCTION
Conversational agents such as ChatGPT, Claude, and Alexa+ (OpenAI, 2025; Anthropic, 2025; Alexa AI, 2025) have become integral to daily life. To manage diverse business rules and enable agentic funetionality, these LLM--baed systems often rely predefined policies, which are structured instructions that specify model metadata, response styles, tool-usage rules, and more. These poli- cies, typically provided in-context prompt prefixes, becoming increasingly long and complex (estimated to range from ~ to ~50K tokens"), imposing substantial fixed computational cost re- gardless of the query size. In contrast. typical user queries usually range from 50 to 200 tokens (Clark et al., 2025), leading to 20x to 250 higher input token cost from the policy prompts compared
"Exact numbers are disclosed due the proprictary nature of system prompts.

---

Published as a conference paper at ICLR 2026
Response not faithfully following policy
Desision Making VQA
Complex Policy
Query
Multimodal Model
Response
Lengthy
Short
Multimodal
Fixed token consumption regardless query
Policy Intemalizationn
Improves policy following Preserves general capabilitiis
Improves prefill efficiency
Supports policy update
Query
Internalized Model
Policy-Complic Compliant Response T
Figure 1:
Motivation of the proposed Multimodal Policy Internalization task. The goal is to enhance the policy-following abilities of large multimodal model without requiring the policy to be provided in-context during inference, thereby improving both performance and efficiency.
with the actual user queries. Moreover, as these policies expand and become more reasoning- inntensive (e.g., requiring the model to follow rules distributed across different sections), models often struggle to adhere to them consistently (Yao et al., 2024; Qian et al., 2024). A natural research question arises: can we internalize the knowledge of policies into the model parameters while also improving the model's 's policy-following abilities?
Previous research on prompt compression (Li et al., 2024) has shown initial success in reducing token usage through hard prompting (Li et al., 2023; Jiang et al., 2023), soft prompting (Mu et al., 2023; Ge et al., 2023), or progressive fine-tuning (Zou al., 2024). However, these approaches primarily focus on compressing templates and demonstration examples, which require minimal reasoning to adhere to the policy. More recent work (Guan et al., 2024) introduced deliberative alignment, which aims to internalize knowledge of more complex safety specifications and emphasizes improving policy-following performance beyond token reduction. Nonetheless, it remains limited to issues of model trustworthiness and has been explored only in text-only models. With the growing trend of multimodal conversational agents, policies are increasingly tied to multimodal tasks and may even include visual instructions such demo images. Yet, prior work has explored how learn and internalize complex policies in multimodal models.
To address this gap, we propose new task, Multimodal Policy Internalization (MPI, which aims to train multimodal models that can generate policy-compliant responses without requiring the policy to be included in-context. Compared with prior work, the MPI task introduces several unique challenges: (1) The target policies focus on reasoning-intensiv multimodal tasks, such as decision
decision-making making and tool usage for conversational agents. (2) There a lack of existing datasets containing multimodal question-answer pairs that adhere to predefined policies. (3) There is a lack of established training paradigms for internalizing multimodal policies. For example, some text-only knowledge injection methods, such as continual pretraining (Ovadia et al., 2025; Maini et al., 2024), may not be directly applicable to MPI due to the presence of visual tokens.
In this work, we contribute from both data and algorithmic perspectives to advance research in multimodal policy internalization. We first construct two new datasets, ClevrPolicy and GTAPolicy, to support training and evaluation across different multimodal policy types. Specifically, ClevrPolicy focuses on internalizing complex decision-making policies that require muuli--hop reasoning. Built upon the synthesized CLEVR dataset (Johnson et al., 2017), ClevrPolicy provides flexible control over policy complexity and dataset size, enabling in-depth investigation of the effectiveness of different algorithms. Curated from the GTA dataset (Wang tool-usage instructions with real-worls images and user queries. GTAPolicy emphasizes a low-data regime, where only limited question- -anseering data is available to demonstrate the expected behavior.
al., 2024a), GTAPolicy targets multimodal
From the method perspective, a key challenge lies in how to align the response with the policy without assuming its existence during inference. Prior methods (Zou al., 2024; Guan al., 2024) focus solely on learning policy-following behavior, leaving it unclear how to better leverage the policy context itself as guidance during training. To address this, we introduce novel three-stage training framework, TriMPI, which incorporates two key ideas: (1) warming up the model through continual pretraining directly on the policy context; and (2) new RL algorithm, Policy Rollout, which enables additional conditioning on the policy context without introducing gap between training and inference. Specifically, TriMPI consists of a essually-masker continual pretraining (VM-CPT) stage, chain-of- supervised finetuning (CoT-SFT) stage, and reinforcement learning (RL) stage with Rollou.. The VM-CPT stage enables language modeling directly on the multimodal policy, thereby explicitly injecting the entirety of the policy knowledge into the
2

---

Published as a conference paper at ICLR 2026
ClerPPolic-T
ClevPulicy-M
pally
ClevPolicy- Example
Decusion Nodes
Cundition I: Chock if there cyan
Condition I: Chock if there is object that.
The fellowing policy proviles docsion- making fnuneseek for proessing visual aupu, You should always stat will caeoking the lightss-eve scodition.
or
Leaf Nodes
Yes
No
-n Palicy: LS-Surieu
Condition check ifttere any cyen abject
Input Image
Decision: Tree
Condition L.I: check any small object
Condition 1.2:
any rubber object
to
ortinee Condition
Otheratec contine with Condition
yooes fhees only exan ebjects, and
Candition
(Surinu), i poovide your resporse for this image
Fullow Pulicy
Query:
Layers
object
Yes
No
yos only cran objects,
GT Answer: 'Caz
Respond "Case 0'
Respond "Case 1*
QA Example (ClevrPoliey)
Policy Creation (ClevrPoliey)
ropund
Figure 2: ClevrPolicy dataset. Left: Illustration of policy generation, where decision tree is first generated and converted into natural language instructions (see Appendix C.I for details on the decision node ontology, and Figures 18, 19 in the Appendix for full policy examples). Right: Example input-output pair corresponding to the policy. The policy is available only during training and not during inference.
model and facilitating reasoning in later stages. The RL stage is essential for internalizing reasoning- intensive policies, as it enables the model to learn from a broader range of policy-related responses through wail-and-errop rather than memorization. PolicyRollout further enhances RL exploration by augmenting the rollout space with policy-grounde responses, serving as a simple yet effective extension to GRPO (Shao et al., 2024) and DAPO (Yu et al., 2025).
We demonstrate that TriMPI yields significant improvements in MPI, approaching 70.7% and 79.4% absolute gains over CoT SFT baseline and in-context setting, respectively. Beyond end-task perfor- mance, we also demonstrate enhanced generalization capability to policy updates and robustness against catastrophic forgetting. Further analysis shows that the improvements of TriMPI are consis- tent across different policy complexities and model sizes, with more pronounced gains observed on main contributions
complex policies. To summarize, internalization of complex policies in the multimodal domain; (2) two new datasets that support both analytical and real-world training and evaluation; and (3) a training algorithm that introduce an effective training paradigm with a customized RL algorithm for policy internalization.
threefold: (1) new task that targets the
PROBLEM FORMULATION
We formally define the proposed task. Multimodal Policy Internalization (MPI), as illustrated in Figure 1. Consider a multimodal conversational agentic task, where, given an input text query Q and visual inputs I, the expected output is textual response A, which can be either free-form natural language or structured code for tool calling. Additionally, the response behavior should follow the instructions predefined in policy context P. Note that the policy may include both textual Pr and visual Pt components. Let Ma denote multimodal model parameterized by The following illustrates the expected inputs and outputs before and after internalization:
A M,
Policy Internalization
= Mo(Q,
(1)
The goal to embed the knowledge of the policy into the model parameters 6, enabling better policy-compliant generation without requiring P in-context during inference. MPI shares the same high-level motivation as deliberative alignment (Guan al., 2024; Zhang al., 2025). where we emphasize improving models' alignment with the policy beyond compressing the prompt. We do not consider training additional special embeddings as in soft prompting, since they are inherently tied to specific tasks (Li & Liang, 2021; Patel et al., 2025) and therefore limit the model's 's ability to maintain general reasoning capabilities and robustness (Fan et al., 2025; Bailey et al., 2023).
DATASET CREATION
3.1 CLEVRPOLICY DATASET
We first create ClevrPolicy, a new dataset focused on sessoning-inteniiee visually dependent decision- making. ClevrPolicy is built upon images and scene graphs from the Clevr dataset (Johnson et al.,
3

---

Published as a conference paper at ICLR 2026
AND Tael Desrriptinss dii +13 Types
Query:
Type: Cuiculatee
RM User Prodik RVp
Nome Yendanc Ciiiioterr
Paear taer 11540 "acocurr, pype" "D", "aye"
Deseription: Aollellter tool. The leput mus be single Pythou expression...
"credii": 259)
Argameats: [ (""ame" "oxprascun", "opp" "aod"y sn i]
Versioning Mechanlsm
Uver Query 4Np prageimaaz
simagee image fimgge jng. simage-
Typec OCR
How many dollars chorld pey for Eigar the pectooo accenday the price iness?
Name Versione OCR, vl, v2, OCR va, OCR
vs
Description: tock can resagrize icci on ieput maage
Inter action History
Arguneats: [""aaze"" "angee" "ope"" "iroge";...l
Assistant:
image deseciption
imsaees.
Product
Price
Image "mmgellmage 203 cogras, huth p ploood
Tael
soutt
flcenesey
BACARD
528 732
Calling
Buies
Unr-conditiaanl Reavaning
bookgrond
Hennery
Taaldj:
APEROR
S24.768
cquablc "Tool Descriptions" and the Colling and query determine appingrisen tool the
stole
crddits 750, "OUR_yf ofr ifoccourt type (A.
TALISRER
$67.272
oteewiie 1OCR
(foor credis 750, "OCR yot irage oterstse "OUR, .i"", "MaaLOC"" isseed
interaction
Input Images
costains matheratical the "OCW a* took...
GT Answer:
QA Example (GTAPolicy)
"rame"" "OUR x5"
Policy Creation (GTAPolicy)
"arguments" ("mmag"" "mregelingge 208 jpt
Figure 3: GTAPolicy dataset. Left: illustration of the policy, consisting of two major parts, tool description and tool calling rules (see Figure 20 in the Appendix for the full policy). Right: input and output example corresponding the policy. The visual input can contain multiple images.
Table 1: Zero-shot in-context performance the ClevrPolicy and GTAPolicy benchmarks. The
metrics are reported as accuracy percentages (%) and are detailed in Appendix C.3. We observe a strong correlation between the number of decision-tree layers (N) and policy complexity. Introducing multimodal demonstrations into the policy (in ClevrPolicy-M) further increases the difficulty.
ClevrPolicy-T
ClevrPolicy-M
GTAPolicy
Model
N=2 N=4 N=6
N=2 Nm4
N=6
Tool Acc
Arg Score Overall
Qwen2,5-VL-3B
36.60 10.85 4.80
33.05
8.05 4.55
18.87
15.44
17.15
Qwen2.5-VIL-7B
72.00 32.20 13.15
51.20
13.90
5.65
23.58
19.44
21.51
Claude-3.7-Soone
97.65
92.60
85.35
95.00
82.75
77.63
42.45
37.13
39.79
Claude-4-Sonnc
98.10 96.70
90.10
93.40
78.55
77.76
60.38
51.53
55.96
2017),, enabling fine- -raiind control over policy complexity and supporting comprehensive evaluation of different MPI algorithms. Here, policy complexity refers to how difficult it is to follow the instructions in the policy when it is provided in-context. To construct the policies, we generate binary decision trees in which decision nodes specify visual conditions and response nodes define actions, with tree depth determining complexity. Each decision tree is then converted into a structured natural language instruction the final policy. We provide two variants of ClevrPolicy: ClevrPolicy-IT where policies are purely bextebased, and ClevrPolicy-M, where policies include image demonstra- tions as part of the decision node conditions. Figure 2 illustrates the policy creation process and shows an input-output example. Further details are provided in Appendix C.I.
3.2 GTAPOLICY DATASET
We further introduce GTAPolicy, which focuses on complex tool-using policies with real-world images and queries. We particularly consider a low-data regime, reflecting a common practical challenge in real-world settings where only limited QA pairs are available for training. The policy in GTAPolicy is constructed from the GTA dataset (Wang et al., 2024a), containing tool descriptions for 13 tools and 24 tool-calling rules with versioning and user-conditional mechanisms to simulate real-world business constraints. The policy can be automatically expanded when additional tools are provided. Training data reformulates multi-turn interactions into single-turn tool-calling tasks, where each instance includes visual inputs, user profile, a query, an interaction history, and an instruction prompt, with the expected output being a JSrN-formatted tool call specifying tool name and arguments. Figure 3 illustrates the policy and an onput-output example in GTAPolicy. Further details on the GTAPolicy dataset are provided in Appendix C.2. The statistics and evaluation metrics for both datasets presented in Appendix Table 7 and Appendix C.3, respectively.
3.3 ZERO-SHOT IN-CONTEXT RESULTS ON CLEVRPOLICY AND GTAPOLICY
We conduct an in-context evaluation on the proposed two benchmarks using off-the-shelf models, where the policy is directly inserted into the inference prompt. This evaluation provides useful
4

---

Published as a conference paper at ICLR 2026
Masking
Pollcy
TrMPP (ours)
Policy
Query
Query
Quiry
Query
Query
Policy
Colf
Reqpunac
Answer
Answer
Answer
Answer
Direet SFT
CoT SFT
CUT
PolicyRollent
Figure 4: Overview of different training algorithms for multimodal policy internalization. The solid purple outlines indicate the parts where the next-token prediction loss is computed. On the right, we illustrate the proposed three-stage training strategy, TriMPI, which enables direct policy knowledge injection through the VM-CPT stage and policy-gnan rolicy-grounde reinforcement learning through PolicyRollout. The PolicyRollout algorithm is detailed in $4.3 and illustrated in Figure 5.
reference for assessing the complexity of the policies. The in-contes. performance also provides guidance on which methods to rely on for generating chain of thought (CoT) data for MPI. As shown in Table 1, on ClevrPolicy, performance decreases as the layer number increases. Even the strongest model, such as Claude-4 (Anthropic, 2025), begins to struggle at N=6 particularly on ClevrPolicy-M. GTAPolicy also poses significant challenges for all models, with the best tool accuracy only at 60%.
4 MULTIMODAL POLICY INTERNALIZATION (MPI) ALGORITTMM
4.1 BASELINES
Inspired by previous work on prompt compression (Zou al., 2024) and deliberative alignment (Guan et al., 2024) in text domains, we consider the following baseline methods for MPI.
Direct SFT. We directly train on the non-CoT data using supervised finetuning (SFT), aiming to learn mapping from the imput to the answer without performing intermediate reasoning.
CoT SFT. We first obtain chain-of- thhouughttt (CoT) data which the model explicitly reasons over the rules in the policy before producing the final answer, and then perform SFT on this CoT data. To generate CoT data for ClevrPolicy and GTAPolicy, we adopt different approaches depending on task complexity and data availability. Further details are provided in Appendix D.
4.2 TRIMPI: A THREE-STAGE TRAINING STRATEGY FOR MPI
We observe key limitation in the baseline methods: they focus solely on learning the expected behavior without direct access to the original policy. This limitation becomes more evident for complex policies, where directly learning output patterns is increasingly difficullt. This raises the following question: can we better augment the learning process with the policy, without introducing gap between training and inference? Empirically, the prompt during training but removing it during inference results in near-random performance. To address this challenge, we propose TriMPI, three-stage training framework that warms up the model via continual pretraining on the original policy and introduces Policy Rollout, a new RL algorithm that enables more policy-aware exploration (detailed in $4.3).
find that simply inserting the policy into
TriMPI consists of three stages: (1) Visually-Maseee Maskedd Continual Pretraining (VM-CPT); (2) Super- vised Finetuning with Chain-of-thouge (CoT SFT); (3) Reinforcement learning (RL). The CoT SFT stage is identical to the baseline described in $4 We present the details on the VM-CPT and RL stage next. An illustration of the baselines and TriMPI stages is presented in Figure 4.
VM-CPT Stage. This stage aims inject policy knowledge directly into the model parameters before performing SFT. Specifically, we construct a new variant of the CoT dataset D. which contains input-output sequences
(Py. P,. I, Q, C, A) defined the concatenation of tokens from the policy P= (Pr, P,), the visual inputs I, the text query Q, the CoT reasoning C., and the final answer A. We then compute the next-token prediction loss all tokens except the visual tokens.
5

---

Published as a conference paper at ICLR 2026
Pollcy Applics
Mininiee
KL
A
Og
Reference Model
Pg
Aq
Policy Model
Rollout
Group Estimate
Image
Query
Answer Verifier
FG
A
A
Policy
dj
dg
A/
Policy Model
Rollout
PolicyRollout (ours)
Image
Query wi
Policy
Policy-Aware Responses
Figure Illustration of the PolicyRollout algorithm (applied to GRPO an example). During the rollout phase, we additionally construct set of input instances with the policy included in-context. These policy-aware responses are added to the rollout space as if they were generated from the original inputs without the policy in-context. The advantage and policy gradient then computed on the combined rollouts, indicated by the thick red outlines. PolicyRollout enables more policy- aware exploration without introducing a gap between training and inference, leading to significant improvements in MPI, especially on complex policies.
((0) EIND
me (x see) m: 1/2 P
(2)
EE
Me tm1
The visual masking enables to adopt CPT (Ovadia et al., 2025; Maini et al., 2024) in this multimodal domain, where continuous visual tokens may appear both the input and the policy Pr. and it has shown empirical success despite its simplicity.
RL Stage. As the target policy becomes more complex and -intonniinintteivee.e SFT baselines face the challenge of insufficient coverage of policy-related behaviors, particularly low-data regimes. Inspired by recent advancements in reinforcement learning with verifiable rewards (RLVR), which can effectively learn from negative samples and exploration, we investigate its effectiveness for the MPI task. We first adopt GRPO (Shao et al., 2024) and DAPO (Yu et al., 2025), following standard response format and reward design. Specifically, we ask the model to generate thinking block enclosed within <think><< thhink></thha </thin>>>>> and answer block enclosed within (boxed(), on which we compute both a format reward and an accuracy reward (detailed in Appendix C.3).
From our initial experiments, observe that although GRPO and DAPO yield substantial empirical gains, exploration in the RL stage remains insufficiently grounded in the policy. This limitation becomes more pronounced with complex policies, where ungrounded exploration rarely produces positive rewards. Simply adding the policy in-context during training is ineffective, since the model lacks access the policy inference time, introducing misalignment between training and inference. In the next section, we present our solution through a modified rollout phase.
4.3 POLICYROLLOUT (PoRo): POLICY-AWARD REINFORCEMENT LEARNING
To further improve the effectiveness of the RI. stage for MPI, we introduce PolicyRollout (PoRo), simple yet effective extension to GRPO-style algorithms that augments the rollout space with policy-aware responses. Specifically, during the rollout stage, for each sampled instance, we construct variant by inserting the policy in-context. We then allow the current policy model to generate an additional set of responses conditioned on the query Q, the image I, and the policy P. These policy-aware responses concatenated with the no no-policy responses to form the rollout space, after which we proceed with group-based advantage estimation. An illustration is provided in Figure 5, and the PolicyRollout objective (applied to GRPO example) is written follows:

---

Published as a conference paper at ICLR 2026
Table Main results and ablations multimodal policy internalization performance. By default, we use Qwen2.5-VL-77 as the base model. PoRo refers to PolicyRollout. The metrics are reported as percentages (%) and are detailed in Appendix C.3. We observe significant improvements of TriMPI over in-context and SFT baselines. Comprehensive ablations demonstrate the importance of each stage and the effectiveness of PolicyRollout. The RL steps indicate the actual update steps (for the three datasets, respectively, separated by "I"). Early stopping (marked with "*") may occur in DAPO (Yu et al., 2025) runs due to its dynamic sampling strategy. Notably, on DAPO, TriMPI achieves competitive stronger performance while using fewer steps.
Stages CPT SFT RL
ClevrPolicy-T
ClevrPolicy-M Ace (N=6)
GTAPolicy
Method
RL Steps
Acc (N=6)
Tool Acc Arg Score
Overall
7ero- Shot with Policy In-Context
In-Context
13.15
5.65
23.58
19.44
21.51
SFT Baselines for MPI
Direct SFT
15.15
14.55
44.34
37.16
40.75
CoT SFT
17.80
14.30
57.55
51.45
54.50
TriMPI Ablation without RL Stage
VM-CPT SET
/
22.75
27.05
69.81
61.13
65.47
TriPP Ablation without VM-CPT Stage
CUT SFT
/
104
64.50
76,42
72.22
CoT SFT
104 150
74.40
76.42
68, 44
72.43
TriMPI without Policy Rollout (PoRo)
TrMPI GRPO
/
104
80.80
83,96
74.70
79.33
/
TrMPI DAPO
70" 770 [50
81.45
79.25
70.85
75.05
THMPI with PolicyRollout (PoRo)
TriMPI w/ PoRo-GRPO
/ / /
/
/
104104500
65.85
84.70 85.00
85.85
76.28
81.06
TrMPPI w/ PoRo-DAPO
90" (75 [50
77.80
80.19
71.82
76.01
26
(PARO-CIRPO(O) - (0100))) (uy] (0)) (010..1.11) 10000000000000000000000000000010 Σ{
min (., (r.) -. a. (,) A.] BDKL [TT) [Tres] r.(0)
To(o:1(11) To.i (os Q)
(3)
The blue part highlights the main modification compared with the original GRPO objective. Note that the policy gradient is applied only to the no-policy path (conditioning solely on and I), thereby ensuring that the training and inference remain aligned.
EXPERIMENTS
We conduct comprehensive experiments multimodal policy internalization, evaluating MPI task performance ($5.1), generalization capability ($5.2), policy knowledge injection ($5.3), and robustness to catastrophic forgetting ($5.5). By default, we use Qwen2.5- - as the base model. We tune all model parameters in the VM-CPT and RL stages, and apply LoRA (Hu et al., 2022) finetuning in the SFT stage. For ClevrPolicy, otherwise specified. Additional implementation details are provided in Appendix B.
the most complex policies (N 6) unless
5.1 MAIN RESULTS
MPI Task Performance. In Table 2, we present evaluation results on ClevrPolicy and GTAPolicy in terms of task performance, Our best-performing model achieves to 70.7% and 79.4% absolute gains accuracy over the CoT SFT baseline and the in-context setting, respectively. We also find that the relative superiority of GRPO and DAPO varies across datasets: DAPO performs better on ClevrPolicy, while GRPO excels on GTAPolicy. We hypothesize that this difference arises because DAPO makes bolder updates due to the removal of the reference KL. This leads to faster learning ClevrPolicy, which contains more abundant and diverse data, but results in overfitting on GTAPolicy, which has very limited data. We further present qualitative example in Figure in the Appendix,
7

---

Published as a conference paper at ICLR 2026
Prompe Taken Reduction
Prefill Inference Time (4)
5490
Clantfucc- (N-5)
88
D:75
Cleenhllees- (N--0) ClevePlliy
5000
ClesPiicy (X
07
GTAPely
GTAPeliy
93,9% Reduction
85,7% Reduction
2000
1208
92
236 506
8.1
Before MP]
After MPI
Before MPI
After MPI
Figure 6: Efficiency metrics before and after MPI.
Table 3: Left: Policy Override results. We show that TriMPI consistently outperforms strong baselines in generalizing to updated policies, demonstrating favorable real-world usage where model behavior can be governed by both internalized policies and in-context instructions. Right: Policy Referral results. We use Claude-4 to rank the consistency between the model's intermediate thoughts and the original policy on scale of 10-00. A higher score indicates better embedded policy knowledge.
ClevrPolicy Override (N=6)
GTAPolicy Override
Policy Referral (0-10 Scores)
Method
ClevrPolicy-T ClevrPolicy-M -
Tool Ace
Arg Score
Overall
CleviPolicy-T ClevrPolicy-M GTAPolicy
In-Context
13.15
6.55
20.75
18.11
19.43
SFT
5,40
33.42
36.99
CoT SFT
16.40
25.20
42.45
34,98
38.71
3.54
3.44
7.68
CoT SFT GRPO 30.80
34.00
50.94
41,60
46.27
5.04
6.74
8,93
CoT SFT DAPO
41.60
37.60
58.49
50.12
53.31
5.70
6.74
8.73
TriMPI PoRo-GRPO
82.70
59.03
63.00
8.72
9.45
TriPPI PaRo-DAPO
59.40
85.15
63.21
54.66
58.94
8.35
9.13
showing that TriMPI achieves better alignment with the policy in both intermediate thoughts and final answers.
Ablation Analysis. In Table 2, we additionally provide a comprehensive ablation study on the key components proposed in TriMPI, including the RL stage, the VM-CPT stage, and the PolicyRollout algorithm. The findings are as follows: (1) The RL stage contributes most of the improvements compared with SFT baselines, highlighting the importance of learning from experience for complex, reasoning-intan intensive policies. We also observe unique benefit of the RI. stage in better leveraging non-CoT data, which is typically more abundant than CoT data, as shown in Table 11 in the Appendix. (2) the VM-CPT stage brings further improvements to both the SFT and RL stages, with the effect being more pronounced in the RL stage due more grounded exploration; and (3) PolicyRollout yields additional gains over the original GRPO and DAPO algorithms.
Efficiency Analysis. We report efficiency metrics in Figure 6, including the number of prompt tokens and the prefill inference time (i.e., the first forward pass on the input prompt) before and after internalization. All metrics are computed on Qwen2.5- VL.-B.. With the policy removed from the prompt, we observe reductions of up 93.9% in prompt tokens and 85.7% in prefill inference time.
Impact of Policy Complexity and Model Size. We further investigate the impact of policy com- plexity and model size on the effectiveness of different MPI algorithms. As shown Table 8 in the Appendix, we conduct additional experiments on (1) N = 4 policies in ClevrPolicy and (2) 3B models. Our findings are twofold: (1) TriMPI yields more pronounced gains over the baselines on complex policies, while the performance gap is smaller simpler policies (e.g.. N = 4). This high- lights the importance of our method for handling increasingly complex policies real-world settings. (2) TriMPI consistently outperforms the baselines on 3B models, demonstrating the generality of the method.
5.2 POLICY OVERRIDE: EVALUATING GENERALIZATION TO POLICY UPDATES
In real-world scenarios, policies may be frequently updated partially overridden by new rules. Ideally, the internalized model should generalize to these updated policies when they are provided in-context. To evaluate this capability, introduce evaluation setting, Policy Override,
8

---

Published as a conference paper at ICLR 2026
Table Policy In- In-Context results. We evaluate models after MPI training with the policies placed back into the context. This aims to demonstrate the effectiveness of the proposed methods regardless of the requirement remove policy inputs during inference for efficiency. We find that TriMPI still achieves superior performance under this setting. The base model is Qwen2.5-VL-7B.
ClevrPolicy (N=6)
GTAPolicy
Method
CleviPolicy-T
CleviPolicy-M
Tool Acc
Arg Score
Overall
In-Context (no MPI)
13.15
23.58
19.44
21.51
CoT SFT CaT SFT + GRPO
10.70
15.80
45.28
39.24
42.26
19.00
20.90
60.38
53.64
57.01
TriMPI w/ PoRo-GRPO
35.15
69.90
70.75
62.86
66.81
where unseen policy content specified in-context during inference with internalized models. As illustrated in Figure 10 in the Appendix, for ClevrPolicy we retain the policy name but modify the conditions, while for GTAPolicy we alter the tool-calling rules, resulting in a different choice of tool version. Table (left) shows that TriMPI consistently outperforms all baselines and ablated settings, demonstrating stronger generalization beyond merely fitting to a specific policy.
5.3 POLICY REFERRAL: EVALUATING POLICY KNOWLEDGE INJECTION
Beyond end-task performance, which measures how well the model generates policy-compliant responses, further investigate how well the model embeds the policy knowledge itself. To this end, we consider a new evaluation setting, Policy Referral, where we leverage LLM-as-a-judde to examine the intermediate reasoning process of the model's responses. The goal is assess whether the referral to the original policy accurate and to assign a score between 0 and 1. Figure in the Appendix illustrates this evaluation setting, and Table 3 (right) presents the results, reporting the average score over subset of 100 test responses from each dataset. We find that TriMPI achieves more accurate policy referral, indicating that not only learns end-task behavior but also internalizes the underlying policy.
5.4 POLICY IN-CONTEXT: EVALUATING DISTINCTIVE IMPACT ON POLICY FOLLOWING
Consider the additional training stage and the rollout computation overhead in TriMPI, an potential alternative is to use baseline internalization methods but put the policy back in-context during inference. this section, we aim to examine if the TriMPI still offers better performance when given the policy in-context during inference. As shown in Table 4, TriMPI consistently outperforms baselines under this setting.
5.5 EVALUATING ROBUSTNESS TO CATASTROPHIC FORGETTING
Similar to related work in continual learning (Yu et al., 2024) and knowledge injection (Song et al., 2025), major challenge in MPI is avoiding catastrophic forgetting, ensuring that policy-related performance improves while general non-policy abilities are preserved. To assess this, we adopt two widely used general reasoning benchmarks, MMMU-Pro (Yue et al., 2024) (multimodal) and MMLU- Pro (Wang et al., 2024b) (textual), to evaluate the model's robustness to catastrophic forgetting after internalization. The results presented in Table 9 in the Appendix. We observe that the baselines exhibit significant performance degradation after MPI on GTAPolicy, while maintaining strong performance after MPI on ClevrPolicy. This discrepancy arises because the small dataset size of GTAPolicy makes it more prone to overfitting. In contrast, TriMPI consistently preserves strong general reasoning abilities across all settings, achieving the best overall performance, Appendix H, we further evaluate on WildGuardTesst (Han et al., 2024) to examine the model's safety-related performance.
5.6 ERROR ANALYSIS
Qualitative Failure Mode Breakdown. We further provide breakdown of failure modes best-periie pessstreformng model to highlight remaining challenges. On ClevrPolicy, we observe both percep- tion and reasoning errors. Perception emrrr include missing occluded objects and misidentifying

---

Published as a conference paper at ICLR 2026
attributes in scenes with multiple similar objects. Reasoning errors primarily involve branching to non-existent Conditions or hallucinating the rules even when the correct section is chosen. GTAPolicy errors are largely reasoning dassoning-driven due to its simpler visual scenes. A typical issue is misusing inter- action history, e.g., incorrectly calling an image-level description tool instead of a region-specifer one after obtaining bounding box. Qualitative error examples are visualized in Appendix M.
Quantitative Analysis on the Branching Error. Branching error is one of the major reasoning errors on ClevrPolicy, where the model hallucinates Condition section that does not exist in the target policy, indicating overfitting and insufficient grounding to the policy. To quantitatively analyze whether the proposed TriMPI effectively reduces this error, we compare the distribution of referenced Condition IDs in model-generated responses against those in synthesized Gold CoT response. As shown in Figure 14 in Appendix, we find that GRPO does not meaningfully reduce the discrepancy in Condition ID distribution relative to the SFT baseline (Average Absolute Difference: 38.72 78.72 vs. 39,74), indicating that GRPO alone does not sufficiently improve referral accuracy to the policy. In contrast, TriMPI achieves a substantial reduction in this discrepancy (Average Absolute Difference: 7.07), demonstrating that TriMPI enables more grounded and policy-aligned reasoning.
RELATED WORK
6.1 PROMPT COMPRESSION AND DELIBERATIVE ALIGNMENT
line of related research (Li et al., 2024) focuses on compressing long prompts into more compact forms that can still effectively guide large language models. Early efforts explored both hard prompts (Li et al., 2023; Jiang et al., 2023; Chuang et al., 2024), where discrete tokens are carefully pruned, and soft prompts (Zhao al., 2023; Wingate al., 2022; Mu et al., 2023; Ge al., 2023), where continuous embeddings are learned to replace verbose instructions or demonstrations. More recent work, such as Promptintern (Zou et al., 2024), adopts a progressive fine-tuning approach that internalizes prompts into the model without introducing additional parameters. Another line of related work is on personalized multimodal models (Nguyen al., 2024; 2025), for which we provide detailed discussion in Appendix L. While promising, these methods focus on prompts limited to task templates and demonstrations that demand little reasoning. Moreover, the introduction of special embeddings confines the model to a specific task, reducing its ability to handle general queries.
Deliberative Alignment (Guan et al., 2024; Zhang et al., 2025) extends this idea to a more general alignment setting without sacrificing the model's overall capabilities. Its goal is to embed the knowledge safety specification into the model parameters while emphasizing reasoning over the internalized knowledge. Our proposed MPI task follows this high-level motivation but further extends the scope to multimodal models and considers diverse conversational agent tasks beyond safety.
6.2 IMPROVING GRPO FROM THE ROLLOUT PERSPECTIVE
We also draw inspiration from recent multimodal reasoning work that investigates improving GRPO- related RLVR algorithms from the rollout perspective. Methods such as NoisyRollout (Liu et al., 2025) and R1-ShareVi (Yao et al., 2025) demonstrate the benefits of diversifying the rollout space with responses generated from altered instances using semantically consistent augmentations, such as applying moderate Gaussian noise to the visual inputs. Our proposed PolicyRollout algorithm follows the same high-level idea of augmenting the rollout space, but instead of introducing noise, it provides the policy in-context to enable more grounded exploration. PolicyRollowe illustrates the potential of more general approach to incorporating additional guidance in the RL stage, while remaining aligned with the original optimization task.
7 CONCLUSION AND LIMITATIONS
In this work, we propose a new task, Multimodal Policy Internalization, which aims to address the emerging challenge of maintaining in-context efficiency while following complex policies in multimodal conversational agents. We introduce two new benchmarks spanning analytical and real- world settings, along with a highly effective training paradigm, TriMPI. The remaining limitations are: (1) scaling up the datasets with more diverse real-world images and tasks; (2) developing more sophisticated continual pretraining strategies beyond simply masking visual tokens; and (3) designing training strategies for internalizing mixtures of tasks with very different response formats.
10

---

Published as a conference paper at ICLR 2026
8 REPRODUCIBILITY STATEMENT
We include anonymous source code archive in the supplementary material, containing training and evaluation instructions for reproducing the results this paper. Details dataset creation and evaluation metries provided in $C and $D. Full policy examples are shown in Figures 18, 19, and 20. Implementation details of the training procedure are provided in $B. Prompts used for the LLM-as-a-judg Policy Referral evaluation are presented in Figure 9.
ETHICS STATEMENT
This work focuses on fundamental research aimed at advancing the understanding of how multimodal models can internalize complex policies more effectively. All experiments are conducted on publicly available datasets, and no human subjects private user data are involved. The GTAPolicy dataset introduced this work contains fully synthesized user profiles and does not include or rely on any real user data. Code and data from this work will be made publicly available for research purposes in the near future.
REFERENCES
Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aberman, and Daniel Cohen-Or. Myvlm: Personalizing vlms for user-specific queries. In European Conference on Computer Vision, pp. 73-91. Springer, 2024. 21
Alexa AI. Amazon alexa. https://deeeloper amazon. www.co////////.e com/en--S//aeexa, 2025. Voice- based intelligent assistant by Amazon.
Anthropic. Claude 4, 2025. URL /ttppp/pt///....attooo....o/oooa 18, 20, 21
5,
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 2025. 18
Luke Bailey, Gustaf Ahdritz, Anat Kleiman, Siddharth Swaroop, Finale Doshi-Velez, and Weiwei Pan. Soft prompting might be a bug, not a feature. 2023.
Yu-Neng Chuang, Tianwei Xing, Chia- Yuan Chang, Zirui Liu, Xun Chen, and Xia Hu. Learning to
compress prompt in natural language formats. arXiv preprint 4rXiv:2402.18700. 2024. 10
Brian Clark, Srividhya Pallay, and Prema Mishra.
Demystifying amazon bedrock pricing for a chatbot assistant. https: awss amazon. com/bloo/ www.b/////agnneeeeeeaeaaaeeeeeee demystifying amazon- camazzon-bedrckt pricing- for- aaatbbot-sssistant. Putin_soure=<chatgee com, 2025.
Sinan Fan, Liang Xie, Chen Shen, Ge Teng, Xiaosong Yuan, Xiaofeng Zhang, Chenxi Huang, Wenxiao Wang, Xiaofei He, and lieping Ye. Improving complex reasoning with dynamic prompt corruption: A soft prompt optimization approach. arXiv preprint arXiv:2503. 13208, 2025. 3
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint WxXiv:208..06111 2022. 21
Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. arXiv preprint arXiv:2307.06945. 2023. 2, 10
Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al. Deliberative aligmment: Reasoning enables safer language models. arXiv preprint arXiv:2412 16339, 2024. 2, 3, 5, 10

---

Published as a conference paper at ICLR 2026
Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang. Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. Advances in Neural Information Processing Systems, 37:8093-81311 2024. 9, 20
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth Inter- national Conference on Learning Representations (ICLR), 2022. URL httpt:ttt/ppenteteteet http::s///peerevieww net/ forum? deneeeeeeFee 7
Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Limlingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:3310.05736
2023. 2, 10
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI. USA, July 21-26, 2017, pp. 1988-1997, IEEE Computer Society, 2017. doi: 10.11099CVP..2017..221 URL https: / doi. org/ 10. 1011///VVP 2017 215. 2,
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint 10t.020000000100 2021. 3
Yucheng Li. Bo Dong, Chenghua Lin, and Frank Guerin. Compressing context to enhance inference
efficiency of large language models. arXiv preprint arXiv:23/0.06200 2023. 2, 10
Zonggian Li, Yinhong Liu, Yixuan Su, and Nigel Collier. Prompt compression for large language models: A survey. arXiv preprint arXiv:2410. 12388, 2024. 2, 10
Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shieh. Noisyrollout: Reinforcing visual reasoning with data augmentation. arXiv preprint arXiv:2504. 250555 2025. 10
Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephras- ing the web: recipe for compute and data-efficient language modeling. arXiv preprint arXiv:2401. 163800 2024. 6
Jesse MU, Xiang Li, and Noah Goodman. Learning to compress prompts with gist tokens. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 19327-19352. Curran Associates, Inc., 2023. URL https: httppe///proeeeiiggg neurips cc/paper file/ htte://277666c011100a32211eee.e.
wttpp/ppeerlleeeeeeeeeeeeeeeseee files/ wtt.e//pappeeee222// Paperr Conference pdf.. 2, 10
Thao Nguyen, Haotian Liu, Yuheng Li, Mu Cai, Utkarsh Ojha, and Yong Jac Lec. Yo'llava. Your personalized language and vision assistant. Advances in Neural Information Processing Systems, 17:40913-409513 2024. 10, 21
Thao Nguyen, Krishna Kumar Singh, Jing Shi, Trung Bui, Yong Jae Lee, and Yuheng Li. Yo' chameleon: Personalized vision and language generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14438-14 14438-14448, 2025. 10,21 21
OpenAL.
Introducing gpt-5, 2025.
URL https: httpss:///peaaccco/// index/
Oded Ovadia, Meni Brief, Rachel Lemberg, and Eitam Sheetrit. Knowledge-instruet: Effective continual pre-training from limited data using instructions. arXiv preprint 2711::2504000055 2025. 2, 6
Oam Patel, Jason Wang, Nikhil Shivakumar Nayak, Suraj Srinivas, and Himabindu Lakkaraju. Towards interpretable soft prompts. arXiv preprint arXiv:2504.02/ 02/45444..2/4, 2025.
Yusu Qian, Hanrong Ye, Jean- Fauconnier, Peter Grasch, Yinfei Yang, and Zhe Gan. Mia- bench: Towards better instruction following evaluation of multimodal llms. arXiv preprint 4000002407.01509, 2024." 2
12

---

Published as a conference paper at ICLR 2026
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceed- ings of the IEEE/CVF conference computer vision and pattern recognition, pp- 22500-22510 2023. 21
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024. 3,
Zirui Song, Bin Yan, Yuhan Liu, Miao Fang, Mingzhe Li, Rui Yan, and Xiuying Chen. Injecting domain-specific knowledge into large language models: a comprehensive survey. arXiv preprint 1000002502..07088, 2025. 9
Jize Wang, Ma Zerun, Yining Li, Songyang Zhang, Cailian Chen, Kai Chen, and Xinyi Le. Gta: a benchmark for general tool agents. Advances in Neural Information Processing Systems, 37:
75749-75790, 2024a. 2, 4, 16
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Aruiraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37: 95266-95290, 2024b. 9, 20
David Wingate, Mohammad Shoeybi, and Taylor Sorensen. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. arXiv preprint arXiv:2210.031622 2022. 10
Huanjin Yao. Qixiang Yin, Jingyi Zhang, Min Yang, Yibo Wang, Wenhao Wu, Fei Su, Li Shen, Minghui Qiu, Dacheng Tao, and Jiaxing Huang. R1-sharevl: Incentivizing reasoning capability of multimodal large language models via share-grpo, 2025. 10
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. tau-bench: A benchmark for tool-agent-usee interaction in real-world domains. arXiv preprint 10XXi:2446.1204/4 2024. 2
Dianzii Yu, Xinni Zhang, Yankai Chen, Aiwei Liu, Yifei Zhang, Philip S Yu, and Irwin King.
Recent advances of multimodal continual learning: A comprehensive survey. arXiv preprint arXiv:2410.05352, 2024." 9
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025. 3, 6,7
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun. al. Mmmu-pre pro: A more robust disti-discipline multimodal understanding benchmark. arXiv preprint 24Xi::2409.02813. 2024. 9, 20
Haoran Zhang, Yafu Li, Xuyang Hu, Dongrui Liu, Zhilin Wang, Bo Li, and Yu Cheng. Reasoning over boundaries: Enhaneing specification alignment via test-time delibration. arXiv preprint arXiv:2509.14760, 2025. 3, 10
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. arXiv preprint arXiv: 1904.09675 2019.
Wenbo Zhao, Arpit Gupta, Tagyoung Chung, and Jing Huang. SPC: Soft prompt construction for cross domain generalization. In Bureu Can, Maximilian Mozes, Samuel Cahyawijaya, Naomi Saphra, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Chen Zhao, Isabelle Augenstein, Anna Rogers. Kyunghyun Cho, Edward Grefenstette, and Lena Voita (eds.), Proceedings of the 8th Workshop on Representation Learning for NLP (RepL4NLP 2023), pp. 118-130, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653/1120223rrphhm 1.10. URL ht https: http://caaaahhhhhocc
org/ 2023. rep14n11p111--. 1. 10/. 10
Jiaru Zou, Mengyu Zhou, Tao Li, Shi Han, and Dongmei Zhang. Promptintern: Saving inference costs by internalizing recurrent prompt during large language model fine-tuning, arXiv preprint arXiv:2407.022111 2024. 2, 10
13

---

Published as a conference paper at ICLR 2026
APPENDIX
Use of Large Language Models
15
Implementation Details
15
C Dataset Creation Details
15
C.1 ClevrPolicy
15
C.2 GTAPolicy
16
C.3 Evaluation Metrics and RLVR Rewards
17
C,4 Full Policy Examples
17
D Details on CoT SFT
18
D.I CoT Data Generation
18
D.2 CoT SFT Objective
18
Additional Results on Varying Policy Complexity and Model Size
19
F Qualitative Analysis
19
Robustness to Catastrophic Forgetting on General Reasoning
20
H Robustness to Catastrophic Forgetting on Safety
20
I Illustration of the Policy Referral Evaluation
20
J Illustration of the Policy Override Evaluation
21
RL Effectively Leverages Non-CoT Data
L Additional Related Work
21
M Error Examples
22
14

---

Published as a conference paper at ICLR 2026
Table MPI training hyperparameters for different stages. EP denotes the number of epochs, LR is the learning rate, and BS is the batch size, expressed as per-device batch size x number of devices. For the RL stage, BS refers the update batch size. Additional configuration details, such as the rollout batch size, are provided in Appendix B. Table 6 shows the actually RL steps taken in different training settings. Table shows the exact dataset sizes.
Stage
Dataset
Data Type
Data Size
LoRA
LoRA Rank
EP
LR
BS
VM-CPT
ClevrPolicy
Policy + CoT
2.5K
x
5
le-5
16x8
VM-CPT
GTAPolicy
Policy + CoT
0.5K
x
le-5
8x8
Direct
ClevrPolicy
Non-CoT
20K
le-4
16x4
Direct SFT
GTAPolicy
Non-CoT
0.5K
/
8
10
le-4
8x4
CoT SFT
ClevrPolicy
CoT
2.5K
/
8
5
1c-4
16x4
CoT SFT
GTAPolicy
CoT
0.5K
10
1c-4
8x4
RL
CleviPolicy GTAPolicy
Non-CoT Non-CoT
20K
X
2
le-6
4x8
RL
0.5K
X
50
le-6
4x8
A USE OF LARGE LANGUAGE MODELS
Large language models, such as ChatGPT and Claude, are used solely for grammar checking during the writing process and not for research ideation. Additionally, LLM-based coding assistants, such as Copilot, employed to aid in code implementation.
B IMPLEMENTATION DETAILS
In-context Experiment Details. The exact model versions used in Table are follows. For Qwen2.5-VL VL, we use the model checkpoints from Huggingface (3B 7B*). For Claude models, we use
and
hosted on Amazon Bedrock
Multimodal Policy Internalization Training Details. Table 5 summarizes the training hyperpa- rameters for different stages. For the RL stages, we the clipping ratio to El 0.2, eh 0.3 for GRPO and 0.2, 0.28 for DAPO. The rollout batch size is 384 for ClevrPolicy and 256 for GTAPolicy. The reference KL coefficient is
0.01 for GRPO. The maximum number of retries for dynamic sampling is set to 20 for DAPO. The actual number of RL steps taken in each run reported in Table 6. We use NVIDIA H100 80GB GPUs for SFT stages and H100 for CPT and RL stages.
C DATASET CREATION DETAILS
C.1 CLEVRPOLICY
C.I.I POLICY GENERATION IN CLEVRPOLICY
As deseribed $3 we first generate a set of binary decision trees to construct the policies. We define two types of nodes: decision nodes and response nodes. A decision node checks whether the current input image satisties sampled condition (for example, the existence of a cyan object), while response node corresponds unique outcome. Each decision node (non-leaf node) samples an attribute type and an attribute value from the following ontology:
Shape: "cube", "sphere", "cylinder"
Color: "gray", "red", "blue", "green", "brown", "purple", "cyan", "yellow"
Sszll" "large"
Inctps://hugginggf wttp:/aaaccccceee/eeeeeeeeeeeeee http:///////gggooooeeee/////////
#https://huggingffaee co/ wttpa/cc....eeeeeeeeeeeeeee htwp://weee......00. ---1,-7-------------------- ruct
Pntpps://docs.antrrgg hwwp:/ttth..ppcccccoooooeoa//a/p btw.aaeee-oo-aaammmbb-eeeeet
15

---

Published as a conference paper at ICLR 2026
Table 6: Detailed statisties of the number of RL steps under different settings. Numbers annotated with (*) denote early stopping, which may occur in DAPO runs due to dynamic sampling.
RL Steps
Methods
ClevrPolicy-T (N=6)
CleviPolicy-M (N=6)
CleviPolicy-T CleviPolicy-M (N=4)
GTAPolicy
(N=4)
Basc Model: Owen2.5-VL-7B
CoT SFT GRPO
104
104
104
104
50
CoT SFT DAPO
104
85
25
45*
50
TriMPI w/ PoRo-GRPO
104
104
104
104
50
TriMPI w/ PoRo-DAPO
90*
75
15a
20*
50
Base Model: Qwen2.5-VL VL-3B
CoT SFT GRPO
104
104
CoT SFT + DAPO
104
104
TriMPI w/ PoRo-GRPO
104
104
TriMPI w/ PoRo-DAPO
100
95*
"atbri,l "metal"
If a child decision node has a parent node and the edge to the parent node is true. the child node is restricted from sampling the same attribute type. For example. if the parent node checks for cyan objects and the edge is true, the child node will not check the color attribute again, to avoid logical conflicts. This results in 16, and 55 unique target responses for policies with N 2, N 4, and N = 6, respectively.
Given a sampled decision tree, we then convert into a textual instruction containing three main components: a general instruction, a unique policy name, and sections of conditions governing the response behavior. This policy challenges the model to perform multi-hop reasoning across different sections in order to faithfully follow the rules.
We further introduce two variants ClevrPolicy, namely, ClevrPolicy-T and ClevrPolicy-M, depend- ing whether the policy itself contains image content. As shown in Figure 2 (left), in ClevrPolicy-M the attribute value decision node may be demonstrated by a small image crop instead of natural language. This introduces additional level of difficulty for policy following.
C.1.2 TRAINING DATA CREATION CLEVRPOLICY
We generate a mixture of 10 unique policies for each the three levels complexity indicated by the layer number N E 2, 4, 6. For each policy, we then sample 2K images as training data. The ground- truth answers can be automatically obtained by verifying the scene graph against the corresponding decision tree. This results in 20K non QA pairs for each training task in ClevrPolicy-T and ClevrPolicyMM. Similarly, we sample another 2K unseen instances for testing. Importantly, the underlying decision trees not accessible to the model during subsequent MPI training. An input-output example is illustrated in Figure (right). Evaluation performed based on exact string matching. and results are reported in terms of accuracy.
C.2 GTAPOLICY
C.2.1 POLICY CREATION IN GTAPOLICY
We manually construct a complex policy based on the GTA dataset (Wang et al., 2024a), which was originally proposed as multimodal tool using benchmark. As illustrated in Figure 3 (left), the GTAPolicy policy consists of two main components: tool descriptions and tool calling rules. The tool descriptions contain the metadata for all 13 types tools, including the tool name, description, and arguments. To ensure that model performance reflects its ability to follow the internalized policy rather than relying on prior knowledge, we further introduce a versioning mechanism and user-condittonal tool-calling rules. Specifically, tool such as "OCR" may have multiple versions, e.g.. "OCR v1" and "OCR v2." A total of 24 tool-calling rules specify which particular version to depending the profile attributes of the input user. This design simulates real-world business
16

---

Published as a conference paper at ICLR 2026
rules in which certain user properties must be considered during decision decision-making and tool calling, such as whether the user is premium member.
C.2.2 TRAINING DATA CREATION GTAPOLICY
We first reformulate the multi-ture pulli--turn data from the GTA dataset into a single-turn tool calling task to avoid overcomplicating the MPI setting with intermediate tool calling errors. As illustrated in Figure (right), each input instance consists of five parts: visual inputs (potentially multiple images), user profile,
query, interaction history containing previous tool calls and returned values, and general instruction prompt. The expected output a tool call formatted in JSON that specifies the tool name (with version) and the corresponding arguments. The final dataset contains 451 instances for training and 106 instances for testing. To evaluate the tool calls, we use exact match on the tool name, and apply different evaluation metries to the argument values depending on their type (e.g.. IoU for bounding boxes, text similarity for free form text queries). Detailed evaluation metrics are provided in Appendix C.3.
C.3 EVALUATION METRICS AND RLVR REWARDS
Evaluation Metries for ClevrPolicy. We use accuracy based the exact match between the predicted and ground truth outcomes, for example, "Case 0". Accordingly, the following accuracy reward is used for GRPO and DAPO on ClevrPolicy:
PAco-CleyPoolicy = Matchy, Ø)]
(4)
where and ú denote the parsed response and the ground truth, respectively.
Evaluation Metrics for GTAPolicy. For the tool name, accuracy based on exact match. For the arguments, we identify four categories of argument types and apply different metrics accordingly. The full list of tools and argument definitions shown Figure 20. We provide the argument names and their corresponding evaluation metrics as follows:
Exact match: "position". "color", "image", "k", "top1"
Text similarity: "attribute", "text", "query", "command", "annotation", "keywords", "in- struction"
Python Eval: "expression"
IoU: "bbox"
Exact match is used for arguments that require precise string matching, for example, the image name when running a tool. The text similarity metric evaluates arguments in free-form text format, such as search queries or object names. We use BertScore (Zhang al., 2019) to compute the semantic similarity between two text sequences. For expressions involving numerical operations, we use the Python "eval" function to compute the final outcome and compare it with the ground truth. For bounding box arguments, use Intersection over Union (IoU) to compute the overlap between the predicted and ground truth coordinates.
All scores are normalized to the range [0. 1]. The final argument score is computed as the average across all argument fields. We then compute an overall score for the entire tool call as the average of the tool accuracy and the argument score. Accordingly, the following accuracy reward is used for GRPO and DAPO on GTAPolicy:
RAcc-GTAPolicy = 0.5 x Tool + 0.5 x Scgrment.Score
(5)
C.4 FULL POLICY EXAMPLES
Examples of the full policy for ClevrPolicy-T. ClevrPolicy-M M and GTAPolicy are presented in Figures 18, 19 and 20, respectively.
17

---

Published as a conference paper at ICLR 2026
Table 7: Detailed dataset statisties.
Train Size
Dataset
CoTT Generator
CoT Strategy
Filtering
Test Size
CoTT
Non-CoT
ClevrPolicy-T (N=6)
Qwen2.5-V1-77
Forward Co'T
Yes
2526
20000
2000
ClevrPolicy-I (N=4)
Qwen2.5-VL-7B
Forward CoT
Yes
2526
20000
2000
ClevrPolicy-M (N=6)
Claude-4-Somne
Forward CoT
Yes
2472
20000
2000
ClevrPolicy-M (N=4)
Claude-4-Sonne
Forward CoTT
Yes
2282
20000
20000
GTAPolicy
Claude-4-Sonnet
Reverse CoT
No
451
451
106
Table 8: Additional results varying task complexity and model size. We show that TriMPI
consistently outperforms strong baselines across different complexities and model sizes. Notably, the performance gain is more pronounced on complex policies.
Varying Policy Complexity
Varying Model Size
Method
ClevrPolicy-T
ClevrPolicy-N N-4
CleviPolicy-T
(N=6)
ClevrPolicy M (N=6)
N=4
N=6
N=6
3B
7B
3B
7B
In-Context
32.20
13.15
13.90
5.65
4.80
13.15
4.55
5.65
Direct SFT
25.65
15.15
37.85
14.55
13.40
15.15
12.90
14.55
COT SFT
49.10
17.80
55.35
14.30
19.00
17.80
9.25
14.30
COT SFT + GRPO
98.25
47.05
87.10
64.50
31.55
47.05
22,40
64,50
CoT SFT DAPO
97.15
67.60
88.75
74.40
48.10
67.60
33.25
74.40
TriPI w/ PoRo-GRPO
99.15
65.85
99.05
84.70
46.65
65.85
78.80
84.70
TrIPI w/ PoRo-DAPO
97.30
77.80
99.40
85.00
68.75
77.80
82.35
85.00
D DETAILS ON COT SFT
D.1 COT DATA GENERATION
Given the practical challenges of accessing APIs for stronger models such as GPT and Claude, key principle follow is minimize reliance on API models whenever possible.
Specifically, on ClevrPolicy, we use forward CoT with answer filtering to generate CoT data, where generator model is asked to solve the task with intermediate reasoning steps given the policy and the input query. The generated instances are then filtered based on answer correctness. For GTAPolicy, due to the small dataset size, adopt reverse CoT. where the generator model is provided with both the inputs and the ground truth answer and asked to generate the intermediate rationales. Reverse CoT is suitable in such settings where filtering is difficult the data samples scarce.
We select the generator model based on the zero-shot in-context performance shown Table 1. For ClevrPolicy-T. we use the base model itself, Qwcn-2.5VL-7B (Bai et al., 2025), as the generator, For more challenging settings, i.c., ClevrPolicy-M and GTAPolicy, we use (Anth--(Anthhopicc, 2025). In the case of ClevrPolicy-Mr with Claude-4, we subsample the input non- non-CoT data to 3K in order to match the CoT dataset size of ClevrPolicy-T. After filtering, we obtain approximately 2.5K CoT instances for each ClevrPolicy setting and 451 instances for GTAPolicy. The full statistics of the CoT and non-CoTT data can be found in Table 7. Figures 15, 16 and 17 provide examples of the CoT data for all three datasets.
D.2 COT SET OBJECTIVE
Given the generated CoT annotation C' for each question-answer pair (Q, A), the training objective for the CoT SFT stage can be written as follows:
101
CAF E(Q.O)
E
log pa (or (Q,0c)) .0=
. [C:]]
(6)
where O is concatenation of C and A.
18

---

Published as a conference paper at ICLR 2026
<hilk
Latasrallee the imaae step by sap sccoeding to policy:
1**Cannttion I: check mere ayy gny objort
TThm are any objocs
image we move Condition
**Candttio check
perple abjoze*
There are no purple objecs the image, So, we nove Condation 1.2.2
**Candtion 1.22 cteck ('es toon ebjezr
Condition quere image. focas
objects
*Condition
are coliader abrect"
-Thm yellow cyltnder imass but browm So, move
Input Image
Oméition
anallateert
Fellow the Policy L6 Vidima, provide your resperse for this image.
bosed]Cuse
CoT SFT Response
Query
<bidk>
Leta sralyoe the imase by sccreding policy:
ICConddtion Cholk of there
chiect-
Them no objocs the imags. So wo mose to Condition 1.2.
**Candtion 1.2: Choce thone any purplc obyeat*
If yes, foces only gaay obsect, and cantinue with Conditon L.I Otherwise, continue with Cooditon 1.2
Conditinn I: check there is any gras abjeet
**Condtton 1.2.2 Check of ohehee is any beews objecte
TThre ne people object the image, So, we move Condition 1.22.
Them lea brown sphere the image So, we fooss brown objecss and contince
False Contition
wddh Condiion
"Coondtion 1.2.1: Check there asy sylinde: objectt
Condition 1.2: check there any purple object
Them yollow syltnikr imags, mita boow unz. So, we move to Cundition 1.2.2.12.
yos boos only purple objects and cantina with Condition 1.3 -therwise, contim:: with Conditon
*CCrndiion 1.2 Chock there is lang object*
Th yellow eglindee lange we Soeee linge brosn obpees contrre
Conditio 122.1:1
evr Condition 1.2.2: check if f'there ans brown abject Otherwise. loows obecis, Candition
<tink>
X hosed[Cise
CoT SFT GRPO Response
Cundition
any cylinder object
uniiyse image by
policy:
any gray object
-There no objocts
**Condition
surple
Candition
**Canhtion pirplo objozs imago,
continue Condito 122121
image
objust
cylinder
#esthed Condition 1.2.1... check thehere small "Coee
1222.12
Otherwisc,
Respend "Case
obieste sbjost* **Gandtion 1.2.1.22 2.2. the smell otjez
sheck
1.22
-Them small bosu opher
with "Coer
Reasoning Trajectory According Policy L6-Vidima
<think
bosed|Cass
TriMPI w/ PoRo-GRPO Response
Qualitative example comparing different MPI algorithms. On the left, we show the inputs and the ground-truth reasoning trajectory annotated with the original policy sections. On the right, the CoT SFT model makes an error in correctly recalling the policy condition, and the CoT SFT + GRPO model makes an incorrect decision at the fifth condition, both leading to an incorrect final outcome. In contrast, the proposed TriMPI correctly recalls all policy conditions and performs reasoning consistent with the input image.
Figure 7:
E ADDITIONAL RESULTS ON VARYING POLICY COMPLEXITY AND MODEL SIZE
Table illustrates the additional results on varying task complexity and model size. TriMPI is especially valuable for handling complex policies, showing larger gains over baselines in these settings while maintaining consistent improvements even on simpler cases (e.g.. N = 4). Moreover, generalizes well across model scales, consistently outperforming baselines on 3B models as well.
F QUALITATIVE ANALYSIS
Figure 7 shows a qualitative example from the ClevrPolicy dataset, comparing the responses of different MPI algorithms. The results demonstrate that the proposed TriMPI algorithm successfully refers to the original policy and produces a policy-commp complitt answer, whereas the SFT and SFT+RL baselines suffer from incorrect policy referral or reasoning errors.
19

---

Published as a conference paper at ICLR 2026
Table 9: Robustness to catastrophic forgetting on general reasoning. We evaluate the internalized
models on MMMU-pro (Yue et al., 2024) and MMLU-pro (Wang et al., 2024b) benchmarks. The results show that TriMPI consistently maintains strong general capabilities across all settings, while most baselines exhibit significant drop the smaller dataset, i.e., GTAPolicy.
MMMU-Pro (Ace)
MMLU-Pro (Ace)
Method
AVG
ClevtPolicy-- ClevrPolicy-M GTAPolicy
ClevrPolicy-TT CleviPoliey-M GTAPolicy
Before MPT Training
Qwen2.5-V1.77B
31.91
31.91
31.91
31.73
31.73
31.73
31.82
After MPI Training
Direct SFT
34.10
33.24
27.86
39.25
37.01
28.91
33.40
CoT SFT
31.85
32.95
10.17
37.67
38.70
10.43
26.96
COT SFT GRPO
32.60
32.08
22.95
37.63
40.13
22.66
31.34
CoT SFT DAPO
34.22
34.16
18.84
38.41
39.79
20.86
31.04
TriMPI w/ PoRo-GRPO
31.45
33.29
30.12
38.57
39.34
35.77
34.76
TriMPI PoRo-DAPO
31.56
32.66
30.52
37.68
39.49
35.63
34.59
Table 10: Robustness to catastrophic forgetting safety. We evaluate the internalized models WildGuardTest (Han et al., 2024). The results show that TriMPI achieves more consistent preservation of safety-related performance compared with other baselines.
WildGuardTest (Acc)
Method
AVG
ClevrPolicy-T ClevrPolicy-M GTAPolicy
Before MPI Training
Qwen2.5-VL-7B
87.24
87.24
87.24
87.24
After MPI Training
CoT SFT
82.04
$2.95
69.22
68.07
CoT SFT GRPO
83.21
47.69
69.87
TriMPI w/ PoRo-GRPO
74.90
76.71
76.77
76.13
G ROBUSTNESS TO CATASTROPHIC FORGETTING ON GENERAL REASONING
Table 9 presents results on general multimodal (MMMU-prO (Yue et al., 2024)) and textual (MMLU- pro (Wang et al., 2024b)) reasoning benchmarks for models after MPI training. This evaluation examines whether the model preserves general reasoning capabilities while internalizing policy- compliant behavior.
H ROBUSTNESS TO CATASTROPHIC FORGETTING ON SAFETY
To further investigate the preservation of safety-related behavior, we include an additional benchmark, WildGuard (Han et al., 2024). Given a prompt and a model response, the task is to determine whether the response harmful (e-g., involving privacy violations, misinformation, harmful language, or malicious uses). Table 10 reports the accuracy (%) on WildGuardfesst for models after training with ClevrPolicy-T. ClevrPolicy-M, and GTAPolicy. We show that the proposed TriMPI method maintains the most consistent performance across different bolicy-intrmallzattion settings, preserving most of the overall capabilities. contrast, the SFT and GRPO baselines exhibit larger variance and, in particular, suffer substantial performance drop under the ClevrPolicy-M setting.
I ILLUSTRATION THE POLICY REFERRAL EVALUATION
Figure illustrates the Policy Referral evaluation setting, where we leverage a strong LLM, Claude- 4 (Anthropic, 2025), to score how accurate does the response cited to the original policy. The exact prompt used for prompting the LLM is presented in Figure 9. This evaluation aims to test whether the model has actually absorbed the policy knowledge beyond merely mimicking the expected behavior.
20

---

Published as a conference paper at ICLR 2026
9
Follow the Policy L3-Surinu, provide your response for this image.
Image
Query
Policy
Internalized Model
(L3-Surinu)
<think> Let's analyze image step by step according to the policy:
LLM (Claude 4)
0-10 Score
I. **onnditoon I: Check if there is any cyan object** </think>
Check referral to the original policy accurate
/boxed (Case 283
Figure 8: Illustration of the Policy Referral evaluation setup. We take the responses from internalized model and ask a strong L.M (Claude-4 (Anthropic, 2025)) to score the consistency between any policy referral in the response and the original policy. Policy referral is designed to evaluate the quality of the embedded policy knowledge beyond end-task performance.
Policy Referral LLM-a--a-udde Prompt
edl Policy HN (Original Pollcy Contect)
Model Response Al [Respense
Models)
the verify whether reference Provide Carefully esamine the final score from to 10, which indicates well the respimse adheres the policy. datailed analysis Put fimal score the following formal:
Whenever
ed Scare #t!
your final scoce here
Figure 9: Prompt to Claude-4 for the Policy Referral evaluation.
J ILLUSTRATION OF THE POLICY OVERRIDE EVALUATION
Figure 10 illustrates the Policy Override evaluation setting, where place the updated policy in-context during evaluation and expect the model to follow the overridden policy. This evaluation aims to test the model's ability to generalize in policy following, beyond memorizing particular policy.
K RL EFFECTIVELY LEVERAGES NON-COT DATA
During CoT data generation, we observe that in real-world settings, non-CoT data is often much more abundant than CoT data. For example, in ClevrPolicy, after filtering, we obtain 2.5K CoT examples versus 20K non-CoT examples. A unique benefit of the RL stage (with GRPO and DAPO) is that it does not require CoT annotations. To investigate the extent to which the RL stage can benefit from non-CoT data relative to CoT data, we conduct the following experiment. We run RL algorithms using only CoT data and additionally include an SFT baseline, where the model first trained on CoT data and then further trained on non-CoT data. The results are shown in Table 1. We observe that the RL stage can more effectively leverage the abundant non-CoT data compared to SFT-only approaches.
L ADDITIONAL. RELATED WORK
Another related line of work from the multimodal community is personalization through textual inversion (Gal et al., 2022). This spans from early studies personalized image generation (Gal et al., 2022; Ruiz et al., 2023) to more recent approaches on personalized multimodal models (Nguyen et al., 2024; Alaluf et al., 2024; Nguyen et al., 2025). The central idea is to learn lightweight embeddings
21

---

Published as a conference paper at ICLR 2026
New Query
New Query
## Tool Calling Rule Override 888
Suppose we have updated Policy: (L3-
rules are updated
Surinu; as follows:
[OCR Tools New Rules]:
-- Policy: L3-Surinu
If user credits - 250, use "OCR_vV v1"
Condition 1:
250 user credits 750, use "OCR v2" account type [A, B). otherwise "OCR y²". If user credits > 750, use "OCR v4" ifage < 21, otherwise use "OCR v5%,
Condition 1.1:
According updated Policy: L3-
Surinu, what should be the response for the given image?
## User Profile #B#
you need to determine the appropriate tool to use the current slep
New Answer
New Answer
"name": "OCR v1".
"Caso !"
"arguments": ("image": "image/imago 203 "pg""
Policy Override on ClevrPolicy
Pollcy Override on GTAPolicy
Figure 10: Illustration of the Policy Override evaluation setup. For ClevrPolicy, provide randomly sampled new policy in-context during inference as updated policy content while keeping its original unique name. For GTAPolicy, model version. Policy Override evaluates the model's capability to generalize to updated or modified policies beyond overfitting. This property is particularly important in real-workd scenarios, where policies are constantly changing and model behavior must be governed by both the internalized policy and the in-context instructions.
modify the tool-calling rules, resulting different
Table 11:
Results of the RL stage with different data source. RL algorithms such as GRPO can better leverage the more abundant non-CoT data compared to SFT.
Data
Dataset Size
ClevrPolicy-T
ClevrPolicy-M Acc (N=6)
Method
Subset (CoT) All (Non-CoT)
Subset (CoT)
All (Non-CoT)
Acc (N=6)
SFT SET
20K
14.30
2.5K
20K
22.25
14.15
SFT GRPO
2.5K
20K
34.10
$4.00
SFT
2.5K
20K
47.05
64.50
TriMPI w/ GRPO
2.5K
20K
39.45
74.55
TriMPI GRPO
20K
55.90
80.80
that represent new visual concepts, enabling models to generate personalized responses. However, the supported concepts typically limited few personal photos, which is far less complex than the policies required for multimodal conversational agents.
M ERROR EXAMPLES
We present the figures supporting the discussion $5.6.
Figures 11, 12, and 13 show qualitative examples of the common types.
Figure shows the branching error analysis compared with basclines.
22

---

Published as a conference paper at ICLR 2026
Reasoning, Example-
Query: ganage-Accurdnng to the Policy: L6 lavete, what should be the response the given image? You first think through the reasoning peucess as an monologar, enclused within <think> tigs. provide your final enclosed ibosed
Response: <think> Let's analyze the image
according policy:
1. **Condition I: Check if there is conn object*
There are cyan objocts
image So,
Conditien
2. *Condition 1.2: two sphere objects: green sphere purple sphere. Condition 1.2.1.2.
Clock there any sphere cbjozt*
sphere objects
3. **Condiion 1.2.1: Check if there is any large objeztee purple spheno
green
also
*CCondition 1.2.1 Check image. There
any objorts
Condition 1.2.1.22.
5. **Condition 2:2 There red the aay objectee
Non-exist condition in Policy: Tavete
Condition 1.2.1.2.2.2.
any ohjecr*s
There is browa cube the image. but not sphere. So. we move to Condition 1.2.1.2.22222
Figure 11: Reasoning error example on ClevrPolicy. The model branches to non-existent
Condition for the current policy.
Perception Error Example- ClevrPolicy
Query: Aamnge=-Accoriiig Policy: Tavese, monologue, enckeed within <think </hink>> tags. your final enclosed within thosed)).
response given image? first through the reasoning pracess
Response:
-think- analyse image step step acconding policy:
*Condition 1: Check Ihehee
objecs**
Therr a eyna eylinder the image. with Condition
eyan objects
any object**
There are other cyan objects (a cube and spshare that are indeed small
cyliader small, continue Coadition
Figure 12: Perception error example on ClevrPolicy. The model incorrectly perceives that there are no small cyan objects.
23

---

Published as a conference paper at ICLR 2026
Reasoingg Ertur Example- - GTAPelicy
Query:
sed User Profile And
type": "B", "age": "onodit": 131)
Bell User Query HWW
imsgerimige 359 jps <image>
the
eggs picture?
nett Interaction History dit
AAssistat: The
indicates that there misumkrssamdnng The tool described and black and white photo, which egzs.
identifying image containing egas, possibly puth
provided does not correspond to an image with white eggs will atcompt to ase the Texboollbox tool to locate the white egas the image specifically and then deseribe that region.
Retun: (543, 645, 792, 993),
You are assistant capable of utilizing extemal tools...
<thakk Looking quary, need analyze
white eggs unage. asking about specific attribute (mood)
specifie object spgs) speeific image. From the intoraction history, can ace that the ToxTToRbox rool was already usod and retumed bounding box for what to be the white repion: (S43, 645, 792, 993), Now noed to understand the mood these white To analyze the mood objects in an image, should use ImsageDeseription tool.
TextToBox has
should
use the Regorsttributeeese betrbbutebesooggttn tool to obtoi descriptions the specific identified rother then the Drsgeiisiiiittionn which captioning the entire image
general
Figure 13: Reasoning error example on GTAPolicy. After correctly using "Tex(ToBbox" localize the region of interest for the query, the model should invoke "RegionAAtrbbutDDeeerpioonn to describe that specific region, but instead erroneously calls "ImageeDescription,, which captions the entire image and disregards the prior tool output.
24

---

Published as a conference paper at ICLR 2026
Coocin
Cordiion Distiace Deforaace N1.2
CoT SFT
Condiiee anthaiee Respemec SOCO Rageerz
COT SFT GRPO
Prod.GM D4f5000
Cendiar
D4B 1.0
TriMPI
Figure 14: Branching error analysis on ClevrPolicy. We compare the distribution referenced Condition IDs (e.g., "1.1") in model-generated responses against those in synthesized Gold CoT response. A distribution closer to the Gold CoT distribution (left) indicates better alignment with the original policy and fewer hallucinations, while smaller difference (right) reflects better overall performance.
CoT Data Example- - ClevrPolicy-T (N-6)
should response given through reasoning process interaal monolague, enclosed within <think cythnkk tags. Thes, provide your final answer enclosed within (bosad
Query: AimaninAccutingg
any eyaa object.
here cyan eylinder the locus only objects and continue with Conditio 1.1.
2. Since there eyan
Condition I.I: check there small object.
1. The eyan eylinder is small.
2. Since there is small eyan object, we focus only small cyan objects and comtinue Condition I.I.I.
any cylinder object.
eylinder. 2. Since there small eylinder, focus only small syan cylinder and continue with Conditiun I.I.I.l.
small eyan
Candition 1.I.I.I: check if there is rubber objeet.
1. The small eyan ey linder appears to be mibberr
2. Since there is rubber object, we respand with "Case (f'.
Answer: lboxed/ Case 0f
Figure 15: CoT data example (ClevrPolicy-T N=6).
25

---

Published as a conference paper at ICLR 2026
CoT Example- ClevrPolicy-
Query: <imagei- Acconding Policy: 1.6 66-aagute what should the resporse for the given image?? think through the ressoning process monnloggae, enclosed within <think- w/hink togs. provide your final enclosed within (boxed
Coll:
-think-
Lel analyze the image and fellow the policy step step.
Finst I need identify objects the image:
A large blue metallic sphere
anall purple metaalii cylander
-A amall yellow cylindar (sopears to be cubberiuutt))
Now follow the policy:
**Condition I: check if there is mny hrge object**
Yes, lurge objsets
purple spheres). focus only large objects Condition
Large objocts: bla metallic sphare, purple metallie sphere
*CCondition 1 check if there
yellow objocst
Among the large objects (biie spbent, purple spóere), there are nu yellow objects. So continue with Condition 1.1.2.
**Condition 1.1:: check if there any cyan objectte
Among large obyects,
ore so Gynn objects, So
Condition
""Condition 112.2.
chock ther is any abjecr that has the came shape as deno
sphere, The largo vbjezts spheres, eylinders So there are lirge objects with the same as Aano contime with Condition 2.2.2.
sphere),
any
"Connittion
there mihter objoct*
Amons dee large obyects (blue metallic sphere, purple metallic sphere), there are no nubber objects they are both metallic. So continue with Condition
"Condition 1. 2.2.2 check if there is any object that his the same material as dano (appears shuny/mttallke Amaou large moterial demo. Since thene lestt <thiink>
sphere are metallic, they with the same materiill demo. 5, snooeer "Cise 23"
Figure 16: CoT data example (ClevrPolicy-M N=6).
26

---

Published as a conference paper at ICLR 2026
Call Data Esampl--GAAPan -TAPulle
Querys
Product
Price
sar Oae Pactie Adl
BACARDI
Tapil: 26, "wwdiet
Henneey
$50.39
skdl Uler Dacyy
line man 52 dhebi pay the Laguo the pount accendingte price on moc..
TALISRER
$67.272
Artiratt Siace ve Aaceeree wilch tmap the lowe and viick rasns use dhe iaus doseit sike bot foe buck kauge Las stant the that imagu langg Taaggh 200 a0g'
but Blicd with dhe mutia, and tuat rapa watle with cota The dhoccp lockgnen
gially liggl
betio ia gliggho doal of llanaoyy hoadd and ligul taid..
imag hewe
tmuge Thrag poepr 714
with
The deiss dibed
nas
wnerge
appne part
pappoes
rmsgest por "mago mage 20li This fill cenun the proc of auan thom ricaue the mage the par taasgc tmnee. 3H4pns hrmaffc will miiallss keall and bactt on the ou Lugad
Lefvoeet by ooant an the encer of quee telle
Austran how dhe haae beause Jaguss acc cersen the braader boaee will do rouggntera uos image a1 perh Thrag 20lippe
221
the dall Les,
394 waaysa Vore
caealls of aciluiin
bf Taul culling Rulee"
The toa saage eveiin bu of l'barery cagr (Siuac)
bovd pras llezacce monu calculas
pruing
real test Srerrte emeinnggo inago, ceelte wose OCR aud reall OCB ted Seuiille
If sue cod 1SA, TOCk ur cotaa If peree en fine fial serooiine Wori culloio ineieel
pritile
Auoeet type
203
The sword keape spes slder owns contiiing webeee of sxproseen, us loekki ssu veniggg DCR the MUROCE
pernnge, MM priceg
timeray
Answer: T'aau" UP"
Figure 17: CoT data example (GTAPolicy).
27

---

Published as a conference paper at ICLR 2026
Policy Example- ClevrPolicyTT (N-6)
bellowing proílesa decison franwod chenid slwaye east with checking the night- axzi conéiton itc. condrice jand fillow the teematten procke a first cuperae Leverized conditora chould colared rebec misty hight--evd
visual lepr You
Canditio check there grace
yes, focta only
sphere abeecs, continas Canditzon
-Oowwie, corine with Conditin 1.2.1
applicble polay
feloww:
New Condition 1.2.1.I: cleck iriiee bra ment obgeet
-n haicy: Lo- lavele
ye'ss resposd "Case 24"
Canddinn
any cyen objec wab Cosditio
-Ohewist cortnne with Conditoon 122.1.3
-Chanois, continae wath Conditio 12
Neve Cordition 1.3
check
any retberrtgee
-f'yr reppod "Cau 25"
focasonly svall ebjecs, ebjeet with Contition 1.11
-Obewwic respeed
-Ohtrrstie, continse wrth Conditon 1.1.
JAN Conditio 121.122: clock if meal ebrect
yE locs only os lasge mead spkere obeecs, and continee witk Canéitice 121:221 -Oleviic omiine with Conditie 1.2.1
oncaconly ovall oyn extoder chjocc and contron Couttion 1.S.1
-Olenniae continae wath Condition 1.1.12
MENV Corditin 1.2
dheckk thee is asy de
-fryrs regurd yos
waat Cooditon I. dock II pose "Case (*
rabber
reepord
-Olleniie, cortinee with Conditin 1.1.1122
Askotel Corditin 1.2 1::22 check of thon is any grey objeet -Iyyrs requid aus
01559 Cooditiss 1.1.1.2 cledi thee rettal olject
respord
yyy ropeedl "Coe
-Oeberetir, requard Case
Aske Conditioe 1.2 chock therr any hos objet
okjeeo, and contine Conditin 1.211.11 Docs only Cooditon 1.2.122
visi Constiion 1.1.2 2 chick of there auy optors object -If yy,, focas only on unall oyon othere object, and ondionnt with Condiion 1.112.1 -Oeberetic, conmiae Condition
-Olawiie
aex Conditio 1.21211 dackk if aay seell dbjist continus with Condition 1.212111
Docs only -Olewwic
ophere with Coodition 122.1.2122
visaa Conditie J. 11122 check ther matal objoct If repeed "Caso -Ohlerwise coctinse Coaditio
Conditin 1.2 check there any rather
CsNeM Goodiice 1.1.1 1 chock (Ttbere is aay rubler ebject yos, reppod "Cooo
-OSarwie, CCac 32"
-Ohherwise, respord
Condition 1.2 1.1.1.1 check there any rubber ubrject
WWAA Condition 1.1. 2.:22:2 check of there any motal object
-OSawwic resporal Ciac 34"
you focas on seall caan meal strecs, and continuz with Condition 1.2222
lory object
-ffyrs Room nodipher dbjcct,
with Coedttoo 12.1222
JWAMM Conditie L.1. 211.1.1 chack if there is cube obiect
-Oheewirr comtre with Condtton 1.1 2..3
you repeedd
any object
-ffyrs requrnd "Cau -Ohurwier repprd "Cee 30*
WAARA Costiiin 1. chock iftherz iry objert repord
-Ohhendtor, "aae or
Corditinn 1.2.1.222 cleck tacee
object
-EE reguadd "Cus 37
Conditiors chock ifther korry rytidde chjoc
-Oleviic ropocal 'iay 35*
on oyen -Ohhemdter, contiae Conditin 11....
313 Cosiliso cleck deree
rabber
yes foos only os nufbe dejoss. and continee will Condition 1.2..1 -Oharviee, amtim with Denditin 1.2222
Clodition dlekk filiddr
any lagge object
costine with Cooditizz 1.1.1.11
-Ohlewiie, codfine wath Conditio 2.122
Asse Conditne 1.2 chack of there
abjoc
yru ficos only os gny obperts and crrtece with Condition 1.22.11 -Oherwie, amtin with Condriie
esttt
cbedi in there cabie object
10"
-Ohbemstee continse wath Condition 1.1 2:111
Asaas Consiton 12 2: chok il thee snall sbjost
-fyeks Doos only os small gno ribber objects, and coofnce with Condition 1.2.1.111
1.1.1.122
isy object
-Oheewiec corinee Coodioon
-If yos, poed "Canc 11"
-Oxberstic, respard "Case 12"
Nevive Condition 1.22..1 cheheck in therr any cylinder object
-ffyrs regoodd "Case
VWAR Conditize
ther
object
-Ohoewiic, rmppod
you road "Cacc 13"
-Oheeriiso, corfinae with Condition 1.1 2.122
Aesevel Conditon 1.3 check tnttoo any lerg object -E'sck respard 4l
WWMM Costitine 1.1.1.222 chick (ftherz ioy nétier -1'ys impord "Cooo 16" -Ohberotion repurdd Case
A4H Conditin 1:2:122 dack if any cake okieet
sE'sk Soco only cubbbe cabe decess, continas Conditin 1.22121
WAM Covttion 1.3 chock of then karr musal objer
1.22
yos locas only on coan recal stoces, and cottinee with Condtton 1.1.2.11 -Otkeriise continee Conditin
Asseval Candition theheck ther any ohjeet repurd -Ohurwie,
00255 Conditoo 1:22:: cleck if thee ebjoc
yes, Roca--olly on reeu cule osaetis and costes with Cooditios 1.122.1.1
-Ctlerviie continee Couditin
Assuval Canditio 1:22::22 check there any grom abjoc -Ohurwic
WAMAA Conditice 1133 chads ther tay large abjec
you ropeed "Coec 18"
repurdd
Conditiin theere is giay object
Coalttion 1.22211
WAMAA Condiiice 1.122:11: chids of is ophens object you roppee "Canc I8"
-Obewisc coninn will Condition 1.2.222
1.2311:
any cyliddrr atjost
-esyss Dous
gylldde objects, -Ohwwisc conime with Condition 1.2.2113
Cooditin 1.2.2211
ettee Coditio 1.222 check there any logg oiseet oncls-ostl on lage cyan ebjects costine with Cooditie 1.12221 -Ohhewise, continee
222.2
Conditon
-88 reqandd "Cau
ennene Coodiice 1.1.222.:: Chcck (fttere asy cobe object iepood -Ohhensiio,
-Ohavwie. ropped CCac 45*
Conditon 1222..12 check dace -EE regaadd "Cau 47"
WAAMAN Coeditien clock there mny obrect
-Ohervirr repord "Ceo
-ohkrwoac, regpand Case 23
ANV Coadition 1333 shock iriin ioy tege etjoct
-ffyrs Roos only on large dhyees, and contioss Condition 1233211
All Condition 1.2:
any vahere deiest
-Ohorvier conire with Condtion 13..333
Conditia
Okkerwaae, continee wath Conditinn 122
Asseve Condition 1,2332 thebe them any now dbject
yr' regund "Cau
line
Obbevisc respeed
-If you focaccnly -Oherroee contine wath Condition 1.313
obycls, contre with Coedition 12.11
3EVE Couditinn 1,2 122222 cleck if thee areaa object
yi reguaal CCase 53"
Ollvvis respoe
Figure 18: Full policy example (ClevrPolicy-T N=6).
28

---

Published as a conference paper at ICLR 2026
Full Policy Example- ClevzPalicy-M M (8-6)
The sallwingg pelloy persitic destace raking frrewookk for proceeting vioud inpe:
otjer dane & and comiase Candions 1.21
-fyys an seodeett with
sutuelees prokic lini copano Lonchics land lood skould beenallaat sllladd be abdu of
-ftervinc contaac Condaae 122
onyuns light
appiicall Buy
Craditis 2 Tichark jri any
the cow dene 7riea
holey 15550a L--aaaa
Ify fioscrlly onlgesss do saw or fono thes come shape an dinto T.add Condiaa 1.2
*D Fiffeer loge objen
you focu only largo digits extiro Coeditio 1.3
-Dlarrii contiac was Condatse 12112
Doas only Lang yellow
my yelloe
continu Condition
.Ifym Rossc by reai eit with the sime color in deders the sane os de
cuntiaa wils Coedate 1...U.
-iiherrr contines Condines
1.1112
New Condtion it t-chect intera any cite street
-Oleviiie
only will Conditin 1.1.2
-Ityak rapsed "Cau 25 -Olerriic rerpoal
object
Condaiinn 1. check iriter iuayy nathee atjact IIf you sopead "Case
enove Conditio 12311.2 check antere is ure nither oiject
Conditin
-nd ai Cvedite 1.2.
If you sopend "Uauc""
Coedaion 211121100 chock thoc ane scall objoct -Ifyr
Oiheriie repood "Cae
New Concitioo 1.1.2.2 cleck there are meca oeeee -ffirviiee coninay with diiiiionn I. 1.122
-Oherraaa regpod
snover Condiice 171 1 cleck d'eer lu shall ebjict
nover Condaice 1.1 1::2:: check rhee If you sogeed "Cine] 3
-Oferrvaac "Cou
1.12212
350 Castiion 1.21
-1fbok racse only on amal oijeca with the coler dana 6 and oves mac with Condaiss
ldgeor
II sogeed "Cuuc" 4
Ciocrionyy oylindo cbjoct
-Ofleriiee centice wilh Conditie 121.22
diherrti repood
enove Condabie 1.21 chide arther rother stjjct
Adual Coedatie 1.1 1.22 cluck Llace aae oyliddeebis Offeraaie coniras Condios 1.55
.ffse (roe on cessoaal tulber odgett vns caler is detn S.nnlvnniie
-Deerviaal contine Condinoe 12112133
Cooditon 1.33 1224 dacck Cocc leary rabler object sos repead "Cow
Aboual Ovediie 12 211211 check thare aubo -Ifyx reveed we
ifceeeiv yllbee objost
1212212 chick .IFsw regvel "ow 13
cube
II you ropeal "Coe
-dherrie rapood "Cau
-Olerriaic rorpad "Cao
Als Condiies 2. whock E'dere is as crea eliedt
stouE Conliiko clecks theer is ophor sljjnt indiomitie
-Oleevive ounkau wilh Condition 11..2
Coedase 121:221
-Oferriie
121122
fruss thes only lurg que yiloten odgens, and cintines Cordirie 1
Condaiee 1. 1: cleck irttrs ary cytede objict
Coedatoe 2.1222 check ihters any rather otjjant
-Olarviac roppud Coocee
enver Condroe 12. 2:1225 cleck drhers sey rather -10x ropsed ax
If you sopend "Lase 22"
dihervii continus wits Condition 1121112
Newer/Caaaa Coediten lo 1.222 dlcck tacc luazy zabber object Ofherviiee repoed "Cae
-11 oecuro snccciinde liske diacc anlventac was Coudatoo 1221
Nover Uondatoe 111.1.2 cleck ther otgect the manatil
Candiaes onleres cy ano bazee ofject
Jena, Or: mege-
lupp with
Of contie Colditio 1.2113
Cesdiin
Coodaion 11 1.1.1
anou Condllio 1.211 cllck l'teer is wall object
Aevert Custizs 1..1t check licayy stjjct that canca sloppaac
1f oo by Conditee 12.1132
crlacee olggess conticno Coeditiee 223.111
-o sopeed "Cue, 13"
Coedtiin 1222 cleck df yyy receed "Cio 36"
1.22 thec iunzy metal sbjict
1I sopead "Can
Cvedttie 1221112 ched. ('ore dijeet -ffys
diferiiee repood Twue
Alw Condatios 11122 cleck there is stued that has ts aane share dane Zikeage> saw shage is dewoy del
-Olerviac ropurd "Coc
1.1 2221 dibberrte continus Coeditin 12222
snowe Conditio 1331 chack dfhoo we rylher dtjpet -Oferrinn cuntnac was Conditee 1:22122
soust Cuedtiio 2:2:: cleck foo ons sfjjett duat has the went coler un done, hrineggo
snouet Coediiie 13.5 1.1 cleck d'hece -1 rocsed "Cas 43
contone with Canvtion 11.33 11.33.3
vive oxdiaa Coedition 12.2122
nevert Conditas 1 check if lecay mart abject
Coedatiee 2.21222 check therr aep ebject yys "Ca
Oilevviee rapked "Cac
-Olerrine reppod "Coc
severt Cuadiowwl 13 th wheck infensssayy atgect that his the cara maniii
Jane +icemp>
enew Cosdtton 1:33 chace enhere any grars abrect
c1 co ve urace yub 12222
Coudition 1.222..
"Cow 1ar -iferiiie reguod
enove Condabee 3:331 cleck tribr neal sdyee -1 on un uroce exal obbecaa and onnine do Coeditie 121211
nover Conditio 11.22 check ('ler lue other sbjort Doceceely laggn oejectt Condinea 112222
Cooditin 11221
Ovedaton 222211 chock thore anp ophere otjust Iffus
Cosdiin 2222 Deserivny whoeet If ses iapead Tow 217
Offerriie, resport
Anpuer Coedase 13 1322112 chick thee ophore sfjjcc .Ifsm requed -Oherzaae,
Condiin dene
inney dijosi ane sataid
No sogead "Cae 13"
about Cundiie 12222 cleck dhe cale cl fo by concefe cbocc and wenii Conditinn 12333.
Coediton 122222 chock lf any mxtal oljien Iffys Now
-btrraaa regood
enover Controe 132225 cledk drhers Mey aiject -liervii "Coc3""
Figure 19: Full policy example (ClevrPolicy-MM N=6))
29

---

Published as a conference paper at ICLR 2026
Full Palicy Exsmplc- GTAPolicy
AN lad Bourgtinnn Ae Coladdo
Aggane ([Tare "tayonaall "ape bury "mmatppaa ead, tapaialf "atadd wxll)
package
Sheiitaes Tie sxfor sood on unl res
Seuge
"ropiiaf" lren Nkdadd xxit)
Sare Youier Creytivoohgg Corethoeniiie wwwetieeeheett.... vS, Cresthonckente darke
Te
Tope" "The Raphih Melad
trancDDetigte
Aspantertd Thene: "rege"" c "ppe "eegg" Toooppann::/ call "Toginod" mn. "detsff:xdt))
Satto Goopliacc vi, Geopkiearh Cnse Croge and stei rilaa reuass De tel
"tcRa"" nalli, ("tane" "I", tope" "Mvcrptie"" "Sdeen fims Eocitii "ragptoc" alle, "dcfia"""
Tyec Begiesker besDeciiies
Siggeeiatteeiheeeee hp Rerkenttikkalassiiiii 92 Ragin weraateeeeeee
opee "celppee aiile "agaiset tne In "dourieker"
Dowrppaer Thcted
woreieg locspiion
Netwe cell, theeee heet topel Yout Mevergte "he opee doctpties Engless "repiref" me. "tehhal"" rafl).
Type: Fhe
you
directy, prareg
pyple
12,41]
(kaclutee)
Sxpreeh([)heee "Voeeso?" "Ape"" "wor. Nnwmipten" "Markkeen foom tyhen "eepoeed Iee, Vilkee
oppeaionn formce intern the tee expenskes
Urpe Saker
reum
prining sawerr eidk intaree fimas
sxpert sycchek Squathe dd Ineedos
e Duttsc cEppties
Sulue
anfiiiiiiiaaal Quateell dap-Tra)
reloe csom]
"ape"" torf "iwrtpier" tphen "eepeee" toc,
Durello
Aspatittd (There" reege Tooh "erge" "euerpte" "reernf" we "ferre" ratti, Careet type "ger "doerpese" "The Mee condinee in the deee (ld. stI. loan seced"
repse
de
State Veuine Addios
(hereasa (hhngge) (niiiigs teeiieeeee (osaerl rekkinayy
"mopihed" Ino. Vochat" mulll, Thame "eaker" hop?" "hot "ooopptoe (eal, "rogrss" the detuer tol'l)
Toctolenge TasTilaage
Cherrcskes The "Rewwaat"" ancedii the best but "covrpsnn" wy openaed by ovene "oceeee tie, "echath nell))
Tyec traadis tegpiiiiiaion sli IragdttsSatee Inapeigliziinn u
well
Trase anaes dosersee on "heapdii riples ute
iraee metie tee aterenecee Traasherrrtiin
(O02 Ttads)
agc Coogliieac 12t Wau oodie 200 otherrac use "Gauliiansh. A2'. 1f Test
1f Test acouet
Toottiinag If codie s0s otiemio TecTstrugs
3pc Shrae 1f east acouet C. taa Wapeeenrraaaaaeeeeeeeee
mndes 200 Tow of oo 105
aoe agit 10, wegoeeaoooeoe.....
Bette ciling onc the htppecttttyycbooooooooooo weceottoiiiiiioooo pols shold fors oe roale TosTeeloss Base. forme wite "ees" arpance www.ceeeeeeeei....
cill
tmagee Soerch
Figure 20: Full policy example (GTAPolicy).
30


Reviews:
[{"review_id": "X9oJT6B4T0", "reviewer": "Reviewer_kmvm", "summary": "The paper introduces Multimodal Policy Internalization (MPI), a task and methods for training multimodal conversational agents to follow complex, reasoning‑intensive policies without carrying those policies in the inference prompt. The authors propose TriMPI, a three‑stage training recipe: (1) Visually‑Masked Continual Pretraining (VM‑CPT) to inject policy knowledge by language modeling the combined (policy + task) streams while masking visual tokens; (2) CoT‑SFT, which teaches explicit policy‑guided reasoning on task data; and (3) RL fine‑tuning with a new rollout augmentation called PolicyRollout (PoRo) that adds policy‑aware trajectories during exploration but keeps training/inference aligned by applying policy gradients only to the no‑policy path. They also release two benchmarks: ClevrPolicy (synthetic, controllable policy complexity; a text‑only variant and an image‑augmented policy variant) and GTAPolicy (real‑world images and queries for tool‑use with versioned, user‑conditional rules; low‑data regime). TriMPI yields large gains over SFT and in‑context baselines, maintains general capabilities, and substantially reduces prompt tokens and prefil time once policies are internalized.", "questions": "My questions to the authors are the following:\n\n1. I could miss something, but you notice that you apply vision masking in the first training stage, isn't it a common practice for almost all adapter-based multimodal training, to mask vision input from calculating cross-entropy loss? Or there is some specific trick applied in your research. Could you, please, provide more detailed explanation for this stage?\n2. Beyond Policy Override, do you evaluate OOD generalization across visual domains or unseen tool types?\n3. How do failure modes break down (policy‑branching vs. perception vs. tool‑argument formatting), especially on GTAPolicy?\n4. What are the most common overfitting patterns you observed during training, and can VM‑CPT or PoRo be adapted to mitigate them further?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strength of the paper is as following:\n\n1. The authors address a practical, under‑explored problem: handling long, reasoning‑intensive policies for decoder‑only models. The motivation is clear and quantified.\n2. Two new datasets are introduced. ClevrPolicy (synthetic, controllable complexity; text‑only and image‑augmented policy variants) and GTAPolicy (real images and queries; tool metadata and versioned rules in a low‑data regime) provide controlled benchmarks.\n3. The three components (VM‑CPT, CoT‑SFT, RL) are well motivated and ablated; PolicyRollout improves over GRPO/DAPO by enabling policy‑grounded exploration. And we can clearly observed that SFT part is not enough to train the model to compress the long prompts.\n4.  The paper evaluates policy updates, policy knowledge referral, catastrophic forgetting, and efficiency, which is crucial for this method, because, generally, on-inference prompt extension with the policy description seems more robust approach in comparison to the specific training.", "weaknesses": "Despite the strong points of the research, I noticed a few weaknesses\n\n1. Risk of overfitting and external validity. While TriMPI maintains general abilities (MMMU‑Pro/MMLU‑Pro) and handles policy updates (Policy Override), broader real‑policy evaluations (e.g., long stylistic/safety guidelines) would further validate external generalization.\n2. The three‑stage pipeline, particularly RL, increases implementation and tuning burden; DAPO vs. GRPO behavior on small data underscores this.\n3. Although the authors argue soft prompts are task‑specific and hurt robustness, a direct empirical comparison to strong prompt‑tuning/gist‑token baselines would strengthen the case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SfH57kfk69", "reviewer": "Reviewer_DXvn", "summary": "The paper tackles the problem of internalizing multimodal policies---complex, reasoning-intensive system prompts that guide agent behavior---into the parameters of large multimodal conversational models to improve efficiency and policy adherence. It introduces TriMPI, a three-stage framework combining Visually-Masked Continual Pretraining (VM-CPT), Chain-of-Thought Supervised Fine-Tuning (CoT-SFT), and PolicyRollout, a policy-aware reinforcement learning stage extending GRPO/DAPO for grounded exploration. Experiments on two proposed benchmarks, ClevrPolicy (synthetic decision-making) and GTAPolicy (tool-use), show significant performance gains over SFT and in-context baselines, while maintaining minimal degradation on general reasoning benchmarks (MMMU-Pro and MMLU-Pro).", "questions": "- *L048:* The estimated range of policy prompt lengths (1K–50K tokens) is broad—are there empirical or open-source statistics supporting this claim?\n- What are practical scenarios where multimodal policy adherence is critical over text-only policy for conversational agents?\n- What’s the distinction with deliberative alignment (Guan et al., 2024) apart from the input modality?\n- Writing\n    - *L072—073:* The phrase “…, which requires minimal reasoning” can be more clear, example,  “…, which requires minimal reasoning to adhere to the policy.”\n    - There’s seems to be inconsistency in Section 4: Line 256 describes two stages while Line 259 and Figure 4 show three. The method description be made more consistent across sections.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is easy to follow, with strong conceptual motivation and clear examples of datasets and methods.\n- The paper addresses an important and relevant challenge of conversational agents under long system prompts.\n- Extensive experiments, ablations, and analysis (including generalization, efficiency, and catastrophic forgetting) lend strong empirical evidence to claims.", "weaknesses": "- It requires multi-stage SFT and RL with explicit SFT data; less straightforward than prompt-based alignment approaches.\n- It is evaluated on self-constructed datasets (ClevrPolicy, GTAPolicy) rather than realistic multimodal policy data.\n- It is unclear how the benefit of “multimodal policy” over “text-only policy” translates to conversation agent use cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7dnqSTsDxF", "reviewer": "Reviewer_2FLG", "summary": "This work addresses the challenge of Multimodal Policy Internalization. Policy internalization is a known problem in LLMs, which involves learning policy information through model parameters, instead of providing it in context during inference. This work highlights that this in-context policy can significantly increase inference costs, more so for simpler user queries. This work concretely explores these issues with Multimodal models and proposes a benchmark and a training paradigm for multimodal policy internalization. Overall, on the proposed benchmark, with their framework TriMPI, the multimodal models are much better at following policy and accomplishing a task, without requiring the policy in context during inference.", "questions": "- L304, it's claimed that the policy remains insufficiently grounded in the policy. Was this a qualitative observation? Is there a way it can be measured?\n- Just as an additional baseline, what if the model did have access to the prompt at inference time?\n  - Does MPI still work better?\n  - Or does having the prompt, essentially, make all the baselines and MPI the same in terms of performance?\n  - This could be a good tradeoff to understand when the cost of MPI tuning may be higher than just putting the policy in the prompt. Note that this does not take any away from the contribution of the work. \n- At what point does the in-context policy tradeoff wrt the user query become negligible?\n  - Specifically, are there scenarios where the user queries may be large enough that the policy tokens are negligible with respect to them?\n- In the policy rollout, the way I understand it, is that the RL algorithm now has more policy-aware options, but a question comes to mind, which might be naive:\n  - How are these policy-aware responses affecting the rollouts? Is it because they’re more correct, hence GRPO will force the Policy model to produce more similar trajectories? \n- Is RAG an alternative? One piece of related work that I get curious about when thinking of the problem of multimodal policy internalization is what if the LLM could leverage a RAG-like system to retrieve necessary policy-based information.\n  - Maybe this is a different way of implementing MPI, but this can maybe reduce the amount of in-context information needed during inference, by only retrieving the relevant one. I would be curious to hear the author’s thoughts on this one.\n- Layers in CLEVR policy: are these referring to the layers in the decision tree? Is that how the complexity of tasks is defined?\n- Minor: There are often references to figures and tables in the supplement; perhaps it might become easier to parse if it were stated, like Table 5 in the Supplement, etc.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Firstly, I think the paper was well written, and most of the content was very clearly understandable. Great job on that!\n- The problem of multimodal policy internalization is a well-motivated one, especially in scenarios like legal or financial applications, where agentics systems might need to reference large documents with text and images to make decisions. \n- The idea of policy rollout, to include policy-aware algorithms in the response space, is a clever one and seems to have good empirical benefits. \n- The authors have ablated various stages of their pipeline, CPT, RL finetuning, and the role of policy rollout. \n- Additional analysis on computational efficiency and evaluating policy referral is interesting and should provide useful advice for future practitioners when adopting the ideas from this work.", "weaknesses": "One key question I have about this work is that, although it is geared towards multimodal models, it seems that most ideas are in some way bootstrapped on ideas in the language domain, and they have, after some tuning, shown benefits. \n- This is not necessarily a bad thing, but were there any explorations as to how using policy images can benefit policy internalization?\n- The lack of discussion on the vision side of things is my only major complaint about this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)