Multimodal Policy Internalization for Conversational Agents
Authors: name: Zhenhailong Wang; name: Jiateng Liu; name: Amin Fazel; name: Ritesh Sarkhel; name: Xing Fan; name: Xiang Li; name: Chenlei Guo; name: Heng Ji; name: Ruhi Sarikaya
Published: 2025-10-10 15:28:30

Abstract: Modern conversational agents like ChatGPT and Alexa+ rely on predefined
policies specifying metadata, response styles, and tool-usage rules. As these
LLM-based systems expand to support diverse business and user queries, such
policies, often implemented as in-context prompts, are becoming increasingly
complex and lengthy, making faithful adherence difficult and imposing large
fixed computational costs. With the rise of multimodal agents, policies that
govern visual and multimodal behaviors are critical but remain understudied.
Prior prompt-compression work mainly shortens task templates and
demonstrations, while existing policy-alignment studies focus only on
text-based safety rules. We introduce Multimodal Policy Internalization (MPI),
a new task that internalizes reasoning-intensive multimodal policies into model
parameters, enabling stronger policy-following without including the policy
during inference. MPI poses unique data and algorithmic challenges. We build
two datasets spanning synthetic and real-world decision-making and tool-using
tasks and propose TriMPI, a three-stage training framework. TriMPI first
injects policy knowledge via continual pretraining, then performs supervised
finetuning, and finally applies PolicyRollout, a GRPO-style reinforcement
learning extension that augments rollouts with policy-aware responses for
grounded exploration. TriMPI achieves notable gains in end-to-end accuracy,
generalization, and robustness to forgetting. As the first work on multimodal
policy internalization, we provide datasets, training recipes, and
comprehensive evaluations to foster future research. Project page:
https://mikewangwzhl.github.io/TriMPI.