ICML 2025

World Model Implanting for Test-time Adaptation of Embodied Agents
Minjong Yoo 1 Jinwoo Jang 1 Sihyung Yoon 1 Honguk Woo 1
Abstract
In embodied AI, a persistent challenge is enabling
agents to robustly adapt to novel domains with-
out requiring extensive data collection or retrain-
ing. To address this, we present a world model
implanting framework (WorMI) that combines
the reasoning capabilities of large language mod-
els (LLMs) with independently learned, domain-
specific world models through test-time com-
position.
By allowing seamless implantation
and removal of the world models, the embod-
ied agent’s policy achieves and maintains cross-
domain adaptability. In the WorMI framework,
we employ a prototype-based world model re-
trieval approach, utilizing efficient trajectory-
based abstract representation matching, to incor-
porate relevant models into test-time composition.
We also develop a world-wise compound attention
method that not only integrates the knowledge
from the retrieved world models but also aligns
their intermediate representations with the rea-
soning model’s representation within the agent’s
policy. This framework design effectively fuses
domain-specific knowledge from multiple world
models, ensuring robust adaptation to unseen do-
mains. We evaluate our WorMI on the Virtual-
Home and ALFWorld benchmarks, demonstrating
superior zero-shot and few-shot performance com-
pared to several LLM-based approaches across a
range of unseen domains. These results highlight
the framework’s potential for scalable, real-world
deployment in embodied agent scenarios where
adaptability and data efficiency are essential.
1. Introduction
In recent years, policies powered by large language models
(LLMs) have demonstrated remarkable success in embodied
1Department
of
Computer
Science
and
Engineering,
Sungkyunkwan University. Correspondence to: Honguk Woo
<hwoo@skku.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR ...