Test-Time Mixture of World Models for Embodied Agents in Dynamic Environments
Authors: Jinwoo Jang
Minjong Yoo
Sihyung Yoon
Honguk Woo
Venue: ICLR 2026 Poster (2026)
Keywords: Embodied AI
World model
Mixture of expert
Test time adaptation
Few-shot expansion
Primary area: applications to robotics, autonomy, planning
TL;DR: Test-Time Mixture of World Models for Embodied Agents in Dynamic Environments

Abstract: Language model (LM)-based embodied agents are increasingly deployed in real-world settings. Yet, their adaptability remains limited in dynamic environments, where constructing accurate and flexible world models is crucial for effective reasoning and decision-making. To address this challenge, we extend the Mixture-of-Experts (MoE) paradigm to embodied agents. While conventional MoE architectures modularize knowledge into expert components with pre-trained routing, they remain rigid once deployed, making them less effective for adapting to unseen domains in dynamic environments. We therefore propose Test-time Mixture of World Models (TMoW), a framework that enhances adaptability to unseen and evolving domains. TMoW updates its routing function over world models at test time, unlike conventional MoE where the function remains fixed, enabling agents to recombine existing models and integrate new ones for continual adaptation. It achieves this through (i) multi-granular prototype-based routing, which adapts mixtures across object- to scene-level similarities, (ii) test-time refinement that aligns unseen domain features with prototypes during inference, and (iii) distilled mixture-based augmentation, which efficiently constructs new models from few-shot data and existing prototypes. We evaluate TMoW on VirtualHome, ALFWorld, and RLBench benchmarks, demonstrating strong performance in both zero-shot adaptation and few-shot expansion scenarios, and showing that it enables embodied agents to operate effectively in dynamic environments.

Published as a conference paper at ICLR 2026
TEST-TIME MIXTURE OF WORLD MODELS FOR EM- BODIED AGENTS IN DYNAMIC ENVIRONMENTS
Jinwoo Jang, Minjong Yoo, Sihyung Yoon & Honguk Woo*
Department of Computer Science and Engineering
Sungkyunkwan University Suwon, Republic of Korea (jinustar, mjyoo2, godboy3752, hwoo]@skku.. edu
ABSTRACT
Language model (LM)-basee) embodied agents are increasingly deployed in real- world settings. Yet, their adaptability remains limited in dynamic environments, where constructing accurate and flexible world models is crucial for effective rea- soning and decision-making. To address this challenge, we extend the Mixture- of-Experts (MoE) paradigm to embodied agents. While conventional MoE ar- chitectures modularize knowledge into expert components with pre-trained rout- ing, they remain rigid once deployed, making them less effective for adapting to unseen domains in dynamic environments. We therefore propose Test-time Mixture of World Models (TMoW), framework that enhances adaptability unseen and evolving domains. TMoW updates its routing function over world models test time, unlike conventional MoE where the function remains fixed, enabling agents to recombine existing models and integrate new ones for contin- ual adaptation. It achieves this through (i) multi-granular prototype-based routing. which adapts mixtures across object- to scene-level similarities, (ii) test-time re- finement that aligns unseen domain features with prototypes during inference, and (iii) distilled mixture-basee based augmentation, which efficiently constructs new mod- els from few-shot data and existing prototypes. We evaluate TMoW on Virtual- Home, ALFWorld, and RLBench benchmarks, demonstrating strong performance in both zero-shot adaptation and few-shot expansion scenarios, and showing that it enables embodied agents to operate effectively in dynamic environments.
INTRODUCTION
Embodied agents powered by language models (LMs) are increasingly deployed in diverse environ-
ments such as households (Song al. 2023 Ahn et 2022). factories (Kang et al., 2024 Zhang et al,, 2023), and virtual games (Zhao et al. 2024: Fan et al.) (022). However, their monolithic architectures restrict adaptation, with capabilities frozen at training and domain knowledge buried in billions of shared parameters. This is especially problematic for embodied agents in real-world settings, where tasks and environments change constantly beyond training. In practice, this lack of adaptability leaves retraining the only option, which imposes significant computational and data collection burdens and hinders real-world deployment. While in-context learning (Song et al. 2023 Kim et al. 2025) attempts to address this through domain- prompting, it merely shifts computational burden to inference with inflated context windows, highlighting the need for truly modular architectures.
Mfxture-of- Experts (MoE) (Jacobs et al. 1991) architectures provide structural modularity by se- lectively activating expert modules per input, thereby supporting scalable inference and domain specialization. Building on this property, recent work has applied MoE to domain adaptation in LM-based agents through techniques such as meta-distillation (Zhong et 2022). vision-language models (Iftee et al. 2024). and anomaly detection (Lei et al. 2024).
In this work, we investigate MoE architectures for LM-based embodied agents, aiming support rapid adaptation to unseen domains in dynamic environments. While MoE can flexibly activate
"oorreponningg author

---

Published as a conference paper at ICLR 2026
Mixture-on Saaerre-ff-xxppee of-Experts
Test- Mixture World Models (TMoW)
Domains
Unssen Domains
Domains
Text Training
Unssen Domains
Observations
Observations
Observations
Observations
Allgned
Not Aligned
Alllnee
Allaned
Prototypes
Refined Prototypes
Routing Function
Routing Function
Protatype based Routing Runction
Uposted Routing Fundiinn
Activate
Activate
III Invalid Routing for Exparts
Acthvate
Activate
Acthvate
Antivate
Experts
Figure 1: Existing Mixture-ou -ace--ff-xxxppp Experts employs a fixed router after training, making adaptation to unseen domains costly and requiring retraining (left). In contrast, Test-time Mixture of World Models (TMoW) overcomes this through prototype-base based routing and test-time training of the rout- ing function to reflect new domain characteristies without demonstrations (right).
domain-specific experts, their routing functions, responsible for selecting appropriate expert mod- ules for each input, remain fixed after training, making adaptation to novel domains possible only through costly retraining. For embodied agents in temporally and spatially changing environments, it is essential overcome this rigidity by adaptively reconfiguring routing functions and extending domain experts to meet these evolving conditions.
To address this need, we present a Test-time Mixture of World Models (TMoW) framework (Fig- ure D. which performs test-time training of a routing function without requiring demonstrations. This test-time training approach enables rapid adaptation to environments with temporal and spatial variations beyond the training distribution, facilitating continuous expansion of the model's s capa- bilities. In the framework, world models act as internal simulators that allow the agent to predict environmental dynamics, reason about future outcomes, and plan appropriate actions, where each model corresponds to a distinct domain within the environment (Zhu et al. 2025]
(025] [aanner et al. 202T).
TMoW employs multi-granulat prototype-based router that adapts world model mixtures by comparing input observations with learned prototype representations across different levels of spa- tial abstraction, ranging from local objects to global scenes. We adopt this hierarchical design in- spired by LMs' proven ability to build effective representations through layer-wise progression from local tokens to global context (Van Aken al. 2019). This router enables test-time prototype finement to unseen domains by refining prototypes through weighted interpolation between existing prototypes based their similarity the current environment. Furthermore, TMoW supports dis- tilled mixture-based model augmentation for data data-efficient creation of new world models. Unlike test-time prototype refinement, this technique requires few-shot demonstrations and distills knowl- edge from existing model mixtures to construct new models for unseen domains, thereby incremen- tally expanding TMoW's repertoire after deployment.
To evaluate TMoW, we conduct experiments on VirtualHome (Puig et al. 2018). ALFWorld (Shrid- har al. 202I). RLBench (James et al., 2020), and real-world robotic scenarios, demonstrating its capability for rapid and scalable adaptation for evolving and expanding environments. Our frame- work achieves 27.21 % improvement over SayCanPay (Hazra et al. 2024), state-of-the-ar base- lines in zero-shot adaptation, and 25.66% gain in few-shot expansion scenarios when constructing new world models.
Our contributions are as follows. (1) We propose the TMoW framework, novel extension of MoE that supports test-time reconfiguration of expert mixtures (i.e., test-time mixture), allowing embod- (2) We develop multi-granula prototype- based routing mechanism that realizes this test-time mixture capability, leveraging pro- totype similarity across spatial levels ranging from local objects global scenes. (3) We devise
ied agents to adapt unseen domains without costly retraining,
2

---

Published as a conference paper at ICLR 2026
distilled mixture-based model augmentation strategy that expands TMoW's adaptability by data- efficiently constructing new world models. (4) We validate TMoW on diverse benchmarks and real-world scenarios, demonstrating that it advances MoE-based adaptation and offers an effective solution for embodied agents operating in evolving physical environments.
RELATED WORK
LM-based embodied instruction following. Embodied instruction following requires agents to ground language instructions in physical environments through sequential action exccution, which in turn necessitates robust world models capable of representing diverse environmental configurations. Several approaches have been introduced. Code-driven policies (Singh al., 2023 Liang et al., 2023) generate executable programs from instructions, while reward-based methods et et al. 2023] Adeniji et al. 2023) learn value functions guide action selection. Furthermore, LM- reason- ing has been combined with domain-specific models to assess environmental affordances (Ahn et al,, 2022 Hazra al. 2024), and in-context learning approaches incorporate relevant demonstrations directly into the agent's decision-making process (Song et al.) 2023). However, these existing ap- proaches often struggle to generalize to new environments without significant retraining or increase inference-time overhead. Our TMoW framework addresses these challenges by introducing test- time adaptation directly into the MoE architecture, enabling efficient adjustment to unseen domains without retraining.
mixttue-off-xperte The Mixture-of-Experts (MoE) provides efficient model scaling by activat- ing only a subset of experts, as shown in sparse MoE transformers like GShard (Leiikhin et al.) 2020) and Switch Transformer (Fedus al. 2022 MoE has also been applied various adapta- tion settings including Meta-DMoF (Zhong et al. 2022) which uses meta-distillation from domain- specific experts, and MoE-TTA (Tftee et al.) (024) which supports adaptation for vision-language models. Yet, existing MoE architectures face limitations in knowledge expansion, experts are tightly coupled within a single training graph. Incorporating new domains typically requires end- to-end retraining or costly knowledge distillation, thereby restricting modular extensibility. This rigid design prevents modular growth, making it unsuitable for evolving physical environments with diverse tasks and domains. Our approach addresses this through prototype-based routing, which directly supports rapid integration of new world models and dynamic mixture adaptation, while also facilitating knowledge expansion through few-shot model augmentation.
Test-time adaptation. Test-time adaptation has emerged as a critical capability for deploying models in dynamic environments where training and test distributions may differ significantly. Meta- learning approaches (Finn et al. [2077 Nichol et al.) 2018) learn optimization procedures that en- able rapid adaptation new tasks with limited data. More recent studies have explored lightweight
methods. L-TTA (Shin Kim 2024) focuses on adapting only the stem layer using uncertainty minimization, while training- free methods such TDA (Karmanov al. (024) leverage key-valur caches to enable progressive adaptation without backpropagation. Our TMoW framework brings test-time adaptation to LM-based embodied agents by supporting both dynamic adaptation and con- tinual expansion of world model mixtures.
APPROACH
3.1
OVERALL FRAMEWORK
We present the TMoW framework, which enables agents to dynamically adapt world model mix- tures and continuously expand their knowledge in response to evolving environments. Similar to unified world model approaches (Zhu et al., 2025 Janner et al. 2021), each world model in the framework captures environment dynamics through state transition function and policy.
As illustrated in Figure 2 TMoW integrates three core procedures into an MoE-based world model structure. These procedures not only allow test-time reconfiguration of world model mixtures by adjusting the mixture strategy itself, but also support efficient expansion through few-shot world model construction. (i) Multi-granular prototype-based routing: At test time, the most relevant
3

---

Published as a conference paper at ICLR 2026
Multi-g Multi-granular Prototype-based Routing
Probabiity
Mixture of World Models
Softmax
3 Domain Embedding
Linear
Graph Processor
D
Top-K Domain Similarity
Normalize
Mixture of World Models
Trajectory Data
Pretrained Weight
World Model
A
Domain Prototype Set
MLP
Mutti-granuaar Prototype
(iii) Distilled Mixture-baeee -eeeeeaaeedeeee Model Augmentation
0.00
(ii) Test- time Protolype Refinement
Guregg
Quary
Pretriined Weight
Now World Model
Instruction
Trajectory
Processor
Prototype
Domain Prototype Set
Figure 2: Overall framework of TMoW.
world models are dynamically selected and composed based on multi gratular prototypes that cap- ture domain semantics at varying levels of abstraction, from local object interactions global scene structures. A hierarchical pesssae--aassing network aggregates these multi-level features, enabling the layer-wise world model mixture where each granularity level can draw knowledge from differ- ent domain experts. This network works as router across world models, allowing the framework leverage partial domain similarities, such as shared object knowledge across different domains. (ii) Test-time prototype refinement: When encountering unseen domains, the router adapts by re- fining prototypes through weighted interpolation between existing prototypes based on similarity to the current environment. Specifically, through limilarity-basee refinement during ongoing en- vironment interactions, the router expands existing prototypes to absorb new domain traits, while preserving core knowledge, thereby enabling efficient adaptation to unseen domains via dynamic re- configuration of world model mixtures. (iii) Distilled mixture-based model augmentation: When encountering environments that differ significantly from seen domains, new world model can be constructed by distilling knowledge from the mixture existing world models based on few-shot demonstrations. This strategy leverages the collective knowledge of existing models, enabling rapid domain expansion while consolidating fragmented knowledge across diverse domains into a coher- representation of the domain. The newly distilled world model integrates seamlessly into the framework, with its multi-granule prototype directly incorporated by the router for future mix- ture decisions. As such, this augmentation supports long-term expansion by introducing new world models, whereas test-time prototype refinement enables immediate adaptation by reconfiguring the mixture weights of existing models.
3.2 MULTI-GRANULAN PROTOTYPE: BASED ROUTER
Datasets and world model. We assume access to pre-trained base model M and demonstrations {D,) collected from domains (D Following parameter-efficien MoF designs (Li et al.
2024), we integrate domain-specific world models into M using lightweight adapters (m, ر (e.g.. LoRA (Hu 2022)), which preserve the backbone parameters while enabling selective activation of multiple world models within each layer. Each adapter my captures the environmental dynamics and policies of domain D, by learning from its corresponding D,. Once trained, they are combined with the base model AM to form a mixture of world models, denoted as M @ {my | Each demon- stration D, = {(in 7,)), contains instruction -rarection-traectooryy pairs, where in specifies the task, and F, = {(o, a,, 9++1) he represents the execution trajectory consisting of observations o,, ot+1 O and actions 0, E A.
Multi-granular prototype. We introduce multi-granulaa prototypes that capture environmental semantics across multiple levels of abstraction through hierarchical graph representations. This
4

---

Published as a conference paper at ICLR 2026
Transformer Layer
Transformer Layer 2
Transformer Layer L
Instruction Context
Instruction
Instruction
Context
Context
Graph representation in multi multi-gramular prototype. As layers deepen, node representations evolve from local object features to global scene structures.
Figure 3:
design exploits key insight: while domains may differ globally, they often share common patterns at specific granularities. For instance, object interactions kitchens and offices follow similar affordance patterns despite different houses. By maintaining representations from fine-grained node features to coarse-grained structural patterns, our multi-granular prototypes enable the layer-wise hierarchical model mixture, identifying which models share relevant patterns each granularity.
To implement this multi-granular representation, we employ a graph processor that embeds ob- servations into hierarchical structures, ensuring comparability each granularity level. Given mini-batch sampled from demonstrations D). the prototype at each layer / for world model j is computed as
((()
Pj
E E (L,T) (o..))EF
where the graph processor (()) transforms observation graphs Glo) conditioned instruction i. We use a Message Passing Neural Network (MPNN) (Gilmer et al., 2017) architecture, introduc- ing a context-aware edge matrix (O) that dynamically adjusts neighbor aggregation based on task requirements.
t (A RO (H-)),
The adjustment function captures observation-contest interactions through
[ = fgate ("
where queries 15; Qi and keys KH, K, are obtained through separate layer-wise learned projec- tions of (t-1) and the embeddings of the instruction ((1) respectively, and d the dimension of queries and keys.
The prototype initially contains only local information but progressively gathers neighbor informa- tion through layers, with Instruction-based adjustment factors controlling this aggregation (Figure This creates natural progression from node-level features in early layers to graph-level patterns in deeper layers, analogous to how LMs progress from token-level paragraph-level reasoning.
Prototype-based router. To select which world models to activate and with what routing scores, the router compares the embedding extracted from the current input data with the prototype of each world model. The router receives an instruction i, and observation o as input and returns expert routing scores (w(",-. ... w for each layer f()) of the base model. Specifically, given input (i, o), the routing score for expert j at 1th layer is computed as
" = sim(E .p))) (()) g(0) = (g(())
(4)
(!) where g()) is domain embedding produced by the 1th layer of the graph processor's MPNN, P; is the prototype of world model at layer and sim(-.-) denotes cosine similarity. The score vector (10(0) ..... (!) at layer / sparsified by retaining only the top- entries and then normalized with softmax and scaling hyperparameter 7 (r 0).
(w]"
() = softmax ((w)) (1) w. W))T))
(5)
Finally, the output at (th layer is calculated by combining the base model with the N adapters, weighted by the normalized scores (1z(0, , (()))
5

---

Specifically, leverage the router's weighted mixture, where the weights indicate which frag- ments of knowledge from models are most relevant to the unseen domain's 's features. Given few-shot demonstrations D = {(i', 7')), we construct combined graph g' from the trajectory and process it Published as a conference paper at ICLR 2026
World Model
0.3
World World Model
3 a 2 www. 20
Rauiing Sonte
0,6
0,6
Rusling
0.0
10
12
14
8 Layer
10
12
Layer
Layer
Layer
(a) Seen domains
(b) Unseen domains
Figure 4: Comparison of the routing score heatmap before (left) and after (right) test-time prototype refinement for (a) seen and (b) unseen domains.
3.3 TEST-TIME PROTOTYPE REFINEMENT
We introduce test-time prototype refinement that dynamically adjusts prototype representations through environment interactions, enabling adaptation unseen domains without retraining. Our refinement expands prototype space and densifies prototype allocation around frequently encoun- tered features, exploiting previously underutilized knowledge within existing world models (Wang
et al. 2023).
Specifically, we obtain the refined prototype (0) from Pj (1) using the environment interaction by
(1) Dj (1 -1-o sim(E ,, )), aasim ,p, ",, (0
(6)
where Ap, is refinement term, g() is a domain embedding for the unseen domain from the envi- ronment interaction, and a is refinement rate. The refinement refinement weights (!)
(1)
then obtained by computing
j,k from prototype-proooyyp similarity, which are used to form the weighted
sum where the weights (Ff.. ... ,(0) J.N) are obtained through similarity measurement and softmax normalization with scaling hyperparameter Tr (Tr > 0):
(1) j.N) softmax((1))) j.1." (TN))/T)); J.k sp P
(1)
(1)
J.
Figure illustrates the comparison of routing scores before and after refinement. In the seen do- main (Figure 4a), the heatmap showed minimal changes, though there is a slight tendency toward utilizing more diverse world models. In the unseen domain (Figure 4b). this diversification is more pronounced, facilitating the utilization of relatively diverse world models. The iterative prototype re- finement enhances shared knowledge across world models by densifying prototype allocation around frequently encountered domains. This expands the effective coverage of the prototype space while constructing dense clusters high-frequency regions, thereby enabling fine-grained routing strate- gies that foster knowledge sharing across world models.
3.4 DISTILLED MIXTURE-BASEE BASED MODEL AUGMENTATION
We introduce a distillation-basse model augmentation that enables rapid expansion to novel domains by constructing new world models from mixtures of existing ones using few-shot demonstrations. When encountering novel domains with fundamentally different characteristics, this approach con- structs new world models by distilling knowledge from weighted mixtures of existing models, where the router's weights indicate which knowledge fragments are most relevant to the unseen domain. Unlike test-time refinement, which adapts unseen domains solely by reconfiguring mixtures of exist- ing models, this augmentation further extends the framework's 's capacity by generating that integrate seamlessly into the routing system. This seamless integration is enabled by prototype- based routing, which allows newly created models to align with existing ones and immediately join the routing process without structural changes.
models
through an if to obtain the layer-wise routing scores (w(4) ... (!) for each layer l. We initialize world model m' mixture of existing models, then fine-tune it on D' before integration into

---

Published as a conference paper at ICLR 2026
Table 1: Zero-shot adaptation performance in VirtualHome, ALFWorld, and RLBench. Throughout the following experiments, we report 95% confidence intervals computed across 5 random seeds.
Seen domains
VorinllHome
ALFWorld
RLBench
Baselines
SR (T)
PS (4)
(T)
PS (4)
SR (1)
PS (U)
(Huang et 2022
10.78%
27.81=007
2.32%
49.34=a.s
11.26%%=1..99%
LLM-FT
61.37% 26.99
17.98
51.78%&+113
13. 13.22
65. 15.53%)a224%
11.01:4..9
L.L.-Planner (Song et 2023
50.98%
17.852055
37.19a025
36.21%
14.77
FLARE Kim al MPS
14.60%
19.9320.1
21.22%
34.40=1144
53.05%:::47799
14.77:00.4
SayCanPay TMoW
2024
14% 14% 17777
15.70+00.5
1174
13.19#n84
68 2:
8.69==0.5
83.61 11.3%%
11.07:0.33
72.05% % 1:062
6.94
71.89% % #2.73%
6.30::0.16
Unseen domains
VärtallHome
ALFWorld
RLBench
Raselines
(?)
(1)
(?)
(1)
SR (t)
PS (L)
7SP (Huany al. 2022
7.32% Shst
28.22:004
2.00%
49.68:0 92
20:4
18.734031
LLM-FTT
14.24%-22602
21.00=:.39
39 2.:441
41. 41.24::047
15 6.5%
17.44:11.0
LLM-PPlaner (Song et
2023
96.05
239
22.93=000
8.16%
43.54=0011
19.799%
17.1941.54
FLARE Kin 21253
40.07% 11
22.57
11.31%
42.85
31.37% 200%
11.37.10.22
SayCanPay (avv é mr 2024
49.53% 4:0-
18.552001
42.04% 245
40.64:043
38. (..555%% a08
10.76:10.4
TMoW
1.45%
13.20=0.22
1.55%
37.44:352
62.75% #2,65%
8.95
the mixture. Then, new world model m' and its corresponding prototype p (0)) are computed
N
m
Σ j=1
11) (1) m
V ml)l)
E [LF))D
Cre Cre (M p()) = E
(8)
(F,F)DD
where 17 is learning rate, and the LTr is teacher-forcing training loss that supervises next action and observation along the trajectory, similar to Janner et al. (202T).
EXPERIMENTS
Environments and datasets. We evaluate TMoW with embodied environments such as Virtu- alHome (Puig et al., 2018), ALFWorld (Shridhar et al. (021), and RLBench (James et al.) 2020). VirtualHome is 3D virtual environment for simulating activities in household, ALFWorld an door task simulation environment that aligns text and embodied robotic manipulation, and RLBench a robotic manipulation benchmark for tabletop tasks, which we adapted to support language- actions. In VirtualHome, 78 tasks (16 seen, 62 unseen) paired with 20 distinct scenes (10 seen, 10 unseen) to form 445 demonstrations. In ALFWorld, following the CL-ALFRED benchmark (Kim) et al. 2024), 6 task categories (4 seen, unseen) are paired with scene categories (3 seen, un- seen) to form 1,304 demonstrations. In RLBeecch, 4 task categories (3 seen, unseen) are paired with 6 scene categories (4 seen, 2 unseen) to form 163 demonstrations from seen domain. Detailed descriptions of environments and datasets are provided in Appendix A
Evaluation Metrics. We adopt evaluation metrics: Success Rate (SR) and Pending Steps (PS). SR denotes the ratio of successfully completed tasks. PS stands for the average timestep taken to complete tasks, akin to cost-fffectiveness in (Hazra et al. 2024).
Baselines. For comparison, we use five baselines. ZSP (Huang et al. 2022) is a zero-shot approach of using pre-trained model to adapt to unknown domains without additional training. LLM+FT is an LLM that is fine-tuned with domain-speciin vironments. I.M--Planner (Song al. 2023) is a few-shot in-context learning method generate and refine high-level plans for new domains. SayCanPay (Hazra et al. 2024) is state-of-the-an model that integrates LLMs with a heuristic cost minimization method to generate cost-effective plans. Lastly, FLARE (Kim al. 2025) is state-of-the-art embodied task planning model that combines environmental perception with adaptive replanning to generate grounded task plans by correcting predictions to align with environment features with few examples. We use Llama-3.2- 3B (AI@ Metal 2024) for ZSP,
demonstrations to improve performance in new en-
FLARE, and the Say model in SayCanPay, while using trainable 1.ama-3.2-11 1B for LLM+FT, the Pay model in SayCanPay, and TMo..
7

---

Published as a conference paper at ICLR 2026
Table Few-shot expansion performance in VirtualHome.
Unseen domains, ViruaaHome
J-Skot
5-Shot
Average
Baselines
SR
PS
PS
SR (t)
PS
LLM-FT
50: 16% 1.0.4%
19. 10.51.
6-1.36%...7.8
18. 5..55.00004
52 11%61338
19.03..0.0
L.M--llanner (Song
2023
97% 02%
20.07:0.19
43.6 43.61%:10992
21.0610.11
43 3.399
21.4510:15
FLARE Kim & al 2025
12.17% 1..37%
22. 1910..9
20. 20.67:10.12
12 2.7777
21.5611..1
SayCanlay
20244
98% 9.8.9:6--4200000
58. 188.8896=1000
16.
56i 36.963%--66.66
17.35++0:3
TMoW
81,56 1.9%
13.20
83.61 % +:333%
12,04=0.64
82.59 1.99%
12.62=0.56
Seen dooain
Can you stack up the boxss that are on the clesk?
Seen Domains
Real-world
Basclines
SR (t)
PS (1)
FLARE Kiim et al 2025
10% sws
7:26 2.000
SayCanPay (Haza aratn 2024
52.90% = 19%
7.00 2:099
Unscen domain
Can you up the trash on the desk?
Unseen Domains
Real world
Baselines
SR (t)
PS (1)
[Kim
36 11.188
7.87
SayCanPay (Haza atal 2024
7 8,80% 4.0%
9.71 2:02
2.99%
4.89
Table Zero-shot adaptation performance in Real-world Scenario.
Figure 5: Examples of Real-world environment.
4.1 MAIN RESULTS
Zero-shot adaptation. We evaluate each method zero-shot adaptation scenarios, where agents must generalize across diverse and unseen domains, without access to any additional demon- strations. Each domain represents a unique combination of task and scene.
As shown in Table I TMoW consistently outperforms all baselines across both seen and unseen domains, particularly showing robust generalization to unseen domains. The results demonstrate the superiority of our test-time mixture approach, where multi-granular prototypes effectively cap- ture shareable domain characteristics and enable immediate generalization through world model mixture, achieving an average improvement of 14.61% in SR and 4.42 reduction (35.31% improve- ment) in PS across environments for seen domains. Furthermore, test-time prototype refinement allows our model effectively process unseen domain data by aligning unseen features with pro- totypes, thereby appropriately exploiting partial knowledge from existing models. This approach particularly excels in unseen domains, where we observe an average improvement 27.21 % SR and reduction of 3.45 steps (14.81% % improvement) in PS across environments for unseen domains.
Few-shot expansion. We further evaluate few-shot expansion scenarios, where each target do- main provides only a few demonstrations at test time. This setup examines how effectively TMoW expands its knowledge through distilled mixture-based augmentation with minimal supervision.
Table compares performance across different few-shot settings (I and 5 shots) in VirtualHome, il- lustrating how the number of available demonstrations affects adaptation quality. shown, TMoW surpasses all baselines, achieving on average 25.66% gain in SR and 4.73 step reduction in PS VirtualHome, These results demonstrate that our distilled mixture-based augmentation efficiently achieves additional performance improvements through knowledge expansion while maintaining the modularity of the overall framework.
Real-world scenario. We conduct experiments in real-world environments similar to RLBench to validate the practical applicability our approach. As shown in Figure 5 we use Franka Research 3 robot arm, and these experiments involve user instruction execution in specific object settings. In Table 3 TMoW achieves an average improvement of 34.36% in SR compared to the best baseline and 2.77 reduction (39.57% improvement) in PS for seen domains. For unseen domains, our method demonstrates even more substantial gains with an average improvement of 38.60% in SR and 2.98 reduction 37.86% improvement) in PS compared the best performing baseline, FLARE
8

---

Published as a conference paper at ICLR 2026
Our TMoW demonstrates superior adaptation capabilities in these real-world scenarios, success- fully handling tasks that require intricate manipulation sequences and environmental understanding. The results confirm that TMoW effectively transfers from simulation to reality, maintaining robust performance despite the inherent uncertainties and variations present in physical environments.
Ablation study. We validate TMoW's components through ablation studies in VirtualHome, Unseen do- mains settings. For multi-granular prototypes, we com- pare against TMoW-OBe -bbject (using only local object fea- tures for routing) and TMoW-Scene (using only global scene features for routing) variants. To assess test-time refinement, T'Mo-NoRReiie uses fixed prototypes with- out adaptation. In Table 4a Object 15.49% improvement in SR and 3.60 reduction in PS, and PMoW-Scent 72.00% SR and 14.26% PS. This confirms that the multi-grane graliiiganularityy approach enables efficient knowledge sharing while preserving an overall understanding of domain structure, thereby contributing to improved performance. Dynamic refinement improves SR by 7.65%, validating its effectiveness for unseen sce- narios. Finally, we evaluate our distilled mixture ap- proach against TMoWtScratch (training from scratch). In Table 4b our method achieves 18.39% higher SR than
Table 4: Ablation study of TMoW
Model
SR (T)
PS (1)
Multi-gramiiar Prototype-based Routing
TMaW Object
65.28%.144%
16.72.19
TMoW-Scene
8.74%
27.38
Test-time Prototype Refinement
TMoW outperforms TMoW-
TMoW-NoRefine 73.30% 475
14.85
TMaW
80.74% 11405
13.12
(a) Zero-shot adaptation scenario
Model
SR (t)
(L)
Distilled Mixture-bused Model Augmentation
TMoW Seratch
59.84%==125
18.164
TMoW
81.56% LAVS
13.20
(b) Few-shot expansion scenario
scratch training while using 40% less data, demonstrating superior efficiency knowledge distilla- tion from mixture of the existing world models.
4.2 ANALYSIS
Analysis for multi-granular prototype. Figure 6a illustrates the routing score entropy across lay- ers, revealing distinct mixture patterns of world models. We observe high entropy in early layers, indicating that multiple world models contribute equally to local object-level representations, max- imizing knowledge sharing for fine-grained features. Conversely, later layers exhibit lower entropy, suggesting that global structural and scene-level embeddings become more specialized to specific world models. This hierarchical transition from shared local features to isolated global represen- tations demonstrates how our multi-granular approach naturally balances knowledge sharing and domain specialization across different levels of abstraction.
Analysis for test-time prototype refinement. Figure 6b shows the routing score entropy before (TMoW-NoRRfine) and after (TMoW) prototype refinement, demonstrating increased average en- tropy across all layers. This entropy increase reveals that refinement corrects the initial misalignment between prototypes and unseen domains, enabling the router to recognize previously underutilized knowledge within existing world models. By expanding prototype coverage to better capture un- seen domain characteristics, the refinement process facilitates more active knowledge sharing across models, allowing previously underutilized models to contribute to unseen domains. This means the agent now leverage more diverse mixture of world model expertise.
Routing Score Ectropy
0,3
10
TMoW NoRefine
Domain
Density
TMoW
0.2
0.1
16
14
15
6m0
0.05
5.10
0155
0.20
0.25
8.70
3.35
Layer
Routing Score Entropy
(a) Layer-wise routing score entropy heatmap
Routing score entropy changes after refinement
Figure 6: Analysis for multi-granular prototype-based routing and test-time prototype refinement.

---

Published as a conference paper at ICLR 2026
Analysis for top-K routing. Table presents the impact of varying K values in our top-K rout- ing mechanism. TMoW-Top-3 (default, same as TMoW) achieves the best overall performance (80.16% SR, 13.20 PS), significantly outperforming single-expert selection (Top-1: 65.43% SR, 17.34 PS). This validates that combining knowledge from multiple world models is essential for handling unseen domains. Notably, increasing K beyond 3 leads performance degradation, with Top-5 showing slight decline (77.52% SR, 15.19 PS) and Top-7 exhibiting substantial drops (66.01 % SR, 17.67 PS), comparable to single-expert performance. This suggests that excessive expert activa- tion introduces noise rather than useful knowledge, as irrelevant world models dilute the contribution of relevant experts. The optimal K configuration balances knowledge diversity with focused expertise, enabling effective mixture without information overflow.
Table 5: Analysis for top-K K routing
TMoW-Top Top-1
TMoW Top-3 (TMoW)
TMoW TMoW-Top-5
TMoW Top-7
(T)
(S)
SR (t)
PS
SR (R)
PS
SR (7)
(S)
65.43%62..29
17.3420.22
80. 16%611488
13:20.082
77.52%20444
15. 192045
66.01%
17.671036
Continuous expansion. Figure 7 presents our TMoW's performance in continuous domain ex- pansion scenarios, As domains are added in new phase, the framework incorporates new world mod- els through distilled augmentation, with refinement facilitating efficient integration. The performance improves not only for new domains but for ex- isting ones, indicating positive knowledge transfer, while maintaining high performance previously encountered domains without forgetting. This validates our prototype-base based routing's ' ability to pre- serve existing knowledge while enabling cross-domain synergy through expanded coverage.
Success Rate
1.0
0.5
Phase
Phase 2
Phase
Pbaso
Damain)
Figure 7: Continuous expansion scenario.
CONCLUSION
We presented the TMoW framework to enable embodied agents effectively adapt to evolving envi- ronments at test time. By extending the MoE paradigm with multi-granular prototype-base based routing and distilled mixture-based model augmentation, TMoW allows embodied agents flexibly recon- figure world model mixtures at test time and efficiently construct new world models from few-shot demonstrations without retraining the entire system. Through extensive evaluation on VirtualHome and ALFWorld, we demonstrated that TMoW achieves strong performance in various adaptation scenarios, validating its effectiveness and scalability dynamic embodied settings.
Future work and limitation. TMoW has two main limitations: (1) its performance is inher- ently bounded by the capabilities of the underlying LLM used for planning, and (2) the world model approach may face challenges in highly non-stationary environments like multi-agent set- tings where other agents' behaviors continuously change the environment dynamics. For future work, we plan to enhance the safety and interpretability of TMoW's s routing decisions while extend- ing to multi-agent systems, enabling more reliable deployment and coordinated decision-making in safety-critte critical applications.
ACKNOWLEDGMENTE
This work was supported by the Institute of Information & Communications Technology Plan- ning (yaluation(IITP)) grant funded by the Korea governmen((MSIT) (No. RS 25522225-5,,4 AI Star Fellowship Support Program(Sungkyunke Insight Reasoning via Continual Learning, 15%, RS-2022-I122 1045 (2022-0-01045), Self-directed multi-modal Intelligence for solving unknown, open domain problems, 15%, RS-2022-11220043 2002-0-00043), Adaptive Personality for Intelligent Agents, 15%, RS-2020-11201822 ICT Cre- ative Consilience Program, 15%, and 2--19-919199421 Artificial Intelligence Graduate School Program (Sungkyunkwan University)). 15%), and HTT--ITRC (Information Technology Research Center) grant funded by the Korea governmen((MIST) (TTP-2202-RS-2224-00022 10%).
Univ.), 15%, 04-2022002288666 Accelerated
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Ademi Adeniji, Amber Xie, Pieter Abbeel, and Archit Sharma. Language reward modulation for pretraining reinforcement learning, 2023.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan. Eric Jang. Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov. Yuheng Kuang, Kuang- -uue Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as i can, not as i say: Grounding language in robotic affordances. In Conference Robot Learning, 2022.
Meta. Llama 3.2 model card. 2024. URL
www./o///meeeee 11ama//
11 1lama mode 1s b www.lo/mmtttoo wwo///aainn wwtp:/ooees...eelo sateelllllaaaaa 1ama3 2/MODEL CARD md
Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang. Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35: 18343-18362, 2022.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1 135, 2017
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry, 2017. URL
01212
Rishi Hazra, Pedro Zuidberg Dos Martires, and Luc De Raedt. Saycanpay: Heuristic planning with large language models using learnable domain knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 20133,, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1,2):3, 2022.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning, pp. 9118-9147. PMIR, 2022.
Md Akil Raihan Iftee, Wahida Mahjabin, Anik Ekka, and Sunanda Das. Moe-tta: Enhancing con- tinual test-time adaptation for vision-language models through mixture of experts. In 2024 27th International Conference Computer and Information Technology (ICCIT) pp. 3278-3283. IEEE, 2024.
Robert A. Jacobs, Michael Jordan, Steven J. Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 1991. doi: 1116///eco.1991.3...
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019- 3026, 2020.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning big sequence modeling problem. Advances in neural information processing systems, 34:1273-1288 2021.
Jeon Ho Kang, Neel Dhanaraj, Siddhant Wadaskar, and Satyandra K Gupta. Using large language models generate and apply contingency handling procedures in collaborative assembly ap- plications. In 2024 IEEÉ International Conference on Robotics and Automation (ICRA), pp- 15585-15592. IEEE, 2024.
11

---

Published as a conference paper at ICLR 2026
Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of lision-language models, 2024.
Byeonghwi Kim, Minhyuk Seo, and Jonghyun Choi. Online continual learning for interactive in- struction following agents. In The Twelfth International Conference on Learning Representations, 2024.
Taewoong Kim, Byeonghwi Kim, and Jonghyun Choi. Multi-modal grounded planning and effi- cient replanning for learning embodied agents with few examples. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 4329-4337, 2025.
Tianwu Lei, Silin Chen, Bohan Wang, Zhengkai Jiang, and Ningmu Zou. Adapted-moe; Mixture of experts with test-time adaption for anomaly detection. arXiv preprini arXiv:2409.05611, 2024,
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding, 2020.
Dengchun Li, Yingzi Ma, Naizheng Wang, Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts, 2024. URL /tttps::/axxiv.....
abs/2404. 15159
Jacky Liang. Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control, 2023,
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. 2018.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Tor- ralba. Virtualhome: Simulating houschold activities via programs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8494-8502, 2018.
Jin Shin and Hyun Kim. L-tta: Lightweight test-time adaptation using versatile stem layer, 2024.
Mohit Shridhar, Xingdi Yuan, Alexx-Ale Cote. Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference Learning Representations, 2021.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models, 2023.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2998-3009, 2023.
Betty Van Aken, Benjamin Winter, Alexander Löser, and Felix A Gers. How does bert answer questions? a layer-wise analysis of transformer representations. In Proceedings of the 28th ACM international conference on information and knowledge management, pp. 1823-1832, 2019.
Qi-Wei Wang. Da- Wei Zhou, Yi-Kai Zhang, De-Chuan Zhan, and Han-Jia Ye. Few-shot class- incremental learning via training-free prototype calibration. Advances in Neural Information Processing Systems, 36: 5560-15076, 2023.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Are- nas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang. Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. Language to rewards for robotic skill synthesis. 2023.
Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hur Sun, and Joseph Lim. Bootstrap your own skills: Learning to solve new tasks with large language model guidance. arXiv preprint arXiv:23100 10021, 2023.
12

---

Published as a conference paper at ICLR 2026
Zhonghan Zhao, Wenhao Chai, Xuan Wang, Boyi Li, Shengyu Hao, Shidong Cao, Tian Ye, and Gaoang Wang. See and think: Embodied agent in virtual environment. In European Conference Computer Vision, pp. 187-204. Springer, 2024.
Tao Zhong, Zhixiang Chi, Li Gu, Yang Wang. Yuanhao Yu, and Jin Tang. Meta-dmoe: Adapting to domain shift by meta-distillation from mixture-of-expers. Advances in Neural Information Processing Systems, 25.22243-22257,22 2022.
Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified world models: Coupling video and action diffusion for pretraining large robotic datasets, 2025. URL http:::://rriii..r//a/asss
02792
13

---

Published as a conference paper at ICLR 2026
A ENVIRONMENTS
A.1 VIRTUALHOME
We conduct our experiments using VirtualHome (Puig et al. 2018), Unity-based simulation plat- form that enables embodied agents to execute complex household tasks through natural language instructions. The environment provides 20 distinet house configurations, each featuring diverse room architectures and object arrangements that capture the beterogeneity of real-world domestic spaces.
Environment structure. VirtualHome represents environmental states directed graphs, where nodes correspond to entities (agents, objects, rooms) and edges encode spatial and functional rela- tionships. Each state observation consists of relational triples (ei, r, e2) encoding relationships such as spatial containment (e.g.. faucet inside bathroom), proximity (e.g., character close plum), adja- cency (e.g., bedroom adjacent kitchen), and agent-object interactions (e.g., character hold bread- slice). This graph-based representation naturally captures the compositional structure of household environments and enables reasoning about object affordances and spatial constraints.
Figure A.1: The example top-view scene of the VirtualHome environment.
Action space. Agents interact with the environment through six primitive action types: walk (navigation), grab (object manipulation). open (state changes), put (placement), putin (con- tainer interactions), and switch (device activation). These actions modify the environment graph according deterministic transition rules, enabling predictable yet complex multi-step behaviors. The constrained action space reflects common houschold interactions while maintaining computa- tional tractability for learning algorithms.
[System]
You are a home robot agent. You can use skills, (walk [object or room], grab [object], switch [object], open [object], putin [target object), put [target object]). You should return only a skill after "Action:". Room: livingroom, bathroom, kitchen, bedroom.
[User]
Instruction: {instruction]
Observation: {observation)
Action:
Figure A.2: System prompt in VirtualHome
Task specifications. We evaluate on 78 household instructions spanning four task categories that test different aspects of embodied reasoning. Each instruction defines success criteria through target graph configurations, requiring agents to transform the initial environment state through appropriate action sequences. Tasks vary from simple object retrieval to complex multi-room activities involving
14

---

Published as a conference paper at ICLR 2026
Figure A.3: The available scenes of the ALFWorld enviromment.
sequential dependencies and state preconditions, providing comprehensive benchmark for evalu- ating adaptation capabilities across varying complexity levels.
A.2 ALFWORLD
ALFWorld (Shridhar et al., 2021 is a text-based embodied reasoning environment that translates the ALFRED benchmark's visual tasks into natural language interactions. This environment chal- lenges agents to exccute multi-step houschold tasks through textual observations and instructions, emphasizing task planning and state reasoning in partially observable settings.
Environment structure. ALFWorld presents environmental states through natural language de- scriptions that include spatial information, object locations, and agent positioning. After initializa- tion, agents receive textual observations (e.g., "You are in the middle of room. Looking quickly around you, you see a armchair 1, a coffeetable I...") alongside the instruction (e.g., "Your task is to: put a pillow armchair").
[System]
You are a home robot agent. You can use 10 skills, (go to [object], take [object] [object], put [object] on [object], open [object], close [object], toggle [object], heat [object] with [object], cool [object] with [object], clean [object] with [object], look). You should return only skill after "Action.". Room: livingroom, bathroom, kitchen, bedroom.
[User]
Instruction: (instruction)
Observation: {observationo)
Action: {actiono)
Observation: (observation))
Action: {action))
Action:
Figure A.4: System prompt in ALFWorld
Action space. Agents interact with the enviromment through structured text commands.
The action space contains navigation (go [location]), perceptual query (look, examine [object]). object manipulation (take [object], put [object] in/on (receptacle)), and state modification (cpen/close [container],
15

---

Published as a conference paper at ICLR 2026
Figure A.5: The example scenes of RLBench simulator.
[appl iancel). Each action triggers deterministic state transitions with corresponding textual feedback, enabling agents to track environmental changes and plan subsequent steps.
Task specification and dataset construction. This enviromment has 6 fundamental task templates that, when instantiated across various object-receptacle-roon combinations, yield 1,304 unique sce- narios. These tasks cover a lot of scenarios from simple object relocation to complex multi-stage procedures involving tool use. Each task requires agents to infer implicit subgoals, manage partial observability, and execute precise action sequences to achieve specified goal states. We utilize 6 task categories seen, unseen) with scene categories (3 seen, unseen) to form 1,304 episodes, and construct episodic dataset which contains task-trajectory pairs.
A.3 RLBENCH
We evaluate in the RLBench simulator (James et al. 2020) for manipulation scenarios. Built on the CoppeliaSim simulator, RLBench provides diverse robotic platforms and rich set of manip- ulation objects with varying complexity. We adapt this framework to support language- based task specification, enabling natural language instructions input for our evaluation benchmark.
Environment structure. RLBench includes tables, shelves, and storage units. Across the scenes vary furniture layouts, object types, and initial states to reflect the heterogeneity of the real en- vironment, Each scene contains 4-6 everyday objects distributed across tables, shelves, and storage units.
Action space. The robot interacts through primitive actions: open [object], pick [objectl, place [object] on [object], close [object], wipe [object], pour [object]. Each action is executed with collision checking and inverse kinematics validation in simulation.
[System]
You are home robot agent. You can use 6 skills, (open [object], pick [object], place [object] on [object], close [object], wipe [object], pour [object]). If the question is "Action:" you should answer with skill. If the question is "Next observation:" you should answer with the next observation. You must answer only with what is requested and nothing else.
[User]
Instruction: {instruction)
Observation: (observation)
Action:
Figure A.6: System prompt in RLBench
16

---

Published as a conference paper at ICLR 2026
Put plates on same color dishracks
Separate cup and straw in bins
cup on cup
Stack
Figure A.7: The examples of the Real-world scenarios.
Task specification and dataset construction. Tasks span home manipulations, object reloca- tion/organization, surface cleaning, and container handling, and specify success as satisfying a target scene graph (e.g.. (yellow plate, is, clean), (yellow plate, on, yellow dishrack)). Tasks embed state preconditions and sequential dependencies (c.g., pick sponge before wipe), encouraging planning that couples perception with action. We construct datasets across 4 task categories: Place (e.g., place knife on chopboard), Pour (e.g., pour water in cup), Clean (e.g., Clean the plate) (3 seen) and Putin (e.g., Put carrot the frying pan) (1 unseen), paired with 6 scene categories (4 seen, 2 unseen), resulting in total of 163 episodes for the seen datasets.
A.4 WORLD ENVIRONMENT
We evaluate in a real-world setup using a Franka Research robot arm for household manipulation scenarios. Our real-world experiments assess robustness, generalization, and closed-loop adaptabil- ity under sensing uncertainty and imperfect actuation.
Environment structure. The workspace includes tables, shelves, and storage units. Across scenes we vary furniture layouts, object types, and initial states to reflect the heterogeneity of real homes. At episode start, the robot waits home pose, then uses RGB-D -GB- perception combined with VLM to extract scene-graph inference (object and relation candidates) to describe targets and constraints (e.g., container relations, surface placement, open/close capability).
Action space. The robot interacts through nine primitive actions: open [object], pick [object], place on [object], , close [object], wipe [object], pour [object], sweep [object], flip [object], push [object]. Each action executed with collision checking and inverse kinematics validation.
Task specification and Dataset construction. Tasks span canonical home manipulations, object relocation/organizztion surface cleaning, and container handling, and specify success satisfying target scene graph (e.g., (yellow plate, is, clean), (yellow plate, on, yellow dishrack)). Tasks embed state preconditions and sequential dependencies (e.g., pick sponge before wipe), encouraging planning that couples perception with action. We construct 8 scenes (4 seen, unseen) reflecting heterogencous household setups, and define 3-5 tasks per scene for total of 29 episodes. Each scene contains 8-12 everyday objects distributed across tables, shelves, and storage units.
B IMPLEMENTATION DETAILS
In this section, provide the implementation details of our proposed framework TMoW and each baseline. Our framework implemented using Python v3.12 and trained on system of an Intel(R) Core (TM) 19- 10980XE processor and two NVIDIA RTX A6000 GPUs.
17

---

Published as a conference paper at ICLR 2026
[System]
You are a home robot agent. You can use 9 skills, (open [object], pick [object], place on [object], close [object], wipe [object], pour [object], sweep [object], flip [object], push sob- ject)). the question "Action:" you should answer with a skill. If the question is "Next observation." you should answer with the next observation. You must answer only with what is requested and nothing else.
[User]
Instruction: {instruction)
Observation: {observation]
Action:
Figure A.8: System prompt in worldworld Environment
B. BASELINES
ZSP. We use the zero-shot policy (7SP) as a non-adaptation reference to assess the improvements achieved by TMoW. A single pretrained LLM(Llama-332-33 1-3B-Instruuc) receives the observation from the environment, which is injected into the prompt described in Figure A.2 and A.4 The model generates the next action step by step without any fine-tuning or additional supervision. Our imple- mentation follows the open-source with only minimal I/O adjustments to match our environment API.
LLM+FT. This baseline represents adaptation via fine-tuning on limited domain-specific data. allows us to compare the efficiency and effectiveness of TMoW against conventional parameter adaptation. For the few-shot adaptation scenario, we further train the already fine-tuned model with the sew-shot data from the target domain.
LLM-PPanner. We evaluate an embodied planner that leverages in-context learning for high-level reasoning. We aim to demonstrate the effectiveness of TMoW over in-context learning through this baseline. Given the current observation and instruction, relevant demonstration snippets are concatenated with the planner prompt, then the pretrained LLM proposes the next action. For data retrieval, we use a DPR-based sentence embedding model that collects relevant data from the same dataset employed in the fine-tuning of other baselines (LLM+FT, SayCanPay). For the few-shot adaptation scenario, we augment the dataset with the target domain examples. Our implementation follows the open-source
SayCanPay. We include a Reinforcement Learning baased planner that integrates LLM reasoning with heuristic cost minimization. By comparing against this approach, evaluate the effective- ness TMoW. This baseline uses three models: (1) pretrained LLM(LLlaa-93.233B-mmm the Say model, (2) environment-proviee proviide optimal affordances the Can model, and fine-tuned LLM 1-B--nsttt))) for the Pay model. The hyperparameters of SayCanPay are listed in We follow the implementation of the open-source
for
Table A.l
This state-of-the-art baseline extends conventional in-context learning with an environment-adaptive replanning module that revises plans based on observed scene states. We demonstrate the superiority of TMoW in cross-domain problems by comparing with the newest method. The retriever, conditioned on the observation and instruction, collects relevant dataset ex- amples and builds the planner prompt. If the agent fails, the model substitutes targets via semantic similarity over observed objects when names mismatch. For the fhw-shot adaptation scenario, target-
FLARE.
18

---

Published as a conference paper at ICLR 2026
domain samples are additionally included in the training data. We follow the implementation of the open-source
Table A.1: Hyperparameter settings and configurations of baselines
Hyperparameter
Value
Trainable model (LLM+FF,, and Pay
Llama-3.2-1B
Reasoning model Simulation (ZSP, LLM-Planner, FLARE and Say
Llama-3.2-3B
Reasoning model for Real-world (FLARE and Say
Llama-3.2-8B
Batch size
Gradient steps Learning rate scheduler Initial learning rate
200
cosine
5 x 10
Learning rate (for fow-shot learning)
x 10-6
Temperature (both of -lama-3.2-1 3.2-IB and 1lmma-3.2-BB
1.0
B.2 TMoW (OURS)
Our framework, Test-time Mixture of World Models (TMoW), builds upon the parameter-efficient MoE architecture (Li al. 2024). We employ Llama-3.2-1B base model with LoRA for parameter-efficient fine-tuning.
Algorithms. Detailed procedures of TMoW are described in [ 3 Algorithm describes the construction the mixture of world models, which ranges from training each world model extracting multi-granular prototypes. The second (Algorithm) [ contains details on time prototype refinement, focusing on how it works while interacting with the environment. The last algorithm (Algorithm 3 provides details of distilled model augmentation, describing the actual construction and training process of the augmented world model.
Detailed implementation of multi-granular prototype-based router. The multi-granular prototype-based router adopts an MPNN (Gilmer et al.) 2017) structure, specifically utilizing GCN models. A key consideration in our design is the oversmoothing phenomenon inherent MPNNs. where node representations become indistinguishable with increasing depth. To address this, we strategically limit MPNN layers while using standard MLPs for the remaining layers. Given that our base model (Llama-3.22 1B contains 16 layers, we implement hybrid architecture where the (hhh 4th 8th, and t2 layers employ GCN to capture graph structure at multiple granularities, while all remaining layers use standard MLPs preserve representational diversity.
Each layer MPNN aggregates information from neighboring nodes through three functions: message(MsG), aggregate(Agg), and update(UPD). These functions recurrently compute the hid- den states for the observation graph (()) = (V,E, state the 7th MPNN layer is computed by
E. R) and the instruction i. Specifically, the hidden
H (t) (g(o), (P (Aco (Mso (H((")) A, R R))
(A.1)
where the initial (0) is the same as V. The three functions are computed by
M Msa (11))) H (11)) M
A ( (M, , A, R) = --
(A.2)
H UPD (1) (Á) 0 (Aw(
where D is a diagonal matrix such that Dj = Σx Ejk. the context-aware edge matrix, and a sigmoid function. (! and w!!) learnable weight matrices.
httpss//gthub..oo//uuumpiata Pay model in SayCanPay
"Say model in SayCanPay
19

---

Published as a conference paper at ICLR 2026
Algorithm 1 Mixture of World Models Construction
Require:
Base model M, demonstrations {D, P/ learning rate 771 and m2. gradient steps T₃ and
72
Ensure: Mixture of world models M @ {my) prototypes {p,") 15-1 for each layer /
1: for each domain {1,..., N] do
2: // Adapter training
3: Initialize the adapter my
4: for step := 1 to Ii do
5: Sample mini-batch B from demonstration D
6: Train adapter m, on mini-batch data B:
7:
m, + m; - 71 V m, [B(.F)esCI((A) (M @ my. F)]
8:
end for
9:
// Prototype extraction
for each layer E (1,... . do
11: Sample mini-batch B from demonstration Dj
12: Extract prototype using MPNN: (0) := Be
13: end for
14: end for
15: // Mixture of world models construction
16: for step := to T₃ do
from a combination of the demonstrations و Dj 17: Sample mini-batch
18: Train mixture of world models M @ (m, 1 on mini-batch data B:
my my - 7/2 m,, [(FF)BCCT (M @ {m.))...)) 7)] Vj E {1, _}3
20: end for
return Mixture of world models M @ {my}) prototypes (p, 1, ||= for each layer
Detailed implementation of context-aware edge matrix. In equation A.2 adjust the amount
of the aggregation based the instruction and context. The context-aware edge matrix É is caleu- lated as
È (A+ R fad (H
(A.3)
where @ represents the Hadamard product and I is the identity matrix.
To incorporate the instruction into prototypes, we introduce an adjustment function fadj that modu- lates neighbor information aggregation based on these inputs:
(H1) fgate ( (Q)
(A.4)
Here, fgate is a gate function (e.g., ReLU) and the embedding dimension. The adjacency function employs a cross-attention mechanism that captures the interaction between observations and context. Specifically, we project the hidden states and context into query and key spaces by
QH = (1- Q# Q: = (i) Q
K == (1-) K K, w(i)
(A.5)
where (!()) wil) w Q21 Q Kn and w are learnable weight matrices, and T denotes embedding functions for instructions, e.g., from language models.
Router pretraining. We pretrain the router using contrastive learning with a compound loss func- tion. For input data X (91, G2...., IN }. we define instance-level contrastive loss that leams innpeentttionnnvvriinn representations:
..,
exp(sim(V.(), (G.). Vz
(x
Σ n=1
log N
(A.6)
ΣN Em=n exp(sim(V))))eeeeeeee (G... V2(G,)))/)
Additionally, we incorporate domain-awand contrastive loss that encourages domain clustering:
N
Σm= [djm]=d[n| exp(sim (Gm.. (2G.)))T
(x( = N
log
(A.7)
Σm= exp(sim(M))) (aaiin/W..G,..... (9m), (2(9.))/T)
20

---

Published as a conference paper at ICLR 2026
Algorithm 2 Test-time Prototype Refinement
Require: Test environment Env (O, A) - O, instruction i
1: Take an initial observation from the environment
2: repeat
3: // Step 1: Prototype-hased routing
4: for each layer E {1..... L} do
5: Extract domain embedding: g() f (g(o),
6:
Compute routing scores: w sim sim(E() (1), Pjj for j
Sparsify and normalize: (w]] wx\ = softmax(topx " ,0/))//)/)))11 end for
7:
8:
9:
// Step 2: Mixture of world model execution
10:
y = (i,
for each layer / E {1,... . do
(1 m (1(-1) y :0:=M'h (y(l-1)) + Σ wj my
12:
13:
end for
Predicted action := y
15: Take next observation o + Cnv(o,a) a)
// Step Test-time prototype refinement
17:
for each layer E {1, _.} do
for j (1, NN do
19:
JjA == (p Pk E (1, ,
(1)
20:
(fg1, Fj,N) == softmax((r).) (rj.1, Tg,N w//r/ TT))
21:
Compute refinement term: Ap) = Ek=r 3.bP%
(1) (1)
Update prototypes: D + (1 - sim(E()) 1/ )p, + aisim p,) end for
22:
23:
24:
end for
25:
until episode done
Algorithm Distilled Model Augmentation
Require: Few-shot demonstrations D', learning rate gradient steps
Ensure: Distilled mixture of world models . 1 prototypes {p)
( N+1 for each layer 7 fj=1
1: if few-shot demonstrations D'' available for unseen domain then
2: Construct combined graph and take instruction: g', i' from observations in D
3: for each layer e {1. ,L} do
4: Forward through MPNN to get routing scores: (w)
5: Initialize new adapter: (1) Σx - np (0()m (1)
m
6: Compute new prototype: (()) (G)
7: end for
8: for step ;= 1 to T do
9: Fine-tune m' on D':
10: m' x- m' 17 Vn m' [EL. (M @ m', #)]
11: end for
12: mE m'; PN+1 (1)) VI (1,....
(1)
13: Add to model mixture: {my | and wase prototype set: {p) (1) 15=1
1, N+1
14:
end if
15:
return Mixture of world models M + {m prototypes (p,
(2) 1 } for each layer
The final training loss Lel combines two contrastive losses:
(x)(X)= 1 (()) + A Cx()
A
(A.8)
21

---

Published as a conference paper at ICLR 2026
where 11, V2 are augmentation functions for creating different views, sim(.,) is the similarity metric (e.g.. cosine similarity), d[m] denotes the domain label of graph Im- T is the temperature parameter for contrastive learning, and À is the balancing coefficient between instance and domain objectives.
Hyperparameters.
The hyperparameters of TMoW are listed in Table A.2
Table A.2: Hyperparameter settings and configurations of TMoW training
Hyperparameter
Value
Common
Base model
1lama-3.2-11
Learning rate scheduler
cosine
Warmup steps Temperature
200
World models
Batch size
16
Rank of LoRA
32
Gradient steps
2000
Initial learning rate
1 10
TMoW
Batch size
Gradient steps
10000
Initial learning rate
1 x 10
Learning rate (for few-shot learning)
1 x 10
C ADDITIONAL ANALYSIS
C.I EXTENSION RESULTS
Table A.3: Few-shot expansion performance in VirtualHome and ALFWorld.
Unseen domains, VirtualHooee
J-Shoot
S-Sikot
Average
Baselines
SR (1)
PS
SR (f)
PS (1)
SR (1)
PS (1)
LLM+FT
30.46781044
19.51.10s
64..6%%.7.18
18.55.0004
52 LA865
19. 19.03:20.4
T.T. Planneer Song al. 2023
10.07%
22.07
43.61 20..9
21.0620.17
43.30% 938
21.4520.15
FLARE Kim al 2023
42. 10.7%
22.19 19.19111
46.611%.77uz
20.67
42 29% 4.447%
21.56.1.18
SayCanPhy
2024
34.95%
17.77:0004
38.88%
16.02.0.22
56. 36.90%
17.3520.1
TMoW
81.56 11.0%
13.2010.48
83.61 % 1.33%
12.04:064
82.59% 1.4%%
12.62.0056
Unseen domains, World
J-Shoo
5-Shor
Average
Baselines
SR (?)
PS (J)
SR (f)
PS (1)
SR (f)
PS (1)
LLM+FT
43. 1.20PS
10.18:10.:3
48 80% 00%%
35.142.2.4
15.99%%....5 1.55%
37. 1.22
LLM-Plaaner (Song et
2023
8.91%
47.43:0064
7.39%
46.79:00.0
8.18%
17.11=0032
FLARE (Kim 2023
12. 12.28%%100 19.25%
43.67
11.46% 10.5%
44.40..0.9
11.87%
44.04110.4
SayCanFay
2024
26.08% 1.33%
39.00:0033
49.3 10.331%
37,5 37.55:00.24
17. 4
38. 0.22
TMoW
71.28 1.08%
34.23-15.56
% L.000s
25.73
71.58% 1009%
19.98
Few-shot expansion scenario. We evaluate few-shot expansion scenarios, where each target do- main provides only a few demonstrations at test time. This setup examines how effectively TMoW expands its knowledge through distilled mixture-based augmentation with minimal supervision.
Table A.3 compares performance across different few-shot settings (I and 5 shots) in VirtualHome and ALrWorld, illustrating how the number of available demonstrations affects adaptation quality. As shown, TMoW surpasses all baselines, achieving average 25.66% gain in SR and 4.73 step reduction in PS VirtualHome, and a 23.87% gain in SR and 8.30 step reduction in PS in ALFWorld, compared to SayCanPay.
22

---

Published as a conference paper at ICLR 2026
These results demonstrate that our distilled mixture-based augmentation efficiently achieves addi- tional performance improvements through knowledge expansion while maintaining the modularity of the overall framework.
Outdoor environment. We conducted additional experiments in CARLA, an outdoor autonomous driving simulator, to validate TMoW's effectiveness outdoor environments. We configured town scenes (Town01-044
shows seen, Town05, 06 unseen). Table showw that TMoW achieves 47.49% SR and 12.03 PS, significantly outperforming baselines (FLARE: 18.34% SR, 27.51 PS). The out- door environment presents different challenges from indoor settings, including varying weather con- ditions, diverse road structures, and multiple moving agents. TMoW's test-time adaptation effec- tively handles these environmental variations.
Table A.4: Performance comparison in CARLA outdoor environment
Method
SR (t)
PS (4)
FLARE
18.34%
27.51
TMoW (Ours)
47.49%
12.03
Vision-language model integration. To enable direct processing of RGB visual observations, we integrated a pre-trained Vision-Language Model (VLM) as graph generation module upstream of our pipeline. The VLM processes raw RGB images into text-based observations, extracting objects, relations, and states that are then converted into our graph structure.
Table | presents the performance comparison between exxt-baed and vision-based inputs in Vir- tualHome. Uision-based TMoW achieves 75.05% SR and 13.93 PS, showing a moderate decrease from text-based input 160.16% SR, 13.20 PS). This gap is attributed to noise in VLM's object de- tection and relation extraction compared to ground-truth text descriptions. However, the results demonstrate that TMoW effectively operates with real visual observations, maintaining substantial performance advantages over baselines even with noisy visual inputs.
Table A.5: Performance with Vision-Language Model integration in VirtualHome
Input Type
SR (t)
PS (4)
Text-based (Ground Truth)
80.16%
13.20
Vision-based (VLM)
75.05%
13.93
These results validate that graph-baser based approach is robust to different input modalities and can be seamlessly integrated with vision systems for real-world deployment, despite the inherent noise in visual perception systems.
C.2 MULTI-GRANULAR ROUTING FOR OVERLAPPING DOMAIN FEATURES
Our multi-granular prototype-based routing addresses domains with overlapping features specific granularity levels. When domains share characteristics at one granularity while differing at others (e.g., office and classroom with distinct objects but similar spatial layouts), ainglegranularrtyy ap- proaches cannot leverage these complementary distinctions. Our layer-wise approach exploits these differences across abstraction levels. Early layers distinguish domains through unique local features, while deeper layers may show convergence when global scene structures similar.
Figure A.9 demonstrates routing score distributions for domains with shared global features but distinct local objects. Initial layers show clear separation different object types (office equipment vs. educational materials) activate distinct experts. As layers progress, routing patterns converge when domains share similar global layouts (rectangular rooms with central tables and peripheral storage). This pattern validates that our multi multi-granular approach captures both distinctive and shared characteristics at appropriate levels.
23

---

Published as a conference paper at ICLR 2026
Routing Score
1.0
Room: 18
Room 20
Room 22
Room 24
Room 26
Room: 28
0.2
Room: 29
12
14
Room 92
Layer
Layer
Figure A.9: The comparison of the routing score distribution.
C.3 CORRELATION WITH ROUTING SCORE ENTROPY AND PERFORMANCE
Our analysis uncovers an insight that the routing score entropy distributions directly impact model performance. We observe that successful episodes demonstrate higher routing entropy than failed ones.
Figure A.10 compares the distribution of the routing on unseen domains. Success cases exhibit higher entropy values (mean = 0.23, blue dashed line) compared failed cases (mean 0.20, red dashed line), demonstrating that distributed routing pat- terns correlate with task success. This indicates that leveraging diverse world models through higher entropy routing enables the framework capture multiple domain characteristics simultaneously, leading to more robust adaptation and improved performance in complex, unseen domains.
entropy between Success and Fail cases
Success Mean
Fail Mean
Success
Fail
Density
5
3
0.00
0.05
0.10
0.35
0.30
0.25
0.30
0.35
0.10
Routing Score Entropy
Figure A.10: The comparison of the routing score entropy distribution between success and failed case.
C.4 ANALYSIS FOR REFINEMENT RATE &
Figure A.II illustrates the impact of the refinement rate a on success rate. When a is small, insufficient refinement occurs, leading to degraded performance compared to the baseline. This suggests that with low a values, refinement acts as noise rather than meaningful adaptation. Due to the test-time refinement nature where updates occur during inference, insufficient learning rates require too many steps to converge, harming performance.
However, once reaches sufficient threshold (a 2 0.5), the model consistently achieves perfor- mance above the baseline (red dashed line) and can rapidly adapt within fewer steps. This demon- strates that while appropriately sized refinement rate is crucial for enabling efficient test-time adaptation, our framework robustly improves performance once this threshold met, validating the effectiveness of approach across range of hyperparameter settings.
24

---

Published as a conference paper at ICLR 2026
Success Rate (%)
60
60
0.002
0.05
0.01
0.02
0.85
0.1
0.2
0.5
Refinement Rate
Figure A.11: The success rate per refinement rate in test-time prototype refinement.
C.5 COMPUTATION OVERHEAD
shown in Table A.6 ZSP and LLM+FT exhibit the lowest latency, followed by our TMoW approach which achieves the next best performance. While our method uses Llama-3.2-18 3.BBBB in ex- periments, the baselines such as LLM-Planner, FLARE, SayCanPay, employ larger Llama-3.2-3B models for reasoning without training. Moreover, in-context learning methods like LLM-Pllnner and FLARE suffer from increased latency due to lengthy prompt processing. SayCanPay shows significantly higher latency it requires inference across multiple models.
In contrast, our TMoW leverages adapter-based MoE architecture, enabling efficient test-time adap- tation while maintaining competitive inference speed through lightweight parameter updates and selective world model routing. The prototype refinement component introduces an average over- head of only 43.40 ms, indicating that the refinement stage remains computationally minor relative to the overall inference process.
Table A.6: Average latency comparison across baselines.
Baselines
Latency (ms)
7SP
115.43
24.91
LLM+FT
115.87 1: 24.88
LLM-Planner
740.46
40.42
FLARE
917.44 = 49.16
SayCanPay
2470.30
106.06
ToRe-ine
656.72 + 15.71
TMoW
700.12
25


Reviews:
[{"review_id": "MvY0vbEAox", "reviewer": "Reviewer_SnZA", "summary": "This paper introduces TMoW, a Test-time Mixture of World Models framework that enables embodied agents to adapt to dynamic environments by dynamically mixing specialized world models. Its core innovation is a multi-granular, prototype-based routing mechanism that leverages hierarchical scene features—from local objects to global contexts—to select experts. The framework supports test-time adaptation through prototype refinement and allows for continual expansion by distilling new world models from few-shot demonstrations. Evaluations on VirtualHome and ALFWorld show significant gains over strong baselines (up to 27.21%), demonstrating robust cross-domain generalization and continual learning capabilities without costly retraining.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1) The paper is well-written, the proposed pipeline is simple and easy to follow.\n\nS2) The framework enables dynamic test-time adaptation of world model mixtures through prototype refinement, allowing rapid adjustment to unseen environments without retraining.\n\nS3) The distilled model augmentation capability supports continuous expansion of the system's knowledge base through efficient few-shot learning from existing model mixtures.", "weaknesses": "W1) The paper emphasizes that multi-granular prototypes capture features from local objects to global scenes for fine-grained routing. However, when multiple distinct domains exhibit significant feature overlap at a certain granularity level (e.g., sharing similar local objects but having vastly different global scene semantics), how does the router effectively prevent expert confusion and erroneous activation?\n\nW2) The test-time prototype refinement relies on online environmental interaction and its performance is sensitive to the refinement rate α. In non-stationary or partially observable environments, where perceptual data can be noisy or delayed, how does this optimization process based on a local interaction sequence guarantee its convergence stability and reliability?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZOc8bPd6Eg", "reviewer": "Reviewer_7Jh2", "summary": "The authors introduce a new framework for adapting language model-based embodied agents to unseen environments, called test-time mixture of world models (TMoW). Through this framework, they address the limitations of vanilla LM-based embodied agents  and the classic mixture of experts paradigm, which has a rigid routing function. When adapting to new domains, existing methods require expensive retraining or knowledge distillation. In contrast, the work here proposes a dynamic test-time adaptation mechanism based on 1) multi-granular prototype-based routing using a hierarchical message-passing network, 2) test-time training of the routing function & 3) a distilled mixture-based model augmentation strategy that constructs new world models based existing ones using few-shot demonstrations.The framework and the effectiveness of the novel components are supported by a comprehensive baseline analysis and ablation studies.", "questions": "*Question*: What is the rationale behind using Llama-3.2-1B for some baselines and 3B for others?\n\n*Suggestion*: In Fig 2, for clarity, it would be good to add a small description of the top right box that links to the mixture of world models.\n\n*Suggestion*:It would be good to reference `Appendix A` in Section 4 to point readers to more details about the the environments, tasks and action space.\n\n*Suggestion*: Reference Table 4 in main body of section 4.2 in the ablation study.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper addresses an important gap in generalization of LM-based agents for embodied tasks and offers a solid alternative to costly retraining for new domains\n-  Authors conduct a thorough evaluation of the framework, on a comprehensive set of environments, tasks and baselines, showcasing significant performance improvement over state-of-the-art methods across both metrics (success rate and pending steps) and a very positive performance especially on unseen domains.\n-  The authors also transfer the framework to real world scenarios and share results of zero-shot performance which are superior to existing SoTA methods.\n- It was good to see that the authors include extensive ablation studies that support the effectiveness of the framework components: multi-granularity, test-time refinement & distilled mixture, where each component shows a clear performance improvement over its respective baseline.", "weaknesses": "I would've liked to see some discussion on the inference time for such an approach compared to the baselines. How much time does the test-time adaptation add to the decision making process and could it be regarded as feasible in a real-world domain from this point of view?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TmUtMSSquh", "reviewer": "Reviewer_AbzQ", "summary": "This paper proposes the Test-time Mixture of World Models (TMoW) framework to enable embodied agents to adapt dynamically to unseen domains through prototype-based routing and mixture refinement.", "questions": "1. There is already a large number of works addressing Dynamic MoE, Adapted MoE, and MoE for Continual Test-time Adaptation. What are the differences between this method and existing methods?\n2. Home environments inherently have low dynamics, making it difficult to accurately evaluate the performance of this method. How can this method adapt to outdoor environments?\n3. The framework introduces non-trivial computational overhead during test-time refinement, which may hinder deployment in real-time or resource-constrained scenarios.\n4. How does the multi-granular prototype mechanism ensure interpretability of routing decisions, especially in failure cases or ambiguous domains?\n5. Why was Llama-3.2-1B chosen as the base model for TMoW, while some baselines used larger models, and how does this affect fairness in performance comparison?\n6. Can the authors justify the choice of hyperparameters (e.g., refinement rate α) and whether they were tuned consistently across all environments?\n7. How does the distilled augmentation strategy ensure that new world models do not overwrite or interfere with previously learned knowledge?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "TMoW demonstrates strong performance gains over state-of-the-art baselines in both zero-shot and few-shot adaptation across multiple embodied benchmarks.", "weaknesses": "1. The motivation of this paper is inaccurate. The abstract states that \"conventional MoE architectures modularize knowledge into expert components with pre-trained routing; they remain rigid once deployed, making them less effective for adapting to unseen domains in dynamic environments.\" However, there is already a large number of works addressing Dynamic MoE, Adapted MoE, and MoE for Continual Test-time Adaptation.\n2. Home environments inherently have low dynamics, making it difficult to accurately evaluate the performance of this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3KWn0Pprgy", "reviewer": "Reviewer_fzQq", "summary": "This paper proposes a framework called \"Test-time Mixture of World Models (TMoW)\" to improve the adaptability of LLM-based embodied agents in dynamic and unknown environments. The core idea of ​​this framework is to extend the traditional MoE by introducing three key mechanisms: (1) Multi-granular prototype-based routing, which dynamically combines multiple experts based on prototype similarity during testing; (2) Test-time prototype refinement, which enables the model to adjust the prototype corresponding to each expert based on real-time data; and (3) Distilled mixture-based model augmentation strategy, which efficiently constructs new experts in few-shot scenarios. The authors conducted detailed experiments on multiple simulation environments such as VirtualHome, ALFWorld, and RLBench, as well as on real robots. The results show that TMoW significantly outperforms existing state-of-the-art (SOTA) models in both zero-shot adaptation and few-shot expansion scenarios.", "questions": "1. The update rule for the prototype $\\mathbf{p}_j^{(l)}$, specifically the update term $\\Delta \\mathbf{p}_j^{(l)}$, is calculated as a weighted average of other prototypes based on their similarity. While the paper includes experiments demonstrating that this method adjusts prototype vectors for unseen scenarios, this adjustment approach seems to inevitably push all prototypes towards a single average value.\n- Won't this degrade the independence between prototypes of different granularities and potentially lead to expert degeneracy? Why is this specific form of update effective?\n- **Recommendation**: I suggest a more in-depth discussion and a theoretical explanation for the validity of this update mechanism, particularly addressing the potential convergence issue.\n---\n2. The experimental results show that the refined model tends to exhibit high entropy in its routing scores. The authors also note that \"success cases have higher entropy in routing scores than failure cases.\" Intuitively, high entropy suggests a uniform distribution of scores (i.e., every branch is activated equally), whereas a good router typically makes a confident, decisive choice (low entropy), activating only a select few experts.\n-  Does this observation stem from the framework fundamentally differing from the standard MoE paradigm?\n\n- **Recommendation**: The current analysis appears somewhat superficial, failing to sufficiently explore the relationship between entropy and other crucial factors such as task type or domain similarity. A deeper investigation is warranted", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- This paper extracts multi-granularity prototypes from datasets divided into multiple domains and trains multiple expert adapters. During inference, the outputs of each adapter are weighted based on feature similarity. This method is logically consistent and effectively decouples the granular details of the scene.\n\n- The paper proposes a method to enable this hybrid expert model to quickly adapt to unseen domains, greatly enhancing the model's generalization ability.\n\n- The paper provides ample experiments, covering multiple simulation platforms and including real-machine experiments, and demonstrates significant advantages compared to multiple strong baselines.", "weaknesses": "- **Naming Convention and Terminology Clarity.** The use of the term \"World Model\" is highly problematic and requires stringent justification. In both Reinforcement Learning (RL) and Generative Modeling, a \"world model\" conventionally refers to a model that predicts the environment's state transition, formally defined as $s_{t+1} \\sim P(s_t, a_t)$. However, the 'world model' in this manuscript appears to be an LoRA adapter integrated into the LLM, designed to learn domain-specific knowledge. Its function is more akin to a domain-specific expert rather than a state transition predictor. Furthermore, the overall architecture (TMoW) strongly suggests an end-to-end policy learning approach. The authors must provide rigorous proof and a compelling rationale for why these LoRA modules qualify as \"world models.\" Failing this, the terminology should be replaced with a more accurate and representative term.\n\n- **Efficiency and Comparison with Traditional MoE.** The claim of leveraging a Mixture-of-Experts (MoE) structure is misleading regarding computational efficiency. Traditional MoE achieves reduced inference computation through sparse gating, which activates only a subset of experts. In contrast, the routing mechanism in TMoW uses a weighted summation, activating all LoRA branches. This design choice inherently increases the computational load during inference, rather than reducing it. The TMoW approach appears to be fundamentally about enhancing performance by increasing model capacity and incorporating multi-granularity perceptive heads. The authors must clarify the trade-offs between inference efficiency and performance improvement and explicitly detail the distinctions between TMoW and conventional MoE architectures.\n\n- **Insufficient Ablation Study on Expert Utilization.**\nThe ablation study focusing on granularity (object vs. scene) is noted, but this primarily equates to selectively enabling shallow or deep routers, which is expected to decrease performance. A critical component is missing: there is no experiment ablating the number of activated LoRA adapters (experts). Consequently, the individual effectiveness of different experts across domains has not been adequately validated. A crucial question remains: If a traditional sparse MoE approach is adopted—activating only a few heads with the highest routing scores—would the model's performance be adversely affected, remain stable, or perhaps even improve? This experiment is necessary to fully validate the expert-based domain specialization.\n\n- **Clarity of Problem Definition and Input Representation.**\nThe introductory and problem definition sections lack essential details regarding the scene observation and its transformation into the input graph. Specifically:\n    - What is the explicit definition of scene observation (e.g., is it standard RGB vision information)?\n    - How is the raw observation converted into a graph structure?\n    - What are the specific components and structure of this graph (nodes, edges, features)?\n\nThese fundamental aspects of the input representation must be clearly and explicitly defined in the main text to ensure the completeness and comprehensibility of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)