COMBO: Compositional World Models for Embodied Multi-Agent Cooperation
Authors: Hongxin Zhang
Zeyuan Wang
Qiushi Lyu
Zheyuan Zhang
Sunli Chen
Tianmin Shu
Behzad Dariush
Kwonjoon Lee
Yilun Du
Chuang Gan
Venue: ICLR 2025 Poster (2025)
Keywords: Embodied AI; Multi-agent Planning; Compositional World Model
Primary area: applications to robotics, autonomy, planning
TL;DR: We learn a compositional world model for multi-agent cooperation by factorizing the joint actions of multiple agents and compositionally generating the video.  In combination with VLMs, a tree search procedure enables online cooperative planning.

Abstract: In this paper, we investigate the problem of embodied multi-agent cooperation, where decentralized agents must cooperate given only egocentric views of the world. To effectively plan in this setting, in contrast to learning world dynamics in a single-agent scenario, we must simulate world dynamics conditioned on an arbitrary number of agents' actions given only partial egocentric visual observations of the world. To address this issue of partial observability, we first train generative models to estimate the overall world state given partial egocentric observations. To enable accurate simulation of multiple sets of actions on this world state, we then propose to learn a compositional world model for multi-agent cooperation by factorizing the naturally composable joint actions of multiple agents and compositionally generating the video conditioned on the world state. By leveraging this compositional world model, in combination with Vision Language Models to infer the actions of other agents, we can use a tree search procedure to integrate these modules and facilitate online cooperative planning. We evaluate our methods on three challenging benchmarks with 2-4 agents. The results show our compositional world model is effective and the framework enables the embodied agents to cooperate efficiently with different agents across various tasks and an arbitrary number of agents, showing the promising future of our proposed methods. More videos can be found at https://umass-embodied-agi.github.io/COMBO

Published as a conference paper at ICLR 2025
COMBO: COMPOSITIONAL WORLD MODELS FOR
EMBODIED MULTI-AGENT COOPERATION
Hongxin Zhang': : Zeyuan Wang²* Qiushi Lyu" Zheyuan , Chen', Tianmin Shu', Behzad Dariush", Kwonjoon Lee², Yilun Du", Chuang Gan'
University of Massachusetts Amherst IIIS Tsinghua University Peking University
Johns Hopkins University Honda Research Institute USA MIT
ABSTRACT
In this paper, we investigate the problem of embodied multi-agent cooperation, where decentralized agents must cooperate given only egocentric views of the world. To effectively plan in this setting, in contrast to learning world dynam- ics in a single-agent scenario, we must simulate world dynamies conditioned on an arbitrary number of agents' actions given only partial egocentric visual ob- servations of the world. To address this issue of partial observability, we first train generative models to estimate the overall world state given partial egocen- tric observations. To enable accurate simulation of multiple sets of actions on this world state, we then propose to learn compositional world model for multi- agent cooperation by factorizing the naturally composable joint actions mul- tiple agents and compositionally generating the video conditioned the world state. By leveraging this compositional world model, in combination with Vi- sion Language Models to infer the actions of other agents, can a tree search procedure to integrate these modules and facilitate online cooperative planning. We evaluate our methods three challenging benchmarks with 2- 4 agents. The results show our compositional world model is effective and the framework enables the embodied agents to cooperate efficiently with dif- ferent agents across various tasks and an arbitrary number of agents, showing the promising future of our proposed methods. More videos can be found https: //umas//msssssss ss embod ed agi g thub. o COMBO/.
INTRODUCTION
Building cooperative embodied agents that can engage in and help humans in tasks requiring visual planning is a valuable yet challenging endeavor. To cooperatively plan in multi-agen scenario, contrast to learning world dynamics in a single-agent scenario, there are additional challenges to simulate world dynamics conditioned on joint actions and partial observations of the world.
Large generative models have brought remarkable advances to various domains, including language understanding and generation (OpenAI, 2023), image understanding and generation (Liu et al., 2023a; Ho et al., 2020), and video generation (Blattmann et al., 2023a). Many have explored how to leverage these powerful foundation models for embodied AI, Ahn al. (2022); Wang et al. Zhang et al. (2023) incorporates
(2023a) leverage Large Language Models for decision-makee LLMs to build modular embodied agents for cooperation and communication, Driess et al. (2023) uses Vision Language Models to build capable vision agents, Du et al. (2023b); Yang et al. (2024) Video Models generate visual plans for robots and explore modeling world dynamics with video models to improve single-agent planning. However, how to leverage Vision Language Models and Video Models to build embodied agents capable of planning under visual cooperation task is under-explored where it's important to accurately simulate the world dynamics conditioned on an arbitrary number of agents' actions given only partial egocentric observations at each step for efficient cooperation.
We propose to learn a compositional world model for multi-agent cooperation by leveraging the nat- ural compositionality of joint actions to generate future frames conditioned on the current world state
denotes equal contribution.

---

Published as a conference paper at ICLR 2025
What may Bob do?
Epocentric RG30 Observation
hamburger patty piate
may.
infer other agents' intents
Whatshould
TDW-Cook
can pick up bread
may.
possible actons
world t-1
cam
simulate how the world may be affected by multple sets actions
TDW-Game (a)
(b)
Figure 1: (a) Two challenging embodied multi-agent visual cooperation benchmarks TDW-Cook TDW- Game, where 2-4 agents cooperate to finish dishes according to recipe or finish puzzles according to visual clue. (b) The agent needs to infer other agents' intents, propose possible actions, and accurately simulate how the world may be affected by multiple sets of actions to make efficient cooperation in the long run.
compositionally. which enables accurate simulation of multiple sets of actions the world state. To address the partial observability issue, we leverage a learned generative model to estimate the over- all world state given partial egocentric observations. By leveraging the compositional world model, in combination with Vision Language Models, we propose COMBO, a novel Compositional wOrld Model-based emBOdied multi multi-agent planning framework facilitate online cooperative planning. COMBO first estimates the overall world state from partial egocentric observations, then utilizes Vision Language Models to infer other agents' intents, and an Outcome Evaluator to evaluate the different possible outcomes. In combination with the compositional world model simulate the effect of joint actions on world state, COMBO uses search procedure integrate these modules and empowers embod- ied agents to imagine how different actions may affect the world with other agents in the long run and plan more cooperatively.
an Action Proposer propose possible actions, an Intent Tracker
We instantiate the challenging embodied multi-agent visual cooperation task in ThreeDWorld (Gan al., 2021), and build TDW-Game and TDW-Cook where 2-4 decentralized agents must cooper- ate to finish several puzzles according to the visual clue or dishes according to the recipe given only partial egocentric views of the world shown in Figure 1, requiring extensive visual cooperation. The agents need to estimate the overall world state given partial egocentric observations, infer other agents' intents, and accurately simulate how the world state may be affected by multiple sets of actions to make efficient cooperation in the long run. Our extensive experiments on these two chal- lenging benchmarks and another adapted cooperation task show our learned compositional world model delivers accurate video synthesis, conditioned on multiple sets of actions from an arbitrary number agents and COMBO enables the embodied agents to cooperate efficiently with different agents across various tasks and an arbitrary number of agents. In sum, our contribution includes:
We propose to learn compositional world model for multi-agent cooperation by factoriz- ing joint actions of an arbitrary number of agents and compositionally generating the video to enable accurate simulation of multiple sets actions on the world state.
We introduce COMBO, Compositional wOrld Model-based emBOdied multi-agent plan- ning framework to empower the agents to imagine how different actions may affect the world with other agents the long run and plan more cooperatively.
We evaluate our methods three challenging benchmarks and conduct ablation studies, the results show our framework enables the embodied agents to cooperate efficiently with different agents across various tasks and arbitrary number of agents.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
2.1 MULTI-AGENT PLANNING
Multi-agent planning has long-standing history (Stone & Veloso, 2000), with various tasks and methods have been introduced (Lowe et al., 2017; Samvelyan et al., 2019; Carroll et al., 2019;
Suarez et al., 2019; Jaderberg et al., 2019; Amato al., 2019; Baker et al., 2020; Bard et al., 2020; Jain et al., 2020; Wen et al., 2022; Szot et al., 2023). For embodied intelligence, Puig et al. (2021) explored the social perception of the agents during household tasks, Zhang et (2023) studied the communication and cooperation of two agents in a multi-room house. However, these works didn't t dig into the explicit challenge of modeling the world dynamics conditioned on an arbitrary number of agents' actions given partial egocentric observations of the world, which is essential for the agents to cooperate efficiently in long-horizon cooperation-extee -extensvv visual tasks. contrast, we learn a compositional world model for multi-agent cooperation and propose compositional world model- based embodied multi-agent planning framework to empower the embodied agents to imagine and plan more cooperatively with only partial egocentric observations.
2.2 LARGE GENERATIVE MODELS FOR EMBODIED Al
With the recent advance of large generative models (Bubeck et al., 2023; Liu et al., 2023a; Driess et al., 2023; Blattmann et al., 2023b), numerous works have explored how they can help build better agents (Wang et al., 2023b; Xi et al., 2023; Sumers et al., 2023), especially in embodied environ- ments (Zhou et al., 2024a; Li et al., 2023; Padmakumar et al., 2022; Kolve et al., 2017; Misra et al., 2018; Xia et al., 2018; Xiang et al., 2020). Specifically, Wang et al. (2023c); Ahn al. (2022); Sharma et al. (2021); Wang et al. (2023a); Park et al. (2023) leverage Large Language Models to help decision-making. Brohan al. (2023); Jiang et al. (2023); Wang al. (2023d) vision lan- guage models to facilitate embodied agents to plan end-to-end in visual worlds. Hong et al. (2024); Black et al. (2024) use diffusion models for decision-making. Xiang et al. (2024); Du et al. (2024); Finn et al. (2016) use video models to help the robot make visual plans. There are two closely related works. VI. (Du et al., 2023b) enables single agent to perform complex long-horizon ma- nipulation tasks by synergizing vision-languag but they neglect the challenge of learning world dynamics conditioned on multiple sets of actions of arbitrary number of agents. RoboDreamer (Zhou et al., 2024b) decomposes language instruc- tions into sets of lower-level primitives and compositionally synthesizes video plans on unseen goals for robot manipulation tasks, however, they only focus the low-level trajectory control, and sume the observation is static and non non-partial. In contrast, we learn a compositional world model for multi-agent cooperation enable accurate simulation of multiple sets of actions the world and leverage a learned generative model to estimate the overall world state given partial egocentric observations address the partial observability issue.
models and text-to-video models with tree search,
PRELIMINARIES
3.1 PROBLEM STATEMENT
Embodied multi-agent cooperation problem can be formalized as decentralized partially observ- able Markov decision process (DEC-POMDP) (Oliehoek et al., 2016; Bernstein et al., 2002; Spaan et al., 2006), defined by (n, S, {A.), {O.),, T, G, h), where:
n denotes the number of agents;
S is a finite of states;
A; is the action set for agent é;
O; is the observation set for agent i, including partial egocentric visual observation the agent receives through its sensors;
T(s, a, s') = p( (s, a) is the joint transition model which defines the probability that after taking joint action A = A x A, in S, the state s' E S is achieved;
G is the final goal state; h is the planning horizon.
3

---

Published as a conference paper at ICLR 2025
Starting from initial state Su S, decentralized agents need act d, A given egocentric RGBD observations or E O, each step to achieve the goal state using a minimal number of steps.
3.2 VIDEO DIFFUSION MODELS
Given an initial image Tg and text prompt tat as the condition, a video diffusion model learns to model the distribution of possible future frames 21...T, denote as Po (xt).. denoising function is trained to predict the noise applied to æ1..7 at diffusion time step t given the noisy frames by optimizing the objective of
MSS [ T, Mro, txt) c||
where e is sampled from a standard Gaussian distribution, and is a randomly sampled diffusion time step, following Ko et al. (2023).
3.3 COMPOSABLE DIFFUSION MODELS
Liu et al. (2022): Du et al. (2023a) shows that diffusion models functionally similar to Emergy-Basee Models, and can be composed to generate images conditioned on a set of concepts {cr, -2.-.. C.) by training diffusion models to learn set of score functions co(22, t/c.), and com- posing them as
E(z,, co)) c0(xe,t) + Σ co(o,, co(
Which corresponds to the production of the probability densities
(o(a))
Pa(x|c1, c2, Cn) Ps(z) II
(3)
Pa(x)
Then the sampling process changes to xt-1 ~ Niz - E(2,, t/o, ... ,.)) of
4 COMPOSITIONAL WORLD MODEL
To cooperatively plan in multi-agent scenario, in contrast learning world dynamics in single- agent scenario, there is an additional challenge to simulate world dynamics conditioned on the ac- tions of arbitrary number of agents. We propose to learn compositional world model for multi- agent cooperation as shown in Figure 2 by factorizing the naturally composable joint actions of an arbitrary number of agents as a of text prompts and predicting the future state by compositionally generating the future frames with a video diffusion model to enable accurate simulation of multiple sets of actions on the world state. Our compositional world model aims to model (s'' [s, a), where represents the current world state, represents the joint action of n agents, and s is the future world states. We first discuss how we learn the composable video diffusion model to act as a com- positional world model in 4.1, then introduce the Agent- Dependent Loss Scaling designed for the challenging situation where we need to accurately model multiple agents' manipulating multiple objects improve the video synthesis performance in 4.2.
4.1 COMPOSABLE VIDEO DIFFUSION MODELS
Our compositional world model can be learned as video diffusion model by treating the current world state as the initial frame To, and the joint actions the text prompt, then modeling the distribution future frames is learning the world dynamics and predicting future states s', The text prompt with multiple agents interacting with multiple objects is challenging to be handled ac- curately. Observing that the joint action a can be naturally factorized into n components a1 corresponding to the action of each agent, we leverage the composable diffusion models introduced in section 3.3 to learn the compositional world model as a composition of video diffusion models conditioned on the initial frame 20 and each text component a;
4

---

Published as a conference paper at ICLR 2025
Alice plik up green-brown piece up the blue brown piece
Charle wait
equa
Current State
oint Action
Compositional World Model
plök grean brownn piace
Ca fik,
Mus the
Compose
Charlie
David pick squa-black
(a %))
Future States
Figure 2: Compositional World Model. Given the current world state xu and joint action of multiple agents the compositional world model predicts the future states by first factorizing into several components corresponding each agent, then generating multiple conditioned the current world state and the components, finally composing them to generate the video.
Po(X o.)
P( Pa(X a) Po ,a,) Px( II
Px(X)
Using Equation 2, the video diffusion models learn a set of score functions EØ e0 1/ a:) and
composes the joint score
([Xe, a) cx( + Σ
(X(,,tt
(5)
We train this composable video diffusion model with two stages. In stage one, we learn to model the distribution of (x(Xazo, a;) by training with text component corresponding to single agent action only using the standard denoising diffusion training objective introduced in section 3.2.
Then in stage two, fine-tune the model to specifically learn compositional generation to model Po(X| a) by training with conditions containing joint actions of multiple agents using the com- posed score function loss
LComposed |E(Xi, a) -cl2
Σ ca ( a.)
After the two-stage training, we can sample from the composed distribution with the composed score function at inference time given current world state zo and joint action -aa1,". -.. (n)) as
é(X,, t/ a) ex
xx,t) t) Σ w(eo f/ro, a.) e (X t))
where w is the guidance weight controlling the temperature scaling on the conditions.
4.2 AGENT-DEPENDENT LOSs SCALING
A well-trained stage model is vital to the final compositional generation performance since the accurate modeling of Pa (X xp a,) is the basis for modeling Px(X[2o, a). We have employed technique named Agent-Dependent Loss Scaling to assist in stage training.
5

---

Published as a conference paper at ICLR 2025
World State Estimation
Environment
Other
Reconstracted
Simulator
Fartial
Egocantric RGBD Observation
place the black bread onto the plate
Rendered Pantial
place the black bread onia the cutting board
Outcome Evaluator
Diffusian Model
Action Proposer
Compositional World Model
bread
the plate
Inpainted
Intent Tracker
World State
llbb may place the burger bottom onto the plate
(a)
(b)
Figure 3: Method Overview, (a) Given partial egocentric RGBD observations. COMBO first reconstructs and inpaints top-down orthographic image as overall world state estimation. (b) COMBO then leverage the planning sub-modules built with Vision Language Models propose actions, infer other agents' intents, and evaluate the outcomes simulated with the compositional world model to plan online with a tree search procedure to cooperate in the long run.
Formally, we define an agent-dependent loss coefficient matrix C RaxHxW for agents and images of H x W pixel, and change the Equation to
LMSE Σ C.
||co((X - c|2
We simply set the loss coefficient matrix based on each agent's reachable region in the image to su- pervise the model to focus more on the related pixels. observed that even this straightforward loss coefficient approach brings a significant improvement to the modeling accuracy of Po( (zo, az).
COMPOSITIONAL WORLD MODEL FOR MULTI-AGENT PLANNING
To plan efficiently in the embodied multi-agent visual cooperation problem, still need address the challenge of partial egocentric observation and model complex world dynamics in the long run. We propose COMBO, novel Compositional wOrld Model- based emBOdied multi-agent planning framework, shown in Figure 3. After receiving the egocentric observations, COMBO first estimates overall world state to plan on, as discussed in 5.1. COMBO then utilizes Vision Language Models to act as an Action Proposer to propose possible actions, an Intent Tracker to infer other agents' intents, and an Outcome Evaluator to evaluate the different possible outcomes, as detailed in 5.2. In combination with the compositional world model to simulate the effect of joint actions on the world state $;+1 CW (ss, a) introduced in 4, we discuss the tree search procedure to integrate these planning sub-module in 5.3. COMBO empowers embodied agents imagine how different plans may affect the world with other agents the long run and plan cooperatively.
5.1 WORLD STATE ESTIMATION WITH PARTIAL EGOCENTRIC VIEWS
Directly planning based on partial egocentric views presents a considerable challenge. To address this, we initially reconstruct partial point clouds from multiple egocentric RGBD views captured from different perspectives. These point clouds are then transformed into unified top-down or- thographic image representation, serving as the world state. This representation is constructed by overlaying the views in chronological order. dynamic, with other agents actively interacting, resulting in top-down orthographic image repre- sentation that is both noisy and incomplete, as depicted (a). To refine this representation, we employ diffusion model inpaint the partial and noisy orthographic image, thereby enhancing
important to note that our world is inherently

---

Published as a conference paper at ICLR 2025
the estimation of the world state s; for subsequent planning. This approach allows us to effectively represent and rectify the world state, enabling more accurate planning in multi-agent environments.
5.2 PLANNING SUB- MODULES WITH VISION LANGUAGE MODELS
Action Proposer Given the estimated world state 8; and the long horizon goal G, the Action Pro- poser first searches over the potential action spaces A,, and then proposes multiple possible actions di.1..pp = (8
e G) to explore on. We implement this module by querying a VLM to generate the possible actions the text given the task goal and encoded image world state as context. We finetune LLaVA on randomly collected rollouts with possible actions labeled to construct this module.
Intent Tracker Inferring what other agents may do given observation history is important for effec- tive multi-agent cooperation. We implement this module a IT(s;, G) by keeping the estimated world state from the last steps and then feed into the VLM together with the task goal to query for the possible actions of other agents a.;. We construct this module by finetuning LLa VA on collected short rollouts with other agents' actions labeled.
Outcome Evaluator It's vital to have way to assess the value of the achieved state from differ- ent plans so the search can be effectively deepened utilizing pruning. We implement an Outcome Evaluator to fulfiil this functionality = OE G)) by generating hcuristic score for each image state s given the task goal G. To construct this module, we finetune LLaVA to describe the state of each object in the image and the corresponding heuristic score representing steps left to achieve the task goal considering all objects.
Algorithm 1 COMBO Planning Procedure for Agent i.
1: Input: Estimated world state so from o,, task goal G
2: Sub-modules: Action Proposer AP,s, G), Intent Tracker [T(s, G), Compositional World Model
CW (WM(s,a) a), Outcome Evaluator E(ss G)
3: Parameters: Action Proposes P, Planning Beams B, Rollout Depths D
4: plans <- [[0o]]
5: new www-plans/ + [[o]]
6: for d 1.....D
7: plans + new ptw.nnnppllee I...s[I...B]
Keeps Only B Plan Beams with Best Scores
8: new-plans +
9:
for plan in plans do
10:
8 [1a
Get the Last Image State in the Plan Beam
11:
G,,1:P < AP(s, G)
# Generate P Different Action Proposals
12:
a-: + (T
# Infer Other Agents' Possible Actions
13:
for I... do
14:
a (ei.p. ai:)
Saeu < CW M(s a)+ Simulate Next State Conditioned on Joint Actions new www.plans.appendeee
16:
+ Suct)
17:
end for end for
18:
19:
new new_plans sorted(new sereddnew.plans, OE(s, G)) Sort Plans by the Score of the Final State
end for
21: plan +- new_plans|]
Return the Plan with the Best Score
5.3 PLANNING PROCEDURE WITH TREE SEARCH
With powerful generative sub-modules implemented so far, we can use an effective tree search algo- rithm to integrate them into achieving the best planning performance. Formally, we need to search for a sequence of actions that compose the plan achieve the task goal with a minimal number of steps. To sample the long-horizon plans, we can chain the sub-modules to expand future states from current state 8; with 8:41 = CWM(ss AP(s;, G), IT (st, G)). By recursively deploying these expand the plans until achieving the goal state. However this is impractical
chained modules due to the large search space, we implement a limited tree search procedure instead by always keep- ing the B best-scored plan beams and exploring P action proposals at each plan step for maximum of rollout steps. We show the complete COMBO planning procedure in Algorithm 1.
Due to the uncertainty of other agents, COMBO replans at every step reflect the changing world state. As similarly observed in (Du et al., 2023b), when searching for the best plan, the Outcome Evaluator may leverage the irregular artifacts of the Compositional World Model to get artificially

---

Published as a conference paper at ICLR 2025
TDW-Game
TDW-Cook
Cooperator 1
Cooperator 2
Cooperator
Cooperator 2
Recurrent World Models
0.00/ /-
0.00/-
0.00/-
0.00 /-
MAPPO
0.00
0.00/-
0.00
0.00
CoELA
0,90/ / 18.4
0.60
0.15/
0,05 / 29.5
LLaVA
0.55 28.4
0.60 29.2
0.35/
0.50/ 35.0
COMBO (w/o IT)
0.65 16.2
17.0
0.80 24.8
0,80 23.8
COMBO (Ours)
1.00 17.5
1.00/ 17.4
0.90 22.8
1.00 22.9
Shared Belief Cooperator*
1.00 / 15.3
1.00 15.9
0.95 24.0
1.00 /014
Table I: Main results on TDW-Game and TDW-Cook. report mean success rate over a horizon of 30 and the average steps of the successful episodes over 20 episodes here. COMBO (w/o IT') denotes COMBO without the Intent Tracker module. Shared Belief Cooperator has access to the Oracle state of the world and other agents" policies. denotes Oracle vision perception.
2D-FetchQ.* * BC-single
BC-GAIL Co-GAIL COMBO (Ours)
Success Rate
21.1
27.2
53.3
81.3
Table Results on 2D-FetchQ.. We report the success rate on replay evaluation over 60 episodes.
high scores, such as where key objects are teleported to desired positions. To mitigate this artificial exploitation issue, the Outcome Evaluator will use the default score from the last state the new state generated from the Compositional World Model suspiciously increases the score above a threshold.
6 EXPERIMENTS
6.1 EXPERIMENTAL SETUP
We instantiate the challenging embodied multi-agent visual cooperation task in ThreeDWorld and build two challenging benchmarks: TDW-Game and TDW-Cook where 2-4 decentralized agents must cooperate to finish several puzzles according to the visual clue or dishes according to the recipe given only egocentric views of the world as shown in Figure requiring extensive visual cooperation. All the agents have an observation space of egocentric 336 x 336 RGBD images and the corresponding camera matrix. We also adapt the 2D-FetchQ challenge from Wang et al. (2022) with visual observation space and high-level action space to evaluate our method. In this challenge, two agents must coordinate their strategy to hold buttons to unlock rooms and fetch treasures from the unlocked room. More details on the tasks can be found in Appendix A. We also verify our method in a real-world human-rohot cooperation task in Appendix
Metrics We evaluate the agent's cooperation ability by task success rate and average steps of the successful episodes when cooperating with different agents across 20 episodes. An episode is done when the goal state is reached and considered successful, or the maximum planning horizon h is reached and deemed failed. Specifically, we implement two different agent policies as cooperators on both tasks reliably evaluate the agents, which are elaborated in Appendix A.1.I.
Baselines We compare our methods to several multi-agent planning methods:
Recurrent World Models (Ha & Schmidhuber, 2018), using VAE to encode the visual observations and a generative recurrent neural network learn the world model. An evolution-trained simple policy is used as controller.
MAPPO (Yu al., 2022), MARL baseline where agents with shared weight Actor and Critic are trained jointly with PPO.
CoELA (Zhang et al., 2023), an LLM agent with access to the Oracle vision perception, re- implemented due to the lack of ground truth segmentation needed to train the Perception Module.
LLaVA (Liu et al., 2023a), an end-to-end VLM (LLaVA v1.5 7B) agent fine-tuned with the same collected rollouts to train the Action Proposer in method.
Shared Belief Cooperator, a planner that shares the same policy as the cooperators and has access to the Oracle state of the environment and other agents' policies, acting strong baseline.
Implementation details The video diffusion model of the compositional world model is built upon AVDC (Ko et al., 2023) codebase, with the architectural modification of introducing a cross attention layer to the text condition in the ResNet block and replacing the Perceiver with MLP to enhance the text conditioning. More details are in Appendix B.
8

---

Published as a conference paper at ICLR 2025
Current State
Action
Current
Joint Action
Alice pick the brown piece
Alice pick up the lettuce
Bob pick the blue-brown piece
Bob pick up the pickle slice
Charlie walt
David pick up aqua black piece
Predicted
Predicted Frame
Predicted
Prodicted Predicted Predicted
AVDC
COMBO
W.o ADLS
COMBO
Figure 4: Compositional World Model learns world dynamics better. compositional world model can simulate world dynamics conditioned on the joint action multiple agents accurately while AVDC struggles with simulating which agents should act, and COMBO w.o ADIS may simulate actions incorrectly.
TDW-Game Multiple
TDW-Cook
Single
Plan
Single
Multiple
Plan
AVDC COMBO w.O ADLS
65%
20%
29.7(80%)
85%
25%
34.5(90%)
70%
55%
26.9(100%)
85%
70%
28.3(100%)
COMBO
95%
75 %
17.5(100%)
100%
100%
21.5(100%)
Table 3: Human evaluation generated videos from different World Models and the corresponding plan performance. Single denotes the accuracy of synthesized videos conditioned on a single agent's action among 20 samples. Multiple denotes the accuracy of synthesized videos conditioned multiple agents' actions among 20 samples. Plan denotes the average steps (success rate) across cooperating with two different cooperators over 10 episodes. Best performances are bold.
6.2 RESULTS
COMBO can cooperate efficiently with different cooperators As shown in Table 1, both operators of different policies achieve the best performance when cooperating with COMBO on TDW-Game and TDW-Cook, finishing all the tasks within the limited horizon with the least steps. Both Recurrent World Models and MAPPO perform poorly due to the difficulty of the multi-agent long-horizon task with only egocentric observations. Compared to LLM Agent CoELA and VLM Agent LLaVA, COMBO with a compositional world model to explicitly model the world dynam- ies and plan accordingly achieves more efficient cooperation. On adapted 2D-FetchQ, COMBO surpasses other behavior cloning or Co-Gail baselines in more challenging setting as shown in Table demonstrating the efficacy and adaptability of our proposed method.
Intent Tracker contributes to efficient cooperation Comparing the results of COMBO (w.o ITT) and COMBO in Table we can see the Intent Tracker important to empower the agent to infer other agents' intents and consider them during planning for better cooperation in the long run. A qualita- tive case demonstrating the adaptability the Intent Tracker Module is shown Appendix C.
Compositional World Model is crucial to make cooperative plans We compared the generated video quality with AVDC (Ko et al., 2023), a video diffusion model designed for robotics tasks, and ablate the effect of the Agent-Dependen Scaling Loss (COMBO and Figure 4. Compared to AVDC, our CWM can accurately simulate which agents are acting and what interactions described in the text condition, leading to large performance gain in the
ADSL) in Table

---

Published as a conference paper at ICLR 2025
Action
Rollout
Plan
Success
Average Steps
4-agent
3-agent
2-agent
Proposals P Depth Beam
Rate
LLaVA
0.58 28.7 0.65 / 27.4 0.80 20.7
3
1
50%
27.8
COMBO
1.00 / 17.5 1.00/ 15.0 1.00/ 10.5
2
2
70%
18.1
100%
17.5
Shared Belief* 1.00 15.9 1.00 / 13.5 1.00/ 8.7
Table 4: Plan performance improves with more compute budgets. We report the mean success rate and average steps of the successful episodes cooperating with two different cooper- ators over episodes on TDW-Game here.
Table 5: Results
with different number agents. report the mean success rate over horizon of and average steps cooperating with two different co- operators over 10 episodes here.
Action Proposals, Rollout Depth,
Plan Beams
Step
Step
Step
Step
Step
Action Proposals, Rollout Depth, Plan Beams
Step
Figure 5: More Computation budgets leads to better plan, With more computation budgets (sccond row), COMBO search for better plan where Alice first clears the common region with David that he can pass the next puzzle piece instead of having to wait, leading to a better state after same number of steps.
accuracy of modeling agent actions on the world state. Removing the peenttDependent Scaling Loss leads considerable accuracy degradation, showing the effectiveness of the introduced technique. Moreover, with the compositional world model, the overall plan performance boosts to 17.5 steps for TDW-Game compared to 29.7 steps with AVDC. See C.3 for additional failure case analysis.
More computation budgets lead to better Plan We study the effect of computation budgets on the long-horizon plan quality in Table 4, where we can see more computation budgets with more action proposals, larger plan beams, and deeper rollout depth leads to higher success rate and less average steps. A qualitative example generated plans with different computation budgets is shown in Figure 5, where more searches can obtain better plans in the long
COMBO can generalize to an arbitrary number agents COMBO trained with only data of four agents playing TDW-Game can surprisingly generalize to three and two agents version well as shown Table 5, showing the strong generalization and promising future of the compositional world model-based modularized planning framework for multi-agent cooperation.
7 CONCLUSION
In this work, we learn a compositional world model for embodied multi-agent cooperation by fac- torizing the naturally composable joint actions of arbitrary number of agents and compositionally generating the video to enable accurate simulation of multi-agent world dynamics. We then propose COMBO, novel Compositional World Model-basee embodied multi-agent planning framework to empower the agents to infer other agents' intents and imagine different plan outcomes in the long run. Our experiments on three challenging embodied multi-agent cooperation tasks show COMBO can enable the embodied agents cooperate efficiently with different agents across various tasks and an arbitrary number of agents.
Future Work Our method leverages tree search combined with Large Generative Models to devise Long-horizon plans. The necessity for multiple inferences using large models leads to relatively slow inference speed, restricting the applicability of method in scenarios demanding rapid sponse. Exploring the development of more efficient models could potentially mitigate this draw- back and enhance the practicality of our approach in time-sensitive environments.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Jeremy Schwartz and Esther Alter for setting up the ThreeDWorld environments, and Chunru Lin for the help with real-world experiments. This project is supported by the Honda Re- search Institute.

---

Published as a conference paper at ICLR 2025
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, al. Do i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0100
2022. 1,3
Christopher Amato, George Konidaris, Leslie P Kaelbling, and Jonathan P How. Modeling and planning with macro-actions in decentralized pomdps. Journal of Artificial Intelligence Research, 64:817-859, 2019.
Bowen Baker, Ingmar Kanitscheider. Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In International Confer- ence Learning Representations, 2020. URL hwwph:tpps.ooo...... http:httss//ppeeeeeeeeee /owp/rppperveeeeeeeeeeeeeeeeeeee ew. net. forum?id d== SkxpxJBKwS. 3
Nolan Bard. Jakob N Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin. Subhodeep Moitra, Edward Hughes, al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280:103216, 2020. 3
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4): 819-840, 2002. 3
Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Rich Walke, Chelsea Finn, Aviral Ku- mar, and Sergey Levine. Zero-shot robotic manipulation with pre-trained image-editing diffu- sion models. In The Twelfth International Conference on Learning Representations, 2024. URL
https: /tpps///ppereve/e wwe.ewt./ee.eee..eeeee....e...ee wtt.eeewweeeeeettiiiiiie ttt.//ommmmmmdmuumeaarrr ddc.mii.sccccccc........
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhom, Seung Wook Kim, Sanja Fidler, and Karsten Kreis, Align your latents: High-resolution video synthesis with latent diffusion mod- els. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22563-22575. 2023b.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-actionn models transfer web knowledge to robotic control. arXiv preprint wrXiv:2307.158/1 2023. 3
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023. 3
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers), pp. 4171 11866, Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: (0.18633/v1/N19. 1423. URL https: https///aaaaaaaooaaa.ooooo org/
N1 M119-4423 1423. 18
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth. Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv.2303.033778 2023. 1,3
12

---

Published as a conference paper at ICLR 2025
Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and memc. In International con- ference on machine learning, pp. 8489-8510. PMLR, 2023a. 4
Yilun Du, Mengjiao Yang. Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B Tenenbaum, et al. Video language planning. arXiv preprint arXiv:2310. 10625, 2023b. 1, 7,7
Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, 29, 2016. 3
Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Ju- lian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Michael Lingelbach, Aidan Curtis, Kevin Tyler Feigelis, Daniel Bear, Dan Gutfre- und, David Daniel Cox, Antonio Torralba, James DiCarlo, Joshua B. Tenenbaum, Josh Me- dermott, and Daniel LK Yamins. ThreeDWorld: A platform for interactive multi-modal phys- ical simulation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round I), 2021. URL https: https:://ppeeeeeeeeeeee /tpp/ppppevveeeeeeeeeeeeeeeeeeee net fant/forunne forum? ?id= db1 InHAWW2T. 2,17
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018. 8, 18
Nikolaus Hansen. The ema evolution strategy: A tutorial. arXiv preprint arXiv:1604.0077) 2016. 18
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Mineui Hong, Minjae Kang, and Songhwai Oh. Diffused task- milestone planner. Advances in Neural Information Processing Systems, 36, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1xXiv:2106.09655. 2021. 20
Max Jaderberg, Wojeiech M Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari Morcos, Avraham Ruderman, et al. Human- level performance in 3d multiplayer games with population-base reinforcement learning. Sci- ence, 364((443))))889-8655 2019. 3, 21
Unnat Jain, Luca Weihs, Eric Kolve, Ali Farhadi, Svetlana Lazebnik, Aniruddha Kembhavi, and Alexander Schwing. A cordial sync: Going beyond marginal policies for multi-agent embodied tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pp. 471-490. Springer, 2020.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. In Fortieth International Conference on Machine Learning. 2023.
Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B. Tenenbaum. Learning to act from actionless videos through dense correspondences, 2023. 4, 8,9
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon. Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint 17Xiv:1712.05474, 2017.
13

---

Published as a conference paper at ICLR 2025
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior- 1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pp. 80-93. PMLR, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304. 2304-08884, 2023a. 1, 8, 20
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference Computer Vision, pp- 423-439. Springer, 2022. 4
Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, and Shenlong Wang. Physgen: Rigid-body physics- grounded image-to- video generation. In European Conference on Computer Vision (ECCV), 2024. 22
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint SeKiv:2303.05499 2023b. 20
Ryan Lowe, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-cometiiii environments. Advances in neural information process- ing systems, 30, 2017. 3
Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi Mapping instructions actions in 3d environments with visual goal prediction. arXiv preprint 1rXiv:1809.00700 2018. 3
Frans A Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
OpenAl. Gpt-4 technical report, 2023. 1
Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. Teach: Task-driven embodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2017-2025, 2022.
Joon Sung Park, Joseph C O'Brien Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442 2023. 3
Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang. Yuan-Hong Liao, Joshua B Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-help: challenge for social perception and human-ai collaboration. In International Conference on Learning Representations, 2021.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma. Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Fe- ichtenhofer. Sam 2: Segment anything in images and vidcos, 2024. URL attps:///axxs org/abs/ /408. 00714. 20
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt. Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon White- son. The starcraft multi-agent challenge. Proceddiggs of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2186-2188, 2019.
Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language. arXiv preprint arXiv: 10, 2021.
Matthijs TJ Spaan, Geoffrey J Gordon, and Nikos Vlassis. Decentralized planning under uncertainty for teams of communicating agents. In Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems, pp. 256- 2006. 3
14

---

Published as a conference paper at ICLR 2025
Peter Stone and Manuela Veloso. Multiagent systems: A survey from machine learning perspec- tive. Autonomous Robots, 8:345-383, 2000. 3
Joseph Suarez, Yilun Du, Phillip Isola, and Igor Mordatch. Neural mmo: A massively mul- tiagent game environment for training and evaluating intelligent agents. arXiv preprint arXiv:1903.00784, 2019. 3
Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas Griffiths. Cognitive architec- tures for language agents. arXiv preprint arXiv:2309.02427, 2023. 3
Andrew Szot, Unnat Jain, Dhruv Batra, Zsolt Kira, Ruta Desai, and Akshara Rai. Adaptive coordi- nation in social embodied rearrangement. In International Conference on Machine Learning, pp.
33365-33380. PMLR, 2023. 3, 21
Chen Wang, Claudia Pérez-D' Arpino, Danfei Xu, Li Fei-Fei, Karen Liu, and Silvio Savarese. Co- gail: Learning diverse strategies for human-robot collaboration. In Conference on Robot Learn ing, pp. 1279-1290. PMLR, 2022. 8, 18
Guanzhi Wang. Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprini arXiv:2305.
2023a. 1,3
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.1 11432, 2023b. 3
Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: interactive planning with Iims enables open-world multi-task agents. In seirth- Conference on Neural Information Processing Systems, 2023c. 3
Zihao Wang. Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. JARVIS-I: Open-world multi- task agents with memory-augmenter multimodal language models. In Second Agent Learn- ing in Open-Eneenn Endedness Workshop, 2023d. URL https: https::///pppeeieeeeeee net forum?idd xzPk 2.PkzyHlOw.
Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is a sequence modeling problem. Advances in Neural Infor- mation Processing Systems, 35: 15509-16551, 20222,2,,,.2.. 3
Sarah A Wu, Rose E Wang, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Kleiman Weiner. Too many cooks: Bayesian inference for coordinating multi-agent collabora- tion. Topics in Cognitive Science, 13(2):414 2021. 21
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et The rise and potential of large language model based agents: A survey, arXiv preprint arXiv:2309.074: 077644 2023. 3
Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9068-9079, 2018. 3
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanx- iao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11097-11107, 2020. 3
Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, et al. Pandora: Towards general world model with natural language actions and video states. arXiv preprint arXiv:2406.0940 09455, 2024.
Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kaelbling, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In The Twelfth International Conference on Learning Representations, 2024. URL https:
/ //openrevies ew. net. fert/formm? dereidddesSyyyyyyssr
15

---

Published as a conference paper at ICLR 2025
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances Neural Information Processing Systems, 112244111242222 2022. 8, 17
Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tian- min Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models, 2023. 3,
Tianyuan Zhang. Hong-Xing Yu, Rundi Wu, Brandon Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William T. Freeman. PhysDreamer: Physics-based interaction with 3d objects via video generation. European Conference on Computer Vision. Springer, 2024. 22
Qinhong Zhou, Sunli Chen, Yisong Wang, Haozhe Xu, Weihua Du, Hongxin Zhang, Yilun Du, Joshua B. Tenenbaum, and Chuang Gan. Hazard challenge: Embodied decision making in dy- namically changing environments, 2024a.
Siyuan Zhou, Yilun Du, Jiaben Chen, YANDONG LI, Dit-Yan Yeung, and Chuang Gan. Robo- dreamer: Learning compositional world models for robot imagination. In Forty-first International Conference on Machine Learning, 2024b. URL https: /wwpereeeevvieeeeeeeoeee ew net / forum? ?id= kH joAAffVe.
16

---

Published as a conference paper at ICLR 2025
A ADDITIONAL EXPERIMENT DETAILS
A.1
TDW TASkS
burger
burger bottom
brown bread slice
black bread slice
top
pickle
lettuce
patty
whole onion
whole tomato
whole cheese
onion
tomato slice
cheese
slice
slice
TDW-Cook
TDW-Game
Figure 6: Objects TDW-Cook and TDW-Game. Left: food items that may occur on table while 10 of them (except for the third row) may occur in the recipe. Right: 4 puzzles containing to pieces each with different colors and shapes.
We instantiate the challenging embodied multi-agent visual cooperation task in ThreeDWorld (Gan et al., 2021), 1), and build two challenging benchmarks: TDW-Game and TDW-Cook where 2-4 de- centralized agents must cooperate to finish several puzzles according to the visual clue or dishes according to the recipe given only partial egocentric views of the world.
In TDW-Game, 3-4 agents cooperate pass and place 6-8 puzzle pieces scattered randomly on the table into the correct puzzle box according to visual clues such as the shape. The location of the puzzle box and pieces randomly initialized across different episodes. The action space includes wait, pick up [obj], place [obj] onto [loc].
In TDW-Cook, 2 agents cooperate pass, cut, and place 6-8 out of possible food items randomly scattered on the table to make some dishes according to the recipe. The location the food items and the recipe are randomly initialized across different episodes. Specifically, there only one cutting board where agents can pass or cut objects, making cooperation vital for efficient plans. The action space is the same TDW-Game with an addition of [obj].
The object assets of the two tasks are illustrated in Figure 6.
A.1.1 COOPERATOR POLICIES
In TDW-Game, Cooperator takes the policy of always passing the unwanted puzzle pieces in clockwise manner, while Cooperator 2 takes the policy of always passing the unwanted puzzle pieces a counter-clockwise manner.
In TDW-Cook, Cooperator takes "selfish" policy of always operating objects in its recipe first, while Cooperator takes an "altruistic" policy of always prioritizing operating objects in the cooperator's recipe.
A.1.2 BASELINES
MАPPO Our task is multi-agent reinforcement learning with decentralized observation and trol, as different agents have disjoint observations to produce their actions. We followed the training procedure in Yu et al. (2022). Each agent is actor-critic network where the actor and critic share
17

---

Published as a conference paper at ICLR 2025
Initial Frame
pick up the green- brown piece
Action
pick up the lettuce
Generated
Frame
Figure 7: Recurrent world models failed to model world dynamics conditioned on single actions accu- rately.
common convolutional backbone. We pre-define set of all possible actions for each environment and let the PPO agent choose among them. We design rewards to reflect the steps left to finish the episode in similar spirit to the Outcome Evaluator's heuristics score design.
However, our setting, the embodied multi-agent simulation samples are relatively slow, creating a higher demand for sample efficiency of the method. We conducted a grid search of learning rate in {le-6, 5e-5, le-3}, batch size in {2. 64). However, the results are poor and the behavior either falls near-random collapses to single action, which verifies the challenge of cooperative planning on ego-centric RGBD) observations.
Recurrent World Models We followed the training procedure in Ha & Schmidhuber (2018), firstly training a variational autoencoder (VAE) on the same data collected for training COMBO to encode the egocentric observations in the TDW-Game and TDW-Cook environments into 32- dimensional latent vector E 1232, Then, we train a mixture density network combined with a recurrent neural network (MDN-RNN) for predicting the future latent vector given textual action which is encoded by BERT (Devlin et al., 2019), the current egocentric observation and the cur- rent hidden state of the RNN, which modeling P(2++1|0.- 2,, h₂). Finally, we train two simple controllers using the covariance-matrix adaptation evolution strategy (CMA-FS) (Hansen, 201 6) to maximize the expected cumulative reward of a rollout by interacting with the actual environments.
The recurrent world model failed to model world dynamics accurately conditioned on a single action, as shown in Figure 7. Specifically, we predict the latent code ž++11 from MDN-RNN and use the decoder of the VAE to generate the image of the next time step. Additionally, the simple controller doesn't work well on a task that requires complex planning as it keeps collapsing to single actions in our experiments.
A.2 2D-FETCHQ TASK
As shown in Figure 8, the 2D-FetchQ environment from Wang et al. (2022) was originally inspired by the Fetch-Quess collaborative game in SuperMario Party where two agents must coordinate their strategy to hold buttons to unlock rooms and fetch treasures from the unlocked room. At the be- ginning of the game, two treasures are locked in the rooms located two corners of the map. For each agent, the only way to fetch treasure is to let its collaborator press the button beside the room to hold the door. After the collaborator presses the button, the agent can enter the room and get the treasure. To win the game, both agents should get treasure and reach the closest destination.
18

---

Published as a conference paper at ICLR 2025
Figure 8: 2D-FetchQ Task. Two agents coordinate their strategy to hold buttons to unlock rooms and fetch treasures from the unlocked room.
Therefore, the agent who has already gotten the treasure will help the other agent to fetch its trea- sure. We adapted this environment with a visual observation space and high-level action space to evaluate our method.
Observation Space 128 128 RGB image.
Action space up, move down, move ifet, move right, wait.
Baselines BC-single refers directly applying behavior cloning learn robot trajectories, BC- GAIL involves using behavior cloning to learn human trajectories first, followed by interactive learning robot trajectories through GAIL (Generative Adversarial Imitation Learning) with the human policy. Co-GAIL leverages data of human-human collaboration demonstrations as guid- ance to concurrently generate simulated interactive behaviors and train a human-robot collaborative policy.
Evaluation
We evaluated COMBO using replay evaluation, where COMBO collaborates with human demonstration trajectories collected through replay and determines whether the task suc- ceeds.
B ADDITIONAL IMPLEMENTATION DETAILS
B. COMPOSITIONAL WORLD MODELS
Implementation Details We use the base resolution 128 x 128 for the video diffusion model and train a super-resolution diffusion model to get the final 336 336 images. We the T5- XXL encoder to pre-process all the text conditions for better contextual representation. All vision language models used the main experiments are LLaVA-v1.5-7B. The planning parameters are set to Action Proposes P=3, Planning Beams B=3, and Rollout Depths D=3 unless other specified.
Dataset Collection We collect random rollouts with scripted planner and generated 107k videos of TDW-Game and 50k videos of TDW-Cook.
Model Parameters The parameters our inpainting diffusion model, super-resolution diffusion
model, and video diffusion model are shown in Table 6.
19

---

Published as a conference paper at ICLR 2025
Inpainting
Super-Resolutee
Video Diffusion Model
num parameters
84M
38M
198M
input resolution
336 336
128 128
128 x 128
336 336
336 x 336
128 x 128
aase.channeh
64
64
128
num_res block
2 (16,)
2 (8, 16)
attention resolutions
(16.)
channel mult
(1, 2, 4, 6, 8)
(1, 2, 3, 4, 5)
(1, 2, 4, 5)
Table 6: Model Parameters.
Agent-dependent Loss Scale We use a simple method to set the loss scale. If agent is located in the upper half of the image, then set 1,1://2,,:. approach is applied to other situations or conditions.
and others are 1. Similarly, the same
We utilize the agents' reachable regions to define the coefficient matrix here, but it's not the only approach. The key idea of this technique to encourage the model to focus more on the pixels related the action prompt. There also other possible ways to leverage this technique when such information of reachability is not available. One promising way is to use advanced object detection and segmentation models like GroundingDINO (Liu al., 2023b) and SAM2 (Ravi al., 2024) to identify the relevant regions in an image (e.g., "the lettuce" in the action prompt "Bob pick up the lettuce") and adjust the loss scale for those regions. Another promising approach is to discover reachability during exploration, such as bootstrapping from zero-shot vision-language models (VLMss, where an agent could learn reachability by interacting with the environment.
Compute We train the world model for 50k steps in the first stage with batch size of 384 on 192 V-100 GPUs in day. Then, we fine-tune the model for 25k steps in the second stage with 120 batch size on 120 V-100 GPUs day. Both the inpainting model and the super-resolution model are trained for 60k steps with batch size of 288 on 24 V-100 GPUs in day.
Samping We use DDIM sampling across the experiments with guidance weight 5 for the text- guided video diffusion model.
B.2 PLANNING SUB-MODULES
Training Data Collection We generated 4k training environments by randomly sampling a recipe in TDW-Cook and a puzzle box in TDW-Game for each agent. All objects were initialized at ran- dom positions on the table. Scripted planners with randomness were then used to play out the episodes and collect the state and action histories. We finetune ALaVA-1.5-78 (Liu et al., 2023a) with LoRA (Hu et al., 2021) for one epoch to obtain one shared model for each submodule across all tasks and cooperators.
For the Intent Tracker, collected 40k short rollouts consisting of three images of consecutive observations and a textual deseription of the next actions of all the agents, converted by a template given the action history.
For the Outcome Evaluator, we collected 138k data consisting of one image of the observation and textual description of the state of each object in the image and the heuristic score, converted by template given each object's 's location.
Compute For each planning sub-module, we finetune LLaVA-1.5-7B with LoRA for one epoch with batch size of 1 144 18 V-100 GPUs in about 3 hours.
20

---

Published as a conference paper at ICLR 2025
C ADDITIONAL RESULTS AND DISCUSSIONS
C.I INTENT TRACKER CAN ADAPT THE PREDICTIONS
The intent tracker module should generalize to unseen cooperators and adapt its predictions dynam- ically based on historical observations. Our approach employs shared intent tracker module across all tasks and cooperators. The test-time cooperator's 's policy is agnostic to our agent during training, meaning the intent tracker must rely on observation history to adapt its predictions and track the intents of diverse cooperators effectively. As shown in Figure 9, given the same current state, the intent tracker accurately predicts the next action of Agent Bob by leveraging its observation history, which reveals preference for preparing his recipe first or helping Agent Alice first. With accurate intent tracking, becomes possible for Agent Alice to select the best action to take.
While the intent tracker module is trained agent TDW-Game setting), it generalizes reasonably well to new behaviors due to its reliance on cooperator-indeeendee observation patterns. As demonstrated in Table 5, the same intent tracker was applied without retraining in 3-agent and 2-agent versions of TDW-Game, where cooperators followed policies unseen during the training phase. The tracker demonstrated adaptability, achiev- ing correct intent prediction rates of 75.8% in the 2-agent cooperation scenario and 64.6% in the 3-agent cooperation scenario. Among the wrong predictions, 20% attributes the intent prediction of "wait" while it shouldn's 't be. This adaptation showcases the model's 's ability to adapt dynamically and predict intents effectively without over-reliance on memorization.
limited set of scripted planners (c.g.. the 4-
Generalizing to more diverse and complex cooperator behaviors remains a challenging yet promising direction for future work. Enhancements such as Population-Bal Based Training (Jaderberg et al., 2019) and Behavior Diversity Play (Szot et al., 2023) could expand the variety of cooperators encountered during training, thereby increasing the robustness
the intent tracker. Furthermore, integrating advanced modeling approaches, such as the Bayesian Theory of Mind (Wu et al., 2021), could facilitate more nuanced inference adaptability in multi-agent collaboration scenarios.
Stap
Alic
Bob
cooperator
up
pick bread.
should
History
Currenstatte
Intent Tracker
Bob will cut onion.
up boof potty
cooperator
Step 1
Step 5
Step
History states
stale
Figure The Intent Tracker Module dynamically adapts its predictions based on historical observations. When collaborating with agents exhibiting either selfish or altruistic policies, the Intent Tracker accurately pre- dicts the cooperator's 's next action by considering their historical behavior. This enables the agent to determine its appropriate next action based the predicted intent, when faced with the same current state but dif- ferent historical contexts.
21

---

Published as a conference paper at ICLR 2025
L.a.VA-155-B
Simpler Supervised Model
Action Proposer
98.3
90.3
Intent Tracker
79.4
73.2
Outcome Evaluator
99.7
62.7
Table 7: Comparison of LLaVA-1.5-7B and the Simpler Supervised Model across different components. The accuracy of the model prediction on the test set.
C.2 OTHER IMPLEMENTATION ALTERNATIVES
These planning sub-modules could also be implemented with other methods. Additionally, we im- plemented simpler supervised model to serve as the planning sub-modules and trained it using the same dataset we employed for fine-tuning the VLM. This model encodes the input image using a VIT-B/I6 CLIP encoder and the textual task description using the SeET-base-uncaseee encoder. The encoded outputs are concatenated and passed through two-layer fully connected network with a hidden dimension of 256.
For the Action Proposer, the output is a multi-hot vector of 157 dimensions, representing all possible actions.
For the Intent Tracker, the output is vector of 4 x 157 dimensions, capturing intents across agents.
For the Outcome Evaluator, the output is a scalar representing the heuristic score.
Compute
For each component, we preprocess the images and the textual prompts to extract the features on 8 V100 GPUs in about 30 minutes. We then train the supervised model for 10 epochs with a batch size of 64 on V100 GPU in about 20 minutes.
For each component, we selected the best result across 10 training epochs. The results shown in Ta- ble 7 indicate that VLMs not only offer better generalizability but also achieve superior performance compared to this supervised model. This can be attributed to the common-scue sense knowledge embed- ded in the VLM during its pretraining on large-scale internet data, as well as its superior ability to integrate the two modalities effectively.
C.3 FAILURE CASE ANALYSIS
As shown Table 3, the compositional world model still fails 25% of the time for predicting the next state conditioned on four agents' actions. We attribute the errors to two primary factors:
Image Quality: This is the most critical issue. accounting for 80% of the errors. We be- lieve this problem arises from the use of CFG (Classifier-Free Guidance) based on multiple textual action conditions. Since CFG strongly guides the diffusion model fulfill textual action conditions, it often results in contradictory behaviors, such ously placing a block on both the left and right sides (see Figure 10) or attempting to pick up block but failing do
agent simultane-
Misunderstanding: This error type involves the agent failing to act according to the textual action conditions (see Figure 11). Compared to image quality issues, misunderstanding errors are less frequent, likely due to the influence of CFG. These errors may stem from data imbalance, insufficient training data, inadequate model generalization capabilities.
Incorporating hidden physical parameters, such as object mass, is an essential direction for making the task closer to real-world scenarios. Extending framework to include physies-aware video generation approaches (Liu et al., 2024; Zhang et al., 2024), a promising avenue for future work and could further improve the capability of world models to handle hidden variables effectively.
22

---

Published as a conference paper at ICLR 2025
Figure 10: An example of the image quality error. The agent at the bottom places the black piece on both sides.
Figure 1I: An example of the misunderstanding error. The textual action condition for the agent the right "place the blue-brown piece onto the right border of the reachable region", but the agent places the piece on the left side.
Human Workspace Shared Regio
Step 1
Step
Step 3
Puzzle Box
Robot Workspace
Initial
Step
Step
Step 6
(a)
(b)
Figure 12: (a) Real-world task up. The reachable region the human the robot shown blue and yellow respectively. Their shared region is shown green, and the puzzle box is shown red. They need to cooperate closely pass and finish the puzzles. (b) An example plan execution. The robot passes the pieces the human needs first and then finishes own puzzle.
D REAL-WORLD TASK: HUMAN-ROBOT COLLABORATION
Task Setup: We instantiate the TDW-Game environment in the real world with two agents. As shown in Figure 12(a), human and an XArm sit on the upper and right sides of the table, each can only reach the objects on his side of the table, there is only one shared region exchange objects, which is located at the upper-right corner of the table. They must cooperate to pass and place 3-5 puzzle pieces scattered randomly on table into correct puzzle box according to visual clues such the shape. In each step, the human the robot only pick place one object within his reachable region.
We fine-tune the Compositional World Model with collected real-world data and use the same VLMs without fine-tuning for other sub-modules. We conducted trials of experiments and reported the number of successful trials and the average steps in Table 8. An example plan execution shown in Figure 12(b), where the robot first helps pass the puzzle pieces the human needs, and finally picks and places its last piece of the puzzle finish the task. We also show two sampled video generations of the compositional world model from the same current state and conditioned on two different joint actions figure 13. The result demonstrates our method is capable in real-world scenarios involving low-level control and non-trivial generalization.
Results:
23

---

Published as a conference paper at ICLR 2025
Generated Videos
Joint Action
Current State
Robot pick the green-purple piece
Human pirk up the green-brown brown piece
Joint Action
Robot green-red
Human pick the green brown piace
Figure 13: Two sampled video generations of the compositional world model conditioned on the same current state but different joint actions.
Success 4/5
Average Steps
COMBO
8.4
Table 8: Real-world experiment results. We report the number of successful trials/ total trials and the average steps of the successful trials over 5 trials here.
24


Reviews:
[{"review_id": "HbmDxipIJI", "reviewer": "Reviewer_yTaz", "summary": "This paper proposes a compositional world model for multi-agent cooperation, leveraging large generative models and compositional diffusion models to build accurate simulations with an arbitrary number of agents. \n\nThe authors present a novel framework called COMBO, which involves the following procedures:\n\n1. **Estimate the Current World State**: Use a diffusion model to infer the state from partial egocentric observations.\n2. **Action Proposal and Evaluation**: Employ a pretrained Vision-Language Model (VLM) to suggest actions, predict other agents’ intentions.\n3. **Future Frame Generation**: Produce future frames based on the current world state image and a text prompt, representing joint actions of multiple agents. This is performed by the Compositional World Model, leveraging a compositional diffusion architecture.\n4. **Evaluate and Plan**: Assess the predicted outcomes from the compositional world model using the VLM, and plan with a tree search algorithm based on these evaluations.\n\nTo assess their model, the authors test their approach on three datasets—TDW-Game, TDW-Cook, and 2D-FetchQ—and compare it to several baselines, including a VAE-based world model, MAPPO, CoELA, and LLaVA. The results demonstrate that the COMBO framework significantly outperforms these baselines, especially in planning capabilities.", "questions": "- Does the action prompt to CWM use a cropped image token along with text, as depicted in Figure 4, or is that just an illustration?\n- As mentioned in the weaknesses, I have concerns about potential scalability. Can this be generalized to continuous action spaces, such as in robotics tasks? If so, it would be beneficial to include experiments on this. Additionally, I wonder if the CWM training setup will be applicable in realistic scenarios. In my opinion, we might only have access to egocentric observations and action history during training.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Novelty**. Suggested framework, named COMBO, offers a unique solution to the multi-agent planning problem by utilizing compositional world modeling for accurate simulation.\n\n**Clear Framework Explanation.** The framework is presented clearly and is easy to follow. Each setting and procedure is understandable through Figure 3 and Algorithm 1. The roles of each module are well-explained in the text and formulation.\n\n**Well-designed Experiments And Clarified Implications of Results.** The experimental setup and baseline choices effectively test the COMBO approach's capabilities. The results demonstrate impressive performance and highlight the necessity of the design choices in their architecture.", "weaknesses": "**Lack of Figure Clarity and Interpretability.** The figures in the paper are unclear and difficult to interpret. Illustrations should enhance understanding, but these require reading the text to decipher them. For instance, Figure 1-(b) displays a random assortment of images without any labels. I believe this can be resolved by adding explicit labels for sequential processes that each frame means. Similarly, Figure 4 presents consecutive frames without explanations. Adding annotations, e.g. state, instruction, prediction t=1~3, would make the figures more informative.\n\n**Limited Scalability and Impact.** Despite its impressive performance, the proposed model may struggle to scale in more realistic scenarios, such as handling low-level controls with continuous action spaces and operating without access to ground truth environmental labels during training. I believe that those compositional world modeling abilities seem reliant on structured inputs through prompts, which aren't typically available in realistic multi-agent cooperation setups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RfkRYaAgYh", "reviewer": "Reviewer_Rt2j", "summary": "The paper propose to use compositional video diffusion model as \"world model\" for an environment with several agents cooperatively solving the task. In addition to learning scores for each agent independently, the diffusion model is trained only on regions reachable by an agent. Such model was used in COMBO agent that is effectively planning by fine-tuning VLM to propose reasonable action given current states while also fine-tuning VLM for intend prediction and for the evaluation of the current state.", "questions": "- Please describe in more details on what data intend and outcome evaluation was fine-tune on? How realistic collection of such data for real-world agents?  \n\n- Potentially more details could be provided on modification of 2D FetchQ environment, as I didn't find any in the appendix. \n\n- Why Co-Gaild baseline is used in this task but not used on the original environments? \n\n- Is usage or fine-tuning of VLM really necessary for these tasks? If so, does this fine-tuning leads to generalizable action generation /agents intends? For example, how robust agents would follow recipes if some ingredients are not seen in the training? Without studying and showing such generalizations, it is not clear why LLM are needed and if more simple supervised models would do the same job.  \n\n- Some formulas could be better connected. E.g in section 4 X is used for world model state, while in 5 s is used.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The problem of join cooperation of several agents is interesting, and the approach for world modeling by merging observations from different agents seems plausible. \n\n- Decomposition of the full state to regions that can be affected by each agent (while not correct in general e.g. turning on light would change the whole image, however it is a reasonable assumption that the overall scene in effected by agents mostly independently)", "weaknesses": "- Scaling loss with the reachability assumes that reachability is provided externally, in real world agents \"reachability\" should be additionally estimated / discovered from the exploration data.  It would be great if the authors would cover better how to discover the reachability regions if they are not provided. Also, what about regions what are not reachable by any agent? In current formulation, it is not clear if those regions are modeled or not in the world model.  \n\n - Fine-tuning of VLM on the \"collected short rollouts\" potentially lead to agent that memories how other agents behave. While this approach can work, it would always require fine-tuning \"intend prediction\" if the agent is changing policy. It would be great if the authors can show that VLM can use some of the initial observations of the agent behaviors to adapt its predictions  on the fly (e.g. with in context learning). \n\n- In both of the TDW-Game and TDW-Cook performance is saturated learning to agents that perform near (and in case of the TDW-Cook somehow better 22.8 vs 24.0) than Oracle Cooperator. Thus, it would be great if authors can increase the difficulty of those tasks and show possible failure cases of their approach. For example, currently world model assumes that visual information and actions is enough to predict the next state. However, for in closer to real world scenarios some parameters are hidden (e.g. objects mass) and thus effective world model would need to deal successfully with estimation of such parameters and using them for the determining of the optimal actions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vyhyIMrHIZ", "reviewer": "Reviewer_Dcsa", "summary": "The authors of this paper explored a compositional workload model for embodied multi-agent cooperation. Unlike previous approaches to embodied multi-agent cooperation, they introduced an explicit world model that simulates the next state by incorporating the estimated actions from the Intent Tracker along with the agent's actions. This model stands out from other single-agent, model-based reinforcement learning approaches by explicitly accounting for multi-agent cooperation dynamics. Utilizing this world model, they implemented a tree search planning strategy similar to Monte-Carlo Tree Search [1]. Their approach demonstrated superior performance in embodied multi-agent cooperation tasks and, due to the compositional nature of their model, exhibited strong generalization capabilities.\n\n[1] Silver, David, et al. \"Mastering the game of Go with deep neural networks and tree search.\" nature 529.7587 (2016): 484-489.", "questions": "- In Figure 1, what means the surrounding figures? Maybe we guess it is to visualize the look ahead planning, but I am not sure it is a good visualization for showing that.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- They designed explicit compositional world model. It is one of the most distinguishable designs in their modeling, which supports the look ahead planning (tree search planning), and they showed it is beneficial through the empirical evaluation results. As a part of this, the world state estimation is good to build the world model for multi-agent setting.\n- It is a well written paper. I can easily follow their discussions without unnecessary questions.", "weaknesses": "- The room of the evaluated benchmarks is too small to show the effectiveness of their proposed modeling. In Table 1, COMBO outperformed previous works. Although, when comparing with LLaVA, it shows comparable success rate except TDW-Cook Cooperator 1 setting. Their efficiency on solving the tasks is clearly better than LLaVA, but we felt it is not good enough to evaluate the effectiveness of their modeling.\n- The generalization performance evaluation is too weak to show that in lines 521-524 and Table 5. They trained the model with 4 agents and applying it to 3 agent only, then what happens if we test with 2 or much more agents such as 10? Additionally, COMBO is more efficient than LLaVA, but LLaVA also showed good success rates for 3 agents setting. The LLaVA is trained on the 3 agents setting? If yes, it should be mentioned in the caption of the figure or the paragraph. If no, then we think this empirical evidence is weak to show the generalizationality of the COMBO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)