Empowering Multi-Robot Cooperation via Sequential World Models
Authors: Zijie Zhao
Honglei Guo
Shengqian Chen
Kaixuan Xu
Bo Jiang
Yuanheng Zhu
Dongbin Zhao
Venue: ICLR 2026 Poster (2026)
Keywords: Model-based Reinforcement Learning
Multi-Agent Reinforcement Learning
Multi-Robot Cooperation
Primary area: reinforcement learning
TL;DR: SeqWM decomposes joint dynamics into sequential agent-wise models, enabling the emergence of advanced cooperative behaviors and achieving successful deployment on physical robot systems.

Abstract: Model-based reinforcement learning (MBRL) has achieved remarkable success in robotics due to its high sample efficiency and planning capability.
However, extending MBRL to physical multi-robot cooperation remains challenging due to the complexity of joint dynamics.
To address this challenge, we propose the Sequential World Model (**SeqWM**), a novel framework that integrates the sequential paradigm into multi-robot MBRL.
SeqWM employs independent, autoregressive agent-wise world models to represent joint dynamics, where each agent generates its future trajectory and plans its actions based on the predictions of its predecessors.
This design lowers modeling complexity and enables the emergence of advanced cooperative behaviors through explicit intention sharing.
Experiments on Bi-DexHands and Multi-Quadruped demonstrate that SeqWM outperforms existing state-of-the-art model-based and model-free baselines in both overall performance and sample efficiency, while exhibiting advanced cooperative behaviors such as predictive adaptation, temporal alignment, and role division.
Furthermore, SeqWM has been successfully deployed on physical quadruped robots, validating its effectiveness in real-world multi-robot systems.
Demos and code are available at:  https://github.com/zhaozijie2022/seqwm

Published as a conference paper at ICLR 2026
EMPOWERING MULTI-ROBOT COOPERATION VIA SE- QUENTIAL WORLD MODELS
Zijie Zhao? Honglei Guo", Shengqian Chen', Kaixuan Xu Yuanheng Zh, Dongbin Zhao
2, Bo Jiang?.,
School of Artificial Intelligence, University of Chinese Academy of Sciences 2SKL-MAIS, Institute of Automation, Chinese Academy of Sciences
ABSTRACT
Model-based reinforcement learning (MBRL) has achieved remarkable success in robotics due to its high sample efficiency and planning capability. However, ex- tending MBRL to physical multi-robot cooperation remains challenging due to the complexity of joint dynamics. To address this challenge, we propose the Sequential World Model (Seq WM), a novel framework that integrates the sequential paradigm into multi-robot MBRL.. SeqWM employs independent, autoregressive agent- wise world models to represent joint dynamics,, where each agent generates its future trajectory and plans its actions based on the predictions its predecessors. This design lowers modeling complexity and enables the emergence of advanced coop- erative behaviors through explicit intention sharing. Experiments on Bi-DexHands and Multi- Quadruped demonstrate that SeqWM outperforms existing state-of-the- art model-based and model-free baselines in both overall performance and sample efficiency, while exhibiting advanced cooperative behaviors such predictive adaptation, temporal alignment, and role division. Furthermore, SeqWM has been successfully deployed on physical quadruped robots, validating its effectiveness real-world multi-robot systems. Demos and code are available at: SM.
INTRODUCTION
Model-based reinforcement learning (MBRL) has been widely applied to robotic systems due to its high sample efficiency (Jiang et al., 2025) and planning capability (Sun al., 2023). However, extending MBRL to multi-robot coop- eration remains challenging (Chai et al., 2025). Farly decentralized approaches built independent world models for each agent (Egorov & Shpilman, 2022), overlooking coupling between agents and hindering coordination. More recent centralized methods, by contrast, assume full observability (Liu al., 2024b), performing dynamics modeling (Zhao al., 2025a) and policy optimization (Zhang al., 2025a) in the joint space. These methods face challenges related to modeling complexity in robotic systems with high-dimensional observation and action spaces, limiting their deployment in real-world scenarios.
Centralized
Decentralized
Distributed Sequential
Robot
Env
2
Communication
(10) Palicy/ Planner
Figure 1: Comparison of Seq WM's distributed sequential
paradigm with existing conitalizae/deccntraiize paradigms.
Between centralized and decentralized paradigms, the distributed sequential paradigm has rapidly developed recent years and demonstrated unique advantages (Khan, 2025). It reformulates multi- agent decision- decision-making as an autoregressive process: agents communicate and act in a certain order, with each updating its policy conditioned on messages and actions from predecessors (Wen et al.,
*oorresponding authors. This work was supported in part by National Natural Science Foundation of China under Grants 621 221360088, 62293541, and in part by Beijing Nova Program under Grant 20240484514.

---

Published as a conference paper at ICLR 2026
2022; Hu et al., 2025). This design enables more consistent joint reasoning (Ding al., 2024) and finer- mner-grained credit assignment (Kuba et al., 2022) without relying on full communication. From a real-world deployment perspective, it reduces the reliance communication synchronization and offers improved robustmess against packet loss or disturbances (Ding et al., 2024).
Motivated by these advantages, shown in Figure 1, propose the Sequential World Model (SeqWMM) which integrates the sequential paradigm into MBRL to structurally decompose dynamics modeling and action planning. For trajectory prediction, SeqWM represents the joint dynamics as sequential agent-wise rollouts, where each agent maintains an independent world model and conditions on the predicted trajectories and actions of its predecessors. For action planning, each agent performs multi-step lookahead conditioned on its predecessors' predictions, thereby preserving cooperative performance while constraining the search to a -iwddimeniooal subspace consistent with the sequential structure. We evaluate SeqWM in two challenging multi-robot cooperation environments: BinDexHands (Chen al., 2024a) and Multi-Quad (Xiong et al., 2024), and further validate its effectiveness on physical multi-robot tasks using two Unitree Go2-W robots. The key contributions are as follows:
(1) By integrating the sequential paradigm, SeqWM decomposes joint dynamics into autoregressive agent- models, reducing modeling complexity, thereby extending MBRL to multi-robot cooperation.
Through explicit intention sharing, SeqWM enables the emergence of advanced cooperative behaviors such as predictive adaptation, temporal alignment, and role division.
(3) Seq WM consistently outperforms all baseline methods on the simulated multi- benchmarks and demonstrates successful real-world deployment on a physical multi-quadruper quadruped platform.
These results collectively demonstrate that Se by leveraging sequential world modeling and planning, offers an effective pathway for multi-robot cooperation, balancing performance, efficiency, and real-world applicability.
2 RELATED WORK
Model-based RL. In robotics, model-based RL. has shown remarkable success due to its high sample efficiency (Jiang et al., 2025), with several approaches (Sun et al., 2023) leveraging learned dynamics models predict trajectories and optimize actions for physical robots. In contrast, existing multi-robot MBRL methods often rely on centralized paradigms (Zhao et al., 2025b), hindering their practical deployment in multi-robot systems. For example, CoDreamer (Toledo, 2024) use transformers or GŇNs to integrate full state-action across all agents. Recent efforts such as MARIE (Zhang al., 2025a) explored decentralized dynamics modeling, but still require communication each prediction step for agent-wise aggregation. Different from these works, SMq assigns each agent an independent world model and predicts trajectories sequentially, which structurally lowers modeling complexity, thereby making it applicable to real-world scenarios.
Sequential Paradigm. Recent studies have highlighted the advantages of the sequential paradigm (Khan, 2025), which enables fine-grained credit assignment (Kuba et al., 2022), effi- cient dynamics modeling (Zhang al., 2025b), and scalable coordination (Xu et al., 2025). For example, MAT (Wen et al., 2022) and PMAT (Hu et al., 2025) model the multi-agent process as a sequence prediction problem, employing transformers to autoregressivelyy predict each agent's 's actions. HARL (Liu et al., 2024a; Zhong et al., 2024) further introduce the sequential update scheme that bring clearer interpretability and ensure monotonic improvement. SeqComm (Ding al., 2024) extends this idea to the communication domain, where agents exchange information in sequential order, effectively mitigating non-stationarity. Motivated by these benefits, we integrate the sequential paradigm into multi-robot world modelling to enhance planning and coordination.
maainng
3 PRELIMINARIES
Problem Formulation. We model the fully cooperative task decentralized partially observable Markov decision process (dee- POMDP). M = (I, S, O A, S2, P, R, 7). where I = {u',.... v" is the set of agents, S is the global state space, O I 1==1 O' is the joint observation space, and
2

---

Published as a conference paper at ICLR 2026
A= I A' is the joint action space. The observation function Q 8 8 - O defines each agent's 's perception of the environment, while the transition function P: A S specifies the environment dynamics. The reward function R S A > IR provides a shared scalar signal, and the discount factor. Fach agent oa learns a local policy x' O + A', which maps its observation of to action a'. The objective is to learn joint policy II x² that maximizes the expected discounted return 7"T-?
Sequential Decision-Making. In many real-world applications, multi-robot systems are often distributed rather than fully decentralized (Negenbom & Maestre, 2014), allowing inter-agent commu- nication to enhance cooperative performance. A Dec-POMDP can thus be extended to a multi-agent POMDP (Oliehoek et al., 2016), where each agent u² receives messages ez from other agents and updates its policy n² Oc x E + A'. To balance efficiency and decision quality, agents adopt communication protocols, defined oi O³ x E' x A' + E. Among them, sequential protocols is especially popular for its simplicity and effectiveness (Ding et al., 2024). It organizes agents in certain order, where each agent acts on its own observation and the message from its predecessor, then passes information forward. Formally, the process is defined as:
af = x (of,e))
et o' (cl, a(),
Such sequential structure naturally motivates us to design world model that predicts trajectories in the same manner, enabling efficient multi-agent planning.
4 METHODOLOGY
In this section, we propose Seq WM, which decomposes the joint dynamics into agent-wise models arranged in sequence. This design substantially reduces modeling complexity, enabling deployment in physical multi-robot systems.
4.1 SEQUENTIAL WORLD MODELLING
Decomposed Joint Dynamics. At each timestep f, the observation-action pair of a single agent (o), a)) can be regarded as a token, and the entire system as sequence of such to- kens. This perspective reformulates joint dynamics as a sequence modeling problem: given the token sequence [[o] a!) ..... (of a.')], the dynamics generates the next-step outcomes [(o|+1, rf+1).... (of (oft simultaneously for prediction, In this setup, agent 1 first predicts (441//41 from its local information ao).a)). and passes the result to 2. Subsequently, each agent conditions on its own observation-action pair (oi,at) to- gether with the predictions of all predecessors ((0(+1:7(<1)) sequential design reduces modeling complexity in a structured, scalable manner, making the approach well-suited for real-world deployment.
1) Unlike existing centralized world models, which fuse all tokens method adopts autoregressive paradigm.
shown in Figure
to produce (of+1: r(+1). Such
World Model. As noted INTRODUCTION, multi-robot cooperative tasks involve high-dimensional observation and action spaces, making it unsuitable use reconstructing raw observations as the learning objective of the world model (Hansen et al., 2022). Therefore we remove the explicit decoder and instead perform dynamics prediction entirely in a latent space. To facilitate distributed deployment, each agent maintains an independent world model without parameter sharing. Let 2f denote the latent state of agent v² at timestep 1, the world model can be defined as follows:
Encoder:
E" (ot)
Dynamics:
$t+1
= D (z), dis et)
Reward:
F
Re (zl, at, ct)
(2)
Communication:
s+1
= ei @ aj
Critie:
O
Q' (zi, ei
Actor:
= n riAd ef)
All modules in SeqWM are implemented using MLPs, ensuring architectural simplicity and consis- tency. For communication function, adopt concatenation operator @ to facilitate modular training. As shown in Section 5.5, this concise design achieves more stable training performance compared to alternatives such as cross-attention and recurrent neural networks (RNNs).
3

---

Published as a conference paper at ICLR 2026
Learning Objective. Let 8g. pD, RR. Qq. and denote the parameters of the encoder, dynamics predictor, reward predictor, critic, and actor, respectively. Following the self-supervised training framework (Hansen et al., 2024), the loss functions can be defined as:
H x (' -
Fe+1 sg( sg(zi+1) + Soft-CE (F..z)) + Soft-CE (of Gi)
(3)
dynamics loss for 9p.. reward loss for #p,Pp
Q Loss for Rg:00
where 0 = {0n, OD. On, 02). H is the prediction horizon, À E (0, 1] is constant that balances the contribution of each rollout step, is the ground-truth reward, G. is TD target, and ei+1 = D' (z), ai, ef) is the predicted latent state. The loss can be backpropagated to the encoder via = F.' (ot). so do the dynamics and reward losses. The latent target 2i+1 F (o(+1) is detached with the cop-gradient operator sg(-) to prevent cyclic gradient flow. Soft Cross-Entron to match the discretized reward and Q-value predictions. Additionally, to ensure modularity and scalability, each agent's world model is trained independently. and the loss can not be backpropagated through the communication channel.
loss is used
Based Eq. (3), the encoder learns compact latent space, while the dynamics and reward predictors minimize prediction errors in this space, ensuring alignment with real environment dynamics. The actor generates initial action estimates in the latent space to warm-start planning and is trained using the Heterogeneous-Agen Soft Actor-Critic (HASAC) (Liu et al., 2024a) algorithm:
((v) = Σ x (o' (e((ziat.e)) atH [rrt.a (+1are))))
(4)
where a is the entropy coefficient, and H[- denotes the entropy function.
We adopt the sequential update scheme (Kuba et al., 2022; Zhong et al., 2024) to train world models in manner aligned with its autoregressive structure. When training the agent vi+1, its inputs are conditioned on the predictions of the first i agents, produced by their most recently updated models. This preserves the sequential dependency, ensuring predictions exploit the most up-to-date outputs, which stabilizes training and improves monotonicity across agent indices.
Inspired by Masked AutoEncoders (He et al., 2022), we randomly permute the order among agents and allow each agent to skip communication with certain probability. This random masking simulates realistic interruptions and forces the world model to robustly adapt uncertainties, significantly enhancing the model's resilience against communication failures.
4.2 SEQUENTIAL PLANNING
Although Eq. (2) includes an actor, it does not serve directly as an explicit decision policy; in- stead, it provides initial action estimates for the planner. We next propose a sequential multi- agent planner based Model Predictive Path In- tegral (MPPI) (Williams et al., 2015) that lever- ages the predictions of world models optimize each agent's 's action. In this framework, the actor contributes by narrowing the action search space to promising regions, while the planner ensures robust long-term suboptimal proposals from the actor.
Lutent Rollout
Planner
#t
Crtlic
P
d:
Fixs
Ne.
Path Integral
Sample
Actor
prirr
Dist.
Updata
dioe outcast
and corrects
Agent yd=
Concat
Agent pir1
At each timestep t, agent v² samples N candi- date action sequences of horizon H, denoted a+ from the initial distribution guided by the actor. Conditioned on its latent state and the received message, the agent performs latent roll- outs with its local world model to predict future
Communication
Figure 2: Sequential planner: agents sequentially opti- mize actions via local world models and share planned trajectories.
4

---

Published as a conference paper at ICLR 2026
trajectories. The value of each trajectory is then estimated as
++ 2h R dh eh)),
(0 +
Σ
(5)
h=t
where 24 H represents the value estimate of sampled action sequence, computed as the sum of predicted rewards over the horizon plus the terminal value given by the critic. Then, candidate sequences are ranked according to their evaluated values, and the highest-scoring subset is selected as the elite set. The action distribution is updated toward the statisties of these elite trajectories, thereby concentrating future sampling around high-value regions and progressively refining the search space. For further details, please refer to Appendix A.I.
After several iterations until convergence, the optimized action sequence and predicted trajectory are transmitted message to the next agent, which repeats the same planning procedure. This sequential planning paradigm substantially enhances multi-agent cooperation efficiency through explicit intention sharing.
Low-pass Action Smoothing. To prevent mechanical wear caused by abrupt action changes, we inte- grate low-pass filtering strategy (Kicki, 2025). In each planning iteration, sampled action sequences are filtered along the temporal dimension to suppress high-frequency fluctuations. This smoothing enforces gradual action transitions across timesteps, reducing control discontinuities and promoting stable, consistent behavior on physical robots. Further details are provided in Appendix A.2.
Heuristic Early-Stopping. Considering the computational constraints of physical robotic platforms, we design a motion- paaiionn heuristic that terminates iterations when the KL divergence between consecutive action distributions falls below a threshold. This early-stopping criterion mitigates diminishing returns (Kobilarov, 2012), reducing computation while preserving plan quality. Further details and experiments are provided in Appendix C.2.
Communication Cache. Inspired by action chunking (Li et al.) which reuses temporally extended action units improve decision efficiency, we introduce a cache that stores the predicted messages from the previous agent, enabling the current agent to retrieve them when communication fails. For instance, communication fails 1++ 1, agent w*+1 retrieves the cached message # = (o()
from agent v instcad of the ideally updated message 2/+1 = (o.+1).
EXPERIMENTS
Environments. We evaluate WM and baselines in two challenging multi-robot cooperative environments: Bimanual Dexterous Hands (Bi-DexHands) (Chen et al., 2024a) and Multi-Quadruper Environment (Multi-Quad) (Xiong et al., 2024). In Bi-DexHands, two agents control pair of dexterous hands to accomplish high-dimensional manipulation tasks (up to O E 229 A E 26),, ). In Multi- Quai, multiple quadruped robots collaborate to solve coordination tasks, and we further deploy WM on real Unitree Go2-W W robots to assess its sim-to-real transfer.
5.1 COMPARISONS
Baselines. We select several competitive baselines, including: HASAC (Liu et al., 2024a), a state-of- the-art model-frea method extending SAC to multi-agent settings; MARIE (Zhang et al., 2025a), model-based method employing Transformer for dynamics prediction; MAT (Wen et al., 2022), method adopting the sequential decision-making paradigm; and MAPPO (Yu et al., 2022), most widely used algorithm, included as a general-purpont bascline.
Results on Bi-DexHands. The representative tasks in Bi-DexHands include object transfer tasks (Over, chcchhbats..... object under different relative positions and grasping postures; and functional manipulation tasks (BottleCap) Pen, Scis scissors), which involve precise bimanual operations to achieve specific functional goals, such opening bottle cap, removing pen lid, spreading pair of scissors.
which require the two hands to transfer an
As shown in Figure 3, SMqWM achieves higher asymptotic returns and faster convergence across all tasks. In several tasks (Over, Metechover2undee rm, Sci ssors,, SeqWM reaches near-
5

---

Published as a conference paper at ICLR 2026
SeqWM (ours)
HASAC
MAT
MARIE
MAPPO
BottleCap
Over
Pushbox
Gate
50
400
0.8
0.8
30
200
0.4
100
10
0.2
0.2
0
Figure 3: Performance comparisons on selected tasks of S with other baselines. Task in Hands report the episode return, while Multi Multi-Quaa (gray background) reports success rate. Bold lines indicate the mean over multiple seeds, with shaded regions denoting the 95% confidence intervals. The results on all other tasks are reported in Figure 12 in Appendix C.I.
optimal performance within 2-4M steps, while baselines require far more interactions fail match it. In more challenging tasks (Pen, catchhesssst) Seq WM steadily improves and achieves the highest final returns with lower variance, demonstrating stability.
Results on Multi-Quad. In Gate, the robots are required to pass through a narrow gate quickly as possible without collision. In PushBox, they jointly push large box to a designated target location. In Shepherd, the two quadruped robots (as sheepdogs) cooperatively guide another robot (as sheep) to a target area (as sheep pen).
In Gate and Shepherd, it rapidly approaches near- 100% success rates within the early phase, significantly surpassing baselines terms of sample efficiency. This superior performance stems from SeqWM's sequential structure, which enables each agent plan actions conditioned its predecessors' intentions, thereby enhancing coordination.
5.2 EMPOWERED COOPERATIVE BEHIVIIRS
We further visualize the behaviors learned by Seq showing that it not only acquires stable policies in high-dimensional state and action spaces, but also achieves advanced cooperative behaviors, including predietive adaptation, temporal alignment, and role division.
ra arm, the throwing hand first performs predic- tion and planning, explicitly transmitting future trajectories to the catching hand. Guided by this message, the catching hand then exhibits predictive adaptation by anticipating the object's 's motion and landing point and proactively adjusting its grasping posture. As shown Frames C-D, the catching hand lowers and opens in advance, aligning its posture with the predicted landing point to enable reliable grasp. In Pen, two hands achieve near- pear-perfect temporal alignment by exchanging predictions of future actions in advance. As result, they grasp the pen body and cap almost simulta- neously Frame D and efficiently complete the extraction Frames E-F, substantially enhancing cooperative efficiency.
Bi-DexHands Behaviors. In
target
A
B
D
Cntch-Over2uddrram
B
(b) Pen
Figure Trajectory visualizations of Cat Catchhover22nddraarr and Fen with SeqWM.
Multi-Quad Behaviors. Figure further shows the role division learned by Seq WM in the Multi- Quad-PushBox. In Frames A-B - 2), the two quadruped robots navigate opposite sides

---

Published as a conference paper at ICLR 2026
of the box, establishing effective pushing configuration. In Frames B-D (t maintain high positive r-axis velocities, indicating continuous forward pushing force. At Frame C (t 3), Robot produces downward y-axis velocity, adjusting the push direction toward the target, while Robot gradually increases its negative y-axis velocity to assist in directional control. As the box approaches the target, Robot reduces its æ-axis velocity avoid overshooting. These behaviors demonstrate that SeqWM supports not only effective force coordination but also fine-grained directional adjustments, resulting precise and efficient task completion.
4)., both
66
Figure 5: Behavior visualizations in PushBox. The first row shows the execution process, where the box is significantly larger than the robots, requiring coordinated efforts from both quadrupeds complete the task. The left side second row visualizes trajectories robots and box, with the right side showing x-axis and y-axis velocities and orientations of each robot.
5.3 SCALABILITY TO MORE AGENTS
We extend the Gate to 5 agents to evaluate the scalability WM, and the behavioral visualiza- tions of 5 robot Gatte are presented in Figure
00
50
B
C
D
Figure 6: Visualization of the learned behaviors on 5- b-rooot-Gate.
As the robots approach the narrow gate, they exhibit predictive adaptation, with certain agents proactively decelerating or waiting to avoid potential congestion. For instance, at Frame (t=#2), Robot 3 maintains near-unity positive x-command, while the other robots moderately reduce their forward commands. In terms temporal alignment, the x-command trajectories reveal a clear wave- like alternation, where the peak sequence (3 + - 5) mirrors the actual passing order, reflecting a dynamic "first-pass parrs-pass-thenn-fe thln-follow" sequence. Overall, the team establishes a coordinated rhythm of "prediction-waiting-- paaiiiggasassiiiiinnnnne which enables efficient multi-robot traversal under constrained environmental conditions.
5.4 REAL-WORLD DEPLOYMENT
The real-world experimental setup is detailed in Appendix C.3, and the results shown in Figure 7.
In PushBox, the two quadrupeds approach the box from opposite sides and coordinate their pushing forces and directions to move toward the target. Between Frames D-F, Robot I moves forward
7

---

Published as a conference paper at ICLR 2026
(a) Push Box
(b) Gate
(c) Shephend
Figure 7: Real-world results of multi-robot cooperation tasks. The trajectories of Robot 1, Robot 2, and the Sheep are marked in different colors.
Dynamics Prediction Error
Reward Prediction Error
con
seq
0.015
0.04
0.02
0.005
Horizon
Horizzn
Figure Dynamies (left) reward (right) prediction errors across horizons.
to provide the main pushing force, while Robot 2 makes slight lateral adjustments steer the box. The overall pushing pattern, including the division of roles and the gradual directional adjustments, closely matches the behavior observed in simulation, confirming a successful sim-to-real transfer.
In Gate, two clear yielding events are observed. Between Frames C-D, Robot slows down and waits for Robot 2 to pass first, demonstrating priority management in constrained spaces. After crossing Frames E-F, Robot 2 actively veers aside to leave sufficient space for Robot 1, enabling smooth passage without collisions. These behaviors reflect SeqWM's trajectory prediction and intention-sharing capabilities, allowing natural, efficient yielding.
In Shepherd, Robot accelerates between Frames A-B, causing the Sheep to move left. To prevent the Sheep from hitting the left gate frame, Robot retreats while Robot advances between Frames C-D. This maneuver drives the Sheep away from Robot 2 and into the target area. The sequence highlights WM''s's capacity for predictive coordination and adaptive role allocation, where the one agent's s motion influences the sheep robot's response and the another agent adapts accordingly to achieve the common goal.
5.5 ABLATION STUDIES
Sequential Sample Generation. To evaluate the contribution of the sequential paradigm in SeqWM's world model, we replace with centralized and decentralized architectures, ensuring all models have an equal number of parameters for fair comparison. Using BottleCap. we collect 50K environment steps with random actions and train each model for 2.5K steps using the loss in Eq. (3). After training, we gather IK additional steps to measure dynamics and reward prediction errors across different horizons. As shown in Figure 8, the sequential and centralized models achieve similarly low errors, both substantially outperforming the decentralized model. The results confirm the advantage of sequential prediction, where each agent conditions its output on the predictions of its predecessors, yielding more accurate and coherent rollouts.
8

---

Published as a conference paper at ICLR 2026
Communication Function. We replaced the concat communication function in SeqWM with alterna- tive fusion mechanisms, including MLP, cross- attn, and RNN, and evaluated them on Bot Bottles EeCap. The results in Figure 9 (left) show that the simplest concat approach achieves the highest and most stable performance. This advantage stems from two factors: (i) concat preserves the complete commu- nication content, allowing the dynamics and reward predictors autonomously identify and exploit the most informative features during training; and (ii) it introduces no additional learnable parameters, thereby maintaining stable gradient propagation in long-horizon prediction. Moreover, we observe that RNN- based fusion even underperforms the no-communication baseline (dec), which we attribute its sensitivity input ordering- an undesirable property in multi-agent communication scenarios lacking fixed semantic sequence.
400
0.8
0.8
300
0.8
0.8
Egloade
MLP
0.4
0.4
cross with
100
RNN
2M
4M
SK
10M
0
1M
2M Steps
1M
2M Steps
3M
4M
Sceps
Figure 9: (Left): Performance ablation of the communication functions, (Middle & Right): Ablation of sequential intention sharing.
Sequential Intention Sharing. A key insight behind SeqWM that cooperation becomes substan- tially easier when an agent can access its partners' future plans. To validate this mechanism, we construct two ablation groups. DecWM removes sequential intention sharing and uses a decentralized world model without inter-agent trajectory passing, while SeqFree removes the world model entirely and allows agents to exchange only single-step messages.
As shown in Figure 9, SeqWM consistently achieves the best performance, followed by De whereas SeqFree performs the worst. This indicates that both the world model for predicting future trajectories (i.c., intentions) and the sequential communication for sharing these intentions indispensable. The gap between SeqWM and DecWM shows that explicit multi-step intention sharing crucial: without receiving communicated future trajectories from others, each agent only plan based on its own rollout, which makes coordinated patterns harder to acquire. At the same time, DecWM still clearly outperforms SeqFree, since the decentralized world model implicitly learns regularities in how other agents tend to respond when predicting future environment evolution under joint actions, enabling more foresighted planning without explicit intention exchange.
CONCLUSION
This paper presented ScqWM, a novel framework that integrates the sequential paradigm into world model learning and planning. By structurally decomposing joint dynamics into autoregressive. agent-wise models, SeqWM offers a principled approach that reduces modeling complexity and naturally enables intention sharing through predicted trajectories. This methodological innovation not only improves scalability but also facilitates the emergence of advanced cooperative behaviors such as predictive adaptation, temporal alignment, and role division. Extensive experiments in Bi-DexHands and Multi- show that SeqWM achieves state-of-the-art performance with superior sample efficiency, while real-world deployment on quadruped robots confirms that these cooperative behaviors transfer reliably from simulation to physical platforms. Beyond empirical results, SegWM WM demonstrates that sequential paradigms provide an efficient and scalable principle for structuring multi-agent cooperation, paving the way for more robust and efficient deployment of cooperation in physical multi-robot systems.
Linitations.
SeqWM is designed for fully cooperative tasks with shared rewards, and its effectiveness in com- petitive mixed-motive settings remains unexplored. Moreover, the current framework relies

---

Published as a conference paper at ICLR 2026
fixed or random order, lacking the ability to dynamically adjust the agent sequence during execution, which may limit performance scenarios where role priorities change over time.
Future Work. Benefiting from the integration of the sequential paradigm and agent-wing world models, SeqWM naturally extends to heterogeneous robot teams and human-robot semantic under- standing. With each agent maintaining independent world model, the framework accommodates diverse dynamics and sensing modalities, enabling cooperation among quadrupeds, manipulators, and aerial robots. Moreover, the explicit trajectory rollouts can be shared not only across robots but also with humans as interpretable intention signals, fostering transparent collaboration, mutual understanding, and trust human-roba roboot teams.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work focuses on advancing multi-robot cooperation through model-based reinforcement learn- ing. All experiments were conducted in simulated environments and on standard quadruped robot platforms, with no involvement of humans, animals, sensitive personal data. The proposed method- ology focuses on technical contributions to the fields of reinforcement learning, multi-agent systems, and robotics, without ethical implications beyond standard academic research practices.
REPRODUCIBILITY STATEMENT
Hyperparameters and training procedures are detailed in Appendix B. All baseline comparisons use publicly available implementations, with the documented parameter settings referenced in the respective sections.
USE OF LARGE LANGUAGE MODELS
Large Language Models (LLMs) were used in the preparation of this paper exclusively for writing assistance and language polishing. The conceptualization of the research, methodology design, experimental implementation, and analysis were conducted entirely by the authors. The authors take full responsibility for the content of this paper.
REFERENCES
Michel Aractingi, Cerre-Alexander Léziart, Thomas Flayols, Julien Perez, Tomi Silander, and Philippe Souères. Controlling the solo12 quadruped robot with deep reinforcement learning. scientific Reports, 13(1):11945, 2023. URL https:/
acm. org/ do abs /10. 5555 3600270.00000 3601471
Jiajun Chai, Zijie Zhao, Yuanheng Zhu, and Dongbin Zhao. A survey of cooperative multi-agent reinforcement learning for multi-task scenarios. Artificial Intelligence Science and Engineering, 1(2)988-121, 2025. URI https:/ //eeee///eeeeepeeeeeeeeeeee ore. leee. org/ abstract/ /wwp///o///////////////eooooe abtttratt/occumeet///////t
077963.
Yuanpei Chen, Yiran Geng, Fangwei Zhong, Jiaming Ji, Jicchuang Jiang, Zongqing Lu, Hao Dong, and Yaodong Yang. Bi-dexhands: Towards human-level bimanual dexterous manipulation. IEEE Transactions Patlern Analysis and Machine Intelligence, 26(5):2804422)) 2024a. URL https:/ http://peeeeeeeeeeeeeeeeeeeeeeee ieee org/
abstract/ ceterrac//dccuuuee htcp///cttcnnnc//ccce/ 10343126.
Zixuan Chen, Xialin He, Yen-Jen Wang, Qiayuan Liao, Yanjie Ze, Zhongyu Li, S. Shankar Sastry. Jiajun Wu, Koushil Sreenath, Saurabh Gupta, and Xue Bin Peng. Learning smooth humanoid locomotion through lipschitz policies. 2024b. URL https: /aw.ivivvv...////////...///.aav. org/ abs/2410 11825.
Guilherme Christmann, Ying-Sheng Luo, Hanjaya Mandala, and Wei-Chao Chen. Benchmarking smoothness and reducing high frequency oscillations in continuous control policies. In 2024 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 627-634, 2024. URL htthhtt///// eeexpl ieee. org/ abst act documentt/////ooooo/mocmmo 0802057.
Gang Ding, Zeyuan I Liu, Zhirui Fang, Kefan Su, Liwen Zhu, and Zongqing Lu. Multi-agent coordination via multi-level communication. Advances Neural Information Processing Systems, 17:11881331188399 2024. net. forum? 312
Vladimir Egorov and Alexei Shpilman. Scalable multi-agent model-based reinforcement learning. In Proceedings of the 2/st International Conference Autonomous Agents and Multiagent Systems, AAMAS '22, pp. 381 390, Richland, SC, 2022. ISBN 9781450392136. URL https: acm. org/ doi/ abs/ 5555/3535850 3535894.
Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. 2024. URL https: httplp:s////peeeveeeee www.eeeeww..eeeeeeeeeeee net wwwpe//ee////eeeeeeeeeeeeeeteeee wtt.ooowwwoooo0000
Nicklas A Hansen. Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. volume 162, pp. 8387-8406. PMLR, 17-23 Jul 2022. URL httptttppp... www.hhttp//////eeeee /pwpoceeddeeeeeeo ngs. ml presss/ v1 62/hansen22a html.
11

---

Published as a conference paper at ICLR 2026
Kaiming He. Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15979-15988, 2022. URL https: 9879206.
https::/icccxppoorr /ttp///ccccccccocoo///oo///o.oce lece wwtpg///.ttttttttaa.aaaa/oeoce /cct//cumee/////////////
Kun Hu. Muning Wen, Xihuai Wang, Shao Zhang, Yiwei Shi, Minne Li. Minglong Li, and Ying Wen. Pmna: Optimizing action generation order in multi-agent reinforcement learning. AAMAS '25, pp. 997-1005, 2025. URL https: / /dl. acm. wtt.o//////ooooo. /abs/ 1 0055555555733333 37436667..7.77
3743619.
Zhennan Jiang. Kai Liu, Yuxin Qin, Shuai Tian, Yupeng Zheng, Mingcai Zhou, Chao Yu, Haoran Li, and Dongbin Zhao. World4rl: Diffusion world models for policy refinement with reinforcement learning for robotic manipulation. 2025. URL https: arxiv org/ abs///ss////0/ 19080.
Nouman Khan. Sequential Decision Making in Cooperativee Mulli-Agent Systems with Constraints. PhD thesis, University of Michigan, 2025. URL https: / httpp/eeeeeeeeeeeeeeeeeeeeeeeeee .ib. umich. edt/hanale/2027.
199212. Accessed: 2025-09-08.
Piotr Kicki. Low-pass sampling in model predictive path integral control. 2025. URL, htt.tttt///axtttt
org/abs 22t.3///55..... 111117.
Marin Kobilarov. Cross-entropy motion planning. The International Journal of Robotics Research, 31(7): 017773369912444443
855-871, 2012. URL /tsttss://oi.... oogg/orgg//o /tlg/10..
Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement lcarning. In International Conference on Learning Representations, 2022. URL https: / hwtp///ppeeeeeeeeeeeeeeeee net fortt/foumm?m fieemiddddcccce ded.dGGGGGGGKNGU....
Qiyang Li. Zhiyuan Zhou, and Sergey Levine. htnp:pl-eeefiiicettiieeioooooooo arXiv preprint arXiv:2507.07969 URL
learning with action chunking. http///axvvvvvvv////////. wwg. hbtp:///ab/a///// 07969.
Jiarong Liu, Yifan Zhong, Siyi Hu, Haobo Fu, QIANG FU, Xiaojun Chang, and Yaodong Yang. Maximum entropy helerogeneoos-ageen reinforcement learning. In The Twelfth International Conference Learning Representations, 2024a. URL https:
http:::///oooeeeee /otp///vcccccccccee/ee/ec not/ / fo rum? Ld=tmqohBC4a5.
Qihan Liu, Jianing Ye, Xiaoteng Ma, Jun Yang, Bin Liang. and Chongjie Zhang. Efficient multi-agent reinforcement learning by planning. In The Twelfth International Conference on Learning Representations. 2024b. URL httptsttt///seeeeeeee http///pppe//iieeeeeeeeeeeeeeeee net forum? ? i C=CpnK
R.R. Negenborn and J.M. Maestre. Distributed model predictive control: An overview and roadmap of future research opportumities. IEEE Control Systems Magazine, 34(4):87-97, 2014. URL https://
leeexplore Leee. org/ ats//tassrcttt //c.//ont////e//////ee /6853439.
Frans A Olichock, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016. URL httph::////ittt springer wtt.ccow..bbceccccoo http///kokk...0//91.10000 3-11779999----1--999------
Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Jan Achterhold, Joerg Stueckler, Michal Rolinek, and Georg Martius. Sample-efficient cross-entropy method for real-time planning. volume 155 of Proceedings of Machine Learning Research, pp. 1049-1065. PMLR, 16-18 Nov 2021. URL wttphhttp//// /ttp:///////ooeeeee ngs.. m1 wtr.ppres//ee v 55/ pinneri221 2 .htll.
Xujie Song, Liangfa Chen, Tong Liu, Wenxuan Wang, Yinuo Wang. Shentao Qin, Yinsong Ma. Jingliang Duan, and Shengho Eben Li. Lipsnet++: Unifying filter and controller into a policy network. In Forty-second International Conference on Machine Learning, 2025. URL https::/p/// /opprrveeeeeeeeeeeeeeeeeeeeeeeee www.ew.eeeee net. forummirmmmm K2o2Xhc8g6.
Jiankai Sun, Yiqi Jiang, Jianing Qiu, Parth Nobel, Mykel J Kochenderfer, and Mac Schwager. Conformal prediction for uncertainty- aware planning with diffusion dynamies model. In Advances Neural Information Processing Systems, volume 36, pp. 80324-80337. Curran Associates, Inc., 2023. URL https:/ acm. org/doi abs/ 10. 5555/36661 22 3669.43.
Edan Toledo. Codreamer: Communication based decentralised world models. In Coordination and Cooperation for Multi-Agent Reinforcement Learning Methods Workshop, 2024, URL forum? f 7A77.
/nt//
Bogdan Vlahov, Jason Gibson, David Fan, Patrick Spieler, Ali Agha- asha-mohammad,, and Evangelos Theodorou. Low frequency sampling in model predictive path integral control. IEEE Robotics and Automation Letters, 955)4454344500 2024. URL https: eeexpl ore. feee. org/ abst ract /ocument/ 104 80553.
12

---

Published as a conference paper at ICLR 2026
Yinuo Wang. Wenxuan Wang. Xujie Song, Tong Liu, Yuming Yin, Liangfa Chen, Likun Wang. Jingliang Duan, and Shengbo Eben Li. ODE-based smoothing neural network for reinforcement learning tasks. In The Thirteenth International Conference on Learning Representations, 2025. URL net/ / forum? 1dd=5YYo63333..
Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang. Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is sequence modeling problem. Advances in Neural Information Processing Systems, 35: 35:16509-16522 2022. URL https: www.oopeevviiiioooo net forum? 2 wtt.:rddwww1wuuuu00.0.e
Grady Williams, Andrew Aldrich, and Evangelos Theodorou. Model predictive path integral control using covariance variable importance sampling. 2015. URL https:
/ 1009.0001111111 01149.
Ziyan Xiong, Bo Chen, Shiyu Huang, Wei-Wei Tu, Zhaofeng He, and Yang Gao. Mqe: Unleashing the power of interaction with multi-agent quadruped environment. In 2024 IEEE/RSJ Internationnl Conference on Intelligent Robots and Systems (IROS). pp. 5918-5924. IEEE, 2024. URL BRLLtttps//ieeexxlooee leee.
org/ absstaastacc..... twtprrac///cccoee dowumentt/////o/c/////ce/cc/ /1000t//2220222
Kaixuan Xu, Jiajun Chai, Sicheng Li, Yuqian Fu, Yuanheng Zhu, and Dongbin Zhao. DipLLM: Fine-tuning LLM for strategic decision-making diplomacy. In Forty-second International Conference on Machine Learning, 2025. URL http::////// http////ppwweeeeeeee/eeee net/ forum?id=hff aOxDDEE.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in neural information processing systems, 35:24611 1--424,,, 2022. URL at htt::/ /twpttss///peeeeeeeeeeeeeeee wwwpeeewwwwwww.eeeeeeeeeeeeeeeee wwt.:eee//oooooeeeee
formm? wttp:/wwnweeeeeeteeeeeoeeeeeettt
Yang Zhang, Chenjia Bai, Bin Zhao, Junchi Yan, Xiu Li, and Xuelong Li. Decentralized transformers with centralized aggregation are sample-efficient multi-agent world models. Transactions Machine Learning Research, 2025a. URL https: http///pppeeeeeeee/eee/ee/eee.e net forum? d=xTd BBggV.....
Yang Zhang, Xinran Li, Jianing Ye. Shuang Qiu, Delin Qu. Xiu Li, Chongjie Zhang, and Chenjia Bai. Revisiting multi-agent world modeling from diffusiin-inspired perspective. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025b, URL https: /pwp///vie/eeeeeeeeee/eeee /tt./eet//tooeeeeeee wwt./orummtmmme forum? id= IRxFIO IOOEeE.
Zijie Zhao, Yuqian Fu. Jiajun Chai, Yuanheng Zhu, and Dongbin Zhao. Meta learning task representation in multiagent reinforcement learning: From global inference to local inference. JEEE Transactions on Neural Networks and Learning Systems, 3698):14908-1499 2025a. URL http::t//seeeepppeeeeeee
leee org/
abstract abstract//ccummet/// http:///ccccccccccceeeeeeeeeseee 109000042.
Zijie Zhao, Zhongyue Zhao, Kaixuan Xu, Yuqian Fu, Jiajun Chai, Yuanheng Zhu, and Dongbin Zhao. Learning and planning multi-agent tasks via moe-based world model. In The Thirty- Annual Conference on Neural Information Processing Systems, 2025b. URL htttp::///peeeeeeeee wwt.e/iwwweeeeeeeeeeeeeeeeeeeeee forum?id f124 11244y0855.
Yifan Zhong, Jakub Grudzien Kuba, Xidong Feng, Siyi Hu, Jiaming and Yaodong Yang. Heterogencous- agent reinforcement learning. Journal of Machine Learning Research, 25(32): 67, 2024, URL http: // /mmrr. org. wwwpp//gppp///////ee
w/tpss/////////// html.
13

---

Published as a conference paper at ICLR 2026
APPENDICES
Multi-Agent Planner
15
A.1 Planning Process
15
A.2 Low-Pass Action Smoothing
15
A.3 Serial-Blocking-Feee Execution
16
Implementation Details
17
B.1 Pscudocode
17
B.2 Hyperparameters.
17
Additional Experiments
18
C.I Additional Comparisons
18
C.2 Inference Time
18
C.3 Sim-to-Red Deployment
18
14

---

Published as a conference paper at ICLR 2026
A MULTI-AGENT PLANNER
A.1 PLANNING PROCESS
At timestep t, the action planning process for agent of can be divided into the following steps:
S1 Communication. Agents are organized to exchange messages in a sequential manner. Specifically, agent v" receives a message ei that aggregates the predicted latent states and planned actions from all its predecessors:
0. ( (2, a.),
1>1, 1,
(6)
where denotes concatenation. To implement this efficiently, employ masking-hased concatenation scheme: fixed-length vector of dimension n x (IA) + dz) is pre-allocated, where n is the number of agents, [A] and da are the action and latent dimensions. Agent u² maintains an empty message, while subsequent agents sequentially fill in their designated slots with their own predictions (šf, a.) in addition to forwarding the received content. This design ensures that information is progressively accumulated along the communication chain with linear complexity in the number of agents.
S2- - Action Sampling, The planner samples N candidate action sequences from two sources. We sample Np candidate action sequences from diagonal Gaussian distribution a H NN (Mia4 (ola= ) where Pi ouI R| represent the mean and standard deviation of the H-step horizon actions. Additionally, we sample Na action sequences directly from the actor module äj (- oh, e, h = t: t + H, and combine these two sets of action sequences to form N candidate action sequences.
S3 World Model Prediction. Following sampling, the world model predicts H-step trajectories for each sampled action sequence using Eq. (2), generating N predicted sequences
{($i. (k). (.))++
S4 Value Evaluation. Each predicted trajectory is assigned value via the H-step return, combining the short-term cumulative predicted reward with the terminal value from the critic:
++
V = 7 Q'
(7)
S5 Action Optimization. The candidate action sequenees are ranked by their evaluated values, and the top M are chosen as the clite set r" The parameters of the action distribution are updated based on the clite set using:
EM Omm
EM=n
(Ti
(((+1)) MEL+H
510+11 PEL4 H
(((+1)) 644H
(8)
Σm
Om
>m= Okm
where the weights are generated based on the evaluated values exp [T (Vrg, maX M Vrr..) fr with being the temperature coefficient.
Iteration. For the default setting, the above process are iterated times to derive the final action distribu- tion. If the carly-stopping heuristic is applicd, after each iteration, we check whether the action optimiza- tion has converged by evaluating the KL divergence between the current and previous action distributions, Dez (Ar(++1) D
([A(()) < 1j. where vy is a small threshold.
The detailed hyperparameters used in the model- based planner are summarized in Table 1.
A.2 LOW-PASS ACTION SMOOTHING
real- world robotics, high-frequency changes in control inputs can cause severe mechanical impacts, accel- erating wear and reducing execution stability. Therefore, many studies in reinforcement learning and motion planning incorporate action-smotthing constraints, such as adding penaltics on differences between consecutive actions during policy updates (Aractingi al., 2023; Christmann al., 2024; Wang al., 2025), introducing regularization in policy networks (Chen al., 2024b; Song ct al., 2025), or filtering noise trajectory optimiza- tion (Pinneri et al., 2021; Vlahov al., 2024; Kicki, 2025), to reduce jitter and improve executability. Inspired by these methods, we apply frequency-domaii low-pass filtering directly to the sampled action noise in our planner, explicitly suppressing the high-frequency components of the actions.
Specifically, during action sampling, we first sample noise from standard normal distribution, apply low-pass filtering, and then add the filtered noise to the action mean generate candidate action sequenees. We use Butterworth filter with øLBF 1, whose transfer function and amplitude- response are given by:
fe
fe
H(s) = 8+ 2m
[H()
(9)
15

---

Published as a conference paper at ICLR 2026
Demeued
-500
-75
Timee
Time
-15.4
14100 155%%
rat10 22%
-175
1020 23%
10"
107
Proquency
Figure 10: Visualization of the low-pass filter. (Left): Amplitude- Prequuedefrequeney response of the low-pass filter. (Middle): Filtering effects random signals at 20% cutoff ratios, the different colors represent different action dimensions. (Right): Effects of low-pass filtering on control commands PushBox, with different colors representing different agents.
where fe is the low-pass cutoff frequency. As show in Figure 10, the amplitude- frequency response shows that the high-frequency components are exponentially attenuated (approaching linear decay in logarithmic coordinates). The corresponding discrete-time difference equation, obtained via bilinear transformation,
s(4 = 1 - 3 (x(4 + x/ 1]) Byll - 1], 8
1- tan (xfe/sa)
(10)
1 tan (xle/1))
where fa the sampling frequency, i.c., the frequency the control signal.
A.3 SERIAL- BERIAL-BLOCKINGG-FREE EXECUTION
common concern regarding the sequential paradigm is that the i-th agent may need to wait for the (i-1) 1)-th agent to finish planning, potentially causing inference time to grow linearly with the number of agents (Wen al., 2022; Hu al., 2025). This issue, referred serial blocking, does not arise in Seq due the of the communication cache.
Message
Message
Message
CLL
et+1
0++1
MPPI
MPPI
Comm. Cache
Comm. Cache
Agent
Agent i+1
Figure 11: Sequential execution without serial blocking enabled by the communication cache.
As shown in Figure 11, the sequential paradigm specifies only the update order of the communication caches, rather than the exccution order of each agent's 's planner. Each agent runs MPPI-based planner at fixed control frequency and simply reads the latest available multi-step trajectory from the cache, without waiting for any other agent complete planning. Under this design, the per-step inference latency satisfies
Tasp Ro max (3)
T
Teomm-
(11)
meaning that ScqWM achieves O(1) decision latency with respect to the number of agents rather than exhibiting linear degradation.
16

---

Published as a conference paper at ICLR 2026
Algorithm 1 Model Training
replay buffer B, parameterized networks Or, On, On, 0g, and db for encoder, dynamics predictor, reward predictor, critic, and actor, respectively; for episode 3......... do for step f = 1, 2, 2.... do
Input:
Get real data ((o]le=1:n [a Tes 10 ili= by interacting with the environment
Add transition into buffer: B B U (foili=1m [afli= Ft, (o) 1m==
end for
for epoch = 1, do
Sample trajectories from B
Update gg. 0p. 0..00.0.0000...... 0g by minimizing Eq. (3)
Update vb by minimizing Eq. (4).
end for end for
Algorithm 2 Model Planning
Input: learned parameters 0g 0p. RR 8q. v. hyperparameters H, K, T, No. M, Na. initial distri-
bution;
for step t = 2, 3.... do
for agent 1,2,....77 do
Get environment observation of and encode it to latent space: 2 (o
if then
Retrieve the message from the previous agent et Djei (2).a))
else
set cj = -
end if
for iteration = 1,., Kp do
Sample Na actions ai N (Pic-H: (o(44H))1
Sample N, actions from actor aj ~ n (..AA((1of e -l:++H
Get predictions by world model rollouts, I { az f=))==tt-
Evaluate the trajectories Eq. (7) and select top-M M elite action sequences Update action distribution following Eq. (8)
end for
end for
end for
B IMPLEMENTATION DETAILS
B. PSEUDOCODE
B.2 HYPERPARAMETERS
We summarize the hyperparameters used in SeqWM in Table and Table 2.
Table The Notations and Values of hyperparameters in the planner.
Hyperparameters Notations
Value
Hyperparameters
Notations
Value
rollout horizon
H
3
sampling actions
N
512
planning iterations
K
6
elites
M
64
temperature
0.5
actor samples
N₂
24
17

---

Published as a conference paper at ICLR 2026
Table The hyperparameters used in the world model.
Hyperparameters
Value
Hyperparameters
Value
Hyperparameters
Value
batch size
1000
buffer size
le6
dynamics coef
20
encoder Ir scale
0.3
entropy coef
le44
5e-4
n-step return
20
num bins
101
q coef
0.1
reward coef
0.1
step
0.5
C ADDITIONAL EXPERIMENTS
C.I ADDITIONAL COMPARISONS
We report additional comparison results on other tasks complement Figure
SeqWM (ours)
HASAC
MAT
MARIE
MAPPO
700
1.2
25
300
0.7
10
700
100
2H
44
64
3H
10M
202
4H
3R
10M
24
4M
6R
8N
30M
24
41
61
8M
10M
04
1M
2M
3M
4M
Figure 12: Comparison results on other tasks.
C.2 INFERENCE TIME
Inference Time Cost. We report the per-step exccution time SeqWM BettleCap using single RTX A6000 GPU on the left side Figure 13. The exccution time increases almost lincarly with the rollout horizon If and the number planner iterations K, which is consistent with the design of SM. With the default settings, WM achieves per-step execution time of 12.8 ms, making it suitable for most real-time robotic tasks
500
H-1
H+]
H+5
400
Time: Step (me)
Episode Return
pollay default
20
16
200
100
Episode Retum
6
Planning Berations
Figure 13: The per-step execution time SeqWM. (Left): Time cost under diffferent rollout horizons H and planner iterations K. (Right): Time cost and performance with and without early-stopping heuristic.
Early-Stopping Heuristic. To further enhance the efficiency of ScqWM, we introduce an carly-stopping heuristic in Section 4.2 that terminates iterations when the change in the action distribution is not significant. The KL divergence used measure of distribution change, and the execution time and performance under different thresholds on Bott cotap are shown Seq WM reduces the exccution time by approximately 57.3% while incurring only about 5.9% performance loss.
right side Figure 13. When the threshold set
C.3 SEA-TO-RELL DEPLOYMENT
We implement all three Multi- Quad tasks in an 8m x 5m indoor space. Each task involves two Unitree Go2- W quadruped robots. The room is equipped with eight Mars cameras, and real-time localization of robots and objects is provided by the NOKOV 3D motion capture system.
18

---

Published as a conference paper at ICLR 2026
PushBox: We cardboard box of 1.2m 1.2m 0.5m and approximately 6 kg The box is
sufficiently large that single robot cannot independently control movement direction, making cooperation essential. The static and kinetic friction coefficients between the box and the ground are both approximately 0.5.
Gate: 1m-wide doorway set up. As shown in Figure 7 (b)-A, the rohots cannot pass through side-hy-side, requiring coordinated navigation.
Shepherd: A DJI EP robot acts as the guided agent (sheep). It is equipped with an omnidirectional chassis simulate sheep behavior: the distance to that robot.
away from the nearest herding robot and its speed inversely proportional
Motion Capturing
EP Robot
Unitree
Figure Real-world setups.
We employ the following strategies enhance the generalization capability Seq WM facilitate sim-to-real transfer:
Positions of other robots transformed from the global frame into the ego-centric frame current robot, reducing observation complexity and improving policy generalization.
Observation transformation:
Domain randomization: Taking PushBox as example, we randomize the initial positions/orientaiinss of both robots and the box, the position and distance of the target, and the friction coefficient between the box and floor improve robustness environmental variations.
Sensor and actuation perturbations: Random noise is added to sensor readings, and small delays with noise are introduced into control commands to emulate real-world sensing errors and actuation inaccuracies.
19


Reviews:
[{"review_id": "aNFlW8pqvy", "reviewer": "Reviewer_wPa1", "summary": "The paper presents sequential world models (SeqWM), a novel multi-agent reinforcement learning (MARL) method that incorporates the sequential multi-agent paradigm into model-based MARL. In SeqWM agents have sequential, independent autoregressive world models, each conditioned on the predecessor, avoiding reliance on centralized stable communication channels. SeqWM is compared with both model-free and model-based baselines on various tasks from two simulation benchmarks, in which SeqWM improves both over final performances and sample efficiency. Additional experiments (ablations, scalability, real-world deployment) further solidify the relevance of SeqWM.", "questions": "- is the y axis of figure 8 reporting accuracy (more is better) or loss (less is better)?\n- how general is SeqWM beyond robotics environments?\n- is SeqWM applicable to environments with very high number of agents (e.g. swarms)? What are its limitations in these scenarios?\n- how are compute costs distributed among the various components of SeqWM? What are the computational bottlenecks in the reported experiments?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Quality:\n- reproducibility is supported by publicly available code and hyperparameters\n- the presented method is compared to relevant model-free and model-based baselines\n- the method is investigated on multiple simulated domains and in the real-world\n\nSignificance:\n- results on the simulated domains suggest that SeqWM not only considerably improves sample efficiency, but can also produce solutions to tasks that baselines cannot solve\n- the role of the core idea (sequential paradigm in model-based MARL) is empirically demonstrated with an ablation\n- the method is empirically validated w.r.t. scalability and real-world deployment\n\nOriginality:\n- the method is an original combination of well-established ideas, namely the sequential multi-agent paradigm and world models\n\nClarity:\n- overall, the manuscript is clear and the layout of the content is predictable\n- core ideas are easily understandable\n- the contribution is properly contextualized both in the introductory and related work sections", "weaknesses": "Quality:\n- although the ablations presented are meaningful, more fine-grained ablations would better support the various contributions to the improved performance of SeqWM w.r.t. baselines; however, this would require substantial additional work which might be out of the scope of this work\n\nSignificance:\n- while computational costs for the planner early-stopping is properly investigated, computational costs of the other compontents is not reported. Discussing this either in the main text or appendix would help framing the applicability of SeqWM\n- while SeqWM is validated on challenging and varied robotics-based domains, it is unclear whether its benefits also apply to different kind of domains (e.g. virtual multi-agent environments like StarCraft); focus on robotics environments is nonetheless properly motivated and sufficiently significant as is\n\nClarity:\n- while the presented method is clear, pseudocode is not present in the main text (but it is present in appendix B.1); moving it or just briefly referencing it in the main methodology section would make it easier to follow\n- the y axis of figure 8 is confusing: it is reported as \"accuracy\", but the main text mentions that the decentralized setting is worse, even though its bars are higher (possibly a typo)\n- appendix B.1 contains no text besides the pseudocode, which is confusing. Adding simple sentence in the main text of appendix B.1 referencing algorithms 1 and 2 would improve its readability", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6hrGsLJnDH", "reviewer": "Reviewer_hYcd", "summary": "This paper successfully integrates the sequential modeling paradigm into model-based multi-agent reinforcement learning and demonstrates both cooperative efficiency and real-world applicability. Addressing the challenging problem of multi-robot cooperation, it introduces a novel Sequential World Model framework. The proposed method not only outperforms existing state-of-the-art approaches in performance and sample efficiency but also achieves real-world deployment on physical robots, demonstrating its ability to handle complex cooperative tasks.", "questions": "1. If this reviewer understands correctly, SeqWM assumes a leader–follower (predecessor–successor) structure where agents act in a predefined order (e.g., Agent 1 -> Agent 2 -> Agent 3). Would this design limit its applicability to tasks that require true simultaneity or are inherently symmetric without a natural ordering? For instance, how would SeqWM handle scenarios where two robots must react simultaneously to catch a fast-moving, unpredictable object, or where multiple robots must lift a shared payload at the exact same time to maintain stability?\n2. How computationally heavy is the proposed planner? how does the runtime scale as the number of agents increases, and what are the practical limits for real-time deployment in physical multi-robot systems?\n3. Based on Table 2, it appears that each agent performs approximately K=6 iterations. If that is correct, does the last agent need to wait for all preceding agents to complete their iterations? For example, with five agents, what is the observed planning delay (in ms) for the final agent? At what point (in terms of the number of agents) does sequential latency become a bottleneck for real-time control?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The central idea of reformulating joint dynamics modeling as an autoregressive sequential modeling problem is both novel and effective. The sequential design significantly reduces communication and modeling overhead, while enabling complex cooperative behaviors with multi-robots.\n2. The paper is well-organized, clearly written, and easy to follow. The motivation and contributions are articulated very clearly.\n3. The method achieves good performance and sample efficiency compared to all baselines. One of the strong contributions is the real-world demonstration. It successfully executed complex cooperative behaviors (e.g., pushing, yielding, shepherding) on physical robots. This reviewer believes that this strongly validates the practicality and robustness of the proposed approach.", "weaknesses": "1. The related work is well written, and it covers well model-based RL and the current sequential paradigm in multi-agent systems. It identifies the gap between centralized and decentralized approaches. However, this reviewer feels that deeper comparative analysis with closely related frameworks such as MARIE, CoDreamer, and SeqComm would clarify the unique technical contribution of SeqWM beyond general sequential coordination. Including reflections on potential limitations, such as dependence on agent ordering and communication latency, would also add balance and critical depth.\n2. Although extensive and well-structured, this reviewer feels that the ablation experiments are not fully sufficient to validate the claimed contributions. First, there is no direct ablation isolating the effect of the sequential structure on emergent cooperative behaviors such as predictive adaptation, temporal alignment, and role division. It is unclear whether these behaviors arise from the sequential design, the MPPI planner, or improved world model accuracy. Second, it appears that the ablations are performed primarily on BottleCap, not on multi-robot coordination tasks such as Multi-Quad or PushBox, limiting generalization to core cooperative scenarios. Third, other design elements, including the planner, early stopping, and communication cache, are under-analyzed. While briefly mentioned in the appendix, the main text lacks quantitative justification of their contributions. For example, there is no comparison showing whether cooperation emerges even without the sequential structure, such as a “Planner-only” baseline.\n3. In sequential planning, with N agents, the Nth agent must wait for all preceding agents to finish planning, which may introduce latency. While the paper demonstrates scalability, it does not analyze how this latency affects responsiveness in larger systems. A discussion on the theoretical convergence and stability of the sequential updates would strengthen the paper.\n4. The proposed method’s reliance on a fixed agent order raises questions about sensitivity. There is no in-depth analysis of how agent ordering affects performance or how accumulated prediction errors propagate over long horizons. (1) How is the order determined, manually or learned? (2) What happens if the optimal order changes during execution (for example, in the Shepherd task when the leading robot should switch)? The paper mentions random masking during training to improve robustness, which is a good idea, but it remains unclear whether a fixed or randomized order is used during testing and how this choice affects robustness.\n5. While the sequential scheme reduces communication frequency, transmitting entire predicted trajectories as messages may increase communication bandwidth. As N grows, the message size received by later agents could scale linearly, potentially becoming a limitation or constraint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DyRU6V0zrq", "reviewer": "Reviewer_VPtz", "summary": "This paper proposes a model-based RL algorithm for multi-agent cooperative problems. The algorithm formulates the world model and the decision making as sequential. This design reduces the complexity of modeling the joint dynamics. The world model is used in a planner to get actions. Experiments conducted in simulated bidexhand and multi-quadruped tasks show that the sequential paradigm leads to more coordinated behaviors. The algorithm also achieves better sample efficiency compared to model-free and non-sequential model-based baselines.", "questions": "1. The sequential paradigm may consume longer inference time since the ${i+1}$-th agent depends on the inference result of the previous $i$ agents. How to overcome this issue when the number of agents is large?\n\n2. Is the action execution of each agent asynchronized? If yes, do you take into account the asynchronous execution during training in simulation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The sequential paradigm is a reasonable design choice to reduce the complexity of a joint world model over multiple agents. Although modeling multi-agent problems as sequential is not new (also studied in one of the baseline methods), applying it to the learning of a dynamic model still has some value to the community.\n\n2. The method achieves strong empirical results in simulation benchmark, achieving higher sample efficiency than all the baselines with a clear margin. The deployment to the real world further strengthened the result.", "weaknesses": "1. There might be some overclaim in contribution (3). Previous work has already demonstrated multi-quadruped pushing in the real world, such as [1].\n\n[1] Learning Multi-Agent Loco-Manipulation for Long-Horizon Quadrupedal Pushing. ICRA 2025. https://arxiv.org/abs/2411.07104\n\n2. Some figures are not self-contained. The captions in Figure 6 are too small to read and the message to convey here is not clear. The meaning of the y-axis in Figure 8 is not explained.\n\n3. In Figure 3, all baselines achieve zero success in the Pushbox environment. Is there any explanation? \n\n4. The ablation study on sequential sample generation seems a little incomplete. As we can see in Figure 8, the centralized architecture achieves even lower prediction errors than the sequential architecture. Does it mean that we should prefer centralized rather than the proposed sequential model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qa2A8tFtMT", "reviewer": "Reviewer_iNPz", "summary": "This work proposes *SeqWM*, a framework for multi-robot cooperative learning. It first trains a Dreamer-style world model that predicts future rewards and observations, and then uses MPPI to derive actions based on the learned model. To facilitate multi-robot communication and cooperation, during the world model training stage, the $i+1$-th model receives the predictions from the first $i$ models. The proposed framework demonstrates its effectiveness across various tasks, both in simulation and in real-world experiments.", "questions": "- A critical question for the proposed framework is: why are a world model and planning necessary in this context? From the demonstration videos, it appears that one major issue with baseline methods is that the robots lack awareness of each other’s positions(xy coordinates in world), leading to coordination failures. However, since motion capture data is available, one could simply provide each robot with the positions of the others and train them to complete the tasks. I would encourage the authors to explore these simpler baseline approaches and compare their performance with the proposed framework.\n- It is also unclear why MAPPO performs poorly on certain tasks, such as PushBox. According to the Multi-Quad paper, MAPPO indeed shows weak performance on similar tasks, consistent with the authors’ findings here. Could the authors provide insights or analysis explaining why MAPPO struggles in these scenarios?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Various experiments demonstrate the effectiveness of the proposed framework in both simulation and real-world settings. The experimental results show that the proposed method outperforms several baseline approaches.\n- The authors provide extensive visualizations to illustrate the performance of their method, along with accompanying code and videos for more intuitive understanding.", "weaknesses": "- The presentation still has significant room for improvement. For example, the authors state that the world model predicts the “environment observation,” defined as the agent’s perception of the environment. However, it remains unclear what these observations actually are — joint angles, absolute positions, or visual inputs? Such important details are not explicitly mentioned in the paper.\n- The discussion of related work is limited. While the authors review prior studies in model-based reinforcement learning and the sequential paradigm, they overlook a substantial body of work in multi-agent learning and multi-robot cooperation.\n- The ablation studies are insufficient. The authors conduct ablation experiments on various communication function designs within the world model (Figure 9) and evaluate performance on the BottleCap task. However, other critical design choices, such as those in MPPI or world-model learning, remain unexplored. A key weakness is the lack of justification for why a world model is necessary for this task—see the detailed question below for further discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)