UniZero: Generalized and Efficient Planning with Scalable Latent World Models
Authors: Yuan Pu
Yazhe Niu
Zhenjie Yang
Jiyuan Ren
Hongsheng Li
Yu Liu
Venue: Accepted by TMLR ()

Abstract: Learning predictive world models is crucial for enhancing the planning capabilities of reinforcement learning (RL) agents. Recently, MuZero-style algorithms, leveraging the value equivalence principle and Monte Carlo Tree Search (MCTS), have achieved superhuman performance in various domains. However, these methods struggle to scale in heterogeneous scenarios with diverse dependencies and task variability. 
To overcome these limitations, we introduce UniZero, a novel approach that employs a transformer-based world model to effectively learn a shared latent space. By concurrently predicting latent dynamics and decision-oriented quantities conditioned on the learned latent history, UniZero enables joint optimization of the long-horizon world model and policy, facilitating broader and more efficient planning in the latent space. We show that UniZero significantly outperforms existing baselines in benchmarks that require long-term memory. Additionally, UniZero demonstrates superior scalability in multitask learning experiments conducted on Atari benchmarks. In standard single-task RL settings, such as Atari and DMControl, UniZero matches or even surpasses the performance of current state-of-the-art methods. Finally, extensive ablation studies and visual analyses validate the effectiveness and scalability of UniZero's design choices. Our code is available at \textcolor{magenta}{https://github.com/opendilab/LightZero}.

Published in Transactions on Machine Learning Research (05/2025)
UniZero: Generalized and Efficient Planning with Scalable Latent World Models
Yuan Pu" Shanghai Artificial Intelligence Laboratory
puguan@xpjllb.orgg
Yazhe Niu"
miuyazhe314 @agaail..mmlll..mmmm com
The Chinese Universiby of Hong Kong
Zhenjie Yang
jayjoung0302@@maall.om
Shonghai Jiao Tong University
Jiyuan Ren Shanghai Artificial Intelligence Laboratory
Syy22@mail..ssinghuuee cn
Hongsheng Lit
hsli@ee. edu..k
The Chinese University of Hong Kong
Yu Liu
liugaisanai@gmmil.com com
Shanghai Artificial Intelligence Laboratory
Reviewed on OpenReview: https: oppnreviee.. net/ forum? Gl6dF9s0oo
Abstract
Learning predictive world models is crucial for enhancing the planning capabilities of reinforcement learning (RL) agents. Recently, MuZero-style algorithms, leveraging the value equivalence principle and Monte Carlo Tree Search (MCTS), have achieved superhuman performance in varions domains. However, these methods struggle to scale heterogencous scenarios with diverse dependencies and task variability. To overcome these limitations, we introduce UniZero, novel approach that employs transformer- effectively learn shared latent space. By coneurrently predicting latent dynamics and decision-oriented quantities conditioned the learned latent history, UniZero enables joint oplimization of the long-horizon world model and policy, facilitating broader and more efficient planning in the latent space. We show that UniZero significantly outperforms cxisting baselines in benchmarks that require long-term memory. Additionally, UniZero demonstrates superior scalability in multitask learning experiments conducted on Atari benchmarks. In standard single-task RI. settings, such as Atari and DMControl, UniZero matches or even surpasses the performance of current state-of-the-art methods. Finally, extensive ablation studies and visual analyses validate the effectiveness and scalability of UniZero's design choices. Our code is available at http://ggihubb.oooooonaaaal///// htwpe//pppennnaiiiieeeeee Teghhiii
world model to
1 Introduction
Reinforcement Learning (RI) has emerged as one of the pivotal approaches for achieving artificial general intelligence (AGI). Despite significant advancements in this field, traditional RL methods often struggle with complex tasks (Ni et al. 2024 Samsami et al. 2024). To address this limitation, researchers have focused developing predictive world models to enhance the planning capabilities and sample efficiency (Janner et al.
2019 Hafner et al. 2020 Hansen et al. 2023 Assran et al. 2023 Sutton et al. 2022 Notably, MuZero-style
These authors contributed equally this work. Corresponding author.

---

Published in Transactions on Machine Learning Research (05/2025)
algorithms (Schrittwieser et al., 2019 Antonoglou et al.
2021 Hubert et al. 2021b Nuu et al. 2024 leverage
the value equivalence principle (Grimm et al. 2020) and Monte Carlo Tree Search (MCTS) (Swiechowski
ett al. 2023) to facilitate planning within learned latent spaces, achieving exceptional performance in domains
such as board games (Schrittwicser et al.) 2019 Silver et al., 2017) and Atari (Schrittwicser et al.) 2019
However, these achievements are largely restricted to settings requiring short-term memory and single-task
learning (Ni et al. 2024 Hausknocht & Stone, 2017). Significant challenges still persist in scaling these methods to heterogeneous scenarios, such as enviromments requiring long-term memory, diverse action spaces, or multitask learning, which limits their applicability to broader and more complex domains.
In the fields of language and vision, the emergence of multi-hoad attention mechanisms Vaswani et al. 2017 has fundamentally transformed the development of general-purpose foundation models. By leveraging large- scale and diverse datasets (Jia et al., 2024) in a simple tokt-token prediction framework, these mechanisms have driven significant advancements across a wide range of applications (Brown et al. 2020 Peebles & Xie 2023). Recently, there has been growing interest in extonding these techniques to docision-making domains. Notably, some studies (Chen et. al., 2021: Janner et al., 2021 Reed al.) 2022) treat reinforcement learning sequence modeling problem, focusing on the supervised offline training of return-conditioned policies. Mcanwhile, other works propose a two-stage online learning paradigm, where the policy and dynamics model
are optimized independently (Hafner et al. 2023 Micheli et al. 2022 Robine et al. 2023
Empirical studies (Parisotto et al. 2019 Ni et al.
2024 demonstrate the effectiveness of transformer- based
architectures in capturing diverse backward memory capabilities within a unified, generalized framework. Similarly, Monte Carlo Tree Search (MCTS) is widely recognized for its efficiency in forward planning. Integrating these two paradigms offers promising pathway for enhancing both retrospective and prospective cognitive functions in artificial intelligence. However, research on the seamless combination of transformers and MCTS remains sparse. This gap raises a critical question: Can transformer architectures enhance the efficiency and scalability of planning in compler, heterogeneous decision-making tasks characterized by diverse dependencies and variability? This paper represents an initial exploration of this question.
We first systematically analyze the limitations of MuZero-style architectures using a representative Atari game under stacked or non-stacked input conditions, simulating varying dependency ranges required for optimal control. Our findings, summarized in Table | and Figure D reveal two core limitations inherent to MuZero-style algorithms. First, the recurrent design introduces an intrinsic enlunglement between latent representations and historical information, resulting in a bottleneck that impedes efficient information propagation. This entanglement further complicates the integration of self-supervised regularization losses (elaborated in Section 2 and 3.1). Second, the architecture suffers from an under-utilization of trajectory data during training, which restricts its ability to fully exploit the accumulated experiential data. Consequently, these limitations reduce data efficiency and scalability, particularly in heterogeneous scenarios characterized by diverse dependencies and task variability. A comprehensive analysis and experimental validation of these limitations is presented in Section 3.1
To tackle these challenges, we introduce UniZero, a novel framework depicted on the right part of Figure 1 UniZero leverages a transformer-based world model to efficiently learn a Lask-agnostic shared latent. space by disentangling latent states from implicit latent histories (see Section 3.2). Specifically, UniZero integrates domain-specific encoders to map diverse inputs-such as images, proprioceptive data, and discrete or continuous actions into a unified latent representation. The unified latent states and actions across varying time steps and tasks are then processed by transformer backbone, enabling temporal and contextual modeling. To support decision-making UniZero employs specialized heads, which model both latent dynamics (e.g., the predicted next latent state and reward) and decision-critical quantities (c.g., policy and value) conditioned on the latent history produced by the transformer backbone. This unified design allows for the joint optimization of long-horizon world models and policies (Schrittwieser et al. 2019 Eysenbach et al. 2022), effectively mitigating the inconsistencies inherent to two-stage learning frameworks (Hafner et al., 2023 With its unified and modularized architecture and training paradigm, UniZero holds the promise of becoming a scalable foundational model for decision-making in various heterogeneous scenarios.
To validate the effectiveness of UniZero, we conduct extensive experiments on the VisualMatch benchmark, which requires long-term memory. UniZero significantly outperforms multiple baseline algorithms across
2

---

Published in Transactions on Machine Learning Research (05/2025)
various memory lengths, demonstrating its capability to model long-term dependencies. Additionally, we investigate UniZero's multitask learning capabilities on 26 Atari games, demonstrating its potential as a versatile and general-purpose agent. In standard RL tasks, such as Atari and DMControl environments, UniZero also achieves comparable performance, underscoring its broad applicability. Comprehensive ablation studies and visual analyses further affirm the effectiveness and scalability of UniZero's design choices.
The main contributions of this paper are summarized as follows:
We identify key limitations in existing MuZero-style architectures and introduce UniZero, a unified and modular approach that addresses these shortcomings. UniZero leverages a transformer- latent world model to learn a task-agnostic and shared latent representation, facilitating more robust and generalizable decision-makiie making across diverse and heterogeneous environments.
UniZero significantly outperforms existing baselines in benchmarks that require long-term memory. To the best of our knowledge, UniZero is the first online MCTS-based agent to achieve performance comparable to single-task settings on the full Atari 100K benchmark using a single model.
UniZero achieves comparable results on standard RL benchmarks, including Atari and DMControl environments, rivaling leading algorithms. Extensive ablation studies and visual analyses validate the effectiveness and scalability of its design choices.
2 Background
Reinforcement Learning (RL) Sutton & Barto (2018) foundational framework for addressing sequential ferision-making problems, typically formalized Markov Decision Processes (MDPs). An MDP is defined by the tuple M = A, 7,7, Po), where S represents the state space, and A denotes the action space. The transition dynamics P : :S SSx A S - [0, 1] specify the probability of transitioning from one state to another given an action. The reward function R : S x A - IR assigns scalar rewards to state-action pairs. The discount factor E [0, regulates the trade-off between immediate and future rewards, while Po defines the initial state distribution. The objective of RL is to derive an optimal policy x* S A that maximizes the expected cumulative discounted return: π² arg max, Eg EE=oo'ra In many real-world scenarios, the Markov property is often violated, nccossitating the of Partially Observable Markov Decision Processes (POMDPs) Sondik (1971). POMDPs generalize MDPs by introducing an observation space O, where the agent receives observations o E O(s) that provide partial information about the underlying state. In environments characterized by terg-term dependencies, optimal decision decision-making requires leveraging the observation history Tıa := (01s,41:-1) Ni et al. (2024). To manage computational complexity, is common to use a truncated history of length H rather than the complete observation history. Consequently, policies and value functions are defined over this truncated history and are expressed as (at T+- and (a, 1+ = B,.M [> "f=2re [z- respectively.
2019 achieves superhuman performance in complex visual domains Bellemare et al. (2013) without requiring prior knowledge of the environment's 's dynamics. It combines MCTS with a learned model comprising 3 networks: 1 Encoder: sy = ha(oi,...,,0) At time step & and hypothetical (also called recurrent/ /urroll) step 0 (omitting when clear), this network encodes past observations (oI..... oz) into a latent representation (or equivalently latent state), which initializes the root node of MCTS. ② Dynamics Network: p*, = go current latent state - and action a 3 Prediction Network: p* v* = fo("s Given latent state st, this network outputs a policy p* (action probabilities) and value ot. MuZero performs MCTS in the learned latent space, with the encoder generating the root node s". Each edge in the scarch tree stores statistics, including (s, a), P(s, a), Q(x, a). R(s a), S(x, a), representing visit counts, policy, value, reward, and state transitions, respectively. The MCTS process consists of three phases: Selection, Exponsion, and Backup (Appendix B.1.1). After scarch, the visit counts N(s, a) at the root node o are normalized to derive an improved policy æ. An action sampled from this policy for interaction with the environment. During training, MuZero optimizes the following end-to-end loss function, incorporating separate terms for policy,
MuZero Schrittwieser et al.
(s a²). This network predicts the next latent state s and reward fk based on the
3

---

Published in Transactions on Machine Learning Research (05/2025)
value, and reward losses (1", 12, and I", respectively). along with a regularization term for weight decay:
K 4() Σ
[P ( m(42) + I' (v z(4R) + 1' (ft,,, (ft, re=x)] + cc|1012.
(1)
Notably, MuZero and its variants (Danihelka et al. 2022 Antonoglou et al. 2021 Hubert et al. b021b Ye et al. 2021) exhibit two key characteristics: (1) during training, only the initial step of the sequence observation is used, (2) predictions at each time step rely on a latent representation obtained recursively. We refer to architectures achering to these principles as MuZero-sbyle architectures.
Self-supervised Regularization. The visualization of latent representations learned by the MuZero agent, presented in de Vries et al. (2021), indicates that in the absence of specific training objective to align latent representations with observations, mismatches may emerge between the latent representations sf predicted by the dynamics network and the observation embeddings 24+- (or equivalently 2/*) generated by the encoder. These discrepancies make the planning process unstable. Moreover, since the primary training objective in RL is based on scalar rewards, this information can be insufficient, particularly in sparse-reward scenarios (Badia et al., 2020). In contrast, observation embeddings, typically encoded as compact tensors, provide richer training signals than scalars. Integrating auxiliary celf-supervised objectives into MuZero to regularize the latent representations is crucial to improving sample efficiency and stability. Specifically, de Vries et al. [2021] proposed a contrastive regularization loss: [(0)= = - || st11 which penalizes the error between the observation embeddings 2 = sg (ho(o)) and their corresponding dynamics predictions st. Inspired by the Sim-Siam framework (Chen & He 2021), ), EfficientZero (Ye et al. (021) introduced a self-supervised consistency loss (SSL), defined as the negative cosine similarity between the projections of predicted latent representations of and the actual observation embeddings z..
3 UniZero
In this soction, we begin by analyzing the two main limitations of MuZero-style architectures, as discussod in Section 3.1 especially in their ability to handle tasks that require capturing long-term dependencies. To address these limitations, in Section 3.2 we introduce novel approach termed UniZero, which is fundamentally modular latent world model. We provide comprehensive description of its architectural design, along with the joint optimization procedure for both the model and the policy. In Section 3.3 we explore how to conduct efficient. MCTS from a long-term perspective within the learned latent space. For more details on the algorithm's implementation, please refer to Appendix
3.1 Main Limitations in MuZero-style Architectures
As shown in left of Figure 1 during training, MuZero-style architectures encode only the initial observation or (which may include stacked frames) and the entire sequence of actions (omilting actions when context is clear) into the first latent state. In effect, no additional information beyond the first step is explicitly provided. This design results in an under-atilizaatio of trajectory data, particularly in scenarios with long trajectories or Long-term dependencies. Additionally, MuZero employs dynamics heads to unroll several hypothetical steps during training, i.e., pk,
s = ( a1-1),, as depicted in the dark gray section of Figure Consequently, the recursively unrolled latent representation sf becomes tightly coupled with historical information, phenomenon we term as entanglement. This entanglement is fundamentally incompatible with the SSL loss, as discussed later. During inference, MuZero encodes the current observation o, into latent representation, which subsequently serves as the root node in the MCTS. While this approach is effective in MDPs, it is likely to fail in POMDP due to the lack of historical information in the root state. Another alternative extension is to employ the recursively predicted latent representation, s as the root node. Here, bt-k (also denoted as =f-x) represents the observation embedding obtained by encoding the true observation at time 1- k. The state * k is then computed by recursively predicting k steps forward using the dynamics network while following the sequence of actions in the trajectory. For clarity, the case of k = 2 is illustrated in Figure However, this design introduces accumulative errors, ultimatoly leading to suboptimal performance. To systematically investigate the limitations of MuZero and the impact of different designs, we propose three variants of the MuZero-style algorithm and compare them with our UniZero architecture:
4

---

Published in Transactions on Machine Learning Research (05/2025)
Mulleo-style Architectures
UniZero Architecture
Entanglement
2tu4
Disentanglement
Head
Dyn. Haad
Hlead
Dyn Hiend
1lend
Dyn
Training
Unified Transformer Backbone
Under-utilization
Full-utilization
Enc
Inference
Enc.
Enc.
Enc,
Enc
Incomplete Context
Complete Context
Lilll...
Figure 1: Comparison between the UniZero (Ours) and MuZero-style architectures during training and inference. Left: In the MuZero-style architecture, the recursinely unrolled latent representation 8 tightly entangled with historical information. During training, it solely utilizes the initial observation of the sequence, resulting in inefficient utilization of information snnder-utilization) During inference, the recursively predicted latent representation o (with à = for clarity) serves as the root node in MCTS, which is prone to inaccuracies due accumulated errors. These issues are particularly pronounced tasks requiring long-term dependency modeling. Right: UniZero employs modular latent world model comprising encoder, unified transformer backbone, and decision /dnamics heads. This design explicitly disentangles latent states from implicit latent history and leverages all observations during training (full-utilization). During inference, the directly encoded latent state is used the root node. By utilizing a more complete and accessible context M = (zt- Huter 4t 3 0t) UniZero improves effective long-term planning the latent space.
prediction accuracy and enables
(Original) MuZero: This baseline does not employ any self-supervised regularization, During inference, the root latent state 2 is generated by encoding only the current observation or via the encoder.
MuZero w/ SSL: As described in Section this variant introduces an auxiliary self-supervised regular- ization loss during MuZero's training process. Theoretically, aims enhance sample efficiency while enabling the latent state to better retain historical information. However, the self-supervised objective focuses accurately predicting subsequent latent states, which deviates from the fundamental requirement of the primary loss function preserving the full historical context necessary for optimal decision-making We refer to this misalignment as incompatibility. As result, while this design performs well in MDP lasks, its effect iveness may be constrained in environments characterized by long-term dependencies or partial observability.
MuZero w/ Context: This variant adopts the same training procodure MuZero but modifies the inference phase by using a k-step recursively predicted latent representation, s as the root node. However, the compounding prediction errors (Janner et. al. 2019 inherent in recurrent unrolling result in

---

Published in Transactions on Machine Learning Research (05/2025)
Table Qualitative comparison of MuZero variants. The original MuZero employs only first observation imput during training, which limits its ability to model long-term dependencies. MuZero w/ SSL incorporates state regularization loss enhance sample efficiency. Both MuZero w/ Content and UniZero (RNN) provide only partially accessible contexts due to their reliance recurrent architectures. In contrast, UniZero, employing transformer- based decoupled architecture, fully leverages the entire observation sequence during training and inference to ensure complete contextual accessibility.
Algorithm
Obs. Full Utilization State Regularization
Context Access
MuZero
X
x
MuZero w/ SSL
MuZero w/ Context
X
Partially Accessible
UniZero (RNN)
Partially Accessible
UniZero
Fully Accessible
PongNoFrameskipp-e (stack=4)
PongNoframeskip-v4 (stack=1)
20
10
Uniiaro (MNN)
Muzero Cansaxd
10-
Muzero Cannuct
(RMN)
Return
Return
-10
-10-
0M
0.1M
0.2M
0.3M
0.AM
0.5M
DM
0.1M
0.2M
0.3M
0.4M
0.5M
Env Steps
Env Steps
Figure 2: Performance comparison of UniZero and MuZero variants in Pong under approximate MDP and POMDP settings. Left: Results in the MDP setting. Right: Results in the POMDP setting. UniZero consistently outperforms all baselines across both scenarios, highlighting its robustness and adaptability. MuZero w/ SSL achieves superior sample efficiency the MDP setting but fails to converge the POMDP setting due representation entanglement issues. Both MuZero w/ Contert and UniZero (RNN) exhibit limited performance in both settings, primarily due prediction errors caused by incomplete context representation.
inaccurate root state. This issue, which we term incomplete context, leads to significant degradation in MCTS accuracy and task performance.
UniZero: Our proposed UniZero (illustrated on Figure I disentangles the latent states 2 from the implicit latent history by leveraging a modular transformer-basee world model. This architecture enables the model fully utilize trajectory data during training while ensuring the prediction of the next latent state at every timestep. During inference, UniZero maintains relatively complete historical context by employing Key- Value (KV) cache mechanism over the most recent Hinter steps. This design not only enriches learning with self-supervised rogularization but also captures long-term dependencies effectively.
UniZero (RNN):
This variant retains the same training scheme as UniZero but replaces the transformer backbone with GRU (Chung et. al. 2014). During inference, the GRU's hidden state is reset every Hufer steps. However, due to the complexity of GRU training Kapturowski al. (2019). this variant also suffers from the incomplete contert problem. Further details are provided in Appendix B.2
For a clearer comparison, Table I summarizes the qualitative differences among these variants. Furthermore, Figure 2 illustrates their performance in Pong under two settings: trame_stack=A and which approximately correspond to MDP and POMDP scenarios, respectively, as described in Hausknocht & Stone (2017). In the stacks setting, MuZero w/ SSL achieves good sample efficiency due the auxiliary self-supervised regularization. However, in the slack1 setting, it fails to converge within 500k environment.
stack=1,

---

Published in Transactions on Machine Learning Research (05/2025)
steps, largely due to the aforementioned entanglement issue. Similarly, both MuZero w/ Contert and UniZero (RNN) struggle to learn effectively because of prediction errors arising from the incomplete context phenomenon. Specifically, for the former, these errors stem from the recurent unrolling of latent states, while for the latter, they originate from the complexity of GRU training Kapturowski et al. (2019). In contrast, UniZero consistently out all other variants, highlighting its robustness and adaptability heterogencous scenarios. Our preliminary experiments also evaluated a hybrid architecture that integrates historical and current observations via a Transformer-based history encoder to generate the initial latent state, while keeping all other training procedures identical to those of MuZero. However, its performance poor, we excluded it from later comparisons. We plan to revisit this variant in future research.
handling
3.2 Scalable Latent World Models
Building on the insights outlined earlier, we propose the UniZero method to address the entanglement of latent representations with historical information and the under-utilization of trajectory data. This section describes the architecture of UniZero and details its training procedure for joint optimization of the model and policy.
3.2.1 Modular Latent World Models
Architecture. As shown in the top-right corner of Figure |I UniZero is composed of four key components: the encoders ha, the transformer-based backbone, the dynamics head ga, and the decision head fa. The encoders include both the observation and action encoders, but for simplicity, the term encoder refers solely to the observation encoder unless otherwise specified. Additionally, the transformer backbone implicitly integrated into the dynamics and decision heads in certain contexts. Formally, at each time step t, the environmental observations and actions are denoted o, and a, respectively. For discrete action spaces, a may refer to action embedding obtained via a learned embedding table, while for continuous actions, it is derived from a two-layer MLP. The latent state is represented as 2t, the predicted subsequent latent state as -, and the predicted reward as År. The policy logits and state value are denoted as and Vr, respectively. These outputs guide the MCTS procedure to enable regularized policy optimization Grill et al. 2020 The UniZero world model W encompassos the following components:
Encoder:
21 = ho:
Maps observations to latent states
Dynamics Head 8, = g0(2g(;
D> Models latent dynamics and reward
(2)
Decision Head: Pt, U2 = Solzz
Predicts policy and value
Training. Each time step is represented by two tokens: latent state and an action. Details on data prepro- cessing and architecture are provided in Appendix [B.2] The dynamics head predicts the subsequent latent state and reward, conditioned on the sequence of prior latent states and actions (Zgr, age). Simultaneously, the decision head predicts the policy and value based on latent states and actions up to time steps and f. 1-1, i.e., (Zge, 1)) ). In MuZero-style methods, the k-th latent state sf is recursively derived from the initial observation via a dynamic network. In contrast, UniZero employs a transformer backbone to model the latent history h = (hū h,"") at each time step. This design explicitly separates the latent state 2, from the latent. history he, mitigating two key limitations of MuZero-like algorithms. Unlike prior approaches (c.g., Hafner et al. (2023)) ) UniZero does not include decoder to reconstruct ž, back into ôz. While reconstruction losses are often used to regularize representations, our empirical results (see Section 4.5 suggest that omitting this decoding loss does not degrade performance. This supports the hypothesis that latent states only need to encode decision-relevant information, rendering reconstruction unnecessary for decision-making tasks.
Inference. During inference, UniZero's latent world model leverages the complete long-term memory stored in the KV cache alongside information from the current observation to produce accurate internal prodictions as root and internal nodes in the tree search. This synergistic use of components significantly improves UniZero's scalability and efficiency. Further details are provided in Section 3.3
7

---

Published in Transactions on Machine Learning Research (05/2025)
3.2.2 Joint Optimization of Model and Policy
In this paper, our primary focus is on online reinforcement learning scttings. Algorithm 1 presents the pseudo- code for the entire training pipeline. This subsection will present the core process of joint optimization of the model and policy (behavior). Unižero maintains a replay buffer that stores trajectories {o,, ar, Tr 0141, Te) (where n is the MCTS improved policy, Section 3.3) and itcrativcly performs the following two steps:
1. Experience Collection: Collect experiences into the replay buffer B by interacting with the environment. Notably, the agent employs a policy derived from MCTS, which operates within the learned latent space.
2. Model and Policy Joint Update: Concurrent with data collection, UniZero performs joint updates on the decision-oriented world model, including the policy and value functions, using data sampled from B.
The joint optimization objective for the mooleyypolicy can be written as:
UuiiZero (0)
E
Σ 1=0
B2 liut1 sg(Mo++2))11
+
CE(fE)))
(r., TNB
next-latent prediction
reward prediction
(3)
+3p
+B
CE(U,, 0t)
policy prodiction
value prediction
Note that we also maintain a soft target world model (Mnih et al. 2013) W = (he, Go, So), which is the training context length, sg is the stop-grad operator, CE denotes cross-entropy loss function, h(o++1) žt+1 is the target latent state generated by the target encoder he, and 6e signifies the bootstrapped n-step TD target: é " So (=gr, agr-1) As the magniludes of rewards across different lasks vary greatly, UniZero adopts reward and value predictions as discrete regression problems (Bellemare et al. 2017) and optimizes by minimizing the cross-entropy loss. Me represents the improved policy through MCTS shown in Scction 3.3 We optimize the dynamics head to predict m., which essentially seems a policy distillation process. Compared to policy gradient methods (Zhang et al. [023] Hafner et al.] 2019 Schulman et al. 2017), this approach potentially offers better stability (Schrittwieser et al. 2019 Grill et al. 2020). The coefficients Bz, 3r, Sp, By are constant coefficients used to balance different loss items. Inspired by Hansen et al. (2023). UniZero has adopted the SimNorm technique, which is implemented after the final layer of the encoder and the last component of the dynamics head that predicts the next latent state, Essentially, this involves applying the L1 norm constraint to regularize the latent state space. As detailed in Section [..5 latent normalization has been empirically proven to be crucial for enhancing the stability and robustness of training.
an exponential moving average of current world model W W In Equation g
(7=eet"re+at +
3.3 MCTS in the Unified Latent Space
RT, agents need memory M (or equivalently comtert) to accurately model future in tasks that require long-term dependencies. To effectively implement this memory mechanism, as depicted in Figure (for simplicity, we use 1 as the starting timestep in this figure), we establish a KV Cache (Ge et al., 2023 for the memory, denoted by: KVM = (KV at ....., 2t, at)]. When the agent encounters a new observation O ,,
and ncods to make a decision, it first utilizes the encoder transform this observation into the corresponding latent state žt, which serves the root node of the search tree. By querying the KV Cache, the keys and values from the recent memory (#--Hashe ...., 22, az) are retrieved for the transformer- based Ot-Hnntrr
latent world model. This model recursively predicts the next latent state ε441, the reward f,, the policy pt, and the value vt. The newly generated next latent state 4t41 functions as an internal node in the MCTS process. Subsequently, MCTS is executed within this latent space. Further details can be found in B.1.1 Upon completion of the search, the visit count set (2 at)} is obtained at the root node 2t. These visit counts are then normalized to derive the improved policy Te: ni = Σ., N(Z.b1))/TT Here, T denotes the temperature, which modulates the extent of exploration (Badia et al. (220). Actions are then sampled from this distribution for interactions with the environment. After each interaction, we save the transition
8

---

Published in Transactions on Machine Learning Research (05/2025)
Monte Carto Tree Search in UniZero
Decssion
Cata
+-
Punsiinn
in Tree
EnC.
Cacw K%
Utrolled
Qynamios Function
query
Accessible Le Memory//ontext
tree
Depl Hoad
Dyn Hoad
Doss Houd
Dyn Muss
Dos Hood
Dyn Hoad
Unified Transformer Backbone
Figure 3: MCTS in the learned latent space. The process begins with new observation o1, which is encoded into a latent state 21- This latent state serves as the root node. The previous keys and values of recent memory are retrieved from the transformer's KV Cache KVM. Subsequently, the search tree utilizes the world model predict the next latent state (which serves as an internal node), reward f, policy and value v, conditioned on the retrieved KV, recursively. These predictions are used to conduct MCTS, ultimately resulting in an improved policy x.
(ot, ae; re, de, 01+1) along with the improved policy into the buffer, with the latter serving as the policy largel. in Eq. 3 By leveraging backward memory and forward search, UniZero demonstrates the potential to perform generalized and efficient long-term planning across a wide range of scenarios.
4 Experiments
To demonstrate the generality and scalability of UniZero, we conduct extensive evaluations across a diverse set of environments characterized by long-term and short-term dependencies, discrete and continuous action spaces, as well as single-task and multitask learning scenarios. Specifically, we evaluate UniZero on the Atari 100k benchmark (short-term dependency, discrete actions) (Bellemare et al. 2013 Kaiser et al., 2024), DMControl (short-term dependency, continuous actions) (Tunyasvvuakool et al. 2020 and VisualMatch (long-term dependency, discrete actions) (Ni et al., 2024). Through comprehensive experiments and in-depth analyses (Appendix and we aim to address the following key questions:
How does UniZero perform on VisualMatch tasks that require long-term memory? (Section 4.2) In multi-task learning on Atari, how does UniZero compare to MuZero? Does it capture meaningful semantie information in the learned embeddings? (Section 4.2p
On single-task settings, can UniZero achieve performance on par with MuZero in the Atari 100k benchmark
and the DMControl continuous control benchmark? (Section 4.4
How effective and scalable are UniZero's 's core design choices? (Section 4.5
4.1 Experimental Setup
Environments and Baselines. (1) Atari 100k: This benchmark comprises 26 Atari games, providing diverse evaluation suite. The agent interacts for 100,000 steps (4 million frames with frame skipping of 4). Sometimes environment steps are abbreviated as Env Steps. (2) DMControl: We consider the Proprio Control Suite, including 18 continuous control tasks with a budget of 500k environment steps. Tasks include classical control, locomotion, and robotic manipulation. (3) VisualMatch: Designed to evaluate long-term dependencies, it tests memory through adjustable memory lengths. These grid-world tasks are divided into exploration, distraction, and reward phases, requiring the agent recall an observed color in the exploration phase to select the correct color the reward phase. Detailed task descriptions are provided in Appendix D.1 For the details of algorithm baselines, please refer to Appendix B
9

---

Published in Transactions on Machine Learning Research (05/2025)
UnDeer
Mudeem
SAC-PT
VisualMatch (memien-60)
VisualMacc memlen-1100
VisualMatch (memlen-25 ()
VisuaPMatch (memlon- 500)
3.2
Sucocs Rate
3.6
3.2
6.7M
6.2M
1.5M
Env
Env
Figure 4: Performance comparison on VisualMatch with increased memory lengths. MuZero consistently underperformed across all tasks, primarily due to insufficient context information. The performance of SAC-GPT significantly deteriorated as the memory length increased. In contrast, UniZero maintained high success rate even with extended memory lengths, demonstrating superior capacity for modeling long-term dependencies. Solid lines denole means; shaded areas represent 95% confidence intervals.
4.2 Visual Match Benchmark
In Figure 4 we compare the performance of UniZero and MuZero on the VisualMatch benchmark, which requires long- bon-term memory. The green horizontal dashed line represents the final success rate of SAC-GPT NNi
ct al. 2024) after training on 3 million environment steps. Due to its lack of contextual information, MuZero performs poorly across all tasks, while SAC-GPT's performance degrades significantly as the memory length increases. In contrast, UniZero achieves consistently high success rates with increasing memory lengths due to its robust long-term dependency capabilities, validating the analysis presented in Section 3.1 Additional analysis of the predictions and attention maps of the trained world model is provided in Appendix
4.3 Multitask Learning on Atari Environments
Table 2: Performance comparison of UniZero and MuZero across eight Atari environments in the
different learning setting. MT means multitask setting, ST means single-task selting. UniZero (MT) outperforms
MuZero (MT) in most enviromments and achieves higher overall human normalized scores than UniZero (ST),
demonstrating its scalability potential. The results on 26 Atari games can be found in Appendix 13
Algorithm
Alien
Boxing
Chopper
Hero
MsPacman
Pong
RoadRunner
Seaquest
Moan
Median
UniZero (MT)
1003
3501
3003
989
19
6300
713
0.4554
0.4085
MuZero (MT)
590
1989
1999
999
5803
600
0.2192
0.0895
UniZero (ST)
580
2802
2991
1012
18
5503
750
0.3223
0.1739
UniZero's yecouplled-yet-unffee architecture proves highly effective across diverse environments with varying dependencics, enabling scamless extension to multitask learning sccnarios. We first evaluate UniZero eight Atari games: Alien, Buxing, ChlopperCommanni Hero, MsPacman, Pong, RoadRunnner and Seaquest.
In the multitask setting (MT), a single model is trained to perform all the considered tasks, where all tasks share a common observation space represented as a (3, 64, 64) image. To ensure consistency across tasks, we set full action space=True (Bellemare et al., 2013), which results a unified action space of 18 discrete actions for each task. In the single-task setting ST), a separate model is trained independently for each task, with full _allaactio___ Daction_sspac=Falll spacceFalse (Bellemare et al. 2013). leading to task-specific action spaces. Unless otherwise specified, the multitask hyperparameters are consistent with those listed in Table g with only minimal adjustments to accommodate the larger model architecture. Specifically, the encoder uses num channel-266, while the transformer backbone incorporates nlayer=12 and nhead=12.
Architecture and Training in Multitask Learning. Unlike single-task configurations, the multitask setup employs independent decision and dynamics heads for each task, following the approach Kumar et al. (2022.. This design introduces minimal additional parameters while preserving efficiency. The shared transformer backbone and encoder promote parameter reuse and generalization across tasks. Each task is
10

---

Published in Transactions on Machine Learning Research (05/2025)
handled by separate data collector that gathers trajectories and stores them in individual buffers. During training, we sample waak_batch___ batch_size=32 size=32 samples from different tasks, aggregate them into a minibatch, and apply the loss function defined in Equation 3 for each task. The task-specific losses are averaged to compute the total loss, which is used to perform backpropagation and network updates.
Results. Table 2 and Figure 10 demonstrate that UniZero (MT) outperforms both UniZero (ST) and MuZero (MT) in lerms of normalized mean and median scores across the evaluated environments within the 400K Eno Steps setting. This result underscores the enhanced efficiency and scalability of UniZero as a latent world model for multitask training. To examine the influence of model size on multi-task learning performance, analyze the effect of varying the transformer backbone size (ulayer-4, 8, 12) across eight Atari games (see Appendix Figure 11 Our findings reveal that increasing model size consistently improves sample efficiency across all tasks, highlighting UniZero's potential in multitask learning.
To further explore the benefits multi-task learning, we conduct T-SNE visualizations of UniZero's latent states (Figure 12 The latent spaces exhibit distinct clustering for each game, reflecting the dynamic variations among environments. Notably, the representations of Alien are more dispersed, likely due to its similarity to other games, such MsPacman, which belongs to the Maze category. This overlap may promote cross-task information sharing, contributing to the substantial performance improvements observed for Alien in the multi-task setting. Morcover, we extended our evaluation to full 26 games in Atai 100k benchmark in Appendix C As shown in Figure 13 and Table 10 UniZero's multi-task model achieves normalized human scores comparable to its single-takk version, whereas MuZero struggles with multi-tase task learning, showing little progress most games. More analysis is provided in Appendix C.3
4.4
Single Task Results in Non-Memory Domains
Human Normalized Median Score (Atari 100k)
Atari We also compare the original MuZero algo- rithm (Schrittwieser et al.) 2019), our reproduced MuZero, and UniZero, on the Atari 100K bench- mark in the single-task setting, as illustrated in Fig- ure 5 Our results show that UniZero achioves a higher human--normalized median score compared to MuZero (Reproduced) with the same code implemen- tation framework, which indicates that UniZero effec- lively models short-term dependencies and demon- strates its versatility across discrete action decision- making tasks. To provide further insights, we present the complete scores and learning curves for 26 games Appendix D.2 Additional analysis of the predic- tions and attention maps of the trained world model on the Pong game is provided in Appendix E.2
0.28
0.23
0.22
Score
0.13
8:10
0.06
0.00
MuZero
MuZero (Reproduced)
UniZero (Ours)
Figure 5: Performance the Atari 100K. UniZero achieves a higher humaan-normalized median score com- pared to MuZero (Reproduced), demonstrating its ability to effectively model short-term dependencics. Detailed scores and curves available in Appendix D.2
4.5 Ablation Study
This subsection and Appendix E. evaluate key Unizero design choices:
Model Size: Varies num layers with training context lengths (Hf 20, keeping inference
context length at Hastr = 4 to handle POMDP in Atari (Mnih et al. 2013).
Latent Normalization: Comparison of normalization techniques that employed in the latent state, including SimNorm (Hansen et al. 2023), Softmaz, and Sigmoid. Details can be found in Appendix B
Decode Regularization: Integrate a decoder on top of the latent state: à, = dg(2)) (where 2, ho(),)) with additional training objective: Ldecode reg = ||o, - do(2)))l 1 + Lpereeptual (Or, do (2)) ). the first term is the reconstruction loss, and the second term is the perceptual loss (iicheli al. 2022
11

---

which combine Published in Transactions on Machine Learning Research (05/2025)
Task
UniZero
Dreamer V3
DMControl UniZero leverages the principles of Sampled Policy Iteration (Hubert et al. 2021b), al- lowing for a seamless extension to continuous action spaces. Training details are provided in Appendix We evaluate UniZero on 18 tasks from the Proprio Control Suite in DMC, which include continuous control tasks with low-dimensional inputs and bud- get of 500,000 environment steps. When compared against the state-of-ther Dreamer V3 (Hafner et al, 2023), UniZero demonstrates superior performance, achieving a higher human-normallize score, and thus showcasing its robust potential in handling diverse action spaces. Detailed learning curves for the DMC Proprio Control Suite are provided in Appendix D.3
acrobot-swingup
400.3
154.5
cartpole-balance
952.2
990.5
cartpole-balanee_sne sparse
1000.0
996.8
cartpole-swingup
801.3
850.0
cartpole-swinupp sparse
752.5
468.1
cheetah-run
517.6
585.9
ball in cu__pcccctcchh
961.6
958.2
finger-spin
810.7
937.2
finger-turn_ent easy
1000.0
745.4
finger-turn hard
884.5
841.0
hopper-hop
120.5
111.0
hopper-stand
602.6
573.2
pendulun-swingup
865.6
766.0
reacher-easy
993.3
947.1
reacher-hare
988.8
936.2
Table 3: Performance between UniZero and D3 across various tasks in the DMCon- trol. The best performance for each task is indicated in bold. UniZero's higher human-normalized scores high- light its strong performance across continuous action spaces. Details are available Appendix D.3
walker-run
587.9
632.7
walker-stand
976.4
956.9
walker- walk
954.6
935.7
Mean
787.2
743.7
Median
875.1
845.5
byert
layer
Pong (H+5)
(H=101
Fong (0#
Pong (Hwd0)
-20
Emy
Figure 6: Effect of model size across training context lengths context length (Hmter = 4) on Pong.
40) with fixed inference
Figure Cand 7 show different performance impacts for several ablation designs on Pong and VisualMatch. Additional results for VisualMalch are in Appendix E.] Based on these results, we can conclude the following key findings: (1) Training Context Length (H): Longer И doesn't always improve performance, likely due to MCTS inference errors. In Pong, larger H needs more layers to maintain performance. Consistent with prior work (Fang & Stachenfeld 2023), longer contexts aid represenlation learning if prediction remains accurate. (2) SimNorm: Outporooms Softmar and Sigmoid, emphasizing effective latent normalization for stable training by enforcing sparsity through fixed L1 norm. (3) Decode Regularization: Negligible effect on both settings, indicating decision-relevant latent information matters more than observation roconstruction.
Further details and visualizations are available in Appendix E. and Appendix E.2
5 Related Work
MCTS-based RL. Algorithms like AlphaGo and AlphaZero (iiverr al. 2016 2017
MCTS with deep neural networks, have significantly advanced board game Al. Extensions such MuZero (Schrittwieser et al.] 2019), Sampled MuZero (Hubert et al.) 2021b). and Stochastic MuZero (Antonoglou et al.] 2021) have adapted this framework for environments with complex action spaces and stochastic dynamics. ZeficientZero (Ye et al. 2021) and GumbelMuzero (Danhelkaa et al. 2022) have further increased the algorithm's 's sample efficiency. MuZero Unpluggod (Schrittwicser et al.] 2021 Xuan al. 2024) introducod reanalyze techniques, enhancing performance in both online and offline settings. LightZero (Niu et al., 2024
addresses real-world challenges and introduces open-source MCTS- + RI. benchmark. Studies like RAP
12

---

Published in Transactions on Machine Learning Research (05/2025)
Latent Nommlization (Ponn)
Decode Regubrttzttion (Pong)
Latent Nomolization Mattn)
Decode Regi Andizalion (Visu Monch))
Noom
Settras
rold
7775
2050
-10
-10
0.0
Cri Steps
Ery Ships
Cnv Stipe
Drv Staps
Figure 7: Effect of Latent Normalization and Decode Regularization on Pong and VisualMatch (0) SimNorm consistently outperforms Softmaz and Sigmoid, emphasizing the critical role of proper normalization in ensuring training stability. Decode regularization exhibits a minimal effect on performance.
(Hao et al.
2023) and SermerFormer (Lehnert et al. 2024) have applied MCTS to enhance the reasoning
capabilities of language models (Brown et al.) 2020). We analyze the challenges MuZero faces in modeling
long-term dependencies in POMDPs and propose transformer-based latent world model to address them.
World Models. The concept of world models, as discussed in Schmidhuber 2015); Chiappa et al. (2117)
Ha & Schmidhuber (2018), enables agents to predict and plan future states by learning a compressed
spatiotemporal representation. Subsequent research (Hafner et al. 2023 Micheli et al. 2022 Robine et al.,
2023 Zhang et al. 2023 Hansen et al. 2023 Schrittwieser et al.) 2019) has enhanced world models in both
architecture and training paradigms. These studics generally follow three main routes based on training paradigms: (1) The Dreamer series (Hafner et al.] 2020 2023) adopts an aclor-critic paradigm, optimizing policy and value functions based on internally simulated predictions. Note that the model and behavior learning in this series are structured in two-stage manner. Building on this, Micheli et al. [2022); Robine et al. (2023): Zhang et al. (2023) leverage Transformer-basee architectures to enhance sequential data processing, achieving significant sample elficiency and robustness. (2) The TD-MPC series (Hansen et. al. 2022 2023
demonstrates substantial performance gains in large-scale tasks by learning policies through local trajectory optimization within the latent space of the learned world model, specifically utilizing the model predictive control algorithm (Kouvarilakis & Cannon| 2016). The model and behavior learning in this series also follow two-stage structure. (3) Research stemming from MuZero (Ye et al., 2021 de Vries et al., 2021 grounded in the value equivalence principle (Grimm et al., 2020), achieves joint optimization of the world model and policy (Eysenbach et al. 2022 Ghugare et al. (202aa) and employs MCTS for policy improvement. Despite these ad advancements, the effective integration of these approaches remains under-explored. In our paper, we provide a preliminary investigation into integrating scalable architectures and joint model-policy optimization training paradigms. A detailed qualitative comparison is presented in Appendix 12
6 Conclusion and Future work
In this paper, we investigate the efficiency of MuZero-style algorithms in heterogeneous scenarios characterized by diverse dependencies and lask variability. Through qualilative analysis, we identify two key limitations of MuZero stemming from its recurrent training paradigm. To address these limitations and enhance the scalability of MuZero, we propose UniZero, a modular approach that integrates a transformer-basel latent world model with MCTS. Experimental results demonstrate that UniZero consistently outperforms baseline methods across a wide range of settings, including discrete and continuous control, single-task and multi-task learning, as well as short- and long-term dependency modeling. Moreover, UniZero exhibits strong potential foundational model for large-scale multi-modal, multi-task learning, paving the way for exciting future research directions, which we aim to explore further.
7 Acknowledgements
This project is supported in part by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (TTC)'s InnoHK, and in part by NSFC-RGC Project N J_CUHK498/244. Hongsheng Li is PI of CPII under the InnoHK. We extend our gratitude to several team-members of the Shanghai AT Laboratory and SenseTime for their invaluable assistance, support, and feedback on this paper and the associated codebase. In particular, we would like to thank Chunyu Xuan, Ming Zhang, and Shuai Hu for their insightful and inspiring discussions at the inception of this project.
13

---

Published in Transactions on Machine Learning Research (05/2025)
References
loannis Antonoglou, Julian Schrittwieser, Sherjil Ozair, Thomas K Hubert, and David Silver. Planning in stochastic environments with a learned model. In International Conference on Learning Regresentations, 2021.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogration, pp. 19619-15629) 2023.
Adrià Puigdomênech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint arXie:2002.0600 2020.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279. 2013.
Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning, 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neurol information processing systems, 33:1877-1901, 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, 2021.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750 15758, 2021.
Silvia Chiappa, Sébastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. arviv preprint ar orxii::704 02254, 2017.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rmn encoder-decoder for statistical machine translation, 2014.
Junyoung Chung, Caglar Gulcehre, Kyungliyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv 2014.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe V. Le, and Ruslan Salakhutdinov, Transformer- xl: Attentive language models beyond a fixed-length context, 2019.
Tvo Danihelka, Arthur Guez, Julian Schrittwieser, and David Silver. Policy improvement by planning with gumbel. In International Conference Learning Representations, 2022.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness, 2022.
Joery A de Vries, Ken S Voskuil, Thomas M Moerland, and Aske Plaat. Visualizing muzero models. arXiv preprint Xiv:22100. 12924, 2021.
Benjamin Eysenbach, Alexander Khazatsky, Sergey Levine, and Russ R Salakhutdinov. Mismatched no more: Joint model-policy optimization for masel-based rl. Advances in Neural Information Processing Systems, 35:23230-2322 -3243,,, 2022.
Ching Fang and Kimberly Stachenfeld, Predictive auxiliary objectives deep rl mimic learning in the brain, 2023.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for 1lss, 2023.
Raj Ghugare, Homanga Bharadhwaj, Benjamin Eysenbach; Sergey Levine, and Ruslan Salakhutdinov.
Simplifying model-based ri: learning representations, latent-space models, and policies with one objective. arXiv preprint ar XXX::2209008466 2022a.
Raj Ghugare, Homanga Bharadhwaj, Benjamin Eysenbach, Sergey Levine, and Ruslan Salakhutdinov. Simplifying model-based rl: Learning representations, latent-space models, and policies with one objective. Xrxiv preprint Xrx:222000 XrX0:2200008460, 2022b.
Jean-Bastien Grill, Florent Altché, Yunhao Tang, Thomas Hubert, Michal Valko, Ioannis Antonoglou, and Rémi Munos. Monte-carlo tree search as regularized policy optimization, 2020.
Christopher Grimm, André Barreto, Satinder Singh, and David Silver. The value equivalence principle for modcl-based reinforcement learning, 2020.
David Ha and Jürgen Schmidhuber. World models. arxiv preprint arXiv:1803.1012. 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arviv preprint arxiv:1912. 016603, 2019.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arxiv preprint arXiv:2010.02193 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023.
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203. 04955, 2022.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for contimnous control. arxiv preprint XrX:::3310.16888, 2023.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992 2023.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps, 2017.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08// 2016.
Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Mohammadamin Barckatain, Simon Schmitt, and David Silver. Learning and planning in complex action spaces. In International Conference on Machine Learning, pp. 4476 4486. PMLR, 2021a.
Thomas Hubert, Julian Schrittwieser, Ioannis Autonoglou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and plamning in complex action spaces. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Fvent, volume 139 of Proceedings of Machine Learning Research, pp. 4476 4486. PMLR, 2021b. URL http::/pproceedingg.ct mir. mlr. wtt.:/swww..rrree..meeeeeeeemeee (tress/v139/ http:///////hubbtla.aaa. html
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
Michael Janner, Qiyang Li, and Sergey Levine. Offlinc reinforcement learning as one big sequence modeling problem, 2021.
Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. arXiv preprint ar arXiv:2406.0387 2024.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Lukasz Kaiser, Mohammad Babacizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model-based reinforcement learning for atari, 2024.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference learning representations, 2019.
Wouter Kool, Herke Van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement. In International Conference on Machine Learning, pp. PMLR, 2019.
Basil Kouvaritakis and Mark Cannon. Model predictive control. Swilzerland: Publishing, 38:13-50 56, 2016.
Springer International
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q-learning on diverse multi-task data both scales and generalizes. arXiv preprint arXiv:221.155144 2022.
Samuel Lavoie, Christos Tsirigotis, Max Schwarzer, Ankit Vani, Michael Noukhovitch, Kenji Kawaguchi, and Aaron Courville. Simplicial embeddings in self-supervised learning and downstream classification. arXiv preprint arXtt:2204.00616, 2022.
Kuang-Huce Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, and Igor Mordatch. Multi-game decision transformers, 2022.
Nayoung Lee, Kartik Sreenivasan, Jason D. Lee, Kangwook Lee, and Dimitris Papalliopoulss Teaching arithmetic to small transformers, 2023.
Lucas Lehnert, Sainbayar Sukhbaatar, Paul Movay, Michael Rabbat, and Yuandong Tian. Beyond a#: Better planning with transformers via search dynamics bootstrapping. arXiv preprint arXiv:2402. 14083, 2024.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advonces in Neural Information Processing Systems, 34:18878 18890, 2021.
L. V. D. Maaten and G. E. Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9: 2579-2605, 2008.
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models, ar preprint XrX:::2209. 00588, 2022.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. ar preprind 1xXim:1312.6002 2013.
Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Pierre-Lue Bacon. When do transformers shine in rl? decoupling memory from credit assignment. Advances in Neural Information Processing Systems, 36, 2024.
Yaxhe Niu, Yuan Pu, Zhenjie Yang, Xueyan Li, Tong Zhou, Jiyuan Ren, Shuai Hu, Hongsheng Li, and Yu Liu. Lightzero: A unified benchmark for monte carlo tree search in general sequential decision scenarios. Advances in Neural Information Processing Systems, 36, 2024.
Emilio Parisotto, H. Francis Song, Jack W. Rac, Razvan Pascamu, Caglar Gulcchre, Siddhant M. Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, Matthew M. Botvinick, Nicolas Heess, and Raia Hadsell, Stabilizing transformers for reinforcement learning, 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 4195-4205, 2023.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth- Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent, 2022.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Jan Robine, Mare Höftmann, Tobias Uelwer, and Stefan Harmeling. Hransformer-based world models are happy with 100k interactions, 2023.
Christopher D Rosin. Multi-armed bandits with episode context. Annols of Mathematics and Artificial Intelligence, 61 (3):2003- 2011.
Mohammad Reza. Samsami, Artem Zholus, Janarthanan Rajendran, and Sarath Chandar. Mastering memory tasks with world models. orXiv preprinl ar Nii:22433.0453,, 2024.
Jürgen Schmidhuber. On learning to think: Algorithmic information theory for novel combinations of reinforcement learning controllers and recurrent neural world models. arXiv preprint arXiu:1511.09449 2015.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. CoRR, hts//911.08266 2019. URL
http://arxiv.orge www.ooggg//.... aaaaa/ba/1999 htabb///9....... http//991...0..111 082655
Julian Schrittwicser, Thomas Hubert, Amol Mandhane, Mohammadamin Barckatain, Ioannis Antonoglou, and David Silver. Online and oflline reinforcement learning by planning with a learned model. Advances in Neural Information Processing Systems, 34:27580- 142259 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.0634F 2017. URL http://arxiv wrg/ abs/ 06347
Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Marc Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference Machine Learning, pp. 30365-30380. PMLR, 2023.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastcring the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
David Silver, Thomas Hubert, Julian Schrittwicser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv: 01811 2017.
Edward Jay Sondik. The optimal control of partially observable Markov processes. Stanford University, 1971.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, Michael Bowling, and Patrick M Pilarski. The alberta plan for ai research. arXiv preprinl WrXii:2208.11172 2022.
Maciej Swiechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Maádziuk. Monte carlo tree search: A review of recent modifications and applications. Artificial Intelligence Review, 56(3):2497-2560 2023.
Saran Tunyasuvunkkool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continuous control. URL
Software Impacts, 6:100022, 2020. ISSN 2665-9638. doi:
https: //tw:///wwww www.wsseeincedirccc..mmmmm wttp:tttccccccciceteeeeeeeeeeeee http:///cc..aaaaaaa.a.aaaa//////
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete represcntation lcarning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annuul Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6306 6315, 2017. URL https://proceednng..oo neurips........... http://ooccaaaaaaooooooooooaap/a wc/paper/2017//aaa..
Trlllldfeeeeaaaaoeeeeeeee22222.. att.raaaosggppbbbaaaaa........ html
17

---

Published in Transactions on Machine Learning Research (05/2025)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Chunyu Xuan, Yazhe Niu, Yuan Pu, Shuai Hu, and Jing Yang. Rezero: Boosting mets-based algorithms by just-in-time and speody reanalyze. arXiv preprint oxXt::2444.1336, 2024.
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in Neural Information Processing Systems, 34:25476-25488, 2021.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-lask learning. Advances in Neural Information Processing Systems, 33:5824 5836, 2020.
Weipu Zhang, Gang Wang, Jian Sun, Yetian Yuan, and Gao Huang. Storm: Efficient stochastic transformer based world models for reinforcement learning, 2023.
18

---

Published in Transactions on Machine Learning Research (05/2025)
A
Environment Details
Phase 1
Phase 2
Phase3
Figure 8: VisualMatch Long-Term Dependency Benchmark: Each task is segmented into three distinct phases- Explorationn, Distraction, and Reward. As illustrated, the duration Phase 2 (Distraction) varies and is denoted by the parameter memory_ length, while the durations of Phases and remain constant. The agent (depicted in purple) operates within a partially observable Markov decision process (POMDP) environment, restricted to a 5x5 grid field of view outlined by white borders and impeded by black walls. During the Eaploration Phase (Phase 1), the agent observes a room with a randomly assigned target color (e.g., blue). In the subsequent Distraction Phase (Phase 2), the environment introduces random distractions such green apples. Finally, the Reword Phase (Phase 3), the agent must navigate to the grid corresponding to the initially observed targel color. Increasing the memory length intensifies the requirement for the agent to retain and utilize long-term dependencies to successfully complete the task.
A.1 VisualMatch Long- Lerm-Term Dependency Benchmark
The VisualMatch benchmark is meticulously designed evaluate an agent's 's capacily for handling long-term dependencies with adjustable memory lengths. As depicted Figure M each task within this benchmark is structured a grid-world environment and is divided into three sequential phases: Exploration, Distraction, and Reward.
Exploration Phase: The agent observes a room exhibiting a randomly assigned RGB color.
Distraction Phase: The environment introduces randomly appearing apples, serving distractions for the agent.
Reward Phase: The agent is required to select a block that matches the initial room color observed during the Exploration Phase.
In our experimental configuration, the duration of Phase 1 (Exploration) is fixed at step, while Phase 3 (Reward) is fixed at 15 steps. The target colors in the Reward Phase are randomly selected from a predefined set of three colors: blue, red, and green.
Our setup diverges from that of Ni et al. 2024 primarily in the reward structure:
Collecting apples during the Distraction Phase yields no reward.
A reward of 1 is granted solely upon the successful completion of the goal in the Reward Phase, ren- dering the environment characterized by entirely sparse rewards. And in the Reward Phase, use f fixed_symbol saaeed__ymmol__olloeeee _oll__ooo_u_u__ee
Additionally, in VisualMatch, the duration of the Exploration Phase is condensed 1 step, compared the 15 steps employed in Ni et al. (2024).
The VisualMatch task thus intricately designed to test an agent's 's proficiency in managing long- term dependencies within its docision-making procosscs. The partially observable nature of the environment, with the agent confined 5x5 grid field of view each step, necessitates strategic soccsion-making based on incomplete information, effectively simulating numerous real-world scenarios.
19

---

The training. of the Published in Transactions on Machine Learning Research (05/2025)
A.2 Benchmark in Non-Memory Domains
Atari 100k Benchmark The Atari 100k benchmark, introduced by SimPLe (Kaiser et al. 2024 is extensively utilized in research focused on sample-eficcient reinforcement learning. This benchmark encompasses 26 diverse Atari games with image-based imputs and discrete action spaces, accommodating up to 18 possible actions per game. The diversity of games ensures a comprehensive and robust evaluation of algorithmic performance across a wide range of environments. In the Atari 100k benchmark, agents are permitted to interact with each game environtent for total 100,000 steps per game, which corresponds to 400,000 environment frames when considering frame skipping at every 4 frames. This setup emphasizes the importance of sample efficiency, agents must learn effective policies with limited number of interactions.
DeepMind Control Suite We utilize collection of continuous control tasks from the DMControl suite, specifically selected from the proprioceptive inputs domain. These tasks cxhibit significant variability in objectives, observation spaces, and action dimensions, providing comprehensive testbed for evaluating continuous control algorithms. All tasks are modeled infinite- horizon continuous control environments; however, for the purpose of evaluation, we imposc fixed episode length of 1,000 steps and climinate any termination conditions to maintain consistency across tasks.
Consistent with the methodology outlined in Hansen et al. 2023), we apply an action repeat value of 2 across tasks. This results in an effective episode length of 500 decision steps. The primary performance metric employed is the cumulative episode return, which quantifies the agent's ability to achieve objectives across the varied control tasks. This metric provides clear and quantifiable measure of agent's 's proficiency in navigating and manipulating the diverse environments presented by the DMControl suite.
Implementation Details
B.1 Algorithm Details
Here, we present the complete training pipcline of UniZero in Algorithm Unizero algorithm consists of two primary procedures:
1.
This procedure gathers experiences (trajectories) {or, az, T2, d and the improved policy Fy derived from Monte Carlo Tree Search (MCTS) into the replay buffer B. The agent interacts with the environment by sampling actions a, from the MCTS policy n,, which is generated by performing MCTS in the learned latent space.
2. Seeate_wordd_om This procedure jointly optimizes the world model and the policy. UniZero updates the decision-oriented world model, policy, and value using samples from B.
collect_steps in Algorithm is defined as num sun_episodes settasddss_eachheeee each_collecttttttteecceeeeeeeece in Algorithm 1 is calculated as pollect_steps x replay_ratal steps and model training steps) [Schwarzer et al. 2023). In our experiments, replay_ratin is usually set 0.25.
x episode_length. In our experiments, num_ samm__pisods_____ each_ collect is typically set to 8. The parameter world (the ratio between collected environment
20

---

Published in Transactions on Machine Learning Research (05/2025)
Algorithm 1: UniZero
Procedure training ():
train_iterations for train iterations do
Create a Key-Value Cache for the memory: KVM = collect collect_expere experience (collect steps)
for world model iterations do
L @erste_wolld_ool
// Due to the variations in the parameters of the world model. We need clear the old Key-Value Cache
Clear the Key-Value Cache: KVM = 0
Procedure collect_ collect_eepereeeeeeeceeeeeeeeeee (epetteeeeeeeeee) (n)) Ou t env env.reset()
for 1= 0 to n do
24 + ho()
Sample a, ~ गK = (a)(2))) which obtained through MCTS(=, KVs, W)
Add the latest Key-Valut Cache (V at) KVM
0r-1, de env. (a,
if de then
L 0++1 + env resete
B ( at. Pt, 0+11, )=) 1n-1
Procedure update_wordd worldwwllllooo sor___oodldd_dd__)__________ model()))))
m)+HH11 ~ B// where H is the training Sample a mini-batch of sequences {(or, ar, re, O1+1, M))tt context length.
Compute larget TD-value i,, and larget next latent state žt41 according to the target world model W Optimize the world model and policy jointly according to Equation
Procedure (2, KVM, sim):
// The following process will repeat sim iteratioons/iimuaatiooss,, where represents the current simulation step.
Require: N),2,),, Qi(2,a), a)( R,(2,),, o((2,))
Initialize root node t 2z repeat
a' + PU,C as Equation until 22(2),a)== =0
Evaluate the leaf root node E using W: pl, of So(2), KVM), a < go(s), KVM) and stored the dynamics and decision quantities into the corresponding tables R,(2,a), Z,(2, a), P.(2,a) for each along the search path do
N,(2.0)+1 Q++1 Q((.)))()
(2, = (N(,,))
return Normalizationl (NtiaiiioonnNn(a a)) A))
B.1.1 MCTS in the Learned Latent Space
As delincated in Algorithm 1 the MCTS procodure (Schrittwicser al. 2019) within the learned latent space comprises 3 phases in each simulation step i. The total iterations/Simulation steps in single search process is denoted sim:
Selection: Each simulation initiates from the internal root state 2t which is the latent state encoded by the encoder hy given the current obscrvation o,, The simulation procoods until it reaches lcaf node 24, where signifies the search root node at timestep t, and indicates it's the leaf node. For each hypothetical timestep = 1,...L l of the simulation, actions chosen based the Predictor Upper
21

---

Published in Transactions on Machine Learning Research (05/2025)
Confidence Bound applied Trees (PUCT)
Rosin 2011
formula:
[ Q(2,a) P(2,a) a)
Σb (2)
+ b2
Es N(,
a= arg max
c1 log
(4)
((,a))
C2
where N represents the visit count, Q denotes the estimated average value, and P is the policy's prior probability. The constants e1 and dg regulate the relative weight of P and Q. For the specific values, please refer to Table 8 For lo < I, the next state and reward are retrieved from the latent state transition and reward table as 2 = S (**, a²)) and p* R aa*).
Expansion: At the final timestep of the simulation i, the predicted reward and latent state are computed by the dynamics network ge: fl, g go (ø', a KVM), and stored in the corresponding tables, R (5', a') rr and S (2t,at a²) = s/+1, The policy and value are computed by the decision network fo: A new internal node, corresponding to state z', is added to the search tree. Fach
p',u' = so (2', edge (3',a) from the newly expanded node is initialized to {N (s', a) 0, Q (s', 0, P (s',a) =p'}.
Backup: At the end of the simulation, the statistics along the simulation path are updated. The estimated cumulative reward at step calculated based on ot, i.e., an (I k)-TD bootstrapped value:
1-1- Σ 1=
(5)
where & are predicted rewards obtained from the dynamics network go, and 8 are obtained from the target decision network fo- Subsequently, Q and N are updated along the search path, following the equations in the MCTS procedure described in
Upon completion of the search, the visit counts N(2, a) at the root node 2, are normalized to derive the improved policy:
म. = (cl2+) = N(z,, 2)2/7
(6)
Σ, b)1,
where is the temperature coefficient controlling exploration. Finally, an action is sampled from this distribution for interaction with the environment. UniZero leverages key-value (KV) caching and attention mechanisms enhance backward memory capabilities and employs MCTS to improve forward planning efficiency. By integrating these two technological directions, UniZero signifieantly advances more general and efficient planning.
B.2 Architecture Details
Encoder. In the Atari 100k experiment, our observation encoder architecture principally follows the framework described in Niu et al. (2024), utilizing the convolutional networks. A notable modification UniZero the addition of linear layer al the end, which maps the original three-dimensionall features to one-dimensional latent state of length 768 (denoted as latent state dim, D), facilitating input into the transformer backbone network. Additionally, we have incorporated a SimNorm operation, similar to the details described in the TD-MPC2 paper (Hansen et al. 2023). Let V (=8 in all our experiments) be the dimensionality of each simplex g, constructed from L (= D V) partitions of z. SimNorm applies the following transformation:
sim NOrM
BL., g:
(7)
j==1 Biilv/
where zz sim__orm the simplicial embodding [Lavoic et al. 2022 z, denotes concatenation, and 7 is temperature parameter that modulates the sparsity of the representation. We set to 1. As demonstratod in 4.5 SimNorm is crucial for the training stability of UniZero.
22

---

Published in Transactions on Machine Learning Research (05/2025)
Table 4: Architecture of the encoder for VisualMatch. The size of the submodules is omitted and can be derived from the shape of the tensors. LeakyReLU refers to the leaky rectified linear units used for activation, while Linear represents a fully-connected layer. SimNorm (Hansen et al.) 2023) operations introduces natural sparsity by constraining the L1 norm of the latent state fixed constant, thereby ensuring stable gradient magnitudes. Conv denotes a CNN layer, characterized by kernel 3, stride = 1, and padding 1. BN denotes the batch normalization layer.
Submodule
Output shape
Input image (o+)
3 x 5
Convl + BN1 LeakyReLU
16 5
Conv2 BN2 LeakyReLU
32
5
Conv3 + BN3 + LeakyReLU
64 X 5 x 5
Adaptive..v@Pool2d
64 x 1 x 1 64
Linear
SimNorm
64
For the encoder used in the Long-Dependenge Benchmark, we employed similar conv. network architecture, with a latent state of length 64. Specifics can be found in the related table (see Table
Dynamics Head and Decision Head. Both the dynamics head and the decision head utilize two-layer linear networks with GELU (Hendrycks & Gimpel 2016) activation functions. Specifically, the final layer's output dimension for predicting value and reward corresponds to the support size (refer to B.4 (Schrittwieser let al. 2019 Bellemare al., 2017). For predicting policy, the output dimension matches the action space size. For predicting the next. latent state, the output dimension aligus with the latent state dimension, followed by additional SimNorm normalization operation. In the context of Atari games, this dimension is set to 768, whercas for VisualMatch, it is configured to 64.
Transformer Backbone. Our transformer backbone is based on the panoGPT project, as detailed in Table For each timestep input, UniZero processes two primary modalities. The first modality involves latent states derived from observations, normalized in the final layer using SimNorm, as discussed above. The second modality pertains to actions, which are converted into embeddings of equivalent dimensionality to the latent states via a learnable nn. Embeddinn layer. For continuous action spaces, these can alternatively be embedded using learnable linear layer. Notably, rewards are not incorporated as inputs in current framework. This choice is based on the rationale that rewards are determined by observations and actions, and thus do not add additional insight into the deeision-making process. Furthermore, our approach does not employ a return-conditioned policy (Chen et al., 2021 Loe ct al. 2022), leaving the potential exploration of reward conditions to future work. Each timestep's 's observed results and correspouding action embeddings are added with a learnable positional encoding, implemented through nn. Embeddiin,g as shown in Table 5 While advanced encoding methods like rotary positional encoding (Su et al.
2023) and innovate architectures of transformer (Dao et al. 2022 exist, their exploration is reserved for future studies. Detailed hyper-parameters
can be found in Appendix B.4
Table 5: Positional encoding module. wi-l is learnable parameter matrix with shape H x D, and H refers to the sequence length and refers to the latent state dimension, 768 for the Atari, for the VisualMatch.
Submodule
Output shape
Input ((zHH, (Lx)) Add ((21:H (1:H) + w1:H)
2H x D
UniZero (RNN). This variant employs a training setup akin to UniZero but utilizes a GRU (Chung et al., 2014) as the backbone network. During training, all observations are utilized. During inference, the hidden state of the GRU is reset every Hiieer steps. The recursively predieted hidden state h, and observation embedding 2t serve the root node of the MCTS. The recursively predicted hidden state he and predicted
23

---

Published in Transactions on Machine Learning Research (05/2025)
Table 6: Details of Transformer block. MHSA refers multi-head self-attention and FFN refers to feed-forward
networks. Dropout mechanism can prevent over-fitting.
Submodule
Module alias
Output shape
Input features (label as x1)
2H D
Multi-head self attention + Dropout(p)
Linearl + Dropout(p)
MHSA
2H x D
Residual (add æ1)
LN1 (label as 222)
Linear2 + GELU
2H X D
Linear3 + Dropout(p)
2H X D
FFN
Residual (add x2))
2H X D
T.N2
X
Table 7: Transformer-based latent world model (Pi.,, Vi.H, fi.H (isH) fa(zu. a1.r). ). The hidden states (Ni.r. kH the final layer of the transformer are referred as the implicil latent history. Positional encoding and Transformer block are explained in Table 5 and G
Submodule
Output shape
Input ((21.1 01-H))
Positional encoding
Transformer blocks xN
2 H D
(implicit) Latent history ((hi. (( Decision head (PI.H, U1.H)
Dynamic head ($1.1, (.d
latent state & serve as the internal nodes. At the root node, due to the complexity of GRU training, the recurrent hidden state h may not fully capture the historical information. At. the internal nodes, the issue is exacerbated by the accumulation of errors, leading inaccurate predictions and consequently limiting performance. For an illustration of the training process, please refer to Figure
B.3 Extension of UniZero to Continuous Action Spaces
To adapt. UniZero for environments with continuous action spaces (Hbbett al. 2021b), we introduce several key modifications to both the network architecture and the MCTS procedure. These adaptations crucial for accurate modeling and effective decision-making when actions are not confined to discrete set but take any value within a continuous range.
B.3.1 Policy Network Modification
The policy network's decision head is redesigned to accommodate continuous actions by outputting parameters suitable for continuous distributions. Instead of producing logits for finite set of discrete actions, the network now generates the mean (μ) and standard deviation (α) parameters of a Gaussian distribution for each action dimension. Specifically, for each dimension of the action space A, the network predicts (s.)(s) and (s), enabling the representation of a continuous action space. Formally, the policy is defined as:
mo(a)s) (s(no(s).
where (o(s) R|AI and (s) E R|A are the mean and standard deviation vectors parameterized by 8, and IAl denotes the dimensionality of the action space.
24

---

Published in Transactions on Machine Learning Research (05/2025)
Doo Hoad
Head
DecL Hlans
Dyn
Training
GRU
Enc
End.
Figure 9: Training pipeline of UniZero (RNN). During training, all observations utilized. The recursively predicted hidden state he and observation embedding 2 serve as the root node. The recursively predicted hidden state he and predict latent state & serve as the internal nodes of MCTS. During inference, the GRU hidden state is reset every Huser steps. However, potential inaccuracies may arise from the recursively predicted hidden state h2 due to the complexity of GRU training
B.3.2 MCTS Node Expansion Adaptation
In continuous action spaces, enumerating all possible actions is computationally infeasible. To address this challenge, we modify the node expansion strategy within MCTS to sample finite set of actions from a proposal distribution derived from the policy network. Specifically, actions are sampled from the Gaussian proposal distribution (als) defined as:
B(als) (((ss)
99)
|A| of actions are sampled from B(als), with K = 20 in our continuous action experiments. Each sampled action a, is associated with its corresponding probability under the proposal distribution, B(a,ss) my ma(asss)
During each node expansion, a finite number K <
B.3.3 PUCT Formula Adaptation
To maintain balanced exploration-cxxpoitttio trade-off in continuous action spaces, the Predictor + UCT (PUCT) formula is adjusted accordingly. In this adaptation, the prior policy distribution transformed from the original prior policy P(λ,a) in Equation 4 to a uniform policy u(²,a). This modification leverages the prior implicitly without introducing explicit bias, ensuring that exploration is not disproportionately influenced by the policy.
B.3.4 Policy Distillation from MCTS Visit Counts
After the MCTS procedure, the visit counts N(s,a) a) at the root node are normalized to derive an improved policy estimate (t(as)):
N(s,
(a(s)) = N,N(s,)
(10)
integrate this improved policy into the policy network, we employ a projection operator P. Inspired by MuZero, this projection minimizes the Kullback-Leibler (KL) divergence between the improved policy and the network's policy output:
LKL. KT. (Ax((())))) mo (-1s))
(11)
25

---

Published in Transactions on Machine Learning Research (05/2025)
B.3.5 Policy Loss Calculation for Continuous Actions
The policy loss for continuous actions is computed by minimizing the KL divergence between the improved policy derived from MCTS and the policy network's output distributtin. The loss function is defined as:
ppoliey = =- - # log mo(a))s),
where a, are the sampled actions from the improved policy. This formulation ensures numerical stability by operating directly with probabilities rather than log probabilities.
B.3.6 Summary of Modifications
Extending UniZero to accommodate continuous action spaces involves the following koy modifications:
Policy Network Redesign: The policy network is modified to output the mean and standard
deviation paramcters of Gaussian distributions for each action dimension, cnabling the reprosentation of continuous actions.
MCTS Node Expansion Adjustment: The node expansion strategy within MCTS is adapted to sample a finite set of actions from the Gaussian proposal distribution, avoiding the infeasibility of enumerating all possible continuous actions.
PUCT Formula Adaptation: The PUCT formula is revised to appropriately balance exploration and exploitation in the context of continuous actions, without introducing bias towards the prior policy.
Policy Distillation via KL Divergence Minimization: The visit counts obtained from MCTS are distilled into the policy network by minimizing the KL divergence between the improved policy and the network's policy distribution.
These enhancements enable UniZero to effectively manage continuous control tasks, ensuring robust perfor- mance across diverse and complex action spaces.
B.4 Baselines and Hyperparameters
Baselines. Our MuZero implementation based on the LightZero (Niu al. 2024 framework. Unless otherwise stated, all references MuZero in this work denote its variant augmented with self-supervised learning regularization (MuZero w/ SSL), as discussed in Section 6 (1) VisualMatch Baselines: We compare against MuZero and the SAC-Discrete variant combined with the GPT backbone, as proposed in Ni et al. (2024), referred to as SAC-GPT. (2) Atari 100k Baselines: The baseline used is MuZero. (3)
DMControl Basclincs: DreamerV3 (Hafnr et al. 2023) is used as the baseline, a model-based approach that optimizes a model-free policy using rollouts generated from learned environment model. For a comprehensive comparison with prior model-based RL algorithms such as TWM (Robine et al. 2023), IRIS
Michhll et al. 2022), DreamerV3 (Hafner et al.) 2023 STORM (Zhang et al. 2023) TDMPC2 (Hansen
el al. 2023 and MuZero (Schriltwieser el al. 2019) please refer to Table 12
Hyperparameters. We maintain consistent set of hyperparameters across all tasks unless explicitly stated otherwise. Table 8 outlines the key hyperparameters for UniZero, which are closely aligned with those reported in Niu et al. (2024). Furthermore, Table 9 provides the critical hyperparameters for MuZero w/ SSL, MuZero w/ Contezt, and UniZero (RNN).
B.5 Computational Cost
Compared MuZero, although UniZero may have a slightly higher computational cost per step, the difference in actual training time not significant due to their different training paradigms UniZero employs parallel training, whereas MuZero requires executing recursive for-loop during training.
26

---

Published in Transactions on Machine Learning Research (05/2025)
Table 8: UniZero Key Hyperparameters. Most hyperparameters are aligned with those in Niu et al. (2024) to enable fair comparisons. For brevity; long-term denotes the long-term dependency benchmark, DMC refers the DeepMind Control Suite, and Ateri refers to the Atari 100k benchmark.
Hyperparameter
Value
Planning
Number of MCTS Simulations (sim)
50
Number of Sampled Actions (K)
20 (DMC only)
Inference Context Length (Hinter)
4 (Atari); 2 (DMC); vemory_lengte + 16 (long-term)
Temperature
0.25
Dirichlet Noise (a)
0.3
Dirichlet Noise Weight
0.25
Coefficient eı
1.25
Coefficient C2
19652
Environment and Replay Buffer Replay Buller Capacity
1, 1,000,000 Uniform
Sampling Strategy
Observation Shape (Atari)
(3, 64, 64) (stack1); (4, 64, 64) (stack4)
Observation Shape (Long-term)
(3, 5, 5)
Observation Shape (DMC)
Varied across tasks
Reward Clipping
True (Atari only)
Number of Frames Stackod
1 (stack1); 4 (stack4; Atari only)
Frame Skip
4 (Atari); (DMC)
Game Segment Length
400 (Atari); 100 (DMC); memory memory_length 16 (long-term)
Data Augmentation
False
Architecture
Latent State Dimension (D)
768 (Atari, DMC); 64 (long-term)
Number of Transformer Heads
8 (Atari, DMC); (long-term)
Number of Transformer Layers (N)
2
Dropout Rate (p)
0.1
Activation Function
LeakyReLU (encoder); GELU (others)
Reward/Valus Bins
101
SimNorm Dimension (V)
8
SimNorm Temperature (T)
1
Optimization
Training Context Length (H)
10
Replay Ratio
0.25
Buffer Reanalyze Frequency
0 (DMC, long-term); 1/50 (Atari); 0 in Figure
Batch Size
64
Optimizer
AdamW
Learning Rate
1 1 10-4
Next Latent State Loss Coefficient
10
Reward Loss Coefficient
0.1 (DMC); (others)
Policy Loss Coefficient
0.1 (DMC); (others)
Value Loss Coefficient
0.1 (DMC); 0.5 (others)
Policy Entropy Coefficient
1 x 10-1
Weight Decay
10-4
Max Gradient Norm
5
Discount Factor
0.997
Soft. Target Update Momentum
0.05
Hard Target Network Update Frequency
100
Temporal Difference (TD) Steps
5
27

---

Published in Transactions on Machine Learning Research (05/2025)
Table 9: Key Hyperparameters for MuZero w/ SSL, MuZero w/ Contert, and UniZero (RNN) on Atari.
Hyperparameter
Value
Planning
Number of MCTS Sinulations (sim)
50
Inference Context Length (Hinter)
0 (MuZero w/ SSL); (for other two algo.)
Temperature
0.25
Dirichlet Noise (a)
0.3
Dirichlet Noise Weight
0.25
Exploration Coefficient (ct)
1.25
Visit Count Coefficient (e2)
19652
Environment and Replay Buffer
Replay Buffer Capacity
1, 000 Uniform
Sampling Strategy
Observation Shape (Atari)
(3, 64, 64) (stack1); (4, 64, 64) (stack4)
Observation Shape (Long-torm)
(3, 5, 5)
Reward Clipping
True (Atari only)
Number of Frames Stacked
(stackl); (stack4; Atari only)
Frame Skip
(Atari only)
Game Segment Length
400 (Atari); memory memory_lenghh + 16 (long-term)
Data Augmentation
True
Optimization
Training Context Length (H)
10
Replay Ratio
0.25
Buffer Reanalyze Froquency
Batch Size
256
Optimizer
SGD
Learning Rate Schedule
0.2
0.02 - 0.002
Ye et al. 2021
SSL Loss Coefficient
2
Reward Loss Coefficient
Policy Loss Coefficient
Value Loss Coefficient
0.25
Policy Entropy Loss Coefficient
Number of Reward /Value Bins
101
Discount Factor (7)
0.997
Target Network Update Frequency
100
Weight Decay
10-4
Maximum Gradient Norm
Temporal Difference (TD) Steps
28

---

Published in Transactions on Machine Learning Research (05/2025)
The following computational overhead experiments were conducted Kubernetes cluster with the following specifications: single NVIDIA A100 80GB GPU, 24-core CPU, and 100GB of memory. Under these computational resources and the hyperparameter settings listed in Table g the training time required for UniZero is as follows:
Training an Atari agent for 100k steps takes approximately hours (see Figure which is comparable to the previously reported training time for MuZero.
On the VisualMatch task (with a memory length 500), completing 1M training steps takes approximately 30 hours (sce Figure
It is important to note that the above results were obtained using single GPU without special computational optinizations. Currently, we have implemented multi-GPU version, and experiments indicate that its speedup ratio is close to linear. In the future, we plan to further explore existing Transformer optimization techniques to enhance computational efficiency.
C Multi-task Learning Details
In this section, we evaluate UniZero's capability to seamlessly extend to a multi-task learning setting. While UniZero demonstrates exceptional performance on single-task problems with varying levels of dependency (Section 4.1p, its decoupleddyett-uiifee design enables to scale effectively to multi-task environments. By leveraging a shared transformer backbone, UniZero adaptively captures diverse dependencies across tasks within unified architecture and training paradigm. To validate its multi-lask learning potential, we present results eight Atari games: Alien, Boring, ChopperCommanad, Hero, MsPacman, Pong, RoadRunner, and Seaguest. Unless explicitly stated, the multi-task hyperparameters remain consistent with those outlined in Table g
C.1 Architecture
The observation space for all tasks is standardized and consists of (3, 64, 64) RGB images. We configure full_ allaaaccctacceeeeee
following Bellemare et al. (013)). which yields unified action space with 18 discrete actions across all tasks. The primary architectural difference from the single-task setup is the introduction of independent decision heads and dynamics heads for each task (as described in Kumar et al. (2022)) ), which requires only minimal increase parameters. The core transformer backbone and encoder, however, are shared across all tasks, enhancing parameter efficiency and enabling shared representation learning. In our initial experiments, we also lested the shared head setting. The results indicate that compared to independent head for each game, a shared head leads to overall performance drop of around 50%. Although single, powerful head should thcorctically handle these tasks, achieving this practically requires a more complex design to address gradient conflict issues. Therefore, this paper mainly explores the independent. head setting.
C.2 Training
During training, each task is assigned its own data collector, responsible for sequentially gathering tra- jectories and storing them in separate replay buffers. For gradient updates, we sample a batch of size task task_batch___ size=32 from each task, aggregate the samples into a larger minibatch, and compute the loss batch_size=32
for each task using the objective function defined Equation g The task-specific losses are averaged obtain the total loss, which is then backpropagated update the shared network parameters.
C.3 Results
Performance Comparison. Table 2 and Figure 10 demonstrate that UniZero (multi-task) significantly outperforms both UniZero (single-task) and MuZero (multi-task) in terms of normalized mean and median scores across the evaluated environments. This improvement underscores UniZero's 's scalability and effectiveness
29

---

Published in Transactions on Machine Learning Research (05/2025)
latent world model for generalized agent training. Notably, UniZero achieves comparable sample efficiency across all tasks relative to single-task learning, highlighting its robust multi-task learning capabilities.
Latent State Analysis. To further investigate the success of multi-task learning, we analyze the latent states learned by UniZero using T-SNE visualizations (Maaten & Hinton 2008) (Figure 12]. Specifically, we sample 40 transitions from each game using the final model trained in the multi-task setup. The observation samples are encoded into 768-dimensional latent states using UniZero's 's representation network, which are then reduced to two dimensions via T-SNE. The visualization reveals well-defined clustering tendencies in the latent state spaces for each game, reflecting UniZero's ability to effectively capture task-specific dynamics. However, certain games, such as Alie,, cxhibit more dispersed latent representations. This dispersion may arise from Alien's similarity to other environments, such as MsPaeman (both belonging to the Maze class), allowing greater information sharing across tasks. This shared information likely contributes to Alien's ' significant performance improvement under the multi-task learning setup compared to the single-task scenario.
Effect of Model Size. To investigate the relationship between model size and multi-task learning per- formance, and to explore whether a scaling law exists, we evaluate the impact of varying the transformer backbone size (nlayer=(4, 8, 12}) on eight Atari games in the multi-task sctting. The encoder and head architectures are kept constant across all configurations, while other hyperparameters remain consistent with those outlined in Table S As illustrated in Figure [1] increasing the model size consistently improves sample efficiency across all tasks, demonstrating the scalability of the model as its size grows. These results highlight the potential of larger models to serve as robust latent world models for training generalized agents.
Extended Evaluation. To assess the scalability of UniZero, we expand our test scope to cover the full evaluation set of 26 Atari games. As shown in Figure B and Table [0] the multi-lask model of UniZero achieves normalized human score comparable to that of UniZero trained on single tasks. In contrast, MuZero fails to scale effectively to a large number of tasks. Notably, MuZero performs poorly in multi-task training across the 26 games, making little to no significant progress in most tasks. This limitation likely stems from the substantial differences in dynamical properties across tasks, which make difficult for its recurrent training paradigm adapt seamlessly to such variations. As result, severe conflicts arise during the learning process, ultimately hindering the model's improvement. In contrast, UniZero effectively leverages shared representations across tasks and models task dependencies through its Transformer backbone, thereby maintaining performance levels comparable to dedicated single-task models and demonstrating strong potential in multi-task learning.
Additional Insights. In our preliminary experiments, we explored several multi-task gradient correction techniques, including PCGrad Yu et al. 2020) and CAGrad (Liu et al.) (021). However, these methods yielded minimal performance improvements in our experimental setup and were therefore excluded from the reported results. Additionally, we tested multi-task minibatch sampling strategies, where the sampling ratio for cach task was inversely proportional to the average episode length of the respective task. We also experimented with augmenting the latent state space by introducing task-specific, learnable embeddings. Unfortunately, neither approach demonstrated significant benefits, and as such, they were omitted from the final analysis.
Future Directions. To address the limitations identified, future work will focus on investigating advanced learning dynamics and for balancing tasks during training, (2) optimizing information reuse in MCTS, and (3) integrating multiple modalities and tasks within a unified framework. Furthermore, we aim to study advanced pretraining and finc-tuning methodologics to enhance UniZero's multi-task performance. These directions are expected to provide deeper insights into the challenges of multi-task reinforcement learning and contribute to more robust and scalable solutions.
techniques. Specifically, we plan to explore: (1) improved strategies
30

---

Published in Transactions on Machine Learning Research (05/2025)
1530
2.90
Return
2003
1000
Hspacmin
feaduuree
Seaquest
30-
400
799
Return
2000
390
189
63M 0.14 12M 624 63N
6.4 3.M 63M 0.d
3.M
63N
000
6:3
aIM
C2N
3.M 02M
00M aIM 6aN aar CaN 1.M 0ON BAM 0.3
Env Stops
UniZero
MuZero
Figure 10: Performance comparison between UniZero and MuZero on eight Atari games multi-task settings. UniZero demonstrates comparable sample efficiency across all lasks relative to MuZero in multi-task learning, underscoring its scalability as latent world model for training generalized agents. The solid line represents the mean of three runs, and the shaded areas indicate the 95% confidence intervals.
Alien
Boxing
Choppercommand
Hero
1250
1000
3000
3000
Return
0
2000
500
-10
1000
250
1000
0
Mspacman
Pong
Roadrunner
Seaquest
1500
800
10000
600
1000
Roturn
5000
400
-10
2500
200
0.2M
0.2M
0.3M
0.3M
0.2M Env Stops
layers 12
layers
layers
Figure 11: Effect of model size of UniZero eight Atari games in multi-task settings. Increasing the model size consistently improves sample efficiency across all tasks (nlayer=[4. (4, 8, 12)). demonstrating the potential of UniZero scalable generalized agents.
D Additional Single Task Results
D.1 Experimental Setup
To evaluate the effectiveness and scalability of the proposed UniZero algorithm, we conducted experiments on 26 games from the image-based Atari 100K benchmark. Detailed configuration settings for the Atari environment. provided in Section B.4 Observations represented as (3, 64, 64) for single-frame
31

---

Published in Transactions on Machine Learning Research (05/2025)
t-SNE Environments
207
LEKE Dimarsion
Preinoment the
Heoo
-203
-150
-100
-50
90
100
150
250
SSEE Dimension
Figure 12: T-SNE visualization of latent states learned by UniZero on eight Atari gamess The latent state spaces for the same game exhibit pronounced clustering tendencies, reflecting UniZero's ability to learn task-specific representations effectively. A representative subset of states is shown for clarity.
Table Performance comparison between UniZero and MuZero both multi-tas task and single-task settings 200k environment steps across 26 Atari games. The results indicate that UniZero exhibits higher scalability for heterogeneous multi-lask learning, achieving comparable performance single-task learning, whereas MuZero fails extend effectively to large number of tasks. MT denotes multi-task learning and ST denotes single-task learning. Bold numbers indicate the best performance each metric under the corresponding settings.
Algorithm
Normalized Mean Normalized Median
UniZero (MT)
0.3133
0.1666
MuZero (MT)
0.0424
0.0004
UniZero (ST)
0.3865
0.2154
MuZero (ST)
0.5696
0.2009
RGB images (stack size as (4, 64, 64) for grayscale images with four stacked frames (stack size 4). This configuration differs from the (4, 96, 96) observation format commonly adopted in previous studies such as Ye ct al. (2021); Niu et al. (2024). All implementations are based on the latest release of the open-source LightZero framework Niu et al. 2024).
Baselines. To benchmark the performance of Unizero on the Atari 100K benchmark, we compare it against the following basclines:
MuZero Schrittwieser et al. 2019 The original MuZero algorithm utilizing stack size of 4.
MuZero (Reproduced): Our reimplementation of MuZero, enhanced with self-supervised learning (SSL) and employing stack size of 4. This variant referred MuZero w/ SSL MuZero (Reproduced) NNu et al. 2024).
To evaluate the capacity different algorithms for modeling short-term dependencies, both MuZero and MuZero (Reproduced) are configured with stack size of 4, while the proposed UniZero operates without stacked frames (stack size 1). All implementations are trained using uniform hyperparameters across all games, with further game-specific tuning.
32

---

Published in Transactions on Machine Learning Research (05/2025)
Assault
Bankhelst
043
145
Retum
330
403
169
Hattlezone
Baxing
Breakout
Choppercommand
Crazyclimber
1430
1390
15300
Return
2030
7330
1503
2.00
003
2003
200
Demonattack
Freeway
Frostbite
Gopher
Hero
143
1222
Return
300
2000
1030
Jamesband
Kangaroo
Kruil
Kungfumaster
Mspacman
300
13:0
4003
Return
200
2003
1090
100
2003
200
300
Pong
Privateeye
Seaquest
10-
314
Return
530
622
-208
Upndown
Return
3000
2008
1800
Steps
UniZero (multi-task)
UniZero (single-task)
Figure 13: Performance comparison between multi-task and single-task settings 26 Atari games. Multi-task training yields comparable sample efficiency across all tasks relative to single-task learning, further validating UniZero's is efficacy as scalable generalized agent.
For Lasks that demand long-term dependency modeling, we exlend the evaluation by comparing UniZero against the original MuZero and the transformer-base SAC-GPT algorithm (Ni et al. (024).. The SAC-GPT framework integrates transformer architectures with actor-critic methods, cnabling the modcling of memory and credit assignment in reinforcement learning, thereby highlighting its potential for addressing long-term dependencies in complex RI. environments.
D.2
Atari 100K Results
Table 11 provides a comprehensive comparison of three methods: UniZero (stack size 1), MuZero (Re- produced) (stack size = 4), and the original MuZero as reported in Schrittwieser et al. (2019). The results demonstrate that UniZero achieves a higher human-normalized median score than MuZero (Reproduced), out- performing the latter in 15 out of 26 Atari games, while maintaining comparable or slightly lower performance in the remaining environments. Notably, both UniZero and MuZero (Reproduced) are implemented within the
33

---

Published in Transactions on Machine Learning Research (05/2025)
same LightZero framework, ensuring a fair and controlled comparison by using identical hyperparameters across all games.
Figure 14 resents the full performance curves, further corroborating that UniZero consistently surpasses MuZero (Reproduced) in terms of human-normalized median scores. These results highlight the ability of UniZero effectively model both short- and long-term dependencies, which a critical factor achieving robust performance on the Atari 100K benchmark.
Performance comparison of UniZero, MuZero (Reproduced), and the original MuZero on the Atari 100K benchmark. UniZerv achieves higher human-normalized median score than MuZero (Reprodueed) oulperforming the latter out of Atari games. The results for the original MuZero are directly taken from Schriltwieser el al. (2019) and are provided for reference. Both UniZero and MuZeru (Reproduced) are reimplemented using the LightZero framework under identical hyperparameter settings, ensuring fairness in comparison. Bold entries denote the superior method between UniZero and MuZero (Reproduced), while underlined values indicate the overall best best-performing approach across all methods.
Table 11:
Game
Raudom
Human
MuZero
MuZero (Reproduced)
UuiZeru (Ours)
Alien
227.8
7127.7
530.0
300
600
Amidar
5.8
1719.5
39
90
96
Assault
222.4
742.0
500
609
608
Asterix
210.0
8503.3
1734
1400
1216
BankHeist
14.2
753.
193
400
BattleZone
2360.0
37187.5
2688
7587
11410
Boxing
12.1
20
Breakout
30.5
48
ChopperCommant
811.0
7387.8
1350
1050
2205
CrazyClimber
10780.5
35829.4
56937
22060
13666
DemonAAtack
152.1
1971.0
3527
4601
991
Freeway
0.0
29.6
22
12
10
Frosthite
65.2
4334.7
255
260
310
Gopher
257.6
2412.5
1256
346
853
Hero
1027.0
30826.4
3095
3315
2005
Jamesbond
29.0
302.8
90
405
Kangaroo
52.0
3035.0
200
1885
Krull
1598.0
2665.5
4891
5191
4484
KungFuMaster
258.5
22736.3
18813
6100
11400
MsPacman
307.3
6951.6
1266
1010
900
Pong
-20.7
14.6
-7
-15
-10
PrivateEye
24.9
69571.3
56
100
500
Qbert
163.9
13155.00
3952
1700
1056
Rondernner
11.5
7845.0
2500
4400
1100
Seaquest
68.4
42054.7
208
620
UpNDown
533.4
11693.2
2897
1213
2823
Normalised Mean (1)
0.000
1.000
0.56
0.44
0.39
Normalised Median (1)
0.000
1.000
0.23
0.13
0.22
D.3 DMControl Results
Figure 15 shows the learning curves for all 18 tasks in the Proprio Control Suite of DMControl. Each solid line represents the mean performance across three seed runs, and shaded regions indicate the 95% confidence intervals. UniZero, leveraging sampled policy iteration (Hubert et al.) 2021b), achieves a higher humau-normalized score compared to the state-of-the-art Dreamer V3 (Halner et al. 2023), highlighting its ability to handle continuous action spaces and diverse control tasks.
34

---

Published in Transactions on Machine Learning Research (05/2025)
Alien
Amidar
Assault
Asterix
Bankheist
100
Recurn
1000
200
Battlezone
Baxing
Breakout
Choppercommaand
Crazyclimber
1200
18000
more
Retum
Freeway
Gopher
1000
8:00
stse
20h
Jameshond
Kangarba
Krull
Kungfumaster
Map Mspacman
1850
Retur
sace
1206
1000
Pong
Privatesye
Qbert
Roadrunner
Seaqu est
J000
Return
2000
600
-20
1000
210
Upndown
1000
28C
Env Stops
MuZero
UniZero
Figure 14: Performance Comparison on 26 Atari Games between UniZero and MuZero (Reproduced) in single-task setting. UniZero achieves higher human-normalized median score. Solid lines represent the mean of three different seed runs, while shaded areas denote the 95% confidence intervals.
35

---

Published in Transactions on Machine Learning Research (05/2025)
Acrozct
Ball Cup Catch
Cartpole Balance
Balance,
1925
1202
Return
942
200
925
212
400
0.50M
0.50M
0.10M
0.40N
Cartaole Swingup
Cartoole Swingup. Sparse
Chestan Run
Finger Spin
1850
302
802
Retumm
900
202
400
0.30M
6.10M
0.6ON
Finger Jam_Eary Easy
Finger Turn, Hard
Hooper Hop
Hopper Stand
002
1050
002
200
350
50-
20c
800
0-
202
6.00M
0.100
0.20
0.20M
DAON
0.50M
6.00
0.10M
0.00
0.20H
C.00M
0.50M
2.00N
0.10M
0.20M
0.30M
6.0
0.52M
0.00N
0.10M
20N
0.32
0.40N
C.50M
Switggup
Easy
000
1250
1000
1000
300
Return
1800
550
0-
230
6.00M
0.10M
0.20
0.20M
SHOM
0.5MM
5.00
0.1M
5.00
0.20M
CAMM 6.5M
5.00
0.CH C2O
0.30M
0.40M
0.50
0.00N
0.10M
0.32M
0.40M
2.50M
Walker Stand
Walker
850
520
0.00M
0.20M
0.50M
0.00
C.O
Env Steps
UniZera
Figure 15: Detailed Training Curves of UniZero on 18 DMControl Tasks. UniZero demonstrates superior performance against DreamerV3, underscoring its effectiveness in continuous control settings. Solid lines denote the mean performance over three different seed runs, while shaded areas represent the 95% confidence intervals.
36

---

Published in Transactions on Machine Learning Research (05/2025)
E
Additional Ablation Study and Analysis
E.1
Ablation Study Details
In Section 4.5 we evaluate the effectiveness and scalability of UniZero's core designs through a series of ablation experiments. These experiments inchude investigations into training context lengths, normalization methods, and decode regularization in the Pong environment and the VisualMatch task.
Additionally, we provide ablation studies on the target world models for both tasks and explore the impact of Transformer depth in VisualMatch (memlen=500).
Below, we present comprchensive experimental details and key observations.
Model Size Across Different Training Context Lengths (H = 5, 10, 40): The number of layers in the Transformer backbone is varied, while the number of attention heads is fixed at 8. We examine how context length affects performance in both Pong and VisualMatch.
Latent Normalization: We compare three normalization methods: the default SimNorm, Softmar, and Sigmoid. These methods are applied to both the encoded latent state and the output of the dynamics network (i.e., the predicted next latent state). Our initial experiments indicate that LayerNorm performs comparably to SimNorm. However, SimNorm's intra-group Softmax mechanism enables more attribute- specific and generalizable representations (Lavoie et al.) 2022), which benefits attribute interpretation and multi-task fine-tuning. Therefore, we default to Sim Norm, while retaining LayerNorm as viable alternative.
Decode Regularization: We introduce decoder function to map latent states back into the observation space:
Decoder: ô = do(; Maps latent states observations for regularization.
Training includes an auxiliary objective:
deccode = llo+ - do(2))llı + (O,, 2, = ho(o))
where the first term represents In reconstruction loss, and the second term is perceptual loss, defined in Ni et al. (2024). For these experiments, the decode regularization loss cocfficient is set to 0.05. Notably, in VisualMatch, only the Lı reconstruction loss is applied.
Target World Model: We evaluate three configurations for the target world model:
Soft Target (default):: Leverages an Exponential Moving Average (EMA) target model (Mnih et al. 2013) for both the target latent state and target. value.
-Hrd Target: Updates the target world model by hard-copying parameters every 100 training iterations.
-No Target: Removes the target world model entirely, using the current world model to generate the target latent state.
Based on these ablation studies, we derive the following key insights:
(1) Training Context Length and Transformer Depth:
In the Pong environment, shorter inference contexts (Hinfer = 4) outperform longer contexts (Hinfer 8) across all Transformer depths. This suggests that shorter contexts are sufficient for Atari tasks. Consequently, we set Hineer = for all Atari experiments.
In VisualMatch, however, the training context length must match the cpisode length to enable the agent retain memory of the target color from the first phase. Accordingly, the training context length is set to 16 + memory menghy_lenghh.
Figure 16 illustrates that deeper Transformer backbones slightly improve performance, indicating that increased capacity better captures long-term dependencies.
37

---

Published in Transactions on Machine Learning Research (05/2025)
Wotta (000
Target: World Model
Target World Model
No: Suged
1.0
Turpet
04
0.8
Sa00ts hate
Return
2.5-
24.4
-10
040
-20
3M
0.3M
0.2M
0.3M
0.4M
0.5M
090
0.1.M
0.29
0.3M
D-4N
0.50
Env Steps
Env Steps
Figure 16: Impact of Transformer depth in VisualMatch (Memory Length 500). Performance im- proves slightly with the number of lay- ers in the Transformer backbone, indi- cating that deeper architectures better capture long-term dependencies.
Figure 17: Ablation results for the target world model. Left: Pong, Right: VisualMatch (Memoryleenghh 60). Soft target models yield the most stable performance. The horizontal axis shows Eav Steps, while the vertical axis represents the Return Success Rate over 10 episodes. Results are averaged over runs, with shaded areas denoting 95% confidence intervals.
(2) Latent Normalization:
The SimNorm method (Hansen et al. 2023) achieves the best performance, followed by Softmaz, while Sigmoid fails to converge.
These results underscore the importance of proper normalization in the latent space for training stability, Specifically, SimNorm enforces fixed L norm latent states, which introduces natural sparsity and stabilizes gradient magnitudes.
Without normalization, gradient explosion is frequently observed.
33) Decode Regularization:
Decode regularization has minimal impact performance in both Pong and VisualMotch.
This suggests that latent states primarily encode task-relevant information, rendering the reconstruction of original observations unnecessary for effective decision-making.
(4) Target World Model:
As shown in Figure 17 the soft target model delivers the most stable performance.
The hard target model exhibits some instability, while removing the target world model leads to non-convergence in Pong and NaN gradients in VisualMatch.
This behavior aligns with the role of target networks algorithms such as DQN (Miih et al. 2013 where the absence of target stabilization mechanisms often causes divergence.
E.2 World Model Analysis
VisualMatch. In Figure 18 and Figure 19 we present the predictions of the learned world model in one success and one fail episode of VisualMatch (Memorggengghh=60) respectively. The first row indicates the predicted reward and true reward. The second row displays the original image frame. The third row outlines the predicted prior policy, and the fourth row describes the improved (MCTS) policy induced by MCTS based the prior policy. For the sake of simplicity, we have only illustrated the first two steps (t = (2)3)) and the last two steps (t =t=(( {60, 61)) of the distraction phase. Please note that at each timestep, the agent performs the action with the highest probability value in the fourth row. As observed, the reward is accurately predicted in both cases, and the MCTS policy has shown further improvement compared to the initial predicted prior policy. For example, in Figure [8] at timestep 75, action 3, which represents moving to the right, identified as the optimal action because the target color, green, is located on the agent's right side. While the predicted prior policy still allocates some probability to actions other than action 3, the MCTS policy refines this distribution, converging more towards action 3.
38

---

Published in Transactions on Machine Learning Research (05/2025)
Thu Souuth
0.7
50
1.4
Acbin1
0--
50
64 61
60
78
Figure 18: Predictions of the world model in one success episode of VisualMatch (MemoryLength=66)) The first row indicates the predicted reward and true reward. The second row displays the original image frame. The third row outlines the predicted prior policy, and the fourth row describes the improved (MCTS) policy induced by MCTS based on the prior policy. For the sake of simplicity, we have only illustrated the first two steps (L = [2,3]) and the last two steps (t = {60, 61]) of the distraction phase. At timestep 75, action 3, which corresponds to moving to the right, is identified as the optimal action because the target color, green, is located on the agent's right side. Although the predicted prior policy assigns some probability to actions other than action 3, the MCTS policy refines this distribution, converging more decisively towards action 3.
Figure 21 shows the attention maps of the trained world model. It can be observed that in the initial layers of the Transformer, the allention primarily focused on the first time slep (which contains the larget color that needs to be remembered) and the most recent. few time steps, mainly for predicting potential dynamic changes. In higher-level layers, sometimes, such in Layer3-Head2, the attention is mainly concentrated the current time step, whereas at other times, such as in Layer4 Headd4, there is relatively broad and dispersed aetetion distribution, possibly indicating the fusion of some learned higher-level features.
Pong. Similarly, in Figure 20 we present the predictions of the world model in one trajectory of Pong. The first row indicates the predicted reward and true reward. The second row displays the original image frame. The third row outlines the predicted prior policy, and the fourth row describes the improved (MCTS) policy induced by MCTS based on the prior policy. Please note that the image in the second row (original image) has already been resized to (64,64) from the raw Atari image, there may be some visual distortion. At each timestep, the agent performs the action with the highest probability value in the fourth row. Throughout all timesteps, the true reward remains zero duc to the absence of score events. Unizcro's world model can accurately predict this, with all predicted rewards consistently remaining zero. Al the 8th timestep, the agent controlling the right green paddle successfully bounces the ball back. At the 7th timestep, the agent should perform the upward action 2; otherwise, might miss the opportunity to catch the ball. The MCTS policy further concentrates the action probability on action 2 compared the prediction policy, demonstrating the policy improvement process of MCTS.
39

---

Published in Transactions on Machine Learning Research (05/2025)
Thu Souuth
0.7
60
Acbin1
0.-
50
41
11
Figure 19: Predictions of the world model in one fail episode of VisualMatch (MemoryLength=660) The first row indicates the predicted reward and true reward. The second row displays the original image frames. The third row outlines the predicted prior policy, and the fourth row describes the improved (MCTS) policy induced by MCTS based on the prior policy.
In Figure 22 we plot the attention maps in one trajectory (train context length is 10, with each time step consisting of two tokens, namely the latent state and the action) of Pong. It can be observed that across various levels, attention is primarily on data from the most recent frames. This is closely related to the short-term dependency characteristic of Pong. Utilizing information from only the recent frames is sufficient. for dynamic prediction and policy-value learning.
E.3 Covariate Shift in UniZero
MuZero and its recent. variants Hubert et al. (2021a); Ye et al. (2021) are founded on the value-oquivalence principle that guides world model training. These models comprise two primary components: a dynamics model that prediets the next latent state and reward, and decision model that estimates both the policy and corresponding value. In the MuZero-style architecture, only the first step directly processes environment. observations (e.g., multiple stacked frames or encoded history), while subsequent steps rely on recurrent predictions of latent states. This design minimizes discrepancies between training- time encoded observations and the predictions employed during Monte Carlo Tree Search (MCTS) at inference.
However, as noted in Ghugare et al. (2022b), MuZero-style latent models experience significant performance degradation with an increasing number of unroll steps during recurrent training. Our proposed UniZero addresses this limitation by employing Transformer-bast based latent model that integrates new information at every step through advanced representation learning techniques, including self-supervised losses. Although this improvement enhances the overall representation and alleviates degradation over long unrolls, it inherently introduces covariate shift, consistent with the "no free lunch" principle.
In UniZero, the root node utilizes a key-value cache (kv-cache) retain historical information, thereby preserving an unbiased process. Furthermore, during the search phase, the depth of the search tree is constrained, with long-term value estimation delegated a dedicated value head. This design effectively
40

---

Published in Transactions on Machine Learning Research (05/2025)
Thu Somuth
60
60
Figure 20: Predictions of the world model in one trajectory of Pong. The first row indicates the predicted reward and true reward. The second row displays the original image frame. The third row outlines the predicted prior policy, and the fourth row describes the improved (MCTS) policy induced by MCTS based on the prior policy.
minimizes biases in value and policy cstimation that may arise from covariate shift. Future rescarch will focus on both quantifying and mitigating the effects of this covariate shift, as well as exploring hybrid architectures that combine the strengths of recurrent and Transformer-based models to balance the trade-offs between extended unroll training and covariate shift.
F
Comparison with Prior Works
Comparison with recent approaches in world modeling. To provide a clear comparison, we present Table 12 outlining the key differences between UniZero and recent approaches (Hansen al. 2023 Schrittwieser et al. 2019 Hafner et al. 2023 Micheli et al. 2022 Robine al.] 2023 Zhang et al. 2023 in world modeling, The attributes considered include the type of sequence model used, the input information introduced during single timestep, the method for obtaining latent representations, the approach to policy improvement, and the training pipeline.
Sequence Model: The architecture employed for modeling sequences.
Input: The type of information fed into the sequence model at each timestep, where "Latent history" refers to the recurrent /hidden state as described in the respective papers.
Latent Representation: This refers to the technique employed to extract embeddings from each obscrvation. For instance, an 'Encoder" might be a neural network such as a Convolutional Network (ConvNet) for processing images or a Multi- Layer Perceptron (MLP) for handling vector observations. The term 'VQ-VAE" (van den Oord et al. 2017) denotes the voctor-quantized VAE, which utilized obtain a discrete code for the observation. Similarly, 'Categorical VAE" (Hafner et al. 2023) represents the discrete VAE, which is used to derive the discrete distribution of the observation.
41

---

Published in Transactions on Machine Learning Research (05/2025)
Ques Maog Layer
58
Query
2
130
110
110
130
121
121
121
132
192
132
143
143
121
177
143
12
Head
Head
Head
11-
22-
22-
22
23-
45-
25-
23
Quany L
64-
Qvery Layer 2
Layer
46-
Guery Layer
64
55
55-
55
55
66
66
66
88-
110
110
113
324
133
172
110
121
132
143
Key - Head
Key Head 2
Key Head
Key - Head
13
11
11
15.
130
130
121
121
121
132
132
132
141
263
145
141
8
= N1
31
R
18
R R.
Key
Head
Key
Head 2
Key
Head
Key Head
Queny 1ayer
Quer Layer
55
Query Layer
66
77
77
m1
59
99
59
130
110
110
130
121
121
121
132
132
122
27
110
121
110
Head
Head
Figure 21: Attention maps in one success episode of VisualMatch (Memorylength=60) (Note that the train contezt length 11+6--0--- 13 76, with each time step consisting two tokens, namely the latent state and the action.). It can be observed that in the initial layers of the Transformer, the attention is primarily focused the first time step (which contains the larget color that needs to be remembered) and the most recend few time steps, mainly for predicting potential dynamic changes. In higher-level layers, sometimes, such in Layer3-Head2, the attention is mainly concentrated on current time step, whereas at other times, such as Layer4-Head4, there relatively broad and dispersed attention distribution, possibly indicating the fusion of some learned higher-level features.
Policy Improvement: The method for enhancing the policy, with "PG" standing for Policy Gradient methods (Schulman et al. 2017 Hafner et al. 2023 and 'MPC' standing for Model Predictive Control
Hansen et al. 2022).
Training Pipeline: The training process involves a 'two-stage" approach, where we first train the world model and then use the learned model to train the policy (bchavior) through imagination. On the other hand, "model-policy joint training' refers to simultaneously learning the world model and the policy (and
42

---

Published in Transactions on Machine Learning Research (05/2025)
Figure 22: Attention maps trajectory (train_contect content Lengtl_eeghhhhheeeeee is 10, with each time step consisting of two tokens, namely the latent state and the action.) of Pong. It can be observed that across various levels, attention is primarily on data from the most recent frames. This is closely related the short-term dependency characteristic of Pong. Utilizing information from only the recent frames is sufficient for dynamic prediction and policy-value learning.
value), rather than following two-stage process. This joint training approach offers several benefits, discussed in Eysenbach al. (2022); Ghugare al. (2022b;; Grill el. al. (2020).
Table Comparison between UniZero and recent model-based RL approaches. The main difference between UniZero and MuZero is highlighted in bold.
reporsentation
Pipeline
TWM
Trasiormes
Nr. Piresin KIT0
Latest observttin, oction, cesund
Curoprial VAE
PG Drvame V2 Raleer di
XW0
teoellge
1815
Mi
Trusdamacs Venammaunn
Lakes, sharaation, sciass Latser, cheraation, actin Latson, cheraation, artun Latern, cherattinn, artun laterr, actio) Latent, observation, aetion
VQ-VAE Gragenaa-VVA Seripnsal-VA..
PG DeoameeV
tnoolige
Dreames
Gill
PG beuners
reneeage
STORM
PG of Derameel
reseaage
TD-MPC4
MILP
Baroder (wzh Stullorm)
MPC Barens 20z
recetage
Milleoo
Emsoder
KTLTS
Modelgatyy Juit traiiing
Trandormme
2013
Rucder (with SheNorr))
MELTS
Midelgolty Join trahiing
Comparison with MuZcro's Extensions. In recent years, MuZero and its extended algorithms have significantly improved efficiency and stability across various scenarios through a series of innovations. To facilitate understanding of the relationship between UniZero and the MuZero family of algorithms, we provide a qualitative comparison across the following key dimensions:
Action Space: The lype of action space supported by the algorithm (continuous or discrete).
Simulation Cost: The computational cost of simulating latent states during the search process.
Sample Efficiency: The ellectiveness of utilizing sampled data for learning.
Explicit Stochasticity Modeling: Whether the algorithm explicitly models the stochasticity in the environ- ment.
The ideas of each extended algorithm summarized follows:
43

---

Published in Transactions on Machine Learning Research (05/2025)
Sampled MuZero (Hubert et al 2021b:: A sample-based policy iteration framework that applies to any type of action space. It computes improved policies over subset of the original action space and probabilistically converges to the optimal policy over the entire action space as the number of samples increases.
Gumbel MuZero (Danihelka et al. 2022 is designed to enhance performance in environments with low simulation costs by leveraging the Gumbel-Top-k trick (Kool et al. 2019) to select actions that guaranlee policy improvement. It seamlessly integrates the original visit counts distribution with MCTS searched values to more informed decision-making and improved performance.
EfficientZero Ye ct al. 2021 Incorporates techniques such as self-supervised consistency loss, end-to- end prediction of value prefixes, and model-based off-policy correction. These enhancements significantly improve sample efficiency, achieving outstanding performance in tasks like Atari 100k.
Stochastic MuZero (Antonoglou et al. 2021): Introduces stochastic modeling (including afterstates) and employs stochastic tree search to effectively handle randomness in the environment. It demonstrates superior performance in tasks such as 2048, Backgammon, and Go.
Table 13
provides a qualitative comparison of UniZero and the aforementioned MuZero family algorithms across key dimensions, It worth noting that the improvements introduced in the MuZero family extended algorithms are largely orthogonal to those UniZero, making them easily transferable to UniZero. We consider integrating these extensions as part of future work.
Table 13: Qualitative Comparison of UniZero and MuZero Family Algorithms Across Different Dimensions.
Algorithm
Action Space
Simulation Cost
Sample Efficiency Explicit Stochasticity Modeling
UniZero
contimuuss/disse /isccete continuous/ /iitoouu//////eeeeee discrete
medium low
medium medium
Sampled MuZero
Gumbel MuZero
discrete
low medium
medium
n1
EfficientZero
discrete discrete
high
Stochastic MuZero
medium
medium
yes
44


Reviews:
[{"review_id": "UDAOz1os7H", "reviewer": "Reviewer_cD6a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- Outperforms MuZero and SAC-GPT on VisualMatch, achieving >80% success rate at memory length=60 vs. MuZero’s ~20%. Transformer architecture mitigates recurrent model limitations (e.g., error accumulation in MuZero w/ Context).\n-  Single-model training on 26 Atari games matches single-task performance (Figure 13), with mean normalized score of 0.455 vs. 0.322 for single-task UniZero (Table 2). Figure 12 show task-specific clustering, indicating effective shared representation learning.\n- Matches MuZero in Atari 100K (Figure 5) and surpasses DreamerV3 in DMControl (Table 3: UniZero mean=787.2 vs. DreamerV3=743.7).\n- SimNorm (latent normalization) proves critical for stability (Figure 7), outperforming Softmax/Sigmoid. No performance drop when omitting observation reconstruction losses, aligning with \"decision-relevant latent encoding\" hypothesis.\n- Authors have discussed limitations of the work which is highly appreciated.\n\n### Weaknesses\n\n- Method has modest scalability in multitask learning, only 13% mean improvement over single-task suggests limited cross-task synergy. Also, method fails to outperform single-task UniZero in 7/26 Atari games (e.g., Breakout: 8 vs. MuZero's 48 [Table 10]).\n\n- The method is claimed to be competitive in standard single-task RL, however, the method Underperforms MuZero in 11/26 Atari games (e.g., RoadRunner: 1100 vs. MuZero's 4400 [Table 10]). Fails short-horizon tasks like Pong (-10 vs. MuZero's -7 [Table 10]).\n\n- Authors suggest that SimNorm is critical for stability, however they don't make comparisons with other norm methods like LayerNorm. The ablation only has the experiment with and without SimNorm.\n\n### Suggestions\n\n- Authors can adopt rotary or relative positional embeddings to enhance temporal modeling.\n- It will be interesting if authors can devise a strategy to adjust  $H_{infer}$ automatically per task.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "UniZero introduces a transformer-based latent world model integrated with Monte Carlo Tree Search (MCTS) to address limitations of MuZero-style algorithms in heterogeneous environments. Key innovations include:\n\n - Disentangled Latent Representation: Separates latent states from implicit history using a transformer backbone, enabling full trajectory utilization during training.\n- Joint Optimization: Combines policy, value, reward, and latent dynamics prediction into a single loss function.\n- Scalable Architecture: Employs domain-specific encoders and a shared transformer for multitask learning.\n- Efficient Planning: Uses a Key-Value (KV) cache for long-term memory and MCTS in latent space.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "While claims are generally supported by the evidence, I request authors to temper down claims about single-task performance and scalability. I have mentioned the reasons in the weakness section (For instance, underperforming MuZero for single-task RL in ~42% of the games).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JP54gt0l0D", "reviewer": "Reviewer_4J8R", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The novelty is not very high, but I do think that exploration of the incorporation of Transformers into MuZero is a topic of interest to the TMLR community. As a result, it seems like a good fit with the venue. \n- The authors perform a good amount of experiments and seem to show consistent improvements or performance matching with MuZero.\n- The results seem very stark as the environment becomes more non-Markovian as in the right side of Figure 2 and Figure 4. This isn't that deep of an insight, but it does makes a lot of sense.  It is also nice to see it does not hurt in Markovian environments. \n\nWeaknesses \n- I thought the discussion around the limitations of MuZero could be a lot clearer. It comes across as proposing some vague intuitions, which aren't always validated. For example, the authors write \"the recurrent design introduces an intrinsic entanglement between latent representations and historical information, resulting in a bottleneck that impedes efficient information propagation\". Can this particular perspective about entanglement be directly measured or validated? \n- The authors also do not engage with the computational efficiency that likely motivated this original design choice in MuZero. It is unclear to what degree the amount of computation and parameters are properly accounted for between models in experiments. My assumption is that UniZero is much more expensive computationally in the experiments provided. The authors also write \"the architecture suffers from an under-utilization of trajectory data during training, which restricts its ability to fully exploit the accumulated experiential data\" which again seems to imply more compute will be used for UniZero. I just think the authors should be much more clear and straightforward about this tradeoff. The authors are motivated by a setting where they want to maximize sample efficiency without regard for compute per step whereas MuZero was trying to computationally efficiently do better in self-play scenarios like Go and Chess. In their setting UniZero would have gotten through fewer games with same computational budget, which could have resulted in being less practical given their set of concerns. \n- The authors do a good job of motivating why UniZero should help for POMDPs, but I was not really clear on what the benefits should be in the multi-task scenario. There should be more discussion of this. It appears like the authors just thought it was an interesting setting (which I agree with) but there is not really a clear expectation on what UniZero uniquely brings to this setting more than any other. \n- I found the Stack = 1 pong experiments very compelling on first glance, but something didn't sit well with me about the idea that every ablation does so much worse than UniZero. I understand this better now looking at Table 1. What I would be interested to see is UniZero -context or UniZero -SSL rather than MuZero +context or MuZero +SSL. Because the usage of the RNN makes such a big difference on its own, we cannot understand the impact of these other ablations in the partially observable context. \n- The related work was a real weak point for me and I believe it must be revised before this paper can be published.  The authors mention relevant baselines in Sampled MuZero, Stochastic MuZero, EfficientZero, GumbelMuZero, and LightZero, but the connections between these works and UniZero are not clearly stated. Maybe something akin to Table 1 would be appropriate here? The authors also say \"The concept of world models, first proposed in Ha & Schmidhuber (2018),...\" this statement is obviously incorrect.   Ha and Schmidhuber 2018 is a weird reference to consider the first as model based RL existed in the literature for decades prior, maybe it can be considered the first \"modern\" model (not even totally sure about this), but the current framing seems to lack scholarly rigor. If you read their paper, you will see that they make no such claims and cite many much older references. \n- There are also some seemingly incorrect statements made when presenting the results in Figure 13. The authors write \"The results in Figure 13, reveal that a single model trained under a online multi-task setting achieves normalized mean scores comparable to those obtained from single-task training\", but the normalized mean score is never actually reported. The authors also write \"Table 2 and Figure 13 demonstrate that UniZero (MT) outperforms both UniZero (ST) and MuZero (MT) in terms of normalized mean and median scores across the evaluated environments within the 400K Env Steps setting.\" This statement appears unsubstantiated in the case of Figure 13 as MuZero is not compared against and the normalized mean scores are not provided. \n- For multi-task learning I felt the use of independent decision and dynamics heads wasn't really justified. Did the authors try the model without this and find that it did not work? I am concerned that the separation we see in Figure 12 is just a direct result of using these different task specific heads, requiring the use of differently synced representation spaces for the different tasks to accommodate the different heads. I believe the observation spaces are sufficiently different for these tasks such that there should be no intrinsic need to have separate heads with a sufficiently powerful monolithic neural network.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new algorithm UniZero to improve MuZero style architectures. The authors identify limitations of the previously established architecture for MuZero and consider what can be gained by incorporating in Transformers for POMDP environments. UniZero compares favorably to MuZero in partially observable settings and experiences bigger gains as the environment becomes more non-Markovian with respect to the observations while maintaining performance in Markovian environments. The authors also take a deeper look into important design choices in this setting.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Below I said the claims made in the submission are supported when forced to choose yes or no. This is not currently totally true, but I believe that the authors can makes some reasonable changes that would make it a fully true statement:  \n\n- Important for substantiating claims: Figure 13 should be updated with comparisons to MuZero and to include normalized mean scores. \n- Important for substantiating claims: The authors should provide empirical evidence directly supporting their claim regarding intrinsic entanglement. Is this just a fancy way of making a standard argument for dynamic attention as opposed to recurrent processing? \n- The authors should provide a detailed look at the computational efficiency of UniZero vs. MuZero. This should be incorporated into the discussion of scaling properties. \n- The authors should provide a direct justification for why UniZero helps in multi-task settings specifically. Or is the argument the same as in the MDP setting i.e. that UniZero doesn't hurt? Their should also be a more detailed comparison with the results with out task specific heads. \n- For the pong experiments, UniZero -SSL nd UniZero -context should be provided as additional ablations. \n- The authors should be very explicit about the differences between UniZero and Sampled MuZero / Stochastic MuZero / EfficientZero, GumbelMuZero / LightZero. \n- The discussion of world models should be updated to accurately reflect the prior literature before Ha & Schmidhuber (2018). \n- The authors should be more clear about what the blue and red coloring implies in and around Equation 3. Does this carry a consistent meaning to the use of the same colors in Equation 2 and Table 1?\n\nComment Post Revisions: After having gone through the author's response to my concerns and revisions to the document, I believe the authors have done a very good job addressing my concerns and incorporating in the suggested revisions.", "broader_impact_concerns": "There is no broader impact statement, but I do not believe it is necessary for this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MnAIHQJZC7", "reviewer": "Reviewer_ijGr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths: \n* The idea of leveraging transformers as dynamics (\"world\") model for MCTS is interesting.\n\n## Weaknesses\n\n### Writing\nIMO, the main weakness of the paper is that it makes a lot of vague, imprecise and confusing claims and I found it on a first pass unnecessarily hard to understand what the actual motivation (and contribution) of the paper was. For example, IMO, the main justification for using a transformer is to allow MuZero-style algorithms to apply to POMDPs where history is important. However, this is not clearly stated in the paper (POMDPs are only tangentially mentioned), instead the authors make claims about \"scale[ing] in heterogeneous scenarios with diverse dependencies and task variability\" or \"intrinsic entanglement between latent representations and historical information\" or \"under-utilization of trajectory data\". \n\n### Experiments\n\n* Baselines: I believe the obvious baseline, which is missing, is to use either a RNN or a Transformer to simply encode the history into a latent state `z_t` and use this as the initial state for the standard MuZero algorithm. In other words, replace the Encoder in MuZero with an RNN. Instead, the authors use \"MuZero w/ Context\" which ignores the last `k` states from the environment.\n* On Figure 2: I find it worrying that UniZero-RNN and and MuZero w/ Context cannot solve the fully observable game Pong.  For \"MuZero w/ Context\", I believe (see above) that this might make sense and this baseline should be replaced. But For UniZero-RNN I don't see why it shouldn't solve Pong.\n* Similarly, I'm not sure I understand why UniZero outperforms the baselines on (standard) Atari and Mujoco, both of which are fully observable environment where the Transformer should not bring any advantages (as historical states are irrelevant).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes \"UniZero\", a variation of MuZero which uses a Transformer as world model. The main motivation is to allow the application of MuZero-style approaches to POMDPs in which the current state does not carry enough information.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please make the motivation, and limitations of UniZero clearer throughout the paper. Furthermore, for non-POMDP environments, please explain why one should expect UniZero to outperform a non-transformer based approach.\n\nBefore those changes, I cannot recommend acceptance. Furthermore, as this would require a sizable change to the current paper, I'd recommend a re-review after those changes have been made.\n\n\nEDIT: After extensive discussions, the authors have added several of the requested changes.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new method to address weaknesses of the MuZero agent. The transformer-based world model is a better fit for POMDP environments. Initially reviewers criticized the presentation of the paper, motivation and lack of experimental results to support the claims. Authors provided additional experiments, rearranged the paper and added significant content through rebuttal phase. All reviewers agreed that concerns are addressed and the work is ready for publication. I recommend authors to make the code public for reproducibility.
Decision: Accept (poster)