DyMoDreamer: World Modeling with Dynamic Modulation
Authors: Boxuan Zhang
Runqing Wang
Wei Xiao
Weipu Zhang
Jian Sun
Gao Huang
Jie Chen
Gang Wang
Venue: NeurIPS 2025 poster ()
Keywords: Model-based reinforcement learning
world model
dynamic modulation
recurrent state space model
Primary area: reinforcement_learning
TL;DR: This paper introduces a novel world model-based reinforcement learning algorithm that leverages the dynamic modulation to enhance the world modeling process, achieving a new record on the Atari 100k, the DMC Suite, and the Crafter benchmark.

Abstract: A critical bottleneck in deep reinforcement learning (DRL) is sample inefficiency, as training high-performance agents often demands extensive environmental interactions. Model-based reinforcement learning (MBRL) mitigates this by building world models that simulate environmental dynamics and generate synthetic experience, improving sample efficiency. However, conventional world models process observations holistically, failing to decouple dynamic objects and temporal features from static backgrounds. This approach is computationally inefficient, especially for visual tasks where dynamic objects significantly influence rewards and decision-making performance. To address this, we introduce DyMoDreamer, a novel MBRL algorithm that incorporates a dynamic modulation mechanism to improve the extraction of dynamic features and enrich the temporal information. DyMoDreamer employs differential observations derived from a novel inter-frame differencing mask, explicitly encoding object-level motion cues and temporal dynamics. Dynamic modulation is modeled as stochastic categorical distributions and integrated into a recurrent state-space model (RSSM), enhancing the model's focus on reward-relevant dynamics. Experiments demonstrate that DyMoDreamer sets a new state-of-the-art on the Atari $100$k benchmark with a $156.6$\% mean human-normalized score, establishes a new record of $832$ on the DeepMind Visual Control Suite, and gains a $9.5$\% performance improvement after $1$M steps on the Crafter benchmark.

DyMoDreamer: World Modeling with Dynamic Modulation
Boxuan Zhang', Runqing Wang', Wei Xiao', Weipu Zhang', Jian Sun', Gao Huang". Jie Chen', Gang Wang'*
School of Automation, Beijing Institute of Technology Department Automation, BNRist, Tsinghua University
Abstract
A critical bottleneck in deep reinforcement learning (DRL) is sample inefficiency, training high-performance agents often demands extensive environmental inter- actions. Model-based reinforcement learning (MBRL) mitigates this by building world models that simulate environmental dynamics and generate synthetic experi- ence, improving sample efficiency. However, conventional world models process observations holistically, failing to decouple dynamic objects and temporal features from static backgrounds. This approach is computationally inefficient, especially for visual tasks where dynamic objects significantly influence rewards and decision- making performance. To address this, we introduce DyMoDreamer, a novel MBRL algorithm that incorporates a dynamic modulation mechanism to improve the ex- traction of dynamic features and enrich the temporal information. DyMoDreamer employs differential observations derived from novel inter-frame di differencing mask, explicitly encoding object-level motion cues and temporal dynamics. Dy- namic modulation is modeled as stochastic categorical distributions and integrated into recurrent state-space model (RSSM), enhancing the model's focus reward- relevant dynamics. Experiments demonstrate that DyMommerrrr sets a new state- of-the-art art on the Atari 100k benchmark with 156.6% mean normannnormalized score, establishes a new record of 832 on the DeepMind Visual Control Suite, and gains 9.5% performance improvement after 1M steps the Crafter benchmark. Our code is released at https://github.com Ctapgittuubbom///ttaaannn Ultraman- Ait.aaaa--iiaaaaayyMooooooo MooDD/ee
Introduction
Deep reinforcement learning (DRL) has achieved significant breakthroughs in sequential decision- making tasks 1
State-of-the-art methods such as DQN (5), PPO (6) MuZero [7], and EfficientZero 8 have demonstrated impressive performance across various benchmarks. However, these methods often suffer from low sample efficiency, requiring hundreds of millions of interactions with the environment [9], thus limiting their practicality for real- world applications where data collection is costly or infcasible. A key approach to addressing this challenge is the general framework of world models, which learns compact latent dynamics for planning and decision-makin One early instantiation of this framework [10,, combines a VAE with an RNN and uses evolutionary strategies to optimize policies in the learned latent space. This framework combines VAE with a recurrent neural network (RNN) and uses evolutionary strategies to optimize the policy within a latent space. By generating synthetic interaction data "imagination trajectories" [11 world models allow agents to learn behaviors entirely within the learned model. This model-based approach reduces the reliance on direct environment interactions, significantly improving sample efficiency.
"Corresoondigg author; gangwang@bit.. edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Traditionally, RNNs have been employed to capture temporal dependencies in world models. For 14 has demonstrated numan-level performance
instance, the Dreamer series (DreamerV1-3) [ on challenging benchmarks like Atari 100k [5 and Minecraft [66] using the recurrent state space world model architectures [18] have revolu- tionized sequence modeling, introducing methods such as GPT-like models [19] and Transformer-XI. [20]. and spatio-temporal transformers [21]. These advances have further refined the capabilities of
model (RSSM) [7]. More recently, transformer-
world models through tokenization techniques [9] 22 23
Despite their success, both RSSM- and transformer- world model architectures face challenges in accurately modeling small dynamic objects [24]. VAEs tend to amplify randomness, disproportionately affecting the pa- rameters of temporal models [25]. Moreover, small dynamic objects often carry higher de- gree of decision relevance compared to static environment backgrounds [26). For example, in Atari's game Pong, the paddle and ball are crit- ical for decision- decision-making. while the static red background is irrelevant. Recent techniques such as IRIS 9 231 and DIAMOND [24] partially mitigate these issues by improving the precision of image reconstruction world modeling, but at the cost of significant com- putational overhead. Similarly, the recent ap- proach OC-STORM [21] integrates vision mod- els with world models, and introduces segmenta- tion masks for object annotation, which require extensive pretraining and prior knowledge of object counts.
Mean Human Norma lized Score (Atari 100k)
Transf men based way model RAN bavsd workd model
160%
156. 6%
145.0%
140%
134
338.5%
322.3%
120%
150%
Median Human Normalizer Score (Atari 100k)
50%
73.3%
64.1%
50%
49.2%
43. 43.8%6
Dreament
STORM (2023)
oc Sc.SToss
Harmor Omann
EMAMORO 414244
byttutramee
(2024)
(oun)
Figure 1: DyMoDreamer surpasses existing state- of-the-art model-based methods.
In this paper, introduce DyMoDreamer, novel world model architecture for model-based RL that improves dynamic feature extraction and enriches temporal information by encoding differential observations. DyMoDreamer integrates dynamic modulation mechanism into RSSM, emphasizing critical environmental variations without relying on high-precision image reconstructions or prior object annotations. Through novel lightweight inter-frame differencing mask, our approach effec- tively prioritizes decision-relevant dynamic features, addressing key limitations of existing RNN- and transformer-based models. Our approach is inspired by the cognitive processes observed in human infants, who naturally focus on dynamic object interactions to infer fundamental principles about their surroundings (27 28 Leveraging this principle, DyMoDreamer achieves superior policy performance, demonstrating that emphasizing dynamic objects and temporal information signifi- cantly improves decision-makiir
RL. Our method establishes new SOTA of the RNN-based world model with 156.6% human-normalizer score on the Atari 100K benchmark, sets a new record with 832 mean score on the DeepMind Visual Control Suite, and delivers a 9.5% performance improvement on the Crafter benchmark after 1M environment steps.
Our main contributions are summarized as follows:
We introduce dynamic modulation mechanism that enhances RSSM by encoding differen- tial observations to focus on key dynamic features and temporal information.
We develop DyMormerrrr a novel world model that integrates dynamic modulation into recurrent states, improving policy efficiency by prioritizing dynamic environmental features.
We achieve state-of-the-an performance of RNN-based world model with 156.6% human- normalized score on the Atari 100k benchmark, 832 average score on DeepMind Visual Control, and 9.5% performance improvement on the Crafter benchmark within 1M steps.
The remainder of this paper is structured as follows. Section details our methodology. Section presents empirical evaluations on the Atari 100k benchmark, the DeepMind Visual Control Suite, and the Crafter benchmark, followed by ablation studies in Section 4 Detailed scores and curves are provided in Appendix [. alongside extensive ablation studies Appendix ! insights and future directions in Section
Sectiondeetailss
We conclude with

---

4r-2
42-1
2x-2
ht-1
h
dg
dynamic modulation
modulation
modulation
Zz
d2
enc
dec
dec
enc
GBC
dec
Figure 2: Overview of DrMamerrmee. The model integrates dynamic modulation derived from differential observations enhance the perception of dynamic features within the environment.
2 Methodology
This section outlines Dreame,,,r, model-based reinforcement learning algorithm designed to enhance dynamic feature extraction for improved agent performance. The problem is formulated as partially observable Markov decision process (POMDP), represented by the tuple (O, T, R. 7), where O denotes high-dimensional image observations, a₂ E A represents discrete actions generated by some policy T(a 01:1, 1). T is the transition function, sociated with particular task, and y is the discounting factor. The objective is to learn a policy that maximizes the expected sum of discounted rewards E, [ rt]. where r; R(0--1, 41-1-
the reward function as-
We present DyMoDremmer model-based RL framework designed to improve world model repre- sentations dynamic temporal features and enhance agent performance. At core, DyMoDremeer employs the RSSM architecture, augmented with novel dynamic modulation mechanisms that se- lectively emphasize dynamic features and temporal information in the environment. This approach draws inspiration from advances in modulated ordinary differential equations (ODEs) [29] where independently trained modulators improve temporal modeling for long-term prediction in complex systems. Moreover, the latent flow method (30 in model free reinforcement learning (MFRL) sug- gests that capturing temporal changes via time-based differencing can enhance transition modeling and improve the performance [3I] Building these insights, DyMoDreamer introduces temporal frame differencing method to refine the expressiveness of dynamic modulators, enabling the agent to leverage richer and more decision-relevant information for improved policy learning and performance.
We mainly base our approach on DreamerV3 [14, introducing dynamic modulation into the RSSM world model architecture. Below, we detail the construction of frter- differencing masks for dynamic modulation (Section 2.1). the integration dynamic modulation into the world model (Section) 2.2 and the end-to-end training process (Section 2.4p.
2.1 Dynamic Modulation
As core component of the world model, we leverage VAE with convolutional neural networks (CNNs) to encode observations into stochastic state representations. The original encoder in Dream- erV3 integrates temporal information, modeled by the hidden states h, (introduced in Section 2.2 along with RSSM), and maps observations or to stochastic representations 24. To focus on the dynamic aspects of the environment, we introduce dynamic modulation, which explicitly captures temporal changes between frames.
Differential observations of are derived by computing inter-frame differences from the original observations, effectively isolating dynamic components. dedicated dynamic encoder processes of to generate dynamic modulators de, which participate in the reconstruction of o, to ensure the
3

---

accurate extraction of dynamic features. The encoding process is as follows:
Stochastic encoder:
2t ~ qo ( he, 0t)
Dynamic encoder:
de ~ q (dz h1g o))
(I)
Decoder:
6. ps (ō; (ò: hes Res de).
The primary claim of this work is that explicitly integrating dynamic modulators, which capture dynamic environmental features, significantly enhances agent performance. By modeling features through differential signals, the approach improves the extraction of critical dynamic patterns and bolsters overall robustness [32] To ensure the dynamic modulators emphasize decision-relevant patterns in the environment, we construct differential observations of using a frame differencing method. Since forward differencing requires access to future information, we instead define temporal backward differential binary function:
D(oihhw,e Ci Ro f1, if |lo,A,,,. o.-khh, others
where is a binary threshold value, set to 0.001 and k: is integer hyperparameter represents the differenced frames' interval (empirically set to 1, increasing k helps suppress potential static background flash), i is the time dimension, f and w are the spatial dimensions, and is the channel dimension. In fact, very few pixels may exceed the threshold e after processing by D(.). To address this sparsity, we expand the regions of interest by filling the O-pixel regions surrounding 1 1-iiell regions with 1's, process achieved using convolution operations. This yields dynamic differential masks M(o), which emphasize both the differing pixels between frames and their surrounding regions. Subsequently, the differential observations are defined as the masked original observations = (o.) and are utilized to determine the dynamic modulators. This design explicitly extracts dynamic features from the observations, effectively capturing temporal changes rather than merely amplifying the influence of the most recent frame in the recurrent information. Unlike approaches that rely on sophisticated vision techniques, computing differential observations imposes negligible computational overhead while yielding notable performance gains (Section (.2). Section (ocction2.300 discusses the motivation for performing differencing in the observation space rather than in the latent space. Furthermore, additional results in Appendix demonstrate that DyMoDremmer can flexibly adapt to different differencing strategies tailored to varying task environments. Differential observations across different benchmarks are provided in Appendix F
2.2 Dynamically Modulated World Model
During interaction with the environment, DrMamrrrrr collects observations, actions, and rewards into experience dataset. After a predefined number of interactions, trajectories are sampled from this dataset to train both the world model and the categorical VAEs 33 34]. The world model is built on RSSM, where the sequence model utilizes the recurrent latent state hr, predict the sequence of stochastic representations and dynamic modulators based on past actions @,-1. The model state is formed by concatenating he 2, and de, enabling the prediction of rewards Êt and episode continuation flags & E {0,1),
Sequence model:
ht fo (fr- (1 1, d,-
Latent predictor:
&. = ps (&e ht)
Dynamics predictor:
d = Pø (d: h2)
Reward predictor:
F. = ps (ft hyy Rc, d,)
Continue predictor:
& = ps (Ce h4, 2t, de)
Since directly modeling the dynamics from raw images computationally expensive and prone to errors, we convert the observation o; into latent stochastic categorical distributions ga (2, h,, oz) Z, for the latent variable 2,, which consists of 32 categories, each with 32 classes. Similarly, the dynamic modulator d, is sampled from D. = 4a (d, h, 0z). where D, also consists of 32 categories, each with 32 classes:
(o( fie (() Zi, de (d4 he, ot) De.
(4)

---

We apply the straight-through gradients trick [35] to maintain the gradients of representations and modulators, as sampling from a distribution inherently lacks gradients for backpropagation. The sequence model f takes as input the hidden state 1 the stochastic representation 2t-1. the dynamic modulator d 1. and the action 0t-1 from the previous step. Using this input, computes a deterministic hidden state h via gated recurrent unit (GRU) [36. Multi-layer perceptrons (MLPs) then utilize h₁s 2t and d, to predict the current reward ft. the continuation flag & the dynamic modulation distribution Dr, and the stochastic distribution 2,, respectively.
By design, the dynamic modulator d, is time-varying, capturing dynamic features that enhance the model's 's focus on decision-related dynamic objects and their surroundings in or. As a result, our world model demonstrates the following advantageous properties:
Dynamic feature embedding: The dynamic modulator embeds dynamic features into the prediction models, compelling the model to prioritize decision-critical dynamic objects.
Temporal information capturing: The dynamic modulator d, predicted by the hidden state h, enables the world model to capture temporal information across time sequences.
End-to-end joint training: The dynamic modulator integrates into the reconstruction process alongside the stochastic representations and hidden states, facilitating end-to-end joint training of all world model components without requiring separate modulator training.
2.3 Intuition Behind Dynamic Modulation
In policy learning, rewards are predominantly influenced by the dynamic parts of observations and their adjacent static parts. For example, in the Boxing environment, the rewards are determined by effective jabs delivered by the agent, which involve dynamic elements (fists and arms) and their immediate static context (the body of the agent and its opponent) in the observations. Therefore, explicitly extracting dynamic local features in temporal sequences enhances the world model's ability to focus decision-critical features, thereby improving policy performance. Moreover, since not all tasks rely exclusively on dynamic features for decision-mmkking For instance, in the Gopher environment, both dynamic elements (e.g., gophers and tunnels) and static objects (e.g., carrots to be protected) are critical for earning rewards. To address this, DyMoDreamer retains the original stochastic representations while augmenting them with dynamic modulators. We provide visualizations in Appendix H. which show that is indeed influenced by the dynamic modulation, leading to increased attention of zz to the static factors.
We choose to perform differencing in the observation space rather than in the latent space (as done in prior MFRL work (30], by adding 6, 24 2t-1 to the sequence model), because the limited precision of latent encoding may hinder the detection of fine-graine tme-grained dynamic features. For example, if a small but important object (such the -ppixel ball in Pong) is not captured in either 2 or # the latent difference would fail to reflect this motion. contrast, frame-level differencing can make such movements explicit in the pixel space, and the dynamic modulation can incorporate this temporal information into modulation, enriching the dynamics modeling. Our ablation in Appendix [.3 further supports this motivation.
2.4 End-to-end Learning
In general, the parameters of the world model, denoted as Φ, are optimized end-to-end to minimize three types of losses: the prediction loss Cpred, the dynamics loss Layn, and the representation loss Crep- This optimization is performed in a self-supervised manner using a batch of sequences comprising observations 01:7, actions d1:7, rewards P1:7, and continuation flags c1.7- To further enhance the model's 's prediction performance, we introduce a differential divergence regularization term Freg, which leverages differential observations o'-1 to capture the dynamical differences between consecutive frames. The total loss function is defined Equation (5), where Wayn = 0.5 and Wrrop 0.1 represent the weight coefficients for the respective loss terms:
[(0) Es Σ (o) + waya Cdyal) (rep
Lo)
(5)
The prediction loss Lpred trains the categorical VAE to encode stochastic representations 2, and dynamic modulators di while reconstructing the input visual observations or Additionally, it enables
5

---

the model to predict environment rewards and episode continuation flags, which are crucial for computing imagined trajectory returns during the behavior learning phase. The prediction loss comprises three components, as defined in Equation (6):
(pred (0) + Frew (0) + (Ø)
(6)
where (Ø) = -In p lh,, d:) denotes the reconstruction loss of the input image, frow (0) In pa (re zt, de) refers to the reward prediction loss, using a symlog two-hot loss, transforming the regression problem into a classification problem with soft targets, and Loon (0) = In Po (po(cel/ss, [/,, 2t, de) corresponds to the continuation flag prediction loss, which is implemented cross-entropy loss.
The dynamics loss Layu trains the sequence model to predict stochastic representations and dynamic modulators by minimizing the Kullback-Leibler (KL) divergence between the predicted and true distributions:
Layn = max (1, (sg(aooz he, 0z)) || Po h+)])
+ max(1, KLlsKKLssgggga he, o()) || (o he)])
modulation's is dynamic loss
Here, sg(+) represents the stop-gradient operator, and the maximum operator clips the loss at nat (approximately 1.44 bits) [37] This "free-bit" operation prevents the loss terms from becoming excessively small, allowing the model focus on other terms during training 38
The representation loss Lrep enforces consistency between the priors and posteriors of 2, and Dr. allowing for factorized dynamics predictor for efficient sampling during imagination training. is defined as:
(0) max (L, he, oz) || (g h))
+ max (1, (T, of) sg(po(d, ht))))
(8)
modulation's 's representation loss
The differential divergence regularization Lreg addresses the limitations of mean- sqeannd error (MSE) loss, which primarily focuses on frtta- differences. Instead, Lreg forces the model to learn fnter-frame variations, improving its awareness of dynamic changes [39]. Backward differences are computed as Δó; ô₂ ō--1 and Ao; = 0t-1. These differences are transformed into probability distributions using the softmax function along the channel, height, and width dimensions:
(Aor)) exp(Aô:/T)
exp(Aor/1
((Do)) = En,m, exp(Aot/T)
hi exp( (Or/T))
where 7 is temperature parameter (empirically set to 0.1) to sharpen the distributions. The regu- larization term is then defined as the average KL divergence between these distributions across the batch B:
1 Σ (KL(((A))l ((())).
Coeg(d)
(10)
B
2.5 Policy Learning
The policy in DrMamre is trained exclusively on abstract trajectories predicted by the model using standard actor-critic framework [40]. In our approach, the actor and critic operate on the concentration states the recurrent world model with dynamic modulation. Notably, due to the adoption of backward differential observations to encode the dynamic modulation, random action is sampled for the initial frame to derive single observation (or le frames when k for starting the backward difference, resulting in of. As we shown in Figure 3 although the reconstruction loss of imagined trajectories shows no significant reduction, we observe that MoMorremrr generates substantially less hallucination on dynamic patterns during imagination compared to DreamerV3, which fundamentally explains the performance gains brought by our dynamic modulation mechanism. The remaining setup closely follows DreamerV3 and is detailed in Appendix D
{/r, =t, ) leveraging the Markovian representations learned by

---

DyMooreemee
Dreamerk's
Bixing
Inagiiation
Difforntal Obscrvttone Ento
Figure 3: DyMoDreamer highlights the dynamic features. The players enclosed in red rectangles represent reward-relevant objects, which can be encoded into the dynamic modulation to enhance the RSSM. Blue rectangles mark the enror regions where DreamerV3 exhibits hallucinations during the imagination reconstruction.
Experiments
3.1 Benchmarks and Results
We evaluate DyMomrr on several widely-used benchmarks for sample-effcient RL: Atari 100k, DeepMind Visual Control Suite and Crafter. Table summarizes the aggregate scores of DyMoDreamer and baselines, with the associated performance detailed per-environment scores and curves provided in Appendix
Atari 100k Benchmark We first evaluate DyMoDremmer on the Atari 100k benchmark, comprising 26 games from the Arcade Learning Environment, widely adopted testhed for RI agents [15] Agents are restricted to 100k actions (equivalent to 400k frames or ~ 1.85 hours of gameplay) for training before evaluation. This constraint emphasizes sample efficiency, critical metric in reinforcement learning. DyMoDremmer significantly improves upon the base method's performance, and sets a new state-of-the-art benchmark with a mean human- nomaaaiied score (HNS) of 156.6%.
To validate DyMomremnr''s capacity for continuous control, conduct experiments on the full set of 20 tasks in the DeepMind Visual Control Suite, challenging environments requiring policy learning from high-dimensional visual observations under 1M step budget. In this suite, DyMoDreamer DrDemmer outperforms the newest and strongest existing baseline TWISTER 41), achieving 5.5% average performance improvement over the DreamerV3 baseline. The results indicate that the dynamic modulation can also yield performance gains for world models in tasks involving high- singh-dmmessonall observations and continuous action spaces.
Mind Visual Control Suite
Crafter Benchmark We finally complete the empirical validation using the Crafter 42 benchmark. which is a procedurally generated environment, inspired by the video game Minecraft, with visual inputs, discrete action space and non-deterministic dynamies. In this benchmark, the background moves along with the agent, causing differential observations to capture the relative motion between the agent and the environmental components. Consequently, dynamic modulation enriches the information available for decision-making, On this benchmark, DyMoDreamer outperforms the IRIS series (the strongest transformer-basee baselines)and achieves 9.5% performance gain the Dreamer V3 baseline.
3.2 Analysis and Implementation Details
compared DyMoDreamer with methods leverage advanced architectures such as transformers and RSSMs, as well as diffusion models. Model-free and search-based approaches (e.g., BBF [3), EfficientZeeo (8) are excluded, as our focus the refinement of world models. Hyperparameters for DyMomrerrer follow DreamerV3 [14), except where explicitly noted, such in the ablation studies.
7

---

Table Aggregate scores of DyMoDremeer and baselines.
Atari 100k
OC-STORM
DIAMOND
DreamerV3
DrMomremmer
(2025)
(2024)
(2023)
(ours)
HNS Mean
134.8%
146% 37%
125% 49%
156.6%
HNS Median
43.8%
71.3%
DeepMind Visual Control Suite
TD-MPC2
TWISTER
DreamerV3 (2023)
DyMoDremmer
(2023)
(2025)
(ours)
Task Mean
720.9
801.8
786
Task Median
795.9
907.6
861
Crafter
IRIS (2023)
A-IRIS
DreamerV3 (2023)
DyMoDreamer
(2024)
(ours)
Return IM
5.5
7.7
10.3
DyMoDre demonstrates superior performance compared to previous methods in environments where the key objects related to rewards are sparse and independent with each other, such as Pong and Finger Spin. This advantage can be attributed to the accurate capturing of dynamic modulators, which explicitly enhances the world model awareness of temporal features in the observations. DyMomremmer also excels in tasks with distinct phases, such as Krull, and Crafter, as the predictable dynamic modulators explicitly guides the world model to capture transitions between task stages. Moreover, DyMoDreamer still yields performance gains by explicitly retaining the stochastic representations, for example, Gopher. It's noteworthy that this enhancement is achieved without leveraging any additional techniques from the computer vision domain, thereby incurring no extra oversized computational overhead. By introducing dynamic modulation into the world model, fully exploit the intrinsic potential of the RSSM framework, thereby enhancing the significance of our contribution. In our experiments, we use machine with an NVIDIA 4090 graphics card with 8 CPU cores and 24 GB RAM. Training DyMomremmer on one Atari game for 100k steps took roughly 5.5 hours in our JAX implementation.
when the reward signal is not fully captured by the dynamic information,
4
Ablation Studies
Our experiments have shown incorporating dynamic modulators into the world model can greatly enhance the final outcomes. To further investigate this, we perform ablation studies on the two novel components introduced in DyMoDrem the dynamic modulation (Section 4.1) and the differential divergence regularization Lreg (Section 4.2). We select four representative game environments: Boxing, featuring large and sparse dynamic objects; Krull, characterized by small and numerous dynamic objects; Pong, with small and sparse dynamic objects; and Road Runner, which includes both small and large dynamic objects.
4.1 Removing Dynamic Modulation
We first perform an ablation study where the dynamic modulators are retained while its integration into RSSM is removed, in order to assess the necessity of incorporating dynamic modulation for achieving performance gains. In this setting, the sequence model in Equation 3 degenerates to h (h, 2t-1. at-1), where the dynamic modulation is only involved in encoding and image reconstruction. Additionally, the constraints on the modulators d, are removed from both Layu and Ldya- As shown in Figure 4 removing the dynamic modulation mechanism causes a notable performance drop. Simply appending differential observations, without integrating them into the RSSM, fails to direct the world model's focus toward dynamic patterns. This result is comparable to the ablation Appendix [.] where increasing the dimensionality of 2, alone offers limited benefits. These findings indicate that the dynamic modulation should be embedded into the sequence model to

---

be effective. By integrating information from the predictable modulators into the recurrent state, the agent gains access richer environment cues during beyond merely increasing the dimensionality of the VAE.
Baxing
Pong
9000
40
6000
8000
S000
-15
50k
100k
0
50k
100k
50c
100k
50k
100k
DyNoDremmer without dynamic modu atton
DyMisBreareer
Figure 4: Ablation study the dynamic modulation.
4.2 Differential Divergence Regularization
The differential divergence regularization Lreg addresses the limitations of mean srean-squrre error (MSE) loss, which primarily focuses intra-frame differences. Therefore, remove this component from DyMoDremmer and observe a performance drop (Figure 5). which demonstrates that the differential divergence regularization acts complementary supervisory signal that works in concert with reconstruction objectives, and contributes improvement through its unique ability to constrain the temporal consistency the reconstruction process. Removing the differential divergence regularization leads to performance drop; however, thanks to the dynamic modulation mechanism, the model still surpasses vanilla D3eamerV3.
Boxing
Knull
Pong
Road Runner
24000
80
9000
15
16000
6000
3000
50k
B00k
%
50k
100k
50k
100k
50k
100k
ByRobreamer w thout cifferent al gence reguirr zation
DyMoGresmrer
Figure 5: Ablation study on the differential divergence regularization.
Related Work
World model-based reinforcement learning enables agents learn via imagination, improving sample efficiency by predicting future states and rewards from compact latent representations. Early works like SimPLe [12 and Dreamer (11) pioneered latent dynamics modeling for decision-making, DreamerV2 13 introduced discrete latents to mitigate compounding errors, and DreamerV3 [14] extended this to diverse domains with fixed hyperparameters. HarmonyDream 44 further refined training via adaptive loss balancing. Recent advances leverage transformers (e.g.. STORM 255 and OC-STORM (2l)) for scalable sequence modeling, with innovations like object-centric prediction. The IRIS series 9 [2] advanced tokenized imagination by employing VQ-VAE (55 to encode observations into discrete codes, compressing visual inputs into a compact symbolic space that facilitates long-horizon predictions. Meanwhile, DIAMOND [24 applies score- bore-based diffusion models in pixel space for detailed visual modeling. Hierarchical world models (e.g.. THICK [46).
HIEROS [47, and Puppeteer (48)) further improve temporal abstraction, enabling long-horizon reasoning and coordination complex environments.
Conclusion
In this work, we propose DyMoDreamer, novel world modeling approach with dynamic modulation, which outperforms previous model-based methods in Atari 100k, DeepMind Visual Control and Crafter benchmarks, setting new record for the RSSM architecture [17]. DyMoDreamer integrates dynamic modulation into RSSM, enabling the agent to effectively capture decision-relevant dynamic features and temporal information within the environment throughout the decision decision-making process.

---

We capture dynamic features and temporal information by encoding differential observations, and enhance dynamic modulation with specialized dynamics and representation losses improve its ability to integrate dynamic information. This design significantly improves the performance of the world model. Our work demonstrates that the RSSM framework remains highly promising and opens numerous avenues for future research. One potential direction is to introduce soft predictive
constraints on future states during end-to-end training could prove beneficial (49 [0]. We believe that designing more human-like world model algorithms by learning from human's cognitive behaviors holds significant potential for impact and represents an exciting path for exploration.
7 Acknowledgments and Disclosure of Funding
We would like thank all anonymous reviewers for their time, constructive comments, and engaging discussions. This work was supported by the National Key Research and Development Program of China under Grant No. 20222D01 19302, and the National Natural Science Foundation of China under Grants U23B2059, 62173034, and 62495090.
References
[1] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schritwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et Mas- tering the game of Go with deep neural networks and tree search. Nature, 529(7587):484 2016.
[2] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Ahdolmaleki, Diego de Las Casas, et al. Mag- netic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414- 419, 2022.
[3] Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. arXiv preprint arXi::2310.1622. 2023.
[4] Zhaohan Feng, Ruiqi Xue, Lei Yuan, Yang Yu, Ning Ding, Meiqin Liu, Bingzhao Gao, Jian Sun, Xinhu Zheng, and Gang Wang. Multi-agent embodied Al: Advances and future directions. arXiv preprint arXiv:2505.055 050000 2025.
[5] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-53) 2015.
[6] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1.XXv:1707.0644. 2017.
[7] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering Atari, Go, Chess and Shogi by planning with learned model. Nature, 588(7839):604-609 2020.
[8] Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering Atari games with limited data. In Advances in Neural Information Processing Systems, 2021.
[9] Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers sample-efficient world models. arXiv preprini arXiv:2209.0000 00588, 2022.
[10] David Ha and Jürgen Schmidhuber. World models. arXiv preprint /803. 10/22, 2018.
[11] Danijar Hafner, Timothy Lillierap, Jimmy Ba, and Mohammad Norouzi. Dream control: Learning behaviors by latent imagination. arXiv preprint arXiv: 19/2001033 2019.
[12] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Model- based reinforcement learning for Atari. arXiv preprint arXiv: 1xXxv:1903.007777 2019.
10

---

[13] Danijar Hafner, Timothy Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering Atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.
[14] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04111 2023.
[15] Mare Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[16] Ingmar Kanitscheider, Joost Huizinga, David Farhi, William Hebgen Guss, Brandon Houghton, Raul Sampedro, Peter Zhokhov, Bowen Baker, Adrien Ecoffet, Jie Tang, et al. Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft. arXiv preprint arXiv:2106. 14876, 2021.
[17] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555-2565, PMLR, 2019.
[18] Vaswani. Attention all you need. In Advances in Neural Information Processing Systems,
2017.
[19] Alec Radford. Improving language understanding by generative pre-training. 2018.
[20] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-a XL: Attentive language models beyond fixed-length context. arXiv preprint arXiv:1901.02860 028660 2019.
[21] Weipu Zhang, Adam Jelley, Trevor McInroe, and Amos Storkey. Objects matter: Object-centric world models improve reinforcement learning in visually complex environments, 2025.
[22] Vincent Micheli, Eloi Alonso, and François Fleuret. Efficient world models with context-aware tokenization. arXiv preprint arXiv:2406. 19320, 2024.
[23] Lior Cohen, Kaixin Wang, Bingyi Kang, and Shie Mannor. Improving token-based world models with parallel observation prediction. arXiv preprint 2..:22022.0663 2024.
[24] Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in Atari. arXiv preprint 0rXiv:2405.112399 2024.
[25) Weipu Zhang, Gang Wang, Jian Sun, Yetian Yuan, and Gao Huang. STORM: Efficient stochastic transformer based world models for reinforcement learning. Advances in Neural Information Processing Systems, 2023.
[26] Quentin Delfosse, Jannis Blüml, Bjarne Gregori, Sebastian Sztwiertnia, and Kristian Kersting, OcAtari: Object-centric Atari 2600 reinforcement learning environments. arXiv preprint arXiv:2306.08649 2023.
[27] Elizabeth S Spelke. Principles of object perception. Cognitive science, 14(1):29-56, 1990.
[28]
Martha Arterberry and Mare Bornstein. Infant perceptual and conceptual categorization: The roles of static and dynamic stimulus attributes. Cognition, 86(1):1-22 2-4, 2002.
[29] Ilze Amanda Auzina, Çağatay Yıldiz, Sara Magliacane, Matthias Bethge, and Efstratios Gavves. Modulated neural ODEs. In Advances in Neural Information Processing Systems, 2024.
[30] Wenling Shang, Xiaofei Wang, Aravind Srinivas, Aravind Rajeswaran, Yang Gao, Pieter Abbeel, and Michael Laskin. Reinforcement learning with latent flow. In Advances in Neural Information Processing Systems, 2021.
[31] Yangru Huang, Peixi Peng, Yifan Zhao, Yunpeng Zhai, Haoran Xu, and Yonghong Tian. Simoun: Synergizing interactive appaaaceeee understanding for vision-based reinforcement learn- ing. In International Conference on Computer Vision. IEEE, 2023.

---

[32] Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer. arXiv preprint arXiv:2410.05258, 2024.
[33] Diederik P Kingma. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
[34] Michat Kempka, Marek Wydmuch, Grzegorz Rune, Jakub Toczek, and Wojeiech Jaškowski. Vizdoom: A doom-based Al research platform for visual reinforcement learning. In Conference on Computational Intelligence and Games, pages 1-8, 2016.
[35] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: (00.::330.340 3432, 2013.
[36] Rahul Dey and Fathi M Salem. Gate-variants of gated recurrent unit (GRU) neural networks. In International Midwest Symposium on Circuits and Systems, pages 1597-1600. IEFE, 2017.
[37] Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Transdreamer: Reinforcement learning with transformer world models. arXiv preprint arXiv: 2022.
[38] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in neural informa- tion processing systems, volume 29, 2016.
[39] Cheng Tan, Zhangyang Gao, Lirong Wu, Yongjie Xu, Jun Xia, Siyuan Li, and Stan Z Li. Temporal attention unit: Towards efficient spatiotemporal predictive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18770-18782. 2023.
[40] Richard S Sutton. Dyna, integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4):160-163, 1991.
[41] Maxime Burchi and Radu Timofte. Learning Transformer-based world models with contrastive predictive coding. arXiv preprint 2rXiv:2503.04416, 2025.
[42] Danijar Hafner. Benchmarking the spectrum of agent capabilities. In International Conference on Learning Representations, 2022.
[43] Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Mare G Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level Atari with human-level efficiency. In International Conference on Muchine Learning, pages 30365-30380. PMLR, 2023.
[44] Haoyu Ma, Jialong Wu, Ningya Feng, Chenjun Xiao, Dong Li, HAO Jianye, Jianmin Wang, and Mingsheng Long. HarmonyDream: Task harmonization inside world models. In International Conference on Machine Learning, 2024.
[45] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, 2017.
[46] Christian Gumbsch, Noor Sajid, Georg Martius, and Martin V Butz. Learning hierarchical world models with adaptive temporal abstractions from discrete latent dynamics. In International Conference on Learning Representations, 2023.
[47] Paul Mattes, Rainer Schlosser, and Ralf Herbrich. Hieros: Hierarchical imagination on struc- tured state space sequence world models. In International Conference on Machine Learning, 2024.
[48] Nicklas Hansen, Jyothir SV, Vlad Sobal, Yann LeCun, Xiaolong Wang, and Hao Su. Hierarchical world models as visual whole-body humanoid controllers. arXiv preprint 2rX.v:2405. 2024.
[49] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05922 2020.
12

---

[50] Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Mare G Bellemare, and Aaron Courville. Sample-efficien reinforcement learning by breaking the replay ratio barrier. In In Advances in Neural Information Processing Systems, 2022.
[51] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improve- ments in deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[52] Jan Robine, Mare Höftmann, Tobias Uelwer, and Stefan Harmeling. Transformer-based world models happy with 100k interactions. arXiv preprint arXiv:2303.07109, 2023.
[53] Lior Cohen, Kaixin Wang, Bingyi Kang, and Shie Mannor. Improving token-based world models with parallel observation prediction. In In International Conference on Machine Learning. 2024.
[54] Lior Cohen, Kaixin Wang, Bingyi Kang, Uri Gadot, and Shie Mannor. Uncovering untapped potential in sample-efficient world model agents. arXiv preprint 1rXx::2522.11337, 2025.
[55] Ho Kei Cheng. Scoung Wug Oh, Brian Price. Joon- Joon-Young Young Lee, and Alexander Schwing. Putting the object back into video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3151-3161, 2024.
[56] Nolan Wagener, Andrey Kolobov, Felipe Vieira Frujeri, Ricky Loynd, Ching-An Cheng, and Matthew Hausknecht. McCapAct: A multi-task dataset for simulated humanoid control. In Advances in Neural Information Processing Systems, 2022.
[57] Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning. In International conference on machine learning, pages 449 449, 2017.
[58] TP Lillicrap. Continuous control with deep reinforcement learning. Xxiv::509.0297 2015.
arXiv preprint
[59] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
[60] Sukjun Hwang, Miran Heo, Seoung Wug Oh, and Seon Joo Kim. Video instance segmentation using inter-frame communication transformers. In Advances in Neural Information Processing Systems, 2021.
[61] Tongzhou Wang, Simon S Du, Antonio Torralba, Phillip Isola, Amy Zhang, and Yuandong Tian. Denoised mdps: Learning world models better than the world itself. arXiv preprint arXiv:2206. 15477, 2022.
[62] Ignat Georgiev, Varun Giridhar, Nicklas Hansen, and Animesh Garg. PWM: Policy learning with large world models. arXiv preprint arXiv:2407.02466, 2024.
[63] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages 1094-1100, 2020.
[64] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 1001::8800000000 2018
[65] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7482-7491, 2018
13

---

A Further Related Work
Model-based RL utilizing world models enables "learning through imagination" to provide greater data efficiency and offers a promising approach [12). These algorithms use self-supervised manners to construct parameterized simulation world models of the environments, leveraging losses such as reconstructing observations with decoders. predicting rewards and states at the next step, and the consistency between prior and posterior of latent states. Agent learns behaviors by the imagination trajectories based on world model predictions, reducing the reliance on environment interactions and improving the sample efficiency. SimPLe [12] applied world models to enable agents to focus on sample efficiency, and solve Atari games with fewer interactions than model-free Rainbow [51]. Dreamer [1 pioneered the reinforcement learning directly from the latent space of recurrent state space model. Building on this, Dreamer V2 [33 demonstrated that employing discrete latent variables mitigates compounding errors, while DreamerV3 [14] further advanced the approach, achieving human-level performance across diverse domains using fixed hyperparameters. HarmonyDream [44] employed harmonious loss dynamically adjust the weights of different loss terms during training. This approach ensures that the losses of various tasks. such as image reconstruction and reward modeling, remain within the same order of magnitude throughout the end-to-end training process.
Following the success of the Dreamer algorithm M. the transformer architectures [8] are also rapidly adopted for its superior training efficiency and favorable scaling properties. TWM (52 utilized the Transformer- XL (20) architecture to learn long-tern nergttemm dependencies while staying computationally
efficient. IRIS 9 and A -IRIS (22) utilized VQ- VAE [45] to build a language of image tokens to represent observations and make decisions based on the reconstructed observations. Building upon the IRIS framework, REM introduced novel parallel observation prediction mechanism to enhance the imagination, and achieves performance surpassing the baseline [53] Simulus [54] proposes a highly modular token-based world model that integrates multi mulli--modality tokenization framework, intrinsic motivation, prioritized replay, and a
strategy for predicting rewards and returns. STORM (25) used GPT-like architecture with a different tokenization approach, which captures long-term dependencies through attention mechanism and achieves impressive performance on the Atari 100k benchmark. OC-STORM extracted object features through a parameter-free frooeen pre- pre-trained foundation vision model "Cutie" (55) to apply both these object features and the raw observations as inputs for training an object-centric world model, which predicts the dynamic objects in the environment while considering the relationships between different objects and the background. Additionally, DIAMOND 24 introduced a diffusion- -based world model that operates entirely within the image space, enabling more accurate modeling of critical visual details in the world model.
Another type of world model adopts hierarchical structure, this architecture operates at different time scales, allowing the agent to learn and make decisions across multiple levels of abstraction. [46] deploys a hierarchy of world models with a look ahead trajectory search approach called THICK, which learns the world model via discrete latent dynamics. The lower level of THICK updates parts of its latent state sparsely over time, forming invariant contexts. The higher level exclusively predicts situations that involve context changes. HIEROS proposed an S5 layer- layer-based world model, which learned time abstracted world representations and imagines trajectories at multiple time scales latent space [47. Puppeteer [48] proposed hierarchical world model in which high-level agent generates commands based on visual observations for a low-level agent to execute, and produces highly performant control policies in 8 tasks with simulated 56-DoF humanoid [56), while synthesizing motions that are broadly preferred by humans.
B Limitations
DyMoDreamer improves the temporal world modeling through dynamic modulation, yet fundamental limitations of MBRL remain. Accurately capturing environment dynamics, especially in visually complex or highly uncertain environments continues to be a key challenge. Compounding errors during long-horizon imagination hinder policy performance, and recurrent latent models offer limited capacity for long-term memory. Although DyMoDreamer partially addresses these issues via archi- tectural enhancements, future work may benefit from integrating memory-augmentee transformers or hierarchical abstractions to improve long-term reasoning. Moreover, despite gains in sample effi- ciency, the trade-off between model fidelity and policy stability persists. Generalization to weal-world domains and robustness under distributional shifts remain critical open problems.
14

---

C
Broader Impacts
Training agents for real-world applications, such robotics and autonomous driving, challenging due to high costs, safety risk, and potential harm to humans. To mitigate these concerns, we conduct all experiments in simulated environments, eliminating risks associated with real-world training. We introduce MoDramerr, world model with dynamic modulation that enables agents to learn through imagination rather than direct physical interactions. By reducing reliance on real- world data collection, our approach enhances the safety and feasibility of deploying model-based reinforcement learning safety-critical domains. This work contributes responsible autonomous systems, addressing key concerns in real-world AI deployment.
development efficient and ethically
D Policy Learning
The policy learning approach closely follows that DreamerV3 [4], with modifications specific to method. To our DyMoDreamer, the actor and critic operate on the concentration states 8t = (2,, h4 de) We select actions by sampling from the actor network and aim to maximize the 0.997 for each state. The critic learns [57 to
return R2 -r= y'rt++ with discount factor approximate the distribution of returns for each state under the current actor behavior:
Actor: at (az st)
(11)
Critic:
(Re R B
Ertet 7=0
where the predicted values from the critic is a expectation the distribution it predicts. We adopt the actor and critic learning settings from DreamerV3. The complete loss of the actor-critic algorithm is described by Equation (7), where fg corresponds to the reward predicted by the world model, and é represents the predicted continuation flag. The critic learns approximate the distribution of the return estimates RS using the maximum likelihood loss:
B
[()) BL
+ (Vo(s)) sg{R)))
sg(V (s)))
(s:)))?]
(12)
where the constant L represent the imagination horizon and VILEA the exponential moving average (EMA) of the critic network for stabilizing training and preventing overfitting (58).
For the actor network, we use the Reinforce estimator for both discrete and continuous actions, resulting in the surrogate loss function:
1 BL
R (o() (1,
((0)
5g
(13)
n=l t=1
where H.)) denotes the entropy of the policy distribution, while constants 1y represent the coefficient for entropy loss. The A-retum R,A [59] is recursively defined as follows
R = r. do [11 A)Vol) + ARAH]
(14)
R Vs(s)).
The normalization ratio S utilized in the actor loss (4 is defined in (15). which is computed as the range between the 5th and thth percentiles of the X-return R2 across the batch:
= percentile( R,A, 95) - percentile( R2,5). 5).
(15)
E Experimental Details
E.1 Atari 100k Experiments
We evaluate agent performance over 100 episodes using the final model checkpoint, saved every 2, 500 steps. The reward function in Freeway is sparse since the agent is only rewarded when
15

---

it completely crosses the road. This poses an exploration problem for newly initialized agents because random policy will almost surely never obtain a non-zero reward with a 100k frames budget. Inspired by the trick in IRIS 19, we opted for the simpler strategy of having a fixed epsilon- greedy parameter and using the policy. However, we reduced the sampling temperature from to 0.0001 in Freeway, in order to avoid random walks that would not be conducive to learning in the early stages of training. The agent's score, R is the score of a random policy, and represents the average human score [3]. To determine H, the human players undergo familiarization with the game under identical constraints. We compare DyMoDre with state-of-the-art model-based RL methods, including OC-STORM 21 DreamerV3 [4], Harmony Table and the curves are shown in Figure
score is calculated as = -- where A is the
A-R
44 and DIAMOND [4 The task scores are shown in
Table 2: Game scores and overall human-normallzze performance on the 26 games in the Atari 100k benchmark. Following the conventions of [14), scores within 5% of the best are highlighted in bold.
STORM (2023)
OC OC.ROORM
DIAMOND (2024)
Dreamer (2023)
Harmony DreamerV3
DoMemrer (ours)
Game
Random
Human
(2025)
Alien
228
7128
984
1101
744
1118
890
971
Amidar
205
163
226
154.2
Assault
222
742
801
1270
1526
683
1003
902.1
Asterix
210
1754
1062
1140
1195.5
Bank Heist
14
753
641
1075
20
398
1069
1222.2
Battle Zone
2360
37188
13540
4590
4702
20300
16456
16240
Boxing
0
12
80
92
87
82
80
93.6
Breakout
2
30
53
132
10
53
40.2
Chopper Command
811
7388
1888
2090
1370
2222
1510
740.4
Crazy Climber
10780
35829
66776
84111
99168
86225
82739
82569.4
Demon Allack
152
165
288
577
248.3
Freeway
0
30
0
0
33
0
0
6.4
Frostbite
4335
274
3377
722
Gopher
258
2413
8240
4457
5898
2160
13043
13456.9
Hero
1027
30826
11044
6441
5622
13354
13378
9874.1
James Bond
29
303
509
347
427
540
317
541.8
Kangaroo
52
3035
4208
4218
$382
2643
5118
4788.8
Krull
9715
8171
7754
Kung Fu Master
256
22736
26182
24988
18714
25900)
22274
25641.4
Ms Pacman
307
6952
2673
2401
1958
1521
1681
1838.6
Pong
-21
15
20.6
20
19
20.9
Private Bye
25
69571
7781
85
114
3238
2932
3041.7
Qhert
164
13455
4522
4546
4499
2921
3933
1736.
Road Runner
12 7845
17564
20482
20673
19230
14646
20971.8
Seaquest
42055
525
712
551
962
5918
Up N Down
533
11693
7985
6623
3856
46910
10874
22321.5
Human Mean
0%
100%
126. 7%
134,8%
146%
125%
136.5%
156.6%
Human Median
0%
100%
58.4%
43.8%
37%
49%
67.1%
71.3%
E.2
DeepMind Visual and Proprio Control Suite Experiments
The DeepMind Visual Control Suite is a widely used benchmark for visual locomotion. We train and evaluate our DyMoDreamer all 20 continuous control tasks where the agent receives only high-dimensional images as observations and has a budget of 1M environment steps. We evaluate the agent performance by conducting 100 evaluation episodes for the final checkpoint and obtained the average score. DyMoDreamer is compared with DreamerV3, TD-MPC2 33. and TWISTER [41], which are recent model-based apporaches applied continuous control. The task scores are shown in Table and the curves are shown in Figure 7
The results demonstrate the effectiveness of DyMomrem DMC visual tasks. Our method greatly unleash the potential of RSSM. Similar to the results in Atari 100k, DyMoDreamer benefits from the dynamic modulation and shows superior performance compared to DreamerV3 in environments where key objects are sparse and independent of each other.
To evaluate the applicability of DyMoDreamer in state-based environments, we conducted exper- iments on DeepMind Proprio, which is a proprioceptive control benchmark where observations consist of structured state vectors such velocity and position. In this setting, observation differenc- ing naturally reduces to state differencing, with temporal dynamics being more explicitly encoded. Consequently, differential modulation provides similar benefits in non-visual settings, since such
16

---

Alien
Amidar
Assault
Asterix
182
1000
1200
200
200
100x
10kk
Sax reekouu
100c
Baxing
24002
50-
29002
0002
10
50c
1004
5ik Crazy Climber
17tk
500
50c Freeway
1004
Chopper Command
Dem on Attack
2400
40002
100
30002
Gophor
5200
12000
750
4000
10002
8002
4000
54k
1034
55k
136k
100
50i Pacman
100
Kangaroo
Krull
Kung Fu Master
9200
0002
2400
6002
15050
2002
100
50c
35k Private Eye
200k
50c Road Runner
Pang
Obert
26000
15:0
-35
50c
200x
sis
10ck
100k
50c
2002
Gamer Mean
Median
1200
120000
3.6
200
1000:
40002
0.5
2.2
0
30c
1002
55k
17tk
10k.
200k
3.0
30c
100k
Dreameed
DyMoorrmmmrr
Figure 6: Evaluation curves of DyMoDreamer the Atari 100k benchmark for individual games (400K environment steps). The solid lines represent the average scores over 5 seeds, and the filled areas indicate the standard deviation across these 5 seeds.
differences also carry meaningful physical semantics similar the differential observations-ffrr instance, differential positions implicitly encode velocity, while the differential velocities reflect acceleration.
E.3
Crafter Benchmark Experiments
In the Crafter mobs, crafting items, and managing health indicators during each episode. Regarding baselines, we compare DyMomremer with IRIS, AIRIS and DreamerV3, and evaluate each run by computing the average return over 100 test episodes every 1M frames, the curve is shown in Figure g
42
benchmark, the agent's goal is to solve many tasks as possible, e.g. slaying
E.4 Deepmind Lab Benchmark Experiments
benchmark features 3D environments that test spatial and temporal reasoning. To the concerns about the evaluation on more explicitly egocentric environments, we conducted additional experiments on the DeepMind Lab benchmark to further validate DyMoDreamer's effec- tiveness under rapid viewpoint changes. We focus several representative tasks with relatively dense
The Deepmind lab 42
17

---

Table 3: Task scores and scores of baselines and DyMomrr
TD-MPC2 (2023)
TWISTER
DreamerV3 (2023)
DyMoDre (ours)
Task
(2025)
Environment steps
IM
IM
IM
IM
Acrobot Swingup
216
239
229
447
Ball In Cup Catch
717
967
972
967
Cartpole Balance
931
998
993
999
Cartpole Balance Sparse
1000
1000
964
999
Cartpole Swingup
808
819
861
864
Cartpole Swingup Sparse
739
735
759
765
Cheetah Run
550
694
836
868.7
Finger Spin
986
976
589
968
Finger Turn Easy
789
924
878
906
Finger Turn Hard
872
910
904
873
Hopper Hop
211
314
227
290
Hopper Stand
803
932
903
943
Pendulum Swingup
743
832
744
808.9
Quadruped Run
362
652
617
770.4
Quadruped Walk
253
905
811
792.0
Reacher Easy
971
933
951
896.8
Reacher Hard
877
566
862
821.9
Walker Run
728
711
684
776.3
Walker Stand
916
977
976
929.6
Walker Walk
945
951
961
962.0
Task mean
721
802
786
832
Task median
796
908
861
871
Table 4: Game scores in the DeepMind Proprio benchmark.
DreamerV3
DyMoDreamer
Acrobot Swingup
134
225
Cheetah Run
614
625
Swingup
931
932
reward signals, which are more likely to exhibit performance improvements within 100M steps to assess DyMoDremme''s performance. The scores shown in Table DyMoDremmer outperforms the baseline on out of tasks, demonstrating competitive performance particularly on tasks with localized motion rewards (e.g., Explore Goal Locations Small, Explore Object Rewards Many) and dynamic visual input (e.g.. Rooms Watermaze). However, Rooms Keys Doors Puzzle, which emphasizes long- horizon memory and discrete planning motion cues, DreamerV3 shows slight advantage, consistent with our analysis of task suitability. In such settings, although the differential observations may not isolate only the task-relevant objects, they nonetheless effectively suppress large portions decision-irreles the agent's attention toward aaward-asooiated regions of the scene. Furthermore, differential obser- vations are not merely a visual processing technique, they also serve as an embedding of temporal information into the model's representation, enriching its capacity to reason over dynamic sequences. These results suggest that DyMoDreamer's dynamic modulation mechanism offers generalizable benefits visually rich and moderately egocentric environments, though may be less effective in domains requiring complex, abstract reasoning or long-term symbolic memory.
background. This facilitates more efficient exploration by directing
18

---

1002
500
200
250
270
500c
300
230k
Carppete Seirqup
Caripote Sw ngun Sporce
Cheeten Bun
Dinger Spt
200
500
500
250
250
100
Hard
750
402
0
500c
14
500<
14
5008 Quadruped Walk
14
5008 Reacher Exsy
14
Pendilum Seingup
Quadinued Run
300
752
300
500
300
Neacher Hard
Welker Run
Wither Stand
Walter Walk
300
750
1002
600
500
232
10k
Drcamotis
DyMotRccamo
Figure 7: Evaluation curves of DyMoDremmer the DMC suite. The solid lines represent the average scores over 5 seeds, and the filled areas indicate the standard deviation across these 5 seeds.
DreamerV3
SyMoDreamen
Figure 8: Evaluation curves of DyMomremer on the Crafter benchmark. The solid lines represent the average scores over 5 seeds, and the filled areas indicate the standard deviation across these seeds.
Table 5: Task scores in the Deepmind Lab benchmark.
DreamerV3
DyMoDreamer
Explore Goal Locations Small
372.8
382.1
Explore Object Rewards Many
58.1
60.0
Lasertag Three Opponents Small
18.8
18.9
Rooms Keys Doors Puzzle
42.4
41.2
Rooms Watermaze
26.1
27.2
19

---

Differential Observations
10 and illustrate the differential observations processed using differential mask con- structed by Section 2.1 Compared to directly employing cramewwise differences, the masks generated through dilated convolutions capture dynamic objects more comprehensively, ensuring more complete representation of motion-related features. Moreover, this demonstrates that the dynamic modulation is effectively encoded from the decision-relate dynamic objects within the observations.
Figure 9
Environment
Boxing
Observation coriginal>
Differential (vanills
Differential lours)
Ae
Environment: Pong
Observation (original)
Differential Ovanila
Differential (ours)
Environment:
Obsenvation
(orginal)
Differential
(venill>)
Differential
cours)
Figure 9: Differential observations for Boxing (the first subfigure), Pong (the second subfigure), and Krull tasks (the last subfigure). Each subfigure comprises three rows, and each illustrates distinct form of observation. The first row displays the raw observations, the second row shows observations processed using vanilla frame differencing (2). and the third row presents the differential observations constructed using our proposed approach, which combines frame differencing with dilated convolutions.
20

---

Environment: Finger Spin
(originai)
Differential (vanilla
Differential (uurs)
Environment: Pendulum Swingup
Observalion coriginal)
Differential (vanila)
Differential lours)
Environment Reacher Hard
Observation
coriginal)
Differential Differential
(vanilia >
(ours)
Figure 10: Differential observations for Finger Spin (the first subfigure), Pendulum Swingup (the second subfigure), and Reacher Hard (the last subfigure) tasks. Each subfigure comprises three rows, and each illustrates a distinct form of observation. The first row displays the raw observations, the second row shows observations processed using vanilla frame differencing (2), and the third row presents the differential observations constructed using our proposed approach, which combines frame differencing with dilated convolutions.
G Alternative Differencing Strategies
naive separation of dynamic and static components may introduce inductive biases in certain environments. To mitigate such issues (dynamic backgrounds or environmental noise), DyMoDreamer designed be highly flexible and accommodates variety of differencing strategies, enabling seamless adaptation to diverse visual environments. While our default implementation adopts frame- wise pixel differencing to highlight dynamic cues, architecture readily supports alternative formulations such as moving average difference,, temporal skip differencing (Appendix (.6). or
21

---

Environment: Crafter
Observation (oiginnal
Differenial
(Ours)
Figure 11: Differential observations for Crafter The first row displays the raw observations, the second row presents the differential observations constructed using our proposed approach, which combines frame differencing with dilated convolutions.
multi-frames differencing. Importantly, frame differencing has long been recognized in computer vision as a lightweight yet effective method for motion detection, foreground segmentation, and event-based visual processing, due its ability isolate dynamic elements from static backgrounds [60] By leveraging this principle within the context of world model, DyMoDreamer inherits its computational efficiency and interpretability, while maintaining the temporal information with the dynamic modulation mechanism. DyMoDremmer demonstrates that the differencing- -baen dynamic modulation not only enriches temporal representation but also offers flexible interface to incorporate various differencing strategies, making it adaptable to a broad range of task scenarios.
Moving Average Difference We formally express the replacement of frame differencing (Equation with moving average differencing as follows:
1.
|oo,,. avg, 112
{0.
(16)
others
where avg, is defined as:
if i21
avg;
(17)
isl
where is the temporal averaging window size. This approach demonstrates superior performance naive frame differencing in egocentric vision tasks, as evidenced in domains like Battlezone (Table 6).
Multi-frame Logical Differencing We further propose a novel multi-frame logical-relational differencing that stabilizes the differencing output ((o.,,,w.e) in Equation by applying logical AND operations (that is, only pixels whose masks were in the past few times will be displayed as in the final matrix.) across temporally displaced frames:
Diog (os,,,,,,..))
A DR. kED())
(18)
where (i A, and A is the symmetric temporal window size. This method demonstrates superior performance environments with higher temporal continuity, as evidenced in domains like (Table 6).
22

---

Table 6: DyMoDremeer with different differencing strategies.
DyMoDreamer
Moving Average Differencing
Multi-frame Logical Differencing
Vanilla
Battle Zone Road Runner
20351
16240
21378
20972
H
Static Factors
extend MyMererr by introducing dedicated decoder for stochastic representations Et, revealing emergent specialization in the learned latent space:
Stochastic encoder:
2t ~ dø (zt l14, 0+)
Dynamic encoder:
d, ~ qo (de he: oi)
(19)
Decoder:
dyn, pddyu he, Ze, de)
Static decoder:
sta, - pe (star Ne, z1)
The reconstructed observations is calculated by:
ô, dýn, + sta,-
(20)
Critically, we maintain the original end-to-end loss 5 without modification, and validate the repre- sentational specialization through natural reconstructions, which is shown in Figure 12] The results demonstrate clear representational dissociation: stochastic representations encoding predominantly capture static environmental factors and passive dynamics (e.g., static backgrounds and the black NPC in Boxing), while the dynamic modulation features specialize in agent-controllast elements (e.g.. the controlled white player in Boxing). This emergent specialization suggests natural segregation within the model's 's representational architecture like the denoised Markov decision process since we do not explicitly assign reconstruction targets to different decoders.
Recoooonoooooottteeeeeee Reconstrution Imaginntion Boxing
Static Dynamic
3
Reconstruction
1
1
Figure 12: The first row shows ground-truth trajectories under the same action sequence. The second
row presents imagined trajectories reconstructed from the predicted priors. The third row shows dynamic decoding results based on dynamically modulated features de The fourth row shows static decoding results based on stochastic representations 2t.
23

---

I
Generality
To further validate the effectiveness of our dynamic modulation mechanism, we integrated it into STORM (25), a recent efficient transformer- based world model. As shown in the results below, this integration also leads to clear performance gain.
Boxing
10000
Kru
Pong
Road Runner
90
15
24000
7500
60
16000
2500
15
50k
100k
50k
100k
50k
100k
50k
100k
STORM (regroduced)
STORM with dynamic modulation
Figure 13: STORM with Dynamic Modulation.
J Ablations
In the following, we provide additional ablation studies to explore the effect of different components of DyMoDremmer. We conduct all ablations on four games: (i) Krull, a game that features multiple levels, with small and numerous dynamic objects, (ii) Boxing, game with a single level and characterized by large and sparse dynamic objects, (iii) Pong, a game with a single level and characterized by small and sparse dynamic objects. (iv) Road Runner, game with both small and large dynamic objects. We use the same hyperparameters as described in Appendix K for all ablations.
J.1 High-dimensional Stochastic Representations
Scaling to larger models leads to higher data efficiency and episode rewards 662 For example, in TDMPC2 [3], a larger parameter count leads to improved results in the Meta-Worke [63] and DeepMind Control (DMC) [64] environments. Therefore, compare the vanilla DreamerV3 that extends 25 to 48 categories with 48 classes for the sake of fairness.
The result is shown in Figure 14 which reveals that only the dimension added in vanilla DreamerV3 2x not achieve the same performance as DyMamrerrer. This once again indicates that dynamic modulation is not equivalent to simply increasing dimensions but rather enriching the information used by agents for decision making.
Boxing
Kru
Pong
Road Runner
90
15
6000
3000
0000
100k
50k
10ok
DreamerV3 (48448 2))
DyMcDreamer
Figure 14: Ablation studies on the high-dimensional Dreamer V3,
J.2
Reconstruction of Differential Observations
To improve the world model's focus on reward-relevant objects, OC-STORM introduces additional object-centric model, Cutie (55), based on target segmentation. This enhancement significantly improves the baseline STORM's performance. OC-STORM encodes both raw and object-centric observations to obtain stochastic representations and reconstructs these separately during the autore- gressive prediction process to train the VAEs. Following a similar approach, we conduct an ablation study on reconstructing differential observations in DyMoDreamer, with results shown in Figure [5] Specifically, we introduce an additional term Lait (Φ) In p hys d,) into the prediction loss function (6. Furthermore, we incorporate a dynamic decoder into (3):
Dynamic decoder: ó' ps (0'. he, de).
(21)
24

---

Baxing
Kru
Pong
Road Runner
90
24000
15
60
16000
3000
0000
-15
S0k
100k
50k
100k
50k
100k
50k
100k
DyMoDroamer with differential reconstruction resostrttin
DyMoGreamer
Figure 15: Ablation study on differential observation reconstruction. Adding a dynamic decoder does not improve performance and increases computational cost.
shows that adding a dynamic decoder does not improve DyMoDreamer's performance. Instead, weakens dynamic modulation's 's effectiveness and significantly increases computational cost. This is because differential observations are not strictly edge segmentations. Their primary function is to isolate and encode dynamic features of the environment rather than improving its reconstruction.
Figure 15
The substantial improvement observed in the Pong environment suggests that when dynamic objects are both small and sparse, differential observations serve as relatively accurate reconstruction target, allowing the agent to match the performance of pre-trained visual models with faster speed. Explicitly constraining the dynamic modulators with a decoder may actually degrade performance due to coarse reconstruction targets. In contrast, implicitly training stochastic representations and dynamic modulators together within a single decoder is more efficient and yields better performance. We claim once again that the goal of dynamic modulation is not to improve the reconstruction accuracy, but to enable the agent to leverage richer dynamic information for decision making.
J.3 Modulation with Latent Difference
We conducted additional experiments using latent-space differencing in place of our dynamic modu- lation mechanism. The latent difference (30] in MFRL is defined ô; 2t 2 where 2, and 2t-1 are latent features extracted at consecutive time steps by the same encoder. Similarly, we also experimented with using 6, in place of de as input to the sequencemodel fo In this setting, the sequence model is changed into:
Sequence model: h, (he-1 11 5 a1-1):
(22)
where the dynamic modulation is retained, coming from & but not the dynamic modulators de. Both the computation and encoding of differential observations are removed. The results show that this approach fails to match DyMoerea'er's performance, particularly in environments like Road Runner, where the critical objects occupy minimal pixels (e.g.. the seed and steel occupying just I pixcl). This occurs because: (1) The original encoding's 's precision limits small dynamic feature capture. (2) The categorical distributions in latent space lose fine-grained positional information. (3) Without the explicit dynamic modulation based on differential observations, frame-to hame-to-frame to-frame differences of small objects become unreliable. DyMoDreamrr's strength lies in its hybrid approach combining pixel-level dynamic detection with latent categorical encoding, which preserves critical small-scale dynamics while filtering noise.
Pong
80
9000
16000
6000
8000
3000
-15
50k
100k
0
50k
200k
50k
100k
50k
100k
Dreamer winh wtent di diferrncce
DyMoDreamer
Figure 16: Ablation study on latent difference RSSM.
We emphasize again that DyMoDremmer is fundamentally different from the latent flow, despite both being inspired by the general idea of differencing. DyMomremmer separates representation pathways: the dynamic modulator d, and the stochastic representation are derived from two
25

---

different encoders (Equation T_. Specifically, 2 is encoded from the full observations or, while is encoded from the differential observations of derived from the naae-differencing and processed by distinct dynamic encoder. Due to the characteristics of CNNs, the encoding of differential observations not equal to the difference between stochastic representations:
2 - * dN ~ go h,),
(23)
where the dynamic modulation mechanism allows DrMomrr to retain subtle yet decision-critical dynamic cues that might be overlooked by 2 and 1 alone.
J.4 DyMoDreamer with Harmonious Loss
An intriguing question whether the inclusion of additional loss terms or adjustments to existing ones during the training of the world model impacts its effectiveness. The answer is unequivocally yes: even without introducing new loss terms, modifying the weights of existing ones can significantly influence the model's performance [4]. Recent advances, such as Harmony Dream [44, highlight the benefits of adaptively adjusting the weights of different loss terms during training. By employing a harmonious loss framework [65]. this approach achieves performance superior to that of the vanilla DreamerV3. In this section we provide a detailed discussion on the impact of incorporating harmonious loss within DyMomremeer demonstrating its potential for improved training efficiency and effectiveness. Adjusting the coefficients of different loss terms has the potential to significantly enhance the performance of world-mode model-ooaaeeeeeeeeee -bsed reinforcement learning algorithms 44] For DyMoereamer, compared to Dreamer 33, we add three additional loss terms during the training of the world model, further complicating the challenge of tuning the initial loss coefficients. To address this, adopt a harmonious loss strategy inspired by HarmonyDream 44, dynamically adjusting the dominance between the observation and the reward modeling during world model training process. In this section, we explore the impact of this approach on the performance of DyMorremerr
To simplify the presentation and avoid excessive complexity, consider the vanilla end-to-end loss as below:
((Φ) Wing (d) + Wrew Crew (rre(()) (Ø) Weon Weon (d) + wdyn Cayn (Ø)
(24)
+ Wrep Crep (1)) + Wrog (re
Then the harmonious loss is constructed as:
(x( = Σ
1 Li + 1m(1 + wi)
(25)
Wi
where E {img, rew, con dyn, rep, reg], and specifically. inspired by bine the dynamics and representation losses into (d) as follows:
we recom-
(a( (o) (1 ax) (1).
(26)
And a is the KL balancing coefficient predefined the same as ( The reason why we use In(1 + wi) as regularization terms is that wi is parameterized as w; exp(w;; 0 to optimize parameters wi free of sign constraint. Since a loss item with small values, such as the reward loss, can lead to extremely large coefficient 1 1/w; Rs L 1 > 1, which potentially hurt training stability. The derivation the harmonious loss scale as follows.
To minimize B[Ca (Ø) wx)], we force the the partial derivative w.r.t. to O:
V.. (.,, Vo
B[C] + In(1 + w))
1
E(C
1+
(27)
-- E[4] + EA + 4E
2
Therefore the learnalle loss weight, in the rectified harmonious loss, approximates the analytic loss weight:
2
(28)
E[A] + /
C
and equivalently, the harmonized loss scale is:
2
B
< 1,
(29)
B[C]
26

---

Baxing
Krull
Pong
Road Runner
80
12000
16000
8000
40-
4000
8000
-15
50k 100k
S0k
100k
50k
100k
50k
100k
Harmonious less
DyMoorrammer
Figure 17: Ablation studies on DyMoDremmer with harmonious loss.
and add the regularization term In(1 + wi) results in the 4/E|C] in the v I 4/18|C] term, which prevents the loss weight from getting extremely large when faced with small E[C]. Regrettably, Figure 17 indicates that while the harmonious loss achieves performance comparable to fixed parameters only in the Pong task, but falls short in Krull, Boxing and Road Runner tasks. A potential explanation for this discrepancy is the increased complexity introduced by additional loss terms. The learning rate inherited from the original settings may no longer be optimal, as it appears overly aggressive for adapting the harmonious coefficients.
J.5 Dimensions of the Dynamic Modulation
The proportion of dynamic objects varies significantly across tasks, evidenced by the differential mask rate. For instance, in the Pong task, the average mask rate (AMR) for single episode reaches as high as 98.5%, reflecting the minimal volume of dynamic components. In contrast, the AMRs for single episodes in Boxing and Krull tasks are 94.3% and 83.1 %, respectively, due to the larger proportion of dynamic elements in these environments. Consequently, the different dimensionality of de can significantly impact the performance world models. For instance, OC-STORM treats number of objects hyperparameter, and manually adjustis it for different tasks. This task-specific modification results in varying dimensions of the stochastic encoding output by the visual model, which partially enhances OC-STORM's performance (21). Furthermore, the relative size between 2 and de could also influences performance. Increasing the dimension of d, effectively enlarges the latent variable space, potentially introducing redundant information that could hinder the agent's decision-making process.
Baxing
Knull
Fong
Road Runner
90
15
60
6000
16000
3000
8000
-15
50k
100k
0
50k
100k
50%
100k
50k
100k
DyMoDreamer (16+16 dj)
DyMoDreamer (32-32 di)
Figure 18: Ablation studies on the dimension of dynamic modulation.
The results in Figure 18 indicate that using 16 x 16 classified dynamic modulators does not fully encode the dynamic information from the differential observations. In other words, low-dimensional dynamic modulation fails to accurately integrate dynamic information into the RSSM, thereby not enriching the decision- decision-making process.
J.6 Longer Difference Interval
Although using single-frame differences already yields substantial performance improvements, include an ablation study on the difference interval & in this section. Specifically, we set k: = to evaluate DymoDreamer, and present the results in Figure 19
The results show that using a longer differencing interval encourages world model to focus on longer-term changes rather than transient noise local perturbations. This often leads to better performance smoother environments, may fail capture critical dynamics occurring in intermediate frames, such as the rapid punching in Boxing.
27

---

Baxing
Kru
Pong
Road Runner
90
24000
15
60
16000
3000
0000
-15
50k 100k
50k
100k
50k
100k
50k
100k
DyNlsrramm whhkk-3
DyMoorraamer
Figure 19: Ablation studies on k = 3.
J.7
Dimensionality of the Stochastic Representations
We additionally examined how the dimensionality of the original stochastic representations žp influences overall performance. The results are summarized in Figure 20] Our findings indicate that low-dimensional leads to degraded performance. As we discussed in Appendix H does not merely encode static content. Visualization suggests that ž tends to capture dynamic elements environment that are beyond the agent's control, whereas the dynamic modulators de focuses more on agent-comee controllable dynamics (despite being decoded from all dynamic components). Therefore, reducing the dimensionality of 2 limits the model's capacity to capture rich environmental dynamics, leading to a notable drop overall performance.
Baxing
Krull
Pong
80
9000
16000
6:00
8000
100k
50k
100k
DyMaDreamer 2:)
DyNoDreamer
Figure 20: Ablation studies on the dimension of stochastic representations.
28

---

K
Hyperparameters
Table details the hyperparameters of the optimization and environment, well as hyperparameters shared by multiple components. The environment will provide "done" signal when losing a life, but will continue running until the actual reset occurs.
Table 7: Hyperparameters
Name
Value
General
Batch size
16
Batch length
64
Activation
LayerNorm + SiLU Adam
Optimizer
World Model
Learning rate
10 4
Adam epsilon
10
Gradient clipping
1000
Actor Critic
Imagination horizon
15
Discount horizon
333
Return lambda
0.95
Critic EMA decay
0.98
Critic EMA regularizer
1
Actor entropy scale
3 x 10
Learning rate
3 10-5
Adam epsilon
10
Gradient clipping
100
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims made in the title, abstract and introduction are limited to the paper's scope, and match results in Section B] We include motivational example from the infant cognition in the introduction but it is clear that this scenario is motivational and not addressed by the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our work in a separate "Limitations" section (see Section B). and analyze the trade-offs in computational efficiency Section 3.2
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
30

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We do not have novel theoretical results, but instead apply the existing theoretical framework of dynamic modulation world modeling. Section provides the necessary theoretical background to dynamic modulation for world modeling in a clearly stated and referenced way.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all the necessary details to reproduce our results. Our method is explained in Section with the full training procedure detailed in Appendix E We provide full details of hyperparameters in Appendix K and reinforcement learning objectives in Appendix D Additionally our entire codebase will be spenesource, and shown in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
31

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The environments consider open-source, and provide our codes and results in the the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ mnips.cc/publiic/
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all hyperparameters, including environment configurations, number of epochs, batch sizes, data collection schedules, optimizer settings, architectures and reinforcement learning details Appendices.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We follow the official evaluation protocol provide confidence intervals for our experimental result.
32

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We detail the type of GPU, memory and time of execution Section 3.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpt///i////////p/pooooooo/oooa http://ppw....pppppiacccccccccaa
Answer: [Yes]
Justification: We do not foresee any harms caused by the research process (there are human subjects involved in our work, and the environments we use are open-source), and also do not foresee any direct harmful societal impact of our work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
33

---

Answer: [Yes]
Justification: Our introduction discuss the general motivation for our research, and position it into broader picture. We consider both positive and negative societal impacts in Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: We only release pretrained models on the benchmarks we evaluated our method, and do not believe these pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We' We've made our best effort to include all relevant citations and licenses in our paper and codebase.
Guidelines:
34

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We will provide detailed documentation alongside our codebase.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing research with human subjects.
35

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs are not important, original, non-standard component of this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy for what should or should not be described.
36


Reviews:
[{"review_id": "hzz0dgNhQZ", "reviewer": "Reviewer_z8ms", "summary": "DyMoDreamer introduces a low‑cost extension and high efficiency to Dreamer‑style (Hafner et al) world models, introducing a scalable way to incoporate dynamic object information without any explicit priors. \n\nThe paper argues that the recurrent state‑space model (RSSM) of Dreamer V3 with an extra categorical \"dynamics modulator\" $d_t$,  that is inferred from frame‑difference masks and concatenated to the usual stochastic latent z_t. This forces the model to attend to motion cues without expensive vision pre‑training and with minimal extra compute. The paper shows state‑of‑the‑art scores on Atari 100k, DeepMind Control (pixels) and Crafter, plus ablations about their design choices.", "questions": "The paper's \"latent‑difference\" ablation simply subtracts the quantised categorical codes: z_t - z_{t-1} and feeds the raw vector into the RSSM, with no re‑encoding, normalisation, or spatial structuring. That choice likely under‑represents what a proper latent‑difference approach can do. It would be good to see more detailed discussion about this. \n\nAdditionally, it would be good to see more discussion or experimantation with more explicit approaches which directly model object dynamics (with physics or 3D priors for example).", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nI think this paper tackles an important problem of introducing object level information without any explicit tracking, bounding boxes etc, and is inspired from how human think about *differences* between scenes and not just scene features. \n\nAdditionally, the paper is clear, well‑explained method. The paper is easy to follow and the algorithmic additions are minimal yet well-motivated. It is also provides strong empirical evidence to support the claims. Across three benchmarks, DyMoDreamer improves on previous state of the art results, such as Dreamer V3. Moreover, The ablations are quite informative for example, removing the modulator or differential divergence regularizer drops Atari performance. Finally, the writing is clear and there are many details which enhance reproducibility, i. e, implementation choices, loss terms, and training schedules are spelled out. \n\nWeakness:\nIn my opinion, there is limited evaluation beyond fixed‑camera games. Unlike DIAMOND (which at least simulates a first‑person CS:GO environment) and Dreamer V3 (which learns on 30 egocentric DMLab tasks), DyMoDreamer reports results only on fixed‑view benchmarks. Therefore, it remains unclear whether its frame‑difference modulation can handle rapid viewpoint changes, parallax, or background motion typical of true ego‑centric domains.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the comprehensive evaluation and respond to the raised concerns below, specifically regarding the generalizability of DyMoDreamer to egocentric domains, the rationale behind the latent-difference ablation design, and the necessity of discussing explicit object dynamics or 3D modeling approaches. All additional experiments referenced in this rebuttal will be included in the final version of the paper.\n\n**1. Evaluation on egocentric DM Lab tasks**\n\nIn response to the concerns about the evaluation on more explicitly egocentric environments, we conducted additional experiments on the DeepMind Lab benchmark to further validate DyMoDreamer's effectiveness under rapid viewpoint changes. To the best of our knowledge, among existing world model literature, only DreamerV3 has been evaluated on this benchmark. Unfortunately, we were unable to locate a readily integrable pip package or a pre-configured Dockerfile for streamlined deployment, therefore setting up the environment and managing dependencies via Bazel incurred substantial overhead during our supplementary experimentation phase. Moreover, conducting a full evaluation across DM Lab benchmark would require approximately **30 (tasks) $\\times$ 2.9 (days per task) $\\times$ 5 (seeds) = 435 (A100 GPU days)**. To make the evaluation tractable, we shorted the evaluation checkpoint at 10M environment steps (100M steps in the DreamerV3 paper) and focused on several representative tasks with relatively dense reward signals, which are more likely to exhibit performance improvements within 10M steps [1] to assess DyMoDreamer’s performance. To ensure the fairness and quality of our results, we also reproduced DreamerV3 results using the official code and configurations, the corresponding results are reported below. \n\n| Task | DyMoDreamer | DreamerV3 (reproduced) | \n|:--------------|------:|------:|\n| Explore Goal Locations Small |**164.5**|156.6| \n| Explore Object Rewards Many |  **28.2**  |  24.1 | \n| Lasertag Three Opponents Small |  **8.3**  | 8.2 | \n| Rooms Keys Doors Puzzle| 24.1 |  **26.2**  | \n| Rooms Watermaze |  **22.0**  |  21.2  | \n\nDyMoDreamer outperforms the baseline on 4 out of 5 tasks, demonstrating competitive performance particularly on tasks with localized motion rewards (e.g., Explore Goal Locations Small, Explore Object Rewards Many) and dynamic visual input (e.g., Rooms Watermaze). However, in Rooms Keys Doors Puzzle, which emphasizes long-horizon memory and discrete planning over motion cues, DreamerV3 shows a slight advantage, consistent with our analysis of task suitability. In such settings, although the differential observations may not isolate only the task-relevant objects, they nonetheless effectively suppress large portions of decision-irrelevant background. This facilitates more efficient exploration by directing the agent’s attention toward reward-associated regions of the scene. Furthermore, as we discussed in lines 95-98, differential observations are not merely a visual processing technique, they also serve as an embedding of temporal information into the model’s representation, enriching its capacity to reason over dynamic sequences. These results suggest that DyMoDreamer’s dynamic modulation mechanism offers generalizable benefits in visually rich and moderately egocentric environments, though it may be less effective in domains requiring complex, abstract reasoning or long-term symbolic memory.\n \n[1] Beattie C, et al. DeepMind Lab. arXiv:1612.03801, 2016.\n\n**2. Latent-difference ablation design**\n\nWe thank the reviewer for the valuable remark. Our initial choice, directly subtracting latent states to construct $\\delta_t=z_t-z_{t-1}$ without re-encoding or spatial structuring was intentional to establish a conservative baseline for comparison with DyMoDreamer. To ensure a more rigorous and fair ablation, we conducted an additional ablation experiment where $\\delta_t$ was not only fed into the sequence model ($h\\_t = f\\_{\\phi}(h\\_{t-1}, z\\_{t-1}, \\delta\\_{t-1}, a\\_{t-1})$), but also participated in the reconstruction ($ p\\_{\\phi}(\\hat{o\\_t}|h\\_t,z\\_t,\\delta\\_{t})$, $ p\\_{\\phi}(\\hat{r\\_t}|h\\_t,z\\_t,\\delta\\_{t})$, and $ p\\_{\\phi}(\\hat{c\\_t}|h\\_t,z\\_t,\\delta\\_{t})$) and the policy learning processes ($s\\_t= \\lbrace h\\_t,z\\_t,\\delta\\_{t}\\rbrace$). In this configuration, the usage of the latent flow $\\delta_t$ aligns with its formulation in the original work, except that it additionally participates in the reconstruction process. As reported in the table below, substituting $d\\_t$ with $\\delta\\_t$ does not lead to improved performance, further underscoring the unique effectiveness of our proposed dynamic modulation signal. As discussed in Section 2.3 (lines 187–194) and Appendix I.3, we empirically confirmed that differencing in latent space will fail to capture subtle but critical motion signals (e.g., 1-pixel ball in Pong) due to quantization and precision loss, ultimately leading to degraded performance.\n\n| Score        | Boxing  | Krull | Pong | RoadRunner |\n|:-------------|:------:|:-----------:|:---------:|:--------:|\n| DyMoDreamer  | **93.6** | **9624** | **21** | **20971** |\n|Replace $d\\_t$ with $\\delta\\_t$|  82.1  |   7461   |  15   |  16932  |\n\n**3. Discussion on modeling object dynamics**\n\nExplicitly modeling object dynamics using 3D priors or physics engines is an important direction. Our work emphasizes an orthogonal strategy: enhancing dynamic feature capture without explicit object segmentation or prior assumptions will improve the world model's performance. To more explicitly explore the incorporation of object dynamics through prior modeling, we conducted an additional experiment in which the differenced observations were replaced with a pretrained computer vision module \"Cutie\". Cutie, informed by prior knowledge, is capable of directly extracting key objects from the scene after pretraining. In this setting, the dynamics captured by $d\\_t$ (where $d\\_t$ are encoded from the Cutie's output in this case) corresponds exclusively to the motion of key objects, with other information effectively excluded. While this approach does yield performance gains comparable to those of DyMoDreamer, it introduces additional training overhead through reliance on supervised pretraining, and incurs a higher computational cost than our lightweight modulation mechanism.The results of this experiment are presented below. \n\n| Score        | Boxing  | Krull | Pong | RoadRunner |\n|:-------------|:------:|:-----------:|:---------:|:--------:|\n| DyMoDreamer  | **93.6** | **9624** | **21** | 20971 |\n|DyMoDreamer with Cutie|  92.1  |   9443  |  21   | **21852** |\n\nWe hope our response has resolved the concerns, and please let us know if any additional clarifications are required."}, {"review_id": "airhOAEFmO", "reviewer": "Reviewer_2wbS", "summary": "The paper introduces DyMoDreamer, an extension of DreamerV3 that aims to decouple dynamic elements from static components in pixel-based observations. Dynamic modulation is achieved by (1) including differential observations as additional inputs, encoded in a dedicated latent space, and (2) incorporating a loss term that emphasizes better prediction of inter-frame variation. The method demonstrates strong performance on three benchmarks (Atari100k, DeepMind Control Suite, and Crafter) achieving or surpassing state-of-the-art results.", "questions": "1. Dimensionality of $z$: In the provided examples $z$ seems to mostly capture static backgrounds. Did the authors experiment with the dimensionality of the static latent $z_t$? Have you considered using a learned but fixed/static $z$ as a baseline?\n2. Clarification on the \"Removing Dynamic Modulation\" Ablation: Could you clarify what is meant by this ablation? Is the decoder predicting both the original and differential images jointly? Are these inputs concatenated, or are separate decoder heads used?\n3. Baseline with $\\mathcal{L}_\\mathrm{reg}$: Is there an ablation experiment comparing against a version of DreamerV3 that incorporates only the differential divergence regularization loss (without modulation)? Comparing against such a baseline would help disentangle the contribution of the modulation mechanism from the loss term.\n4. Applicability Beyond Images: DreamerV3 has also been used for environments with state-based inputs. Could DyMoDreamer extend to such cases? Do the authors believe that dynamic modulation would provide similar benefits in non-visual settings?\n5. Broader Impact of Architectural Biases: Does separating dynamic and static latents introduce inductive biases that may hurt performance in environments where this assumption doesn’t hold (e.g., rapidly changing backgrounds)?\n\n\nMinor comments:\n- Line 27: “A key approach to addressing this challenge is the world model [...]. This framework combines a VAE with a recurrent neural network (RNN) and uses evolutionary strategies to optimize the policy within a latent space.” The first sentence reads as a general statement about world models, but the second appears to describe the specific approach of Ha & Schmidhuber. It would be helpful to clarify which parts refer to the general framework versus this specific implementation.\n- Equations 2 & 9: The indices $i,h,w,c$ should be introduced in natural language before or alongside the equation. Clarifying what these dimensions represent (e.g., batch index, height, width, channels) would improve readability and help readers unfamiliar with the notation.", "limitations": "Limitations are not clearly addressed. The paper would benefit from a dedicated discussion of limitations. For example, a brief analysis of scenarios where dynamic/static separation could fail would strengthen the contribution.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- an original extension of the Dreamer line of work, emphasizing the modeling of dynamic structure in visual inputs. \n- strong empirical performance across a diverse set of benchmarks, which showcases the method's significance to the model-based RL community.\n- the paper is generally well-written, and the figures and illustrations clearly communicate key ideas.\n\nWeaknesses:\n- certain ablation details are either missing or under-explained, e.g., the role of the dimensionality of the static latent component ($z_t$) and the inclusion/exclusion of regularization terms like $\\mathcal{L}_\\mathrm{reg}$ from Dreamer.\n- the method is closely tied to Dreamer-style architectures and pixel-based inputs; its generality or applicability to other world models or state-based inputs is unclear, which limits broader impact.\n- the related work section focuses exclusively on world models, even though the approach appears to draw from modulated ODEs and latent flow methods. Including a discussion of these inspirations would help clarify the method’s novelty.", "was_revised": true, "final_justification": "The dynamic modulation mechanism introduced in this paper leads to strong performance across several benchmarks. While some reviewers raised concerns about the narrow scope of environments (e.g., egocentric views), the authors addressed this by including additional evaluations in DMLab. My own concerns regarding missing baselines (regularization term, applicability beyond pixel-based inputs) were addressed through new ablations. The authors also demonstrated how dynamic modulation can be applied to another world model architecture (STORM), supporting the generality of the approach, albeit on a model structurally similar to Dreamer. They additionally clarified the paper’s limitations, which I believe should be stated more clearly in the final version. Overall, my concerns have been addressed and my questions answered, so I recommend acceptance. However, as the paper is mainly of interest to the model-based RL community, I did not assign a higher score.", "rebuttal": "We sincerely thank the reviewer for the insightful comments. Below, we address each point in detail, and we will incorporate clarifications and additional results in the final version.\n\n**1. Dimensionality of stochastic representations $z\\_t$**\n\nWe analyzed the effect of dynamic modulators' dimensionality in Appendix I.5, and additionally examined how the dimensionality of the original stochastic representations $z\\_t$ influences overall performance. The results are summarized in the table below. Our findings indicate that a low-dimensional $z\\_t$ also leads to degraded performance. As we discussed in Appendix H, $z\\_t$ does not merely encode static content. Visualization suggests that $z\\_t$ tends to capture dynamic elements in the environment that are beyond the agent’s control, whereas the dynamic modulators $d\\_t$ focuses more on agent-controllable dynamics (despite being decoded from all dynamic components). Therefore, reducing the dimensionality of $z\\_t$ limits the model’s capacity to capture rich environmental dynamics, leading to a notable drop in overall performance. This is why we don't use a completely static fixed $z$. In fact, in our early experiments, we have explored a variant where a fixed static vector $z$ replaced $z\\_t$ to encode/decode only the background (achieved by averaging over the batch length dimension), while only the dynamic modulators $d\\_t$ were used to drive the sequence model. Under this configuration, the performance was comparable to or worse than vanilla DreamerV3, as the static background can shift over the course of an episode (e.g., the level transitions in Krull). A fully fixed  $z$ lacks the flexibility to adapt to such stage-wise changes in the static background, ultimately hindering representation quality and decision-making.\n\n| Score        | Boxing  | Krull | Pong | RoadRunner |\n|:-------------|:------:|:-----------:|:---------:|:--------:|\n| DyMoDreamer  | **93.6** | **9624** | **21** | **20971** |\n| DyMoDreamer with $16 \\times 16$ $z\\_t$| 79.0 |   7612   |  16.3   |  16732  |\n\n**2. Clarification on the “Removing Dynamic Modulation”**\n\nIn this ablation, a single decoder is used to reconstruct only the original observation $o\\_t$, without reconstructing the differential observations $o'\\_t$. The decoder receives a concatenation of $z\\_t$ (encoded from  $o\\_t$) and $d\\_t$ (encoded from  $o'\\_t$) as input, which is fed through a shared decoding head (the reconstruction process is formulated the same as Equation (1)). In this configuration, $d\\_t$ is not explicitly embedded into the RSSM as in our dynamic modulation design.\n\n**3. Baseline with $\\mathcal{L}\\_{reg}$**\n\nWe thank the reviewer for this insightful comment. In response, we implemented a baseline using DreamerV3 with the same divergence regularization term $\\mathcal{L}\\_{reg}$ as DyMoDreamer but without dynamic modulation. This variant showed minor improvement over DreamerV3 but underperformed compared to DyMoDreamer (in the following table). The results demonstrate that the differential divergence regularization acts as a complementary supervisory signal that works in concert with reconstruction objectives, and contributes improvement through its unique ability to constrain the temporal consistency in the reconstruction process of vanilla DreamerV3.\n\n| Score        | Boxing  | Krull | Pong | RoadRunner |\n|:-------------|:------:|:-----------:|:---------:|:--------:|\n|DreamerV3 (reproduced)  |80 | 7834 | 18 |**15248**|\n|DreamerV3 with $\\mathcal{L}\\_{reg}$ |**82** |**8240**|**18.1** |13649| \n\n**4. Applicability beyond images and generality to other world models**\n\nTo evaluate the applicability of DyMoDreamer in state-based environments, we conducted experiments on DeepMind Proprio—a proprioceptive control benchmark where observations consist of structured state vectors such as velocity and position. In this setting, observation differencing naturally reduces to state differencing, with temporal dynamics being more explicitly encoded. Consequently, differential modulation provides similar benefits in non-visual settings, since such differences also carry meaningful physical semantics similar to the differential observations—for instance, differential positions implicitly encode velocity, while the differential velocities reflect acceleration.\n\n| Score        | Acrobot Swingup  | Cheetah Run | Finger Spin |\n|:-------------|:------:|:-----------:|:---------:|\n|DyMoDreamer |**225**|**625**|**932**|\n|DreamerV3|134 |614|**931** |\n\nTo further validate the effectiveness of our dynamic modulation mechanism, we integrated it into STORM [1], a recent efficient transformer-based world model. As shown in the results below, this integration also leads to clear performance gain. \n\n| Score        | Boxing  | Krull | Pong | RoadRunner |\n|:-------------|:------:|:-----------:|:---------:|:--------:|\n|STORM (reproduced)  |81 | 6824 | 18 |13866|\n|STORM with dynamic modulation |**85** |**8563**|**19.1** |**19337**|\n\n[1] Zhang W, et al. STORM: Efficient stochastic transformer based world models for reinforcement learning. Advances in Neural Information Processing Systems, 2023, 36: 27147-27166.\n\n**5. Broader impact of architectural biases**\n\nA naive separation of dynamic and static components may introduce inductive biases in certain environments. To mitigate such issues (dynamic backgrounds or environmental noise), we provide several alternative differencing strategies in Appendix G, including multi-frame logical differencing and moving average differencing. Importantly, the dynamic modulation mechanism is highly compatible with a wide range of advanced, difference-based computer vision techniques. This extensibility suggests that DyMoDreamer possesses strong generalization potential, as differencing is a foundational principle in many visual detection methods.\n\n**6. Discussion of modulated ODEs and latent flow**\n\nWhile modulated ODEs also employ modulation operators to enhance the temporal modeling in neural ODEs, our approach differs in several key aspects. In modulated ODEs, modulators are **time-invariant** and trained as a global statistic across the entire sequence, whereas DyMoDreamer extracts modulation explicitly from differential observations at each timestep. Additionally, modulated ODEs encode modulators from the full input sequence rather than isolating dynamic cues via differential observations. As a result, modulated ODEs are primarily suited for modeling passive, uncontrolled dynamics, whereas our dynamic modulation framework enables explicit modeling of environment transitions conditioned on agent actions.\n\nThe latent flow methods largely focused on **model-free reinforcement learning** and do not introduce dedicated encoders or decoders for the latent flow  $\\delta\\_t$. The latent flow $\\delta\\_t$ are not designed to be predictive but serve as auxiliary information injected into decision-making. In contrast, DyMoDreamer explicitly incorporates the dynamic modulators into the reconstruction process to mitigate hallucinations in dynamic regions (as shown in Fig. 3). Moreover, the modulators $d\\_t$ are fully predictive and plays a central role in improving the world model’s capacity to model temporally coherent dynamics.\n \n**7. Revision in the manuscript**\n\nWe gratefully appreciate your valuable comments and suggestions, and we will revise the manuscript accordingly:\n \n$\\bullet$ We will clarify the distinction by rewriting the sentence in line 27 as:\n“A key approach to addressing this challenge is the general framework of world models, which learns compact latent dynamics for planning and decision-making. One early instantiation of this framework, proposed by Ha & Schmidhuber, combines a VAE with an RNN and uses evolutionary strategies to optimize policies in the learned latent space.”\n\n$\\bullet$ Equations (2) & (9): We will introduce the indices in natural language before the equations to improve clarity. Specifically, we will indicate the meaning of each index (e.g.,$i$ is the time dimension, $h$ and $w$ are spatial dimensions, and $c$ is the channel dimension) to assist readers unfamiliar with the notation.\n\nWe appreciate the reviewer’s thoughtful feedback and hope that our rebuttal has addressed your concerns. Please kindly let us know if further details are required."}, {"review_id": "IzLsRPUvKD", "reviewer": "Reviewer_zbN2", "summary": "This paper proposes DyMoDreamer, a new model-based RL (MBRL) algorithm with a dynamics modulation mechanism to extract and utilize dynamic features of the observation. The model builds on DreamerV3 and combines a masking function to extract dynamical parts of the observation and a regularization loss to motivate the model to capture dynamic changes of the environment better. In experiments, DyMoDreamer outperforms other MBRL algorithms, as the model can attend better to parts of the observation with dynamical changes.", "questions": "- Related to Weakness 2, you mention that dynamic parts of the observation are often correlated with rewards (line 175). Given this, have you considered conditioning the dynamics modulation mechanism on the agent's actions? It seems this could be a possible direction to help the model focus on extracting parts of the image that are task-related as well?\n- While I agree that both dynamic and static parts of the observations are important for task solving (line 182), could you elaborate on the design choice not to fully disentangle these two information channels in the model's representation (e.g., by creating separate static (1-M(o_t)) ⋅ o_t and dynamic M(o_t) ⋅ o_t pathways)?\n- I think one of the motivation behind this paper was to improve the model’s capability to capture dynamics of small, moving objects. To further strengthen this claim beyond the helpful visualizations, have you considered a more quantitative evaluation? For example, using object segmentation or tracking metrics on the extracted dynamic masks could provide a powerful, objective measure of the model's capability in this area.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper proposes a novel MBRL model, DyMoDreamer, that improves capturing dynamical changes in the environment by encoding temporally differential observations. DyMoDreamer can predict small dynamic objects’ movements better, which is also useful for the policy learning as well.\n- Quantitative results show DyMoDreamer outperforms other RSSM- and Transformer-based MBRL algorithms on 3 benchmark datasets. Qualitative results show that the differential observations extracted by DyMoDreamer are the objects that dynamically change in the environment.\n- The paper is clearly written, well-structured, and includes a thorough set of ablation studies that provide valuable insights into the contributions of the different architectural components. The figures are informative and effectively support the paper's claims.\n\nWeaknesses:\n\n- The experiments are only conducted in allocentric environments. The environments that this paper uses all have a third-view perspective, allowing the model to “see” the agent itself and its relationship with other players/objects. However, in egocentric environments, such as maze navigation, all the pixels will dynamically modulate due to changes in the agent’s perspective. This will likely lead to the model failing to extracting objects dynamics important in solving the task. This raises the question of the applicability of DyMoDreamer in broader contexts.\n- As I understand, the main assumption of the paper is that all dynamically changing aspects of the environment is important for solving the task. However, in more realistic environments, there may be more noises such as moving foreground objects or background noises which are not important. As the extraction is conducted using a simple binary function with unlearnable threshold, the model may fail to distinguish the important agent and objects’ changes and noises that change unrelated to the agent’s actions. This raises a question about the model's robustness in more complex, noisy settings.\n- The paper has several minor grammatical errors and typos\n    - The sentence in line 130 “Considering that …” seems to be grammatically incorrect\n    - In line 667, progcess should be process", "was_revised": true, "final_justification": "Overall, my concerns have been largely addressed and my questions answered, and I recommend borderline accept.", "rebuttal": "We sincerely thank you for the constructive feedback on our paper. Below, we address each concern raised, and all new experiments conducted in the response will be incorporated into the final version of our manuscript.\n\n**1. Generalization to egocentric environments DeepMind Lab**\n\nTo evaluate DyMoDreamer's generalizability, we conducted additional experiments on the DeepMind Lab benchmark, which is an egocentric 3D game platform designed for RL and contains difficult tasks (e.g.,maze navigation and objects exploration). Classical methods (IMPALA and R2D2+) typically require over 1B environment steps to achieve non-trivial reward performance. Since our main goal is to improve the sample efficiency of RL algorithms, we shorted the evaluation checkpoint at 10M environment steps (100M steps in the DreamerV3 paper) and focused on several representative tasks with relatively dense reward signals, which are more likely to exhibit performance improvements within 10M steps [1] to assess DyMoDreamer’s performance. To ensure the fairness and quality of our results, we also reproduced DreamerV3 results using the official code and configurations, the corresponding results are reported below. \n\n| Task | DyMoDreamer | DreamerV3 (reproduced) | \n|:--------------|------:|------:|\n| Explore Goal Locations Small |**164.5**|156.6| \n| Explore Object Rewards Many |  **28.2**  |  24.1 | \n| Lasertag Three Opponents Small |  **8.3**  | 8.2 | \n| Rooms Keys Doors Puzzle| 24.1 |  **26.2**  | \n| Rooms Watermaze |  **22.0**  |  21.2  | \n\nDyMoDreamer outperforms the reproduced DreamerV3 baseline on 4 out of 5 tasks, showcasing strong performance particularly in environments with localized motion-dependent rewards (e.g., Explore Goal Locations Small, Explore Object Rewards Many) and dynamic visual input (e.g., Rooms Watermaze). In contrast, Rooms Keys Doors Puzzle—which prioritizes long-horizon memory and discrete, symbolic planning over visual motion cues—favors DreamerV3, aligning with our analysis of environment suitability.\n\nWhile the differential observations in DyMoDreamer may not perfectly isolate task-relevant entities, they significantly reduce decision-irrelevant background clutter (e.g., the sky background). This suppression enhances exploration efficiency by directing the agent’s attention toward regions more likely to yield reward. Furthermore, as we discussed in line 95-98, differential observations are not merely a visual processing technique, they also serve as a direct embedding of temporal information into the model’s representation, enriching its capacity to reason over dynamic sequences. Collectively, these results indicate that DyMoDreamer’s dynamic modulation mechanism also provides robust and generalizable benefits in visually complex and moderately egocentric domains, though its advantage may diminish in settings dominated by abstract reasoning or delayed, non-perceptual dependencies.\n\n[1] Beattie C, et al. DeepMind Lab[J]. arXiv:1612.03801, 2016.\n\n**2. Robustness in noisy or dynamic background**\n\nOur default differencing mechanism employs a binary mask with a fixed threshold to highlight DyMoDreamer's generality and compatibility with a wide range of differencing strategies. In practice, DyMoDreamer can be readily extended to incorporate more robust alternatives, such as logical differencing to mitigate flickering backgrounds and exponential moving average differencing to suppress noise, as detailed in Appendix I.5. More importantly, the architecture is fully compatible with advanced dynamic extractors, including optical flow and attention-based motion segmentation modules, enabling a flexible and extensible framework suitable for complex visual scenes. Furthermore, our experiments in DeepMind Lab demonstrate that foreground object motion often conveys egocentric relative dynamics, which can be informative and beneficial for downstream policy learning.\n\n**3. Dynamical modulation and agent actions**\n\nWe thank the reviewer for the thoughtful suggestion to condition the dynamic modulation mechanism on the agent’s actions. We agree that this is a promising direction, and in fact, we believe that the current version of DyMoDreamer already implicitly incorporates this idea. As our model is trained end-to-end, the dynamic modulators participate in the temporal inference process ($f\\_{\\phi}$  in Equation (3)) alongside the agent’s action, contributing to the prediction of future latent states. The world model is optimized jointly via a reconstruction and policy loss on fixed replay sequences, which inherently couples the modulation signal with the action trajectory. While we do not explicitly encode the action $a\\_t$ into the computation of $d\\_t$ , the action influences the next latent state $h\\_t$, which in turn affects the encoding and interpretation of subsequent modulators $d\\_t$, since the encoding process of  $d_t$ is $d_t \\sim q_\\phi\\left(d_t \\mid h_t,  o'_t\\right)$. Thus, DyMoDreamer performs implicit action-conditioned modulation through temporal feedback in the recurrent dynamics model. We provide supporting empirical evidence for this interaction in Appendix H, although we did not explicitly constrain $d_t$ to focus on the part controlled by the agent, it naturally focuses more on agent-controllable dynamics (despite being decoded from all dynamic components).\n \n**4. Partial vs full dynamics-static disentanglement**\n\nWe intentionally avoid explicitly separating dynamic and static components using hard static masking $(1-M(o\\_t) \\cdot o\\_t)$ for the following reasons. \n\nFirst, our differential observations are derived from frame-differencing followed by dilated convolutions, but not from sophisticated edge-aware segmentation techniques. As a result, the resulting dynamic masks $M(o\\_t)$ highlight motion-correlated regions rather than clearly delineated object boundaries. The edges of these masks may not align with semantic object contours, and emphasizing such boundaries through hard segmentation may overemphasize potentially irrelevant edge pixels while neglecting the truly decision-relevant patterns within the masked regions $M(o\\_t) \\cdot o\\_t$.\n\nMoreover, since the differenced observation includes not only the changed pixels but also their local neighborhoods (as we metioned in lines 131-134), directly enforcing a binary split could disrupt the integrity of reward-relevant static features. For instance, partial inclusion of an important static object across both streams could prevent either $z\\_t$ or $d\\_t$ from capturing it in full, ultimately degrading decision quality. Our joint encoding approach circumvents this issue by allowing the model to learn a soft, task-driven decomposition guided by reconstruction and policy optimization signals.\n\n**5. Quantitative evaluation of dynamic feature modeling**\n\nStandard RL benchmarks typically lack precise real-time annotations, and more complex environments such as DMLab introduce additional visual challenges including stage-wise transitions and shifting viewpoints. As a result, there are no established segmentation or tracking metrics readily applicable for evaluating dynamics modeling within RL settings. In this work, we rely on task performance, imagination accuracy, and qualitative visualizations (Fig. 3, Appendix F) to assess the model's capacity for dynamic feature modeling. We believe these provide sufficiently comprehensive and task-relevant evidence in the absence of standardized evaluation protocols.\n\n**6. Grammatical errors and typos**\n\nWe gratefully thank you for the careful check on our manuscript, and we will revise the manuscript accordingly:\n\n$\\bullet$ Line 130: In fact, very few pixels may exceed the threshold $\\epsilon$ after being processed by $D(\\cdot)$.\n\n$\\bullet$ Line 667: This emergent specialization suggests an natural segregation within the model’s representational architecture like the denoised Markov decision process [59], since we do not explicitly assign reconstruction targets to different decoders.\n\nWe hope we have addressed all your raised points. If there are anything unclear, we would be happy to provide further clarifications."}, {"review_id": "CYsgun9iyA", "reviewer": "Reviewer_Whue", "summary": "This paper presents a novel mechanism that leverages pixel-level inter-frame differences to generate stochastic categorical modulators, which dynamically influence the recurrent state-space model to focus on task-relevant motion cues . This approach sidesteps the need for high-fidelity reconstructions or external object masks, resulting in a lightweight world model that boosts sample efficiency in high-dimensional environments .", "questions": "1.\tCould you report some results on environments like Procgen to demonstrate out-of-distribution generalisation?\n2.\tHow does the modulation branch behave in photorealistic or cluttered environments?\n3.\tPlease provide an aggregated ablation table so that the effect of each architectural component is immediately clear.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Elegant and Lightweight Design.\n2. Significant Empirical Improvements on Atari 100k\n\nWeaknesses:\n– All reported experiments use identical train/test distributions. Without a held-out-level benchmark such as Procgen, it is hard to tell whether the dynamic-modulation mask over-fits to textures seen during training.\n– Simple frame differencing can break down under motion blur, camera shake or illumination changes that typify photorealistic simulations or real-world robotics. The current evidence, limited to low-resolution games, leaves the method’s applicability to such settings uncertain.\n– The paper motivates pixel-space modulation with intuitive arguments and an empirical ablation, yet offers no information-theoretic or representational analysis explaining why pixel differencing consistently outperforms latent differencing.\n– Ablation studies are presented only as per-game evaluation curves for a handful of Atari titles. Without an aggregated table, readers cannot gauge the average or worst-case contribution of each component.\n- While the authors motivate a focus on world-model variants, readers will inevitably ask how the approach stacks up against strong model-free and search-based methods under the same data budget.", "was_revised": true, "final_justification": "Questions were answered.", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. Below we address each of the raised concerns in detail.\n\n**1. Results on the held-out-level benchmark**\n\nWe agree that generalization to unseen levels is crucial for assessing model robustness. Our current evaluation benchmarks (Atari 100k, DMC, and Crafter) are aligned with prior work focusing on sample efficiency and reward-driven dynamics modeling. Notably, the Crafter benchmark shares important characteristics with Procgen, such as procedurally generated content, stochastic transitions, and distributional shifts across episodes. Crafter already introduces distributional shift through its procedurally generated world, in which agent-centric motion relative to a dynamic background challenges traditional frame-differencing approaches. DyMoDreamer's superior performance on Crafter (9.5% improvement over DreamerV3) offers preliminary evidence that our dynamic modulation generalizes beyond static backgrounds. \n\nMoreover, conducting a full evaluation across Procgen benchmark with DreamerV3's settings (the hard difficulty setting and the\nunlimited level set) would require approximately **16 (games) $\\times$ 16.1 (days per task) $\\times$ 5 (seeds) = 1288 (A100 GPU days)**. Since reducing the evaluation checkpoint to shorter environment steps often results in underdeveloped policies to Procgen benchmark, we conducted additional experiments on the DeepMind Lab benchmark, which is an egocentric 3D game platform designed for RL and contains difficult tasks (e.g.,maze navigation and objects exploration). Considering that our main goal is to improve the sample efficiency of RL algorithms, we shorted the evaluation checkpoint at 10M environment steps (100M steps in the DreamerV3 paper) and focused on several representative tasks with relatively dense reward signals, which are more likely to exhibit performance improvements within 10M steps [1] to assess DyMoDreamer’s performance. To ensure the fairness and quality of our results, we also reproduced DreamerV3 results using the official code and configurations, the corresponding results are reported below. In response to concerns about out-of-distribution generalisation, we ensure that all evaluations are conducted with randomized environment seeds.\n\n| Task | DyMoDreamer | DreamerV3 (reproduced) | \n|:--------------|------:|------:|\n| Explore Goal Locations Small |**164.5**|156.6| \n| Explore Object Rewards Many |  **28.2**  |  24.1 | \n| Lasertag Three Opponents Small |  **8.3**  | 8.2 | \n| Rooms Keys Doors Puzzle| 24.1 |  **26.2**  | \n| Rooms Watermaze |  **22.0**  |  21.2  | \n\nDyMoDreamer outperforms the reproduced DreamerV3 baseline on 4 out of 5 tasks, showcasing strong performance particularly in environments with localized motion-dependent rewards (e.g., Explore Goal Locations Small, Explore Object Rewards Many) and dynamic visual input (e.g., Rooms Watermaze). In contrast, Rooms Keys Doors Puzzle—which prioritizes long-horizon memory and discrete, symbolic planning over visual motion cues—favors DreamerV3, aligning with our analysis of environment suitability.\n\nWhile the differential observations in DyMoDreamer may not perfectly isolate task-relevant entities, they significantly reduce decision-irrelevant background clutter (e.g., the sky background). This suppression enhances exploration efficiency by directing the agent’s attention toward regions more likely to yield reward. Furthermore, as we discussed in lines 95-98, differential observations are not merely a visual processing technique, they also serve as a direct embedding of temporal information into the model’s representation, enriching its capacity to reason over dynamic sequences. Collectively, these results indicate that DyMoDreamer’s dynamic modulation mechanism also provides robust and generalizable benefits in visually complex and moderately egocentric domains, though its advantage may diminish in settings dominated by abstract reasoning or delayed, non-perceptual dependencies.\n\n[1] Beattie C, et al. DeepMind Lab. arXiv:1612.03801, 2016.\n\n**2. Modulation in photorealistic or cluttered environments**\n\nWhile our current experiments are conducted in controlled simulators, the architectural design of DyMoDreamer emphasizes temporal change detection rather than reliance on appearance or segmentation priors. This property lends itself naturally to generalization in photorealistic or cluttered scenes (e.g., the Rooms Watermaze task in DeepMind Lab). Our default differencing mechanism employs a binary mask with a fixed threshold to highlight DyMoDreamer's generality and compatibility with a wide range of differencing strategies. In practice, DyMoDreamer can be readily extended to incorporate more robust alternatives, such as logical differencing to mitigate flickering backgrounds and exponential moving average differencing to suppress noise, as detailed in Appendix I.5. More importantly, the architecture is fully compatible with advanced dynamic extractors, including optical flow and attention-based motion segmentation modules, enabling a flexible and extensible framework suitable for complex visual scenes. \n\n**3. Benefits of pixel differencing over latent differencing**\n\nSimilar to most prior work on world models, we provide both intuitive reasoning and experimental evidence for pixel-space differencing in Section 2.3. Specifically, pixel differencing maintains spatial precision for small but critical features (e.g., 1-pixel objects in Pong) that may be lost in latent encoding due to compression. Moreover, our ablation studies in Appendix I.3 confirms that latent differencing ($\\delta_t = z\\_t − z\\_{t−1}$) underperforms pixel-based modulation, particularly when small motions are not captured by $z\\_t$ in the experimental tasks.\n\n**4. Aggregated ablation table**\n\nThank you for pointing this out. First, we would like to clarify that, as discussed in lines 294–297, the four Atari tasks selected for ablation were chosen because they exhibit distinct dynamic characteristics (e.g., sparse motion, and small/large object dynamics). These settings are closely aligned with the design goals of DyMoDreamer and allow for efficient and targeted evaluation of each architectural component’s contribution to dynamic modulation, since conducting all ablation studies across 26 Atari tasks are resource-intensive and infeasible effort given our time and funding budget at present. \n\nIn the final version, we will include an aggregated ablation table summarizing the performance of different ablation studies across the four Atari games (Boxing, Pong, Krull, Road Runner). A preliminary version of the table is:\n\n\n| Ablation Variant                   | Boxing | Krull  | Pong  | Road Runner |\n|-----------------------------------|--------|--------|-------|-------------|\n| DyMoDreamer      | **93.6** | **9624.8** | **20.9**  | **20971.8**  |\n| Removing dynamic modulation            | 80  | 7969   | 18.5  | 12536  |\n| Removing  $\\mathcal{L}_{reg}$ | 90  | 8961  | 20  | 20918   | \n| High-dimensional $z\\_t$ (48x48)        | 76   | 7325   | 18.2  | 16320       |\n| With differential reconstruction  | 71  | 7895   | 19.9  | 17323      |\n| Latent difference  |81   | 7855  | 20  | 12266   | \n| Low-dimensional $d\\_t$ (16x16)  |73  | 7423  | 19  | 17465   | \n\nThese results reinforce that the dynamic modulation mechanism is crucial for robust performance across diverse game types, and that simply appending difference images without architectural integration leads to suboptimal results (Appendix I).\n\n**5.Comparison with MFRL and search-based methods**\n\nCompared to model-free approaches, DyMoDreamer exhibits significantly higher sample efficiency and achieves substantially better performance on a number of Atari tasks (compared with BBF). This highlights the strength of world model–based reasoning in low-data regimes. Furthermore, unlike model-free methods that often require millions of environment interactions to converge, DyMoDreamer reaches competitive or superior performance within a constrained budget, showcasing its practical advantages in sample-limited settings. Search-based methods such as MuZero requires 8 TPUs for training per task, whereas our JAX implementation of DyMoDreamer achieves competitive performance with just 1 GPU for 5.5 hours, demonstrating significantly better computational efficiency while maintaining the benefits of world model learning.\n\nWe hope that our responses have adequately addressed all your concerns. Please feel free to let us know if any further clarification is needed."}]

Author rebuttal:

Meta-review: This paper introduces DyMoDreamer, a model-based RL algorithm that explicitly models dynamic objects via inter-frame differencing and dynamic modulation. The method is simple yet effective, achieving state-of-the-art results on Atari100K, DMC, and Crafter. The paper is well-motivated and clearly written.

The main limitation is that experiments are confined to relatively simple simulated domains, where its applicability to realistic scenes and tasks questionable. 

The rebuttal addressed reviewer concerns satisfactorily. 

Overall, I recommend acceptance: the contribution is novel, well-presented, and empirically strong.
Decision: Accept (poster)