Title: DyMoDreamer: World Modeling with Dynamic Modulation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Model-based reinforcement learning

world model

dynamic modulation

recurrent state space model
Abstract: A critical bottleneck in deep reinforcement learning (DRL) is sample inefficiency, as training high-performance agents often demands extensive environmental interactions. Model-based reinforcement learning (MBRL) mitigates this by building world models that simulate environmental dynamics and generate synthetic experience, improving sample efficiency. However, conventional world models process observations holistically, failing to decouple dynamic objects and temporal features from static backgrounds. This approach is computationally inefficient, especially for visual tasks where dynamic objects significantly influence rewards and decision-making performance. To address this, we introduce DyMoDreamer, a novel MBRL algorithm that incorporates a dynamic modulation mechanism to improve the extraction of dynamic features and enrich the temporal information. DyMoDreamer employs differential observations derived from a novel inter-frame differencing mask, explicitly encoding object-level motion cues and temporal dynamics. Dynamic modulation is modeled as stochastic categorical distributions and integrated into a recurrent state-space model (RSSM), enhancing the model's focus on reward-relevant dynamics. Experiments demonstrate that DyMoDreamer sets a new state-of-the-art on the Atari $100$k benchmark with a $156.6$\% mean human-normalized score, establishes a new record of $832$ on the DeepMind Visual Control Suite, and gains a $9.5$\% performance improvement after $1$M steps on the Crafter benchmark.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel mechanism that leverages pixel-level inter-frame differences to generate stochastic categorical modulators, which dynamically influence the recurrent state-space model to focus on task-relevant motion cues . This approach sidesteps the need for high-fidelity reconstructions or external object masks, resulting in a lightweight world model that boosts sample efficiency in high-dimensional environments .\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Elegant and Lightweight Design.\n2. Significant Empirical Improvements on Atari 100k\n\nWeaknesses:\n– All reported experiments use identical train/test distributions. Without a held-out-level benchmark such as Procgen, it is hard to tell whether the dynamic-modulation mask over-fits to textures seen during training.\n– Simple frame differencing can break down under motion blur, camera shake or illumination changes that typify photorealistic simulations or real-world robotics. The current evidence, limited to low-resolution games, leaves the method’s applicability to such settings uncertain.\n– The paper motivates pixel-space modulation with intuitive arguments and an empirical ablation, yet offers no information-theoretic or representational analysis explaining why pixel differencing consistently outperforms latent differencing.\n– Ablation studies are presented only as per-game evaluation curves for a handful of Atari titles. Without an aggregated table, readers cannot gauge the average or worst-case contribution of each component.\n- While the authors motivate a focus on world-model variants, readers will inevitably ask how the approach stacks up against strong model-free and search-based methods under the same data budget.\n\nQUESTIONS:\n1.\tCould you report some results on environments like Procgen to demonstrate out-of-distribution generalisation?\n2.\tHow does the modulation branch behave in photorealistic or cluttered environments?\n3.\tPlease provide an aggregated ablation table so that the effect of each architectural component is immediately clear.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes DyMoDreamer, a new model-based RL (MBRL) algorithm with a dynamics modulation mechanism to extract and utilize dynamic features of the observation. The model builds on DreamerV3 and combines a masking function to extract dynamical parts of the observation and a regularization loss to motivate the model to capture dynamic changes of the environment better. In experiments, DyMoDreamer outperforms other MBRL algorithms, as the model can attend better to parts of the observation with dynamical changes.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper proposes a novel MBRL model, DyMoDreamer, that improves capturing dynamical changes in the environment by encoding temporally differential observations. DyMoDreamer can predict small dynamic objects’ movements better, which is also useful for the policy learning as well.\n- Quantitative results show DyMoDreamer outperforms other RSSM- and Transformer-based MBRL algorithms on 3 benchmark datasets. Qualitative results show that the differential observations extracted by DyMoDreamer are the objects that dynamically change in the environment.\n- The paper is clearly written, well-structured, and includes a thorough set of ablation studies that provide valuable insights into the contributions of the different architectural components. The figures are informative and effectively support the paper's claims.\n\nWeaknesses:\n\n- The experiments are only conducted in allocentric environments. The environments that this paper uses all have a third-view perspective, allowing the model to “see” the agent itself and its relationship with other players/objects. However, in egocentric environments, such as maze navigation, all the pixels will dynamically modulate due to changes in the agent’s perspective. This will likely lead to the model failing to extracting objects dynamics important in solving the task. This raises the question of the applicability of DyMoDreamer in broader contexts.\n- As I understand, the main assumption of the paper is that all dynamically changing aspects of the environment is important for solving the task. However, in more realistic environments, there may be more noises such as moving foreground objects or background noises which are not important. As the extraction is conducted using a simple binary function with unlearnable threshold, the model may fail to distinguish the important agent and objects’ changes and noises that change unrelated to the agent’s actions. This raises a question about the model's robustness in more complex, noisy settings.\n- The paper has several minor grammatical errors and typos\n    - The sentence in line 130 “Considering that …” seems to be grammatically incorrect\n    - In line 667, progcess should be process\n\nQUESTIONS:\n- Related to Weakness 2, you mention that dynamic parts of the observation are often correlated with rewards (line 175). Given this, have you considered conditioning the dynamics modulation mechanism on the agent's actions? It seems this could be a possible direction to help the model focus on extracting parts of the image that are task-related as well?\n- While I agree that both dynamic and static parts of the observations are important for task solving (line 182), could you elaborate on the design choice not to fully disentangle these two information channels in the model's representation (e.g., by creating separate static (1-M(o_t)) ⋅ o_t and dynamic M(o_t) ⋅ o_t pathways)?\n- I think one of the motivation behind this paper was to improve the model’s capability to capture dynamics of small, moving objects. To further strengthen this claim beyond the helpful visualizations, have you considered a more quantitative evaluation? For example, using object segmentation or tracking metrics on the extracted dynamic masks could provide a powerful, objective measure of the model's capability in this area.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces DyMoDreamer, an extension of DreamerV3 that aims to decouple dynamic elements from static components in pixel-based observations. Dynamic modulation is achieved by (1) including differential observations as additional inputs, encoded in a dedicated latent space, and (2) incorporating a loss term that emphasizes better prediction of inter-frame variation. The method demonstrates strong performance on three benchmarks (Atari100k, DeepMind Control Suite, and Crafter) achieving or surpassing state-of-the-art results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- an original extension of the Dreamer line of work, emphasizing the modeling of dynamic structure in visual inputs. \n- strong empirical performance across a diverse set of benchmarks, which showcases the method's significance to the model-based RL community.\n- the paper is generally well-written, and the figures and illustrations clearly communicate key ideas.\n\nWeaknesses:\n- certain ablation details are either missing or under-explained, e.g., the role of the dimensionality of the static latent component ($z_t$) and the inclusion/exclusion of regularization terms like $\\mathcal{L}_\\mathrm{reg}$ from Dreamer.\n- the method is closely tied to Dreamer-style architectures and pixel-based inputs; its generality or applicability to other world models or state-based inputs is unclear, which limits broader impact.\n- the related work section focuses exclusively on world models, even though the approach appears to draw from modulated ODEs and latent flow methods. Including a discussion of these inspirations would help clarify the method’s novelty.\n\nQUESTIONS:\n1. Dimensionality of $z$: In the provided examples $z$ seems to mostly capture static backgrounds. Did the authors experiment with the dimensionality of the static latent $z_t$? Have you considered using a learned but fixed/static $z$ as a baseline?\n2. Clarification on the \"Removing Dynamic Modulation\" Ablation: Could you clarify what is meant by this ablation? Is the decoder predicting both the original and differential images jointly? Are these inputs concatenated, or are separate decoder heads used?\n3. Baseline with $\\mathcal{L}_\\mathrm{reg}$: Is there an ablation experiment comparing against a version of DreamerV3 that incorporates only the differential divergence regularization loss (without modulation)? Comparing against such a baseline would help disentangle the contribution of the modulation mechanism from the loss term.\n4. Applicability Beyond Images: DreamerV3 has also been used for environments with state-based inputs. Could DyMoDreamer extend to such cases? Do the authors believe that dynamic modulation would provide similar benefits in non-visual settings?\n5. Broader Impact of Architectural Biases: Does separating dynamic and static latents introduce inductive biases that may hurt performance in environments where this assumption doesn’t hold (e.g., rapidly changing backgrounds)?\n\n\nMinor comments:\n- Line 27: “A key approach to addressing this challenge is the world model [...]. This framework combines a VAE with a recurrent neural network (RNN) and uses evolutionary strategies to optimize the policy within a latent space.” The first sentence reads as a general statement about world models, but the second appears to describe the specific approach of Ha & Schmidhuber. It would be helpful to clarify which parts refer to the general framework versus this specific implementation.\n- Equations 2 & 9: The indices $i,h,w,c$ should be introduced in natural language before or alongside the equation. Clarifying what these dimensions represent (e.g., batch index, height, width, channels) would improve readability and help readers unfamiliar with the notation.\n\nLIMITATIONS:\nLimitations are not clearly addressed. The paper would benefit from a dedicated discussion of limitations. For example, a brief analysis of scenarios where dynamic/static separation could fail would strengthen the contribution.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDyMoDreamer introduces a low‑cost extension and high efficiency to Dreamer‑style (Hafner et al) world models, introducing a scalable way to incoporate dynamic object information without any explicit priors. \n\nThe paper argues that the recurrent state‑space model (RSSM) of Dreamer V3 with an extra categorical \"dynamics modulator\" $d_t$,  that is inferred from frame‑difference masks and concatenated to the usual stochastic latent z_t. This forces the model to attend to motion cues without expensive vision pre‑training and with minimal extra compute. The paper shows state‑of‑the‑art scores on Atari 100k, DeepMind Control (pixels) and Crafter, plus ablations about their design choices.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nI think this paper tackles an important problem of introducing object level information without any explicit tracking, bounding boxes etc, and is inspired from how human think about *differences* between scenes and not just scene features. \n\nAdditionally, the paper is clear, well‑explained method. The paper is easy to follow and the algorithmic additions are minimal yet well-motivated. It is also provides strong empirical evidence to support the claims. Across three benchmarks, DyMoDreamer improves on previous state of the art results, such as Dreamer V3. Moreover, The ablations are quite informative for example, removing the modulator or differential divergence regularizer drops Atari performance. Finally, the writing is clear and there are many details which enhance reproducibility, i. e, implementation choices, loss terms, and training schedules are spelled out. \n\nWeakness:\nIn my opinion, there is limited evaluation beyond fixed‑camera games. Unlike DIAMOND (which at least simulates a first‑person CS:GO environment) and Dreamer V3 (which learns on 30 egocentric DMLab tasks), DyMoDreamer reports results only on fixed‑view benchmarks. Therefore, it remains unclear whether its frame‑difference modulation can handle rapid viewpoint changes, parallax, or background motion typical of true ego‑centric domains.\n\nQUESTIONS:\nThe paper's \"latent‑difference\" ablation simply subtracts the quantised categorical codes: z_t - z_{t-1} and feeds the raw vector into the RSSM, with no re‑encoding, normalisation, or spatial structuring. That choice likely under‑represents what a proper latent‑difference approach can do. It would be good to see more detailed discussion about this. \n\nAdditionally, it would be good to see more discussion or experimantation with more explicit approaches which directly model object dynamics (with physics or 3D priors for example).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-23"}