R2-Dreamer: Redundancy-Reduced World Models without Decoders or Augmentation
Authors: Naoki Morihira
Amal Nahar
Kartik Bharadwaj
Yasuhiro Kato
Akinobu Hayashi
Tatsuya Harada
Venue: ICLR 2026 Poster (2026)
Keywords: model-based reinforcement learning
world models
representation learning
Primary area: reinforcement learning
TL;DR: R2-Dreamer is a decoder-free agent that replaces data augmentation with a self-supervised objective to excel on challenging visual tasks.

Abstract: A central challenge in image-based Model-Based Reinforcement Learning (MBRL) is to learn representations that distill essential information from irrelevant visual details. While promising, reconstruction-based methods often waste capacity on large task-irrelevant regions. Decoder-free methods instead learn robust representations by leveraging Data Augmentation (DA), but reliance on such external regularizers limits versatility. We propose R2-Dreamer, a decoder-free MBRL framework with a self-supervised objective that serves as an internal regularizer, preventing representation collapse without resorting to DA. The core of our method is a \emph{redundancy-reduction} objective inspired by Barlow Twins, which can be easily integrated into existing frameworks. On DeepMind Control Suite and Meta-World, R2-Dreamer is competitive with strong baselines such as DreamerV3 and TD-MPC2 while training 1.59$\times$ faster than DreamerV3, and yields substantial gains on DMC-Subtle with tiny task-relevant objects. These results suggest that an effective internal regularizer can enable versatile, high-performance decoder-free MBRL. Code is available at https://github.com/NM512/r2dreamer.

Published as a conference paper at ICLR 2026
R2-DREAMER: REDUNDANCY-REDUCH WORLD MODELS WITHOUT DECODERS OR AUGMENTATION
Naoki Morihira 1.2
Amal Nabar
Kartik Bharadwaj!
Yasuhiro Kato²
Akinobu Hayashi !.2 Honda R&D Co., Ltd.
Tatsuya Harada The University of Tokyo
RIKEN AIP
ABSTRACT
A central challenge in image-based Model-Based Reinforcement Learning (MBRL) to learn representations that distill essential information from irrel- evant visual details. While promising, reconstruction--aase capacity on large task-irrelevant regions. Decoder- free methods instead learn ro- bust representations by leveraging Data Augmentation (DA), but reliance such external regularizers limits versatility. We propose R2-Dreamer, a decoder-free free MBRL framework with self-supervised objective that serves larizer, preventing representation collapse without resorting to DA. The core of our method a redundancy-reducciion objective inspired by Barlow Twins, which can be easily integrated into existing frameworks. On DeepMind Control Suite and Meta- Meta-World, 22-Dreamer is competitive with strong baselines such as Dreamer and TD-MPP MPCC2 while training x.59 faster than DreamerV3, and yields substantial gains on DMC-Subtle with tiny task-relevant objects. These results suggest that an effective internal regularizer can enable versatile, pirf-mceformance decoder-free MBRL. Code available at withttps::.. / thub. wwt.cmomm.m//... NM01//MM12//// 22d
methods often waste
internal regu-
INTRODUCTION
Learning effective latent representations is cornerstone of world models in Model-Based Rein- forcement Learning (MBRL), yet this poses a significant challenge: representations must capture task-essential information without overfitting to irrelevant details. While architectures like the Re- current State-Space Model (RSSM) have achieved remarkable success (Hafner al.) (025),, fun- damental question remains open: What is the optimal objective function for learning the rep- resentation itself? This question is particularly important in image-based settings, where high- dimensional observations make representation learning inherently challenging.
In practice, many leading methods learn representations by optimizing pixel-level reconstruction
objectives (Micheli et al. 2023 Zhang al. [023 Seo et al. 2023) Micheli et al. 2024 Alonso
et al. 2024; Hafner et al. spatially large but task-irrelevant parts of the observation, such the background. Consequently,
2025 This creates a critical issue: the learning signal is dominated by
the model is incentivized to meticulously reconstruct these details, wasting representational capacity and computational resources at the expense ignoring small but task-critical objects.
To address the limitations of pixel-wise reconstruction, decoder-free methods learn representations via self-supervised losses (Deng al. 2022, Okada & Taniguchi] 2022 Burchi & Timofte 2025).
To prevent the representation collapse common in such approaches. they depend critically on Data Augmentation (DA) as an external regularizer. This reliance on DA is significant bottleneck for general agents [Laskin et al. 2020 Ma et al. 2025), as the choice of transformation is task- dependent: random shifting can discard crucial small objects, while color jittering can detrimental when color itself is key feature.
In this work, we focus on the representation learning objective within the widely used RSSM frame- work and propose R2-Dreamer that breaks the dependency decoders and DA. To isolate the impact of the learning objective itself, we build upon the well-established Dreamer architecture. introduce redundancy- reduuctoon objective
Inspired by Barlow Twins (Zbontar al. 2021). between image embeddings and latent states to prevent representation collapse without external reg- ularizers, providing versatile and robust baseline capable of achieving competitive performance.

---

Published as a conference paper at ICLR 2026
Our main contributions are:
A new representation learning paradigm for RSSM-based decoder-free MBRL that replaces heuristic DA, which risks distorting task-critical information, with internal redundancy reduction objective.
Competitive performance across standard benchmarks, including DeepMind Control Suite (DMC) and Meta-World, and superior performance on our new, challenging DMC-Subtle benchmark, while enabling faster training by removing the decoder.
The release of our unified PyTorch codebase, including implementations of our method and baselines built on our Dreamer implementation, along with the DMC-Subtle benchmark to facilitate future research.
2 RELATED WORK
Our work is positioned at the intersection of MBRL and Self-Supervised Learning (SSL). We con- textualize our approach by reviewing representation learning strategies in MBRL. and how they address the challenge of regularization.
2.1 REPRESENTATION LEARNING IN WORLD MODELS
Decoder-Based World Models
dominant paradigm in MBRL, popularized by the Dreamer series (Hafner et al. 2025), learns representations by reconstructing observations from latent state. While successful, this reconstruction- based objective often forces the model to waste capacity on task-irrelevant details, such backgrounds, motivating shift towards decoder-free methods.
Decoder-Free World Models and the Reliance on DA To address the limitations of reconstruc- tion, recent decoder-free methods learn representations through auxiliary objectives that do not in- volve pixel-wise reconstruction, such as predicting future rewards or learning via contrastive losses. However, despite the diversity in their learning signals. these prominent examples Ye al. 2021 Deng et al. [022]] Hansen et al. (2022 Wang et al. [024 Burchi & Timofte (025) all critically rely on typically random shifts as an external regularizer prevent representation collapse. This fundamental dependency augmentations that may distort task-relevant details limits their
versatility, key bottleneck we address.
Aside from DA, some methods mitigate visual distractions through architectural mechanisms; for instance, VAI (Wang et al., (02T) introduces additional attention modules relies on motion cues, which can overlook static yet task-critical visual cues. Several works regularize representations more directly by injecting Gaussian noise into latent features (Shu et al. 20200] Nguyen et al.) [022]. In contrast, we show that single information
cormmattiooothoettieee theoretic principle for redundancy reduction sufficient for stable and effective representation learning in RSSM-based models without any DA.
2.2 FROM INVARIANCE TO INFORMATION BASED REGULARIZATION
DA-Driven Invariance Many popular self-supervised representation learning methods, including those used in existing decoder-free agents, are invariance-basee They rely on DA to create positive pairs (e.g., augmented views of the same image) and train the model produce similar represen- lations for them, as seen in contrastive (Chen et al. 2020 He et al., 20200 Caron el al.) 2020) and non-contrastive (Grill al., 2020; Chen He 2021 learning. In this paradigm, DA is essential to prevent collapse to trivial solutions.
DA-Free Internal Regularization Our work adopts a different approach from the information- based SSL literature (Zbontar et al.) 2021 Bardes et al. 2022), which focuses on reducing feature redundancy. While these methods still use DA the computer vision domain, we adapt this prin- ciple to serve as complete replacement for DA-based regularization in the reinforcement learning domain. Specifically, we apply the redundancy reduction objective between the image encoder's output and the RSSM's latent state. This yields internal regularizer sufficient to prevent rep- resentation collapse, thereby allowing us to build a more versatile and robust learning framework without task task-specific augmentations.
2

---

Published as a conference paper at ICLR 2026
3 METHOD
Our method, R2-Dreamee redesigns the representation learning mechanism of the powerful Dream- erV3 (Hafner et al. 2025) framework to be decoder-free and DA-free. We achieve this by replacing its reconstruction- based objective with a self-supervised objective based on redundancy reduction, inspired by Barlow Twins (Zbontar al. 2021). To isolate the impact of our proposed learning ob- jective, other components of the world model and the actor-critic implementation are kept identical the original DreamerV3. This single change demonstrates notable improvements in computa- tional efficiency and robustness. This section first details the latent dynamics model, introduces our new world model learning objective, and reviews the actor-critic learning process.
Gr11
a2
a-11
ar11
a2
11
Yl
S
S2-1
f1
ht
Z
Projector
Lyemp
Encoder
LBr
Projector
Encoder
Decoder
Encoder (EMA)
Lswav
Encoder
Lrecon
0
O
6,
aug(or)
02
aug(O,)
R2-Dreamer (ours)
(b) Dreamer
DreamerPro
Figure 1: Comparison of representation learning mechanisms in world models. (a) R2-Dreamer learns representations without a decoder or DA. uses an internal redundancy reduction objective LIT that aligns the latent state (via projector) with the embedding of the observation o,, (b) Dreamer relies on decoder to learn representations by reconstructing the observation o; from the latent state St. guided by a reconstruction loss Lrecon. (c) DreamerPro removes the decoder but depends on DA. It enforces consistency between augmented views of the observation aug(o:) using spatial loss LswAv and temporal loss Lemmp that leverages an Exponential Moving Average (EMA) of the encoder weights.
3.1 LATENT DYNAMICS MODEL
Following D3 we use the RSSM (Hafner et al. (019) with composite latent state (h,, =x), consisting of deterministic state he and stochastic state 2, This state acts as the agent's memory, aggregating the sequence of observations and actions to model dynamics, and it is also used to predict rewards Ft and continuation flags Ct.
The key architectural distinction of R2-Dreamer is the complete removal of the image decoder and the introduction of a lightweight linear projector head. While DreamerV3 relies on decoder (o 8t) for representation learning, projector simply maps
to reconstruct observations the latent state &, to the feature space of the image embedding et. This design avoids the computa- tional cost of pixel-level reconstruction while enabling effective representation learning through the objective described in the next section. The components of our model are defined as follows:
Image Encoder:
C fo(
Sequence Model:
h a0(0-11,04-0
Dynamics Predictor:
2 ~ p=(2 he)
Representation Model:
21 ao (2,
h
Reward Predictor:
Pr ~ ps (ft
8t)
Continue Predictor:
& ps pc, 8t)
Projector:
k = fo(
3

---

Published as a conference paper at ICLR 2026
3.2 WORLD MODEL LEARNING
Our core contribution is a new learning objective for the world model that replaces the reconstruction loss of DreamerV3.
theoretically motivated in Appendix [ this new objective tractable surrogate for an extended Sequential Information Bottleneck objective. We now detail the practical implementation of this objective, adhering to the original loss components from DreamerV3 where applicable.
D3eamer Objective
The world model in DreamerV3 is trained by optimizing four distinct objectives: reconstruction, prediction, and two KL--divergence terms for regularizing the latent dy- namics. The overall loss, shown in Eq. equation [
is weighted sum of these components.
Σ
Creron (1) L(r CaynLayn(1) + Brophrep (1)
(1)
Dreameer V3 Egg
The reconstruction and prediction losses are negative log- liglliikss The dynamics and represen- tation losses regularized using KL balancing (Hafner al. (02T) and free bits (Kingma al., 2016 Each component is defined as:
Lrecou (t) = log Po(oe(sE)
Spred (L) log Dogpo(ralae (x)) - log po(e)s))
Cayn (t) = max (L, (sg(go ex)) (s(
frep = max (1, KT. 2o (2, hh, ea) (2,
where sg denotes the stop-gradient operator.
R2-Dreamer Objective We remove the reconstruction term Crecon and replace it with our pro- posed loss, CBT. The other components, including the KL balancing scheme and loss coefficients (Bdyn = 1, Srep 0.1). are adopted from DreamerV3:
Σ (Cproa(t) + Bayn (t) + Beop Crep PropCrep(tt) (t)
L (0) = Egg BRTLET +
(4)
This formulation isolates the contribution of our method to the new representation learning signal provided by LBT-
Representation Learning via Redundancy Reduction (LBT) We adopt the Barlow Twins ob- jective as redundancy reduction mechanism. Compared to other methods like VICReg (Bardes) ct al. 2022). it chosen for its minimal implementation footprint and fewer hyperparameters, which reduces tuning effort. The objective is defined as:
CBT Ca) ta
Σ C iF1
(5)
Invariance Term
Redundancy Term
The indices and j refer to the feature dimensions. Here, Cs the cross-correlation matrix computed over a mini-batch between projector output k; and the image embedding e, (see Appendix G for details). This loss is governed by single hyperparameter. reduction term. Instead creating artificial views via DA, we form a natural pair of views from the model's 's internal signals: the image embedding e, and the projected latent state k.-
which weights the redundancy
In practice, we compute over all time steps in a mini-batch (i.c., over x samples) after standardizing both k; and along the batch dimension.
In our implementation, we detach the target e, to enhance stability, similar to strategies TD- MPC2 (Hansen et al.) 2024). Despite this, the encoder receives rich gradients backpropagating through the projector and RSSM, while the reward, episode continuation, dynamics, and value ob- jectives provide task-relevant supervision identical DreamerV3.
4

---

Published as a conference paper at ICLR 2026
3.3 ACTOR-CRITIC LEARNING
To ensure our performance gains are attributable to the world model's representation quality, the actor-critic learning process remains unchanged from DreamerV3. The critic optimized on both imagined rollouts and replay trajectories, whereas the actor is optimized only on imagined trajecto- rics. Specifically, imagined rollouts start from latent states inferred from replayed trajectories and unrolled using the learned dynamics model under the current policy.
The critic is trained to predict the distribution of A-returns, robust estimate of future rewards. The critic's loss is the maximum likelihood of predicting these returns:
Leritie (v)) = -Eps To
Σ log log (R)
(6)
where the A-return RtA is computed recursively as R. = r. re((L - ()uw(s + AR with discount y and continuation flag es. As in the original setup, this objective is applied both imagined rollouts and replay trajectories from the buffer.
The actor is trained to maximize these returns using the REINFORCE estimator (Williams 1992
incorporating entropy regularization with a fixed scale yi and robust return normalization:
R oi (81) log (oggo(())s)) + +=([(m)(((ss)))) max
Lactor (0) -Epo,
Σ
sg
where S is a dynamically scaled normalizer computed as an exponential moving average of the 5- 95th percentile range of returns, i.e., S = EMA (RA, 95) Per( R, R,, (), 0.99), which improves robustness to outliers across diverse environments.
EXPERIMENTS
In this section, we conduct a series of experiments to validate the core claims our work: that R2-Dreamer learns high-quality representations in decoder-free and DA-free manner, leading to framework that is not only computationally efficient but also highly performant. Our evaluation is structured answer the following key questions:
1. How does R2-Dreamer perform against leading recoderrbassed and decoder-free agents standard continuous control benchmarks? (Sec. 4.2 Sec. 4.3)
2. How does our internal regularization handle challenging scenarios where task-relevant in- formation subtle and casily missed by competing methods? (Sec. 4.4)
3. How does the learned representation qualitatively differ from baselines in focusing on task- relevant information? (Sec.) 4.5)
4. What is the direct impact of our proposed redundancy reduction objective compared to other design choices, particularly DA? (Sec. (4.6)
5. What are the computational benefits of its decoder-free and DA-free design practice?
(Sec. 4.7
We report task scores DMC and DMC-Subtle and success rates Meta-World summarizing results with mean and median across tasks, and provide detailed per-task curves in the appendix. In all experiments, and, unless otherwise stated, use the same hyperparameter configuration (see Appendix F) across all tasks and benchmark suites.
conduct training five random seeds, with 10 evaluation episodes per seed,
4.1 EXPERIMENTAL SETUP
Baselines We compare ---rammer against a carefully selected set of competitive baselines to cover the main paradigms of image-hased reinforcement learning:
5

---

Published as a conference paper at ICLR 2026
R2-Dreamer (ours): Implemented top of our PyTorch-basee DreamerV3 reproduction. This unified codebase is used for all decoder-free variants to ensure that performance dif- ferences directly attributable to the representation learning objective.
DreamerV3 (Hainer et al.) 2025): leading and highly competitive decoder-based world model. To provide one of the strongest and most credible baselines, we use the author's official JAX implementation as our primary point of comparison, utilizing the latest version which includes several algorithmic improvements made in April 2024
Dreamer-InfoNee contrastive learning baseline using the InfoNCE loss (van den Oord et al. 2019) to investigate performance in the absence of DA, implemented on our Dream- erV reproduction.
(Deng et al. 2022): ; A leading decoder-free free method that relies on DA, specifi- cally random image shifts, to prevent representation collapse. Since the original implemen- tation is based on Dreamer 2, we re-implemented its core mechanism on our DreamerV3 reproduction ensure fair comparison. This re-implementation also improved its perfor-
DreamerPro
mance.
DrQ-v2 Yaratt al. 2021 A strong and widely-used model-free agent for image- based RL, included as representative model-free baseline for performance reference. relies on DA as key component of the method. We use the author's 's official implementation
2024): A strong decoder-free model-based method that com- TD-MPC2 (Hansen al.
bines TD learning with planning in latent space. and uses DA as external regularizer to prevent representation collapse. We use the author's official implementation
Environments All our benchmarks focus on continuous control from pixels. We evaluate our method on three benchmark suites:
DeepMind Control Suite (DMC) Tassa et al. 2018): A widely adopted benchmark suite for continuous control tasks from pixels, encompassing both locomotion and manipulation domains.
Meta- Woold (Yu al. 2021): benchmark suite for evaluating performance diverse manipulation tasks using robotic arm. We use the MTI benchmark, where agents are trained 50 distinct tasks individually. These tasks involve interacting with various ob- jects, including small ones, and require precise fine-graine manipulation.
DMC-Subtle: A new benchmark designed
controlled stress test for representation learning in pixel-based control, where task-critical objects are scaled down to make task- relevant visual cues subtle. For example, Figure 2 illustrates the Reacher task, where the target is scaled down to one-third of its original size. This benchmark demands higher level of representational precision. Detailed modifications for all tasks are provided in Appendix
Target
Target
Figure 2: Examples drawn from benchmark tasks. Left: Meta-World Assemble. Center: DMC Reacher (hard). Right: DMC-Subtle Reacher with significantly smaller target.
laterr//aaaaalccm/mmmmiaa/eeerrr

---

Published as a conference paper at ICLR 2026
4.2 PERFORMANCE ON DEEPMIND CONTROL SUITE
We first evaluate R2-Dreame on 20 standard DMC tasks, Figure summarizes performance across tasks using both the mean and the median. Our method is competitive with the decoder-based, decoder-free, and model-free baselines on average. This result indicates that our internal redundancy reduction objective an effective learning signal, capable of achieving competitive performance without decoder or an external regularizer like DA. Detailed per-task curves are in Appendix C
Task mean
Task median
800
R2-Dreamer
800
600
Dreamer
Return
600
Dreamer-InfoNCE
400
DreamerPro
DrQ-v2
200
TD-MPC2
S00K
SC0K
Env steps
Env steps
Figure 3: Mean and median performance over 20 DMC tasks, with standard deviation across seeds. R2- R2-Dreamer is competitive with the baselines average without requiring decoder DA.
4.3 PERFORMANCE ON META-WORLD
We evaluate R2- 12-Dreamer on Meta-World MT1, which contains 50 robotic manipulation tasks trained independently. Figure reports the mean and median success tion across seeds. R2- Dreaaeme is competitive with the baselines in mean success rate across tasks on average, even on contact-rich manipulation tasks involving small objects. Detailed per-task curves are in Appendix D
tasks, with standard devia-
Task mean
Task median
1.0
1.0
R2-Dreamer
Success Rate
0.8
0.8
Dreamer
0.6
0.6
Dreamer-InfoNCE
DreamerPro
0.4
DrQ-v2
TD-MPC2
1M
500K Env steps
Env steps
Figure 4: Aggregated performance on Meta- World 50 tasks using the mean and the median across tasks. R2-Dreamer achieves strong results even contact-rich manipulation tasks, remaining com- petitive with the baselines on average.
4.4 ROBUSTNESS IN CHALLENGING ENVIRONMENTS
We now highlight the benefits our approach on the DMC-Subtle benchmark, a challenging testbed designed to penalize methods that either overfit to irrelevant backgrounds or discard small, critical objects. We hypothesize that our redundancy reduction objective is particularly well-suited for these meeision-demanding tasks. By not being driven by reconstruction signal dominated by task- irrelevant backgrounds and avoiding the potential distortion critical features from DA, our method should learn more focused representations. The results in 5 confirm this hypothesis, showing a substantial performance gap over the baselines and demonstrating that R2- D-Dreamer can effectively isolate and attend to task-critical information, crucial capability for real-world applications where salient cues may be sparse. We further analyze the learned representations to understand the source of this robustness.

---

Published as a conference paper at ICLR 2026
Ball In Cup Catch Subtle Cartpole Swingup Subtle
Finger Turn Subtie
Point Mass Subtle
1830
002
030
Return
403
400
600
203
200
Reacher Subtle
Task
Task median
002
602
R2-Dreamer
Dreamer
Retur
920
DreamerInfoNCE
DreamerPro
DrQ-v2
230
203
202
TD-MPC2
520K
06
595K
500k Env steps
Env steps
Env steps
Figure 5: Performance on five challenging DMC-Subtle tasks. R2-Dreamer substantially outper- forms the baselines, demonstrating its robustness subtle but critical visual information.
4.5 ANALYSIS OF LATENT REPRESENTATIONS
We visualize the policy's ' focus using occlusion-base based saliency method (Greydanus et al. 2018 to assess how well the learned representations capture task-relevant information. For this analysis on the DMC-Subtle Reacher task, compute saliency maps the first frame of each episode to isolate the spatial focus from temporal dynamics. The results in Figure reveal a clear distinction: the saliency map for R2-Dreamer is sharply focused the target, indicating its policy is grounded in task-critical visual evidence. In contrast, baselines exhibit more diffuse saliency, suggesting less precise understanding of the task. This finding provides strong qualitative evidence that our redundancy reduction objective encourages learning compact and relevant representations.
R2- Dreamer
Dreamer
Dreamer-InfoNCE
DreamerPro
Target
Target
Target
+ Targpt
Figure 6: Policy saliency maps on the DMC-Subdl Reacher task. For clarity, the target location is marked with yellow dot. The two rows show results from two different environment seeds, which identical across all methods.
4.6 ABLATION STUDIES
To isolate core contributions, conduct targeted ablation study the effectiveness of our redundancy reduction objective against DA. We compare six variants: R2-Dreamer (our complete method), R2-Dreamer (half batch), R2-Dreamer with DA (adding random shift), DreamerPro (DA-reliant baseline), DreamerPro without DA, and Dreamer without Decoder (no visual auxil- iary objective).
8

---

Published as a conference paper at ICLR 2026
First, Figure shows that adding DA R2- R2-Dreamer yields only marginal gains. In contrast, Dream- erPro collapses without DA, confirming its critical dependency on the external regularizer. Perfor- mance degrades close to that of Dreamer without Decoder, which has no explicit objective to learn visual representations.
We also test batch-size sensitivity, SSL objectives be affected by correlation estimation. Con- sistent with the reported robustness of Barlow Twins (Zbontar et al. 2021), halving the batch size (B 8 B 16) does not yield a significant performance drop.
Detailed results are in Appendix
Task mean
Task median
800
R2-Dreamer
R2-Dreamer + DA
600
Return
R2-Dreamer (half batch size)
400
400
DreamerPro
DreamerPro (no DA)
Dreamer (no recon. loss)
OK
500K Eny steps
0K
500K
Eny steps
Figure 7: Ablation results on 20 DMC tasks using the mean and the median across tasks. Our internal redundancy reduction objective proves more effective and robust than reliance on heuristic DA.
Second, examine the same design choice in setting where preserving fine-grained spatial infor- mation essential. On the precision-demanding DMC-Subtle benchmark, DA proves detrimental. As shown in Figure adding DA significantly degrades method's performance. This highlights a key risk of external regularizers: while generally applicable, they can distort subtle, task-critical information. Our DA-free, internal mechanism provides more robust solution in such cases, rein- forcing its effectiveness as a principled regularizer for RSSM.
Ball In Cup Catch Subtle Cartpole Swingup Subtle
Finger Turn Subtle
Point Mass Subtle
1000
1000
Return
600
400
400
200
200
0
500K
Reacher Subtle
mean
Task
1000
002
Retum
600
R2-Dreamer R2-Dreamer + DA
100
200
200
260
500c
1M
0c
50TK
1N Env steps
500C
14
Erw steps
Env steps
Figure 8: Comparison of R2-Dreamer with and without DA on the DMC-Subtle benchmark. The results highlight that DA can be detrimental in tasks requiring high precision, as it may distort subtle but critical information. underscoring the importance of DA-frec approach for such environments.

---

Published as a conference paper at ICLR 2026
4.7 COMPUTATIONAL EFFICIENCY
A core advantage of our decoder-free design is its computational efficiency. To ensure a fair com- parison, we measure the wall-clock training time of our method against baselines implemented on our unified DreamerV3 reproduction. As shown in Table I over our DreamerV3 reproduction by eliminating the computationally expensive image generation process. Furthermore, it demonstrates 2.36x speedup compared to DreamerPro, which involves processing different augmented views of the input and subsequent relatively complex logic. We also include the training time of the original, highly optimized DreamerV3 JAX implementation. These results highlight that R2-Dreamer offers more practical and scalable solution.
R2-Dreamer achieves 1.59x speedup
Table I: Computational efficiency comparison on the DMC Walker Walk task. Wall-clock time is measured for I million environment steps single NVIDIA GeForce RTX 3080 Ti GPU.
Method
Training Time (hours)
R2-Dreamer
4.4
Dreamer
7.0
DreamerPro
10.4
Dreamer (Author's JAX impl.)
6.6
CONCLUSION
We demonstrated that a principled internal regularization objective can supplant the need for image reconstruction in MBRL. Our framework, R2-Dreamer, learns representations focused on salient features without decoders or task-specific DA.
The strength of this approach is most evident our challenging DMC-Subule benchmark, where R2-Dreamer substantially outperforms leading decoder-based and DA-reliant agents by isolating tiny, critical objects. On standard benchmarks across locomotion and manipulation domains, it is competitive with DreamerV3 while achieving 1.59x faster training.
An important direction for future work is evaluate R2-Dreamer in environments with dynamic and irrelevant backgrounds, such as the Distracting Control Suite (Stone et al.) (02T). Our results on DMC- Subtle suggest that our internal redundancy reduction objective naturally avoids wasting representational capacity on irrelevant pixels, which may imply robustness to such dynamic distrac- tors. Verifying this hypothesis would further establish the efficacy of DA-free internal regularization for complex visual control tasks. Scaling to high- nigh-dimensional tasks like Humanoid is also a future direction.
By shifting the focus from visual fidelity to informational efficiency, our work provides a scalable foundation for building agents where heuristic augmentations risk distorting task-critical informa- tion. This study opens new inquiry into internal regularization as principled path toward more general and capable learning agents.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide detailed methodology (Sec. 3) experimental setups (Sec. 4.1 and hyperparameters (Appendix (F). Pseudocode is available in Appendix(Ci and we release the uni- fied Torch codebase including R2-Dreamer, baselines built on our DreamerV3 implementation, and the DMC-Subde benchmark.
REFERENCES
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. btLhttps: //openreviess ew. nwt. www.://ttt.......oooooooo http://rrrrmmm.ttyyyyyyyraaeaee
10

---

Published as a conference paper at ICLR 2026
Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos J Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37:58757-58791, 2024.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: litaanc-invariance-oon for self-supervised learning. International Conference on Learning Representatiins, 2022.
regularization
Maxime Burchi and Radu Timofte. Learning transformer--aased world models with contrastive pre- dictive coding, 2025. URL wttp:iiii..rg/bb///55000 2
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 9912-9924. Curran Associates, Inc., 2020. URL.
neurips ttt.cc//pppeeee wttpc//ppppppeeeeeeeeeeeeeeeeeee filesspaper/20222 Paper.app... pdf
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning. 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15745-15753, 2021. doi: 10. C0WCCVVPRR4443222210000155
Fei Deng, Ingook Jang, and Sungjin Ahn. DreamerPro: Reconstruction-free model-based rein- forcement learning with prototypical representations, In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th Inter- national Conference on Machine Learning, volume 162 of Proceedings of Machine Learning mmlr.
Research, pp. 4975-4975. PMLR, 17-23 Jul 2022. URL https::/proceeeeeeee
press/v162/deng222 html
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding Atari agents. In Jemnifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.
1792-1801. PMLR, 10-15 Jul 2018. URL
mlr
greydanus18a htm1
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-21288 2020.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2555-2565. PMLR, 09-15 Jun 2019. URLEttpa httt:ss:/proceeed..ttt http:///pcceeeeig...// mlr wttppppsss/////aaee/rr333
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
net htt:://ourmiddreeee URL http:///geeeaneeeeeeeeeeeeeeeea. fow.mmmiddd1110000S
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with
discrete world models. In International Conference on Learning Representations, 2021. URL
https:://ppenre//
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640088059) Apr 2025. ISSN 1476-4687, doi: 10. 1038/ 941586-025-08744-
025
Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for contin- uous control. In The Twelfth International Conference on Learning Representations, 2024. URL
https::/openreww wtt.:tewwwweeeeeeeee.........
net www..ettooorrmmiooooootttcttt.t.

---

Published as a conference paper at ICLR 2026
Nicklas A Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 8387-8406. PMLR, 17-23 Jul 2022. URL htt.tttts://ppoceeeeeee htt.eeedsd.......es.ess/s//////
ml www.s/ressviii.... http:///vv6ohaaaa///ee/e
html
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9726-9735, 2020. doi: 10. 1009/CCPR420000220000000
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural In- formation Processing Systems, volume 29. Curran Associates, Inc., 2016. URL
https::/proceedingg neuri cc wecpppapr__tee lesl weeapprrffllsppperrreeeee httppapr//011////1110 Paper .ddf
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srini- vas. Reinforcement learning with augmented data. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Pro- cessing Systems, volume 33, pp. 19884-19895. Curran Associates, Inc., 2020. URL
Guozheng Ma, Zhen Wang, Zhecheng Yuan, Xueqian Wang, Bo Yuan, and Dacheng Tao. A com- prehensive survey of data augmentation in visual reinforcement learning. International Journal doi: 10. 1 URL
of Computer Vision, jul 2025. ISSN 1573-1405.
https://doo wwo.or////o 10. 1007 10100/01111100 s1166311163111 025-02472 02472-w
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. In The Eleventh International Conference on Learning Representations, 2023. URL
wttp//eeeweeww.eee/eeeemeo h www.:riddowwuuuaaaaaaa http:///.....peeee....ee..//oota
https:/
Vincent Micheli, Eloi Alonso, and François Flcuret. Efficient world models with context-aware tokenization. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 35623-35638. PMLR, 21-27 Jul 2024. URL httphhtts/ss/et hottps::/prroceedngg...tt press// v235 htt.//mmcheee222.. htmml
Tung D Nguyen, Rui Shu, Tuan Pham, Hung Bui, and Stefano Ermon. Temporal predictive coding for model-based planning in latent space. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine I, 18-24 Jul 2021. URL httph:ttps//ie //tp::/:///coceednggs.
Learning Research, pp. 1130-8139. PMLR, mlr. tttess/v139//guueee ht1.3/9/nnuun22132 html
Masashi Okada and Tadahiro Taniguchi. Dreamingy2: Reinforcement learning with discrete world models without reconstruction. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 985-991, 2022. doi: 10. 199//R0047662..22299999
Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Conference on Robot Learning, pp- 1332- 1344. PMLR, 2023.
Rui Shu, Tung Nguyen, Yinlam Chow, Tuan Pham, Khoat Than, Mohammad Ghavamzadeh, Stefano Ermon, and Hung Bui. Predictive coding for locally-lines control. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8862-8871. PMLR, 13-18 Jul 2020. URL httpp:://pooceedinggg
mll. .ww.ppress...... ww19//// sh/2 Ca. html
Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting control suite- a challenging benchmark for reinforcement learning from pixels, 2021. URL https:
//arxiv.crg//bs//2/2.. 2210.002222 02722
12

---

Published as a conference paper at ICLR 2026
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li. Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap. and Martin Ried- miller. Deepmind control suite, 2018. URLhttts://aaxxi crg.abbs.rg/bbs///o. 1801. 00690
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method, 2000.
URL
abs/ wttp:///w.....p.....o....o.oocc/ 00/004057
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, 2019. URL URL httpp::/aaxxii.......mm http://:///.aaiioooooaaa/a////// httpg//bss18077 03748
Shengjie Wang. Shaohuai Liu, Weirui Ye, Jiacheng You, and Yang Gao. EfficientZero v2: Mas- tering discrete and continuous control with limited data. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Machine Learning, volume 235 of Pro-
Proceedings of the 41st International Conference ceedings of Machine Learning Research, pp. 51041-51062. PMLR, 21-27 Jul 2024. URL https://proceedings ml www.srpres///ee www.rsss/v33/// wang2
http://///////a//tt...aa .ttmmmllll
Xudong Wang, Long Lian, and Stella X. Yu. Unsupervised visual attention and invariance for reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 6677-6687, June 2021.
Satosi Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of Re- search and Development, 4(1):66 82, 1960. doi: 10.1147/rd.41.000 10066.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine learning, 8(3):229-256 1992.
Denis Yarats, Rob Fergus, Alessandro Lazaric. and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning, 2021. URL htll:ttts:///xxiv.orrr abs/2107.0 006455
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Master- ing atari games with limited data. In M. Ranzato. A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Pro- cessing Systems, volume 34, pp. 25476-25488. Curran Associates, Inc., 2021. URL https::/prrocedings...tt new.eeennrrppp cctpppp.ccppppppppppppppppppp httpc//aapee..ieeeeepppeeeee//ee fiLes//operr222///1ee dbccssccssccaaateeeaaaaaaaaaaaaa
pdf
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning, 2021. URLhhttss://axxiv
abs/1910. 10897
Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In Proceedings of the 38th International Conference Ma- chine Learning, 2021.
Weipu Zhang, Gang Wang, Jian Sun, Yetian Yuan, and Gao Huang. STORM: Efficient stochastic transformer based world models for reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URLhttts: htt.tttsss//peeeeeeeeee opwnreviev net. forum?
Xd=#xn=X42rms
13

---

Published as a conference paper at ICLR 2026
A CONNECTING REDUNDANCY REDUCTION TO THE SEQUENTIAL INFORMATION BOTTLENECK
Our World Model's loss function optimizes variational bound on extended Sequential Informa- tion Bottleneck (SIB) objective (Tishby et al. 2000). Building on DreamerV1 (Hafner et al.) 2020))
formulation incorporates spatial compression term that encourages disentanglement by mini-
mizing the Total Correlation (TC) (Watanabe) 1960) of the latent states. The full objective is defined
as:
max 1($1:T: (OWTT T1.T, C1T) ai.T)
3 [1sIrrii (I.T)
TC(s))
(8)
Fidelity
Temporal Compression
Spatial Compression
where $1:7 is the latent state sequence, (01-T, T1:7, C1-T) are the observation, reward, and contin- uation sequences, and 01.7 is the action sequence. Following (Alemi et al., 2017), is denotes the underlying data-generating index for the observation. The objective of compressing $1:7 with re- spect to 11T is to discard predictable information from the past, thereby encouraging the latent state to capture only novel information. Below, we derive tractable variational bounds for each term and demonstrate how our proposed loss function optimizes them.
Lower Bound on Fidelity The fidelity term ensures that the latent state 81.7 retains predictive information about observation, reward, and continuation. As this term is intractable, maximize a variational lower bound. The derivation begins with the chain rule for mutual information. For simplicity, considering only observations 01:7:
T
1(31:1) 01. (1:1)
181:110 01:-1, 01:T)
1(st or 01:-11,
(9)
1(s & (1:-11.11))
T
Σ I(see:)
The first inequality holds as information cannot increase with subset of variables, and the second follows from the data processing inequality (e, enc(o,)). The final step drops the conditioning on history under the common assumption that 8; sufficient statistic of the past (Hafner al. 2019). under which the approximation preserves the inequality direction. A similar derivation, omitting the data processing inequality step, can be applied to rewards and continuation signals. This yields the final surrogate objective for the fidelity term:
I(51:1 (OI:T, To:T, F1.r,C1:11 (1.T))
E1(s + EI(sn)) + EI(sno) E
(10)
t=1
1=1
Upper Bound on Temporal Compression Following prior work (Hafner al. 2020). the tem- poral compression term is upper-bounded by the KL divergence between the posterior and prior dynamics:
I(S1:1: (1r))
E
(11)
Unification via Barlow Twins Crucially, the extended SIB objective's fidelity and spatial com- pression terms can be jointly optimized by single surrogate loss based the Barlow Twins objec- tive (Eq. (5) This loss is applied to the image embedding e, and the projected state k, and consists of two components:
14

---

Published as a conference paper at ICLR 2026
Invariance:
The loss penalizes the deviation of the diagonal elements of the cross- correlation matrix from 1. This encourages the projected state k to predict the image embedding ct2 surrogate for maximizing the fidelity term [(s;;e))
Redundancy Reduction: The loss penalizes the off-diagonal elements of the cross- correlation matrix. This encourages the dimensions of ly to be uncorrelated. Since ky is linear projection of the latent state (he, 2t), i.c., h Wha; 2r], this pressure to decorrelate k directly incentivizes the model to learn a factorized representation. This, in turn, aligns the optimization minimize the Total Correlation, thus approximating the spatial compression objective.
This unified objective provides practical and theoretically motivated mechanism for representation learning. While the SIB framework (Eq. equation 8 uses theoretical coefficients 3 and our prac- tical loss function (Eq. equation 4) implements these compression terms as a collection of weighted surrogate losses, including the KL balancing (Hafner al. 202T).
B DETAILED DESCRIPTIONS OF DMC-SUBTLA TASKS
This section details all five tasks in the DMC-Subtle benchmark introduced in Section 4.3. Figure compares the standard version of each task with our modified version, where task-critical objects have been intentionally scaled down to just few pixels. The specific modifications are as follows:
The agent must swing a tethered ball into a cup. The ball size and string width are reduced to 1/12 of the original.
Ball in Cup Catch:
Cartpole Swingup: The agent must swing up and balance pole on a cart. The pole width is reduced to 1/20 of the original.
Finger Turn: The agent must spin two-link finger touch target. The target size is reduced to 1/2 of the original.
Point Mass: The agent must move a point mass to a target. The goal removed as is always at the center, and the point mass size is reduced to 1/6 of the original.
Reacher: The agent must guide two-link arm to reach target. The target size is reduced to of the original.
Target
Point mass
Ball
Pole
Target
Target
Point mass
Ball +
Pole
Target
original versions of the five tasks (left to right: Ball in Cup Catch, Cartpole Swingup, Finger Turn, Point Mass, Reacher). Bottom: modified versions with downscaled task-critical objects in the same order.
Figure 9: DMC-Subtle benchmark. Top:
15

---

Published as a conference paper at ICLR 2026
C DETAILED RESULTS ON DEEPMIND CONTROL SUITE
This section provides the individual learning curves for 20 tasks in the DMC benchmark (Fig-
ure [Op. The corresponding aggregated results (mean/median across tasks) are shown in Figure 3 in the main text.
Acrobot Swingup
Ball In Cup Catch
Cartpole Balance
Cartpole Balance Sparse
1230
3002
150
200
032
030
153
602
100
432
202
200
250
SO0K
04
SC0K
1M
04
500K
04
500K
Cartpole Swingup
Cartpole Swingup Sparse
Cheetah Run
Finger Spin
1002
832
603
603
400
202
05
500K
IM
04
S00K
1M
05
300k
14
0%
100k
Finger Turn Easy
Finger Turn Hard
Hopper Hop
Hopper Stand
1.00
2002
1000
D00
803
Return
603
602
400
200
202
200
0.
500K
3M
0.
SCOK
1M
0.
500k
1M
0K
500K
Pendulum Swingup
Quadruped Run
Quadruped Walk
Reacher Easy
1002
1000
502
602
402
302
402
200
105
202
BOOK
00
SCOK
1M
0%
200k
500K
Reacher Hard
Walker Run
Walker
Walker
002
1220
693
602
000
603
402
600
400
203
203
0.
500K
0c
SC0K
0C
500K
500K Enw steps
Env
Env
22-Dreamer
Dreamer
Dreamer-InfoNCE
DreamerPro
DrQ-v2
TD-MPC2
Figure 10: Per-task learning curves for all 20 DMC tasks.
16

---

Published as a conference paper at ICLR 2026
D DETAILED RESULTS ON META-WORLD
This section provides the individual learning curves for all 50 tasks Meta- Woold (Figure 11 The corresponding aggregated results (mean/median success rate across tasks) are shown in Figure 4 in the main text.
Assembly
Basketball
Picking
Button Topdown
Sioce
Button Press Topdown
Button Press Well
Coffee
Coffee
Coffee Fush
Dial Tumm
134
204
Disessemble
Door Close
Door Lock
Door Open
Door Unlock
Drawer Close
Drawer Open
Faucet Close
Paucet Open
Hammer
Hand Insert
Handle Press
Handle Press Side
Handle Pull
Hondle Pull Side
Lever Pull
Peg Insert Side
Peg Unplug Side
Pick Dut Of Htte
Pick Place
Pick Place Wall
Plate Stide
Plate Slide Back
Plate Slide Back Side
Plate Slide Side
Push
Push Back
Prush Wall
Reach
Pa-ach Wall
Shalf Place
Seccer
Stick Pull
Stick Push
Sevep
Sweep Into
Window Close
Window Open
2012
R2-Dreamer Dreamer
Dreamer-InfoNEE
DreamerPro
TD-MPC2
Env stepa
stage
Figure 11: Per-task learning curves for 50 BeradWorld tasks.
17

---

Published as a conference paper at ICLR 2026
E DETAILED RESULTS ON ABLATION STUDIES
This section provides the individual learning curves for all 20 tasks our ablation study (Figure 12 The corresponding aggregated results (mean/median across tasks) shown in Figure in the main lext.
Acrobot Swingup
Ball In Cup Catch
Cartpole Balance
Cartpole Balance Sparse
12:0
1200
350
1000
1000
Return
8c0
850
200
600
600
150
400
100
200
200
3K
500K
3M
0K
5:0K
1M
0K
500K
3M
DK
5O0K
1M
Cartpole Swingup
Cartpole Swingup Sparse
Cheetah Run
Finger Spin
1000
800
Return
600
600
6:0
400
410
400
400
200
200
500K
0
SO0K
500k
50ck
Finger Turn Easy
Finger Turn Hard
Hopper Hap
Hopper Stand
3000
1000
800
810
800
Retur
600
600
600
400
200
6:00
200
200
100
200
S00K
SOOK
500K
1M
500K
Pendulum Swingup
Quadruped
Quadruped Walk
Reacher Easy
1000
1000
1000
800
800
8:0
Actum
600
500
400
400
400
200
200
200-
200
S00K
5:0KK
1M
500K
5.0K
Reacher Hard
Walker Run
Walker Stand
Walker Walk
2000
300
620
8100
Return
000
400
400
200
200
200
200
520K
Env sleps
Env steps
Env steps
Env steps
R2-D 12-Dreamer R2-Dreamer + DA
R2-Dreamer (half batch size)
DreamerPro (no DA)
DreamerPro
Dreamer (no recon. loss)
Figure 12: Per-task learning curves for the ablation study across all 20 DMC tasks.
18

---

Published as a conference paper at ICLR 2026
F HYPERPARAMETERS
Table summarizes the hyperparameters used in this study. These settings are based on those of DreamerV3, with minimal modifications related the proposed representation learning objec- tive. We used a single fixed hyperparameter configuration across all benchmarks, without task- or environment-speciine adjustments unless otherwise stated.
Table 2: Main hyperparameters. Our settings are identical DreamerV3, with key changes to the representation learning loss.
Name
Symbol
Value
General
Replay capacity
5 10
Batch size
B
16
Batch length
64
Activation
RMSNorm + SiLU
Learning rate
x 10 5
Gradient clipping
AGC(0.3)
Optimizer
LaProp(e = 10 10)
World Model
BT loss scale
0.05
Redundancy loss scale
5 x 10 1
Dynamics loss scale
3dyn
Representation loss scale
Srep
0.1
Latent unimix
1%
Free nats
1
Actor-Critic
Imagination horizon
H
15
Discount horizon
1/(1-1)
333
Return lambda
0.95
Critic loss scale
Byal
1 0.3
Critic replay loss scale
Brepwal
Critic EMA regularizer
1
Critic EMA decay
0.98
Actor loss scale
3pol
1
Actor entropy regularizer
3 x 10-4
Actor unimix
1%
Actor RetNorm scale
S
Per(R, 95) - Per RR 5)
Actor RetNorm limit
1 0.99
Actor RetNorm decay
19

---

Published as a conference paper at ICLR 2026
G PSEUDOCODE FOR REPRESENTATION LOss
Algorithm provides a PyTorch-style pseudocode for the core representation learning objective.
Algorithm R2-Dreammer Representation Loss (Pyforch-style Pseudocode)
alpha: weight on the off-diagonal terms
B: batch size, I: time steps, D: feature dimension
h: deterministic state from sequence model, [B, T, H_dim]
stochastic state from representation model, [B, T, 2_dim]
embeddings from image encoder, [R, T, R_dim]
projector: linear layer to project concatenated state to embedding space
Project features from dynamics model state carch.cat(ll z], dim=-1)
k projector (state) (state) [B, T, D]
Reshape for loss computation
rerhae (B T, = e.detach reshapee (B T, D) Stop gradient to encoder
Normalize features along the batch dimension
norm (k k.mean (dim=0)) st.ddd(iim=)) 1e-5) e_norm (e e.mean (dim-0)) (e. 1e-5)
Cross-correlatioe matrix
C = (k nerm. T e {_norm) / (B
T) # ID, DJ
Invariance loss invariance_lcss = ((torch. diaghhaiiggaalalc (C) 1) x+)) sum ()
Redundancy reduction loss off_dias = C.clone()
off off_diag..o wit.fffll__a. [alll_diagonal_00 (0)
redundancy_loss = (off_diagus)) sum
Total less
loss invariance loss
aipha
redundancy_
H THE USE OF LARGE LANGUAGE MODELS
We utilized large language models to improve the grammar and readability of this manuscript.
20


Reviews:
[{"review_id": "AizcEr5AeM", "reviewer": "Reviewer_Hody", "summary": "The paper proposes R2-Dreamer, a Model-Based Reinforcement Learning (MBRL) agent based on the DreamerV3 architecture. It addresses two main limitations in current image-based world models: the computational expense and potential for task-irrelevant overfitting in decoder-based models, and the reliance on brittle, task-specific Data Augmentation (DA) in existing decoder-free models. R2-Dreamer replaces the reconstruction objective with a redundancy-reduction objective inspired by Barlow Twins, applied between the image embeddings and the projected latent states of the RSSM.", "questions": "1. Clarification on Gradients: In Algorithm 1, you have e = ...detach(). Does this mean your image encoder $f_\\phi(x_t)$ receives NO gradients from $\\mathcal{L}_{BT}$? If so, what is the primary learning signal for the encoder? Is it solely the $KL(q(z_t|h_t, e_t) || p(z_t|h_t))$ term?\n2. Batch Size Sensitivity: Have you evaluated the sensitivity of R2-Dreamer to the batch size (specifically the total samples $B \\times T$ used for the correlation matrix)? Barlow Twins often degrades rapidly with small batches.\n3. Complex Backgrounds: How does R2-Dreamer perform when the background is not just static (like DMC) but dynamic and irrelevant (e.g., \"distracting control\" suite)? This would be a stronger test of the claim that it avoids wasting capacity on irrelevant details compared to reconstruction.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Principled approach to removing DA: The paper successfully identifies a major pain point in current decoder-free methods—the reliance on heuristic DA. Proposing an information-theoretic internal regularizer (redundancy reduction) as a replacement is a sound and theoretically motivated direction, nicely grounded in the Sequential Information Bottleneck framework in Appendix A2.\n2. Strong empirical validation of the core hypothesis: The ablation study (Figure 6) provides compelling evidence. It shows that while DreamerPro collapses without DA, R2-Dreamer maintains performance, proving that the proposed $\\mathcal{L}_{BT}$ effectively prevents collapse without external heuristics\n3. Effective stress-testing (DMC-Subtle): DMC-Subtle cleanly isolates failure modes of reconstruction-based (wasted capacity on background) and DA-based (distortion of small features) methods, highlighting the specific advantages of R2-Dreamer in precision-demanding tasks\n4. Computational Efficiency: The reported speedups are significant and practically important for scaling MBRL.", "weaknesses": "1. Batch size concerns for Barlow Twins: Redundancy reduction objectives often require large batch sizes for stable covariance estimation. The paper uses standard Dreamer batching ($B=16, T=64 \\implies N=1024$)8. While this appears sufficient for DMC, it raises concerns about stability in higher-dimensional or more diverse visual environments where 1024 samples might not sufficiently estimate the cross-correlation matrix.\n2. Ambiguity in Encoder Training: The pseudocode (Algorithm 1) indicates that the image embeddings $e$ are detached before entering the $\\mathcal{L}_{BT}$ loss9. If accurate, this means the image encoder does not receive gradients from the primary representation learning objective, relying solely on gradients flowing back from the RSSM via $KL$ terms, which is highly unusual for this class of SSL objectives.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jvd6Q8O1J5", "reviewer": "Reviewer_z3Lo", "summary": "This paper proposes a data-augmentation-free learning method for RSSM-based decoder-free MBRL. The core of this method is is a feature redundancy reduction objective inspired by Barlow Twins and therefore has no need for pixel-wise reconstruction or data augmentation. Emprical study shows the superior performance on standard DMC benchmark and more chanlleging DMC-Subtle benchmark.", "questions": "1. Could the authors compare their method with state-of-the-art decoder-free MBRL methods like TD-MPC2?\n2. Could the authors evaluate different methods on other benchmarks like Meta-World?\n3. Could authors explain why data augmentation is a problem?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written, allowing readers to follow the main arguments.\n\n2. It provides a comprehensive experiments and ablations to demonstrate the effectiveness of their proposed new representation learning paradigm.\n\n3. Releasing codebase is good, which can facilitate future research.", "weaknesses": "1. The authors didn't compare TD-MPC2 in their experiments, which is a strong state-of-the-art baseline for decoder-free methods.\n2. Though authors evaluated different methods on many tasks on the DMC benchmark, I think evaluating only on these locomotion tasks is kind of not comprehesive and it would be better to evaluate on other types tasks like Meta-World.\n3. One of claims in this paper is that their method doesn't need hand-engineered data augmentation like other decoder-free MBRL methods. But I question whether this is really a significant problem. For example, TD-MPC2 only uses very simple random shift augmentation, which I don't think it is troublesome for algo implementation. So I'm confused that why the data augmentation for decoder-free MBRL methods is a drawback.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TfK15YbPMh", "reviewer": "Reviewer_Eb9R", "summary": "This paper proposes R2-Dreamer, an enhanced variant of DreamerV3 that introduces an internal regularizer to improve latent state representation learning. Specifically, the authors replace the traditional image reconstruction loss with a Barlow Twins loss, aiming to encourage more informative and less redundant latent features. Theoretical analysis shows that this new objective is equivalent to optimizing a variational bound on an extended Sequential Information Bottleneck. Experimental results demonstrate that the proposed method achieves more robust latent representations and improved computational efficiency.", "questions": "1. Minor: Why are the ablation results of R2-Dreamer+DA different between Figure 7 and Figure 10? In Figure 7, the performance of the DA version shows a significant drop, while in Figure 10, the drop is much smaller. This discrepancy seems inconsistent with the claim that data augmentation is harmful for precision-demanding tasks.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper clearly identifies a key limitation of DreamerV3: its latent representations can be overly influenced by reconstructing input images, leading to a focus on irrelevant pixels rather than task-relevant features. By replacing the reconstruction loss with a self-supervised objective defined in the latent space, the proposed method encourages more compact and task-relevant representations.\n2. The paper provides solid theoretical analysis to explain the impact of the new learning objective and includes extensive experimental validation. Results on the challenging DMC-Subtle benchmark demonstrate the superiority of the proposed representation. Moreover, the ablation studies convincingly show both the necessity of the proposed objective and the redundancy of data augmentation under this framework.\n3. The proposed approach is conceptually simple yet empirically effective, making it a strong candidate for a new baseline in model-based reinforcement learning.", "weaknesses": "1. The experimental environments are not sufficiently diverse. The paper evaluates only on DMC-Subtle, whereas DreamerV3 has also been tested on other benchmarks such as Atari and DMLab. Including experiments on these additional environments would make the evaluation more comprehensive and the conclusions more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oeMywGMJEI", "reviewer": "Reviewer_zUBH", "summary": "This paper  presents a decoder-free agent that introduces a self-supervised objective acting as an internal regularizer to prevent collapse. Experiments on DMC and DMC-Subtle validate the effectiveness of the proposed method.", "questions": "Have the authors tried conducting experiments on environments with distractors, such as Distracting Control Suite [1]?\n\nReference:\n\n[1] Stone et al. \"The Distracting Control Suite — A Challenging Benchmark for Reinforcement Learning from Pixels\", arXiv, 2021.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper proposes a decoder-free MBRL agent that adopt internal regularizer to avoid reconstructing observations from a latent state.\n- This paper introduces DMC-Subtle, a modified DMC benchmark where task-critical objects’ sizes are significantly reduced, demanding a higher level of representational precision.\n- The proposed approach outperforms baselines in DMC and DMC-Subtle.", "weaknesses": "- There is no comparison with competitive baselines such as TD-MPC2 [1].\n- The proposed DMC-Subtle benchmark seems somewhat questionable or not well-justified.\n- More challenging benchmarks are needed to better validate the effectiveness of the proposed algorithm.\n- The DMC tasks used are relatively simple. How does the method perform on more complex tasks like *DMC Dog* or *DMC Humanoid*?\n- As shown in Figure 3, the improvement appears to be marginal.\n- There is no comparison with VAI [2], which is a baseline that adopts unsupervised visual attention.\n\nReferences:\n\n[1] Hansen et al. \"TD-MPC2: Scalable, Robust World Models for Continuous Control\", ICLR, 2024.\n\n[2] Wang et al. \"Unsupervised Visual Attention and Invariance for Reinforcement Learning\", CVPR, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)