Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
Authors: Brett Barkley
David Fridovich-Keil
Venue: ICML 2025 poster (2025)
Keywords: model-based reinforcement learning
online reinforcement learning
deep reinforcement learning
Primary area: reinforcement_learning->deep_rl
TL;DR: Dyna-style synthetic data helps in OpenAI Gym but hurts in DeepMind Control Suite, despite both using MuJoCo and identical hyperparameters, highlighting key challenges in model-based RL across benchmarks that can't be alleviated via modern methods.

Abstract: Dyna-style off-policy model-based reinforcement learning (DMBRL) algorithms are a family of techniques for generating synthetic state transition data and thereby enhancing the sample efficiency of off-policy RL algorithms. This paper identifies and investigates a surprising performance gap observed when applying DMBRL algorithms across different benchmark environments with proprioceptive observations. We show that, while DMBRL algorithms perform well in control tasks in OpenAI Gym, their performance can drop significantly in DeepMind Control Suite (DMC), even though these settings offer similar tasks and identical physics backends. Modern techniques designed to address several key issues that arise in these settings do not provide a consistent improvement across all environments, and overall our results show that adding synthetic rollouts to the training process --- the backbone of Dyna-style algorithms --- significantly degrades performance across most DMC environments. Our findings contribute to a deeper understanding of several fundamental challenges in model-based RL and show that, like many optimization fields, there is no free lunch when evaluating performance across diverse benchmarks in RL.

Stealing That Free Lunch:
Exposing the Limits of Dyna-Style Reinforcement Learning
Brett Barkley'
David Fridovich-Keil'
Abstract
implies that performance will vary depending on the envi- ronment and problem characteristics. that light, this paper begins with a simple, yet novel and unexpected observation: Model-Based Policy Optimization (MBPO) (Janeer et al.) 2019), popular Dyna-style (Sutton 1991) model-based reinforcement learning (DMBRL) algorithm, demonstrates strong performance across tasks in OpenAl Gym (Brockman et al. 2016), but performs significantly worse than its base off offfpolicy algorithm, Soft Actor Critic (SAC) (Haarnoja et al. 2019a), when trained in DeepMind Control Suite (DMC) (Tassa et al. 2020 cf. Figure !
Dyna-style off-policy model-based reinforcement learning (DMBRL) algorithms are a family of techniques for generating synthetic state transition data and thereby enhancing the sample efficiency of off-policy RI algorithms. This paper identifies and investigates surprising performance gap ob- served when applying DMBRL algorithms across different benchmark environments with proprio- ceptive observations. We show that, while DM- BRL algorithms perform well control tasks in OpenAI Gym, their performance can drop signifi- cantly in DeepMind Control Suite (DMC), though these settings offer similar tasks and iden- tical physics backends. Modern techniques de- signed to address several key issues that arise in these settings do not provide a consistent improve- ment across environments, and overall our re- sults show that adding synthetic rollouts the training process the backhone of Dyna-style algorithms significantly degrades performance across most DMC environments. Our findings contribute to deeper understanding of several fundamental challenges in model-based RL and show that, like many optimization fields, there free lunch when evaluating performance across diverse benchmarks in RL.
Interestingly, the Gym and DMC benchmarks feature sim- ilar tasks and identical physics backends (Todorov et al. 2012). Yet. despite the popularity of MBPO, with around 1k citations (as of late 2024), only two studies report eval- uating MBPO in DMC (Wang et al. 2024 Voelcker et al., 2024), and performance gap has only been noted in the hopper-howd task (Voelcker et al. eralize these findings, and call into question the robustness of MBPO, and of Dyna-style algorithms more broadly.
2024 Our results gen-
OpenAl Gym
DeepMind
Suite
ALM
(No Dyna)
MBPO (No
1. Introduction
Colloquially, the "no free lunch theorem" states that no optimization algorithm can be universally optimal across all problem instances. In reinforcement learning (RL), this
Figure 1. Comparison of normalized final return for two different DMBRL algorithms with and without Dyna-style enhancements. Results are averaged across 6 random seeds per task, with 6 tasks
Department of Computer Science, University of Texas Austin, Austin, TX, USA 'Department of Aerospace En- gineering and Engineering Mechanics, University of Texas at Austin, Austin, TX, USA. Correspondence to: Brett Barkley <bbarkley edu>>
from OpenAl Gym (Brockman et al. 2016) and 15 from DMC Tassa et al.
2020
For training curves, cf. Figures and 17
Beyond robustness, these discrepancies raise critical ques- tions for us in the RL community. First, is the perfor- mance gap fundamental, or merely an artifact we can "fix" with modern techniques applied to predictive models, other model-based RI. components, and/or the base off-policy
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforvement Learning
algorithms?
Latent Models (ALM), model-based algorithm known for its high wall-clock speed, as shown in Figure 2 These improvements over previous model-based RL implementa- tions empower researchers with limited resources to conduct extensive DMBRI. studies using just single GPU.
Our extensive results indicate: "yes they are fundamental, and no we cannot." Figure shows that this gap persists more broadly than in just MBPO, but also in another re- cent DMBRL algorithm, Aligned Latent Models (ALM) (Ghugare et al. 2022). These algorithms have contrasting design philosophies: iBPO built on SAC (Haamooj et al. 2019a). a stochastic policy algorithm leveraging entropy- based exploration, while ALM is built on DDPG (Lillierap et al. 2019). deterministic policy algorithm that relies on additive Gaussian noise for exploration. These differ- ences exploration strategies, combined with their distinct architectures and objectives, make MBPO and ALM com- plementary test cases for assessing whether the observed gap is inherent Dyna-based methods rather than specific to particular implementation. While our findings raise serious concerns about these instantiations, our intent is not to challenge the theoretical soundness of the Dyna frame- work itself, but rather to evaluate how well its practical realizations hold up under realistic, high-fidelity conditions.
0.6
0.3
Seconds per Erw
0.4
0.3
0.2
0.1
Figure 2. Comparison of seconds per environment step across mul- tiple DMBRL implementations when deployed OpenAl Gym environments. Compared to Pytorch MBPO and ALM, our imple- mentation takes ~40 and ~4x less time, respectively.
Using MBPO as a surrogate for subclass of DMBRL algorithms that underperform in DMC, we analyze the Gym/DMC performance gap and try to close it by address- ing some possible sources of this discrepancy including overestimation bias, neural network plasticity, and environ- ment modeling fidelity. Despite these efforts, there remains a substantial gap in performance on these two benchmarks, which suggests that the generalization challenges of MBPO and, by extension, subclass of DMBRL algorithms more deeply rooted.
In summary, our main contributions are:
1. Demonstrating that DMBRL methods suffer from significant performance gap when training from scratch in OpenAl Gym versus DMC environments.
2. Analyzing potential causes for this discrepancy and ap- plying modern mitigation approaches, which ultimately fail to consistently resolve these problems. These re- sults provide insight into the factors affecting perfor- mance across different environments.
Second, this oversight remaining unnoticed for several years highlights sigmificant, pervasive issue which has also noted in several other Al subfields (Haibe-Kaing et al., 2020 Ben- der et al., 2021 Agarwal et al., 2021 Ahmed et al. 2023 Jordan et al., 2024): a lack of reasonable access which pre- vents scientists from reproducing and verifying key results. For instance, per the calculations in Appendix [ gener- ating just Figure 4 of this manuscript using the original MBPO implementation (Janner 2019) would require over 100 GPU-days, assuming optimistic runtime estimates from prior work (Ghugare et al., 2022 Xu et al. 2022).
3. Dramatically accelerating the DMBRL experimen- tation process with a new JAX-based implemen- tation, which achieves up to a ~40x decrease wall-clock time. This acceleration lowers the computational barrier for researchers to de- velop (and comprehensively evaluate) DMBRL al- gorithms. We have released our code here:
2. Background
Such computational barriers limit experimentation and cross-comparison, leaving critical issues, like those high- light, unexamined. To address this, we developed MBPO implementation built on nigh-performance SAC imple- mentation Oro et al., 2023), leveraging JAX (Bradbury et al. 2018 for efficient parallelization. This new implemen- tation reduces training time for the experiments in Figure 4 to approximately 4 GPU-days, a 26x speedup compared to the original Pytorch implementation (Janner al. 2019)
2.1. Reinforcement Learning, Model-Based RL, and Dyna-style Algorithms
Reinforcement learning (RL) models agent-environmen in- teractions as a Markov Decision Process (MDP), defined by the tuple (S, A. A.p,,, p. r.))) Here, S is the state space, A is the action space, probability distribution p(s'|s, a) represents the transition dynamics, r(s, a) is the reward function, and ty E [o, 1) is the discount factor. The goal of the agent is to learn a policy (a]s) that maximizes the expected cumula- tive reward.
Our implementation also compares favorably to Aligned
2

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
Learning an effective policy often requires extensive inter- action with the environment, which can be prohibitively costly and time-consuming. To address this, model-based RL algorithms aim to learn a model of the environment's 's dynamics, po r|s, a), to reduce the need for direct envi- ronment interaction. Within this family of algorithms exists the Dyna architecture (Sutton ]991). which aims to learn a world model pa. predict hypothetical transitions under that world model, and then use those imagined experiences to ac- celerate training. One such method is Model- Based Policy Optimization (MBPO) (Janner et al. 2019), which trains an ensemble of probabilistic neural networks to approximate the environment's 's dynamics and rewards.
the policy and latent model, DreamerV3 separates these processes and learns latent world model using multiple objectives. Please refer to Appendix A for further details.
2.2. Benchmarks: OpenAl Gym and DeepMind Control
OpenAI Gym (Brockman et al., 2016) is widely-used benchmark suite for RL algorithms, providing variety of environments, including continuous control tasks with and without the MuJoCo physics engine (Todorov et al.) 2012) as the physics backend (in this paper, we restrict our attention to Gym environments which use MuJoCo). The DeepMind Control (DMC) Suite (Tassa et al. 2020 provides larger set of continuous control tasks based on MuJoCo and is designed to provide more challenging and comprehensive evaluation of control algorithms.
Like most off-policy methods, MBPO maintains a replay buffer of the transitions it has experienced in the environ- ment. MBPO supplements this real world data by gener- ating synthetic on-policy rollouts that branch from states in the replay buffer, thereby augmenting the replay data with "imagined" transitions. This partially synthetic dataset is then used to train both the actor, a neural network that determines the optimal action based on the current state, and the critic, neural network that estimates the expected return given a choice of action and state. For MBPO this process is done using the model-frec algorithm Soft Actor- Critic (SAC) (Haarnoja et al.] (2019)), a widely used, reliable method for continuous control tasks. MBPO has demon- strated strong performance on OpenAI Gym benchmarks, often greatly surpassing traditional model-free approaches in sample efficiency. Due to its reputation for strong per- formance, many recent algorithms are based on MBPO, e.g.
Significant differences exist between DMC and OpenAl Gym in terms of physical parameters, reward structures, and termination conditions. We present an extended discussion of modeling differences and their possible connections to the performance gap we investigate in this paper in Appendix C
3. Performance Gap of Dyna-based MBRL Across Benchmarks
In this section, we put forth the following hypothesis: incor- porating Dyna-style modifications into otherwise successful off off-py algorithms can prevent them from improving be- yond their performance at initialization - that of a randomly initialized policy. In support of this hypothesis we present empirical evaluations of MBPO's performance across six OpenAI Gym tasks and six challenging robotics tasks (Nik- ishin et al. bury et al.
2024 Dong al. 2024| Zheng (Lai et al. 2020 Li et al.
et al.
2023 Wang et al.]
2023 Lai et al. 2021).
2022) from DMC using our JAX-based (Brad-
Another recent extension of the Dyna line of model-based RL is Aligned Latent Models (ALM) (Ghugare et al. 2022). Rather than generating synthetic data in the true state-action space like MBPO. ALM jointly learns observation repre- sentations, world model that predicts next representations given the current representation, and a policy that acts in the representation space. It then uses the Deep Deterministic Policy Gradient (DDPG) algorithm (Liliicrpp et al./ (2019) for off- policy learning and uses synthetic trajectories to train the latent policy only. Like MBPO, ALM has demonstrated strong performance across OpenAI Gym tasks, while also requiring substantially less wall clock time than MBPO for training. shown in Figure 2
2018 implementation of MBPO. For the OpenAl Gym benchmark, we demonstrate that our implementation matches the performance produced by the original PyTorch (Paszke et al. [2019) MBPO implementation (Janner et al.) 2019) and ALM (Ghugare et al., 2022). Furthermore, we show that MBPO dramatically outperforms its "no Dyna" base off-policy algorithm in OpenAl Gym, Soft Actor-Critic (SAC) (Haarnoja et al. 2019a), even when both share iden- tical hyperparameters and architectures.
In contrast, in DMC (Tassa et al. 2020) we observe that MBPO struggles to achieve any policy improvements in six out of fifteen of the challenging environments that we tested on when training from scratch. Because MBPO differs from SAC only due to Dyna-style "enhancements," these results demand deeper investigation into whether the default SAC hyperparameters, the model ensemble, or other factors in- duce an otherwise healthy SAC implementation to fail.
Finally, particularly notable DMBRL approach is Dream- v3 (Hafner et al. (2025), which, unlike the other DMBRL methods presented in this work, was not tested in Gym, but has state-of-the sample efficiency in DMC. Like ALM, DreamerV3 optimizes a policy in the latent space using imagination-based trajectories and actor-critic learning. Un- like ALM, which uses a single objective for optimizing both
'raining curves for MBPO in all fifteen challenging DMC environments are provided in Appendix G in Figure II
3

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
Happerey3
Wallierdd-33
Amtro2
quadrupeed walk
hopper-hop
humanold-wok
Eaosude Notur
1000
Eo Retum
750
150
Toscon Betum
150
2020
2000
2500
500
250
500
250
250
85 0.5
63
02
63
64
Envrrommnn Step: (xlor)
Erwi onment Stops 104)
Enviromment 1t Stope 1x1
Endiiommnt Stens (xl0f)
End commnnt Steps (X209)
Endiroemant Steps (x109)
Humenosd-ee
FiverteedPerdullm--
Halff
hepper-saadd
humannidd
quadruped-run
Epannde Saturn
750
150
Roasote Saturs
750
Excol:
2000
200
5050
500
Crande
100
:30
100
0.02
Environment Steps Ox
Erufromment Steps 101)
Endirommnt Steos 1x10
Envirommnn Steas (x20f)
Endironment Steps (x10f)
Endrcnmmnt Steas (x10f)
1-step
tescap
Figure 3. 1-step MBPO consistently achieves higher final episodic returns and demonstrates faster convergence compared to SAC. Likewise, 1-step MBPO either matches surpasses ALM in terms of sample efficiency across all but one task. Solid curves cor- respond to the mean and shaded regions represent the standard deviation across six trials. The dotted line represents the episodic return the original n-step MBPO at the end of training.
Figure 4. Performance of MBPO and SAC six DMC bench- marks. MBPO exhibits no policy improvement across these envi- romments when Dyna-style enhancements used, but can imme- diately make policy improvements without them (i.c. SAC).
3.2. Results in the DeepMind Control Suite
In contrast, MBPO's performance when training from scratch drops significantly in the DMC environments despite using the exact same hyperparameters and model structure as for the successes in OpenAl Gym. Full results across all fifteen challenging DMC environments are provided in Figure I1 in Appendix G These results show the complete range of MBPO's performance, including scenarios where MBPO solves tasks more slowly than SAC, matches SAC's sample efficiency, or fails to make any policy improvements.
This performance gap extends beyond MBPO into other model-based RL algorithms in the Dyna family (Sutton 1991). To this end, we deploy the recently developed ALM (Ghugare et al. (202)) method across the same six challeng- ing DMC tasks and observe the same phenomenon.
3.1. Results in OpenAl Gym
In this section, we focus six continuous control tasks where MBPO fails to improve the policy when training from scratch, cf. Figure 4 These results are particularly striking because several of these tasks have high-level analogues in OpenAl Gym when accounting for reward structures, termination conditions, and physics parameter differences.
As depicted in Figure 3 when training from scratch our implementation of MBPO using 1-step synthetic rollouts consistently outperforms SAC across six OpenAl Gym en- vironments, and achieves the same sample efficiency as the original n-step Pytorch MBPO implementation. These findings agree with those of the original MBPO paper- augmenting policy training with single-step synthetic roll- outs produces an algorithm with strong performance (Janner let al., 2019). This affirms both the accuracy of our re- implementation and the sample efficiency gains of 1-step MBPO in OpenAI Gym tasks.
These results indicate that the recent observation of (Voel: oker al. 2024) - i.e., that MBPO cannot reliably solve the hopper heoper-hop-eee environment in DMC - indicates a much broader trend. Moreover, unlike prior work, when we consider these findings alongside the results in Section 3.1 we see that across multiple environments and two seemingly similar benchmarks, MBPO consistently struggles to im- prove upon random policy, let alone train a competent one. Because one can recover SAC from MBPO by remov- ing the Dyna-style actor/critic updates based on synthetic data, we can conclude that these Dyna-style "enhancements" are the culprit behind MBPO's failure in the DMC environ- ments shown in Figure
Here, 1-step and n-step refer to the length of synthetic roll- outs simulated using the learned world model. It is common to prefer shorter rollouts, however, in order to mitigate com- pounding model errors (Janner et al., 2019 Sikchi et al.) 2022); indeed, we demonstrate that these errors are quite substantial in Section 4.1 Combining the results in Fig- ure 2 and Figure 3 we can conclude that our JAX-based implementation of 1-step MBPO matches the original im- plementation's 's (Janner, 2019) n-step MBPO's performance with much greater wall-clock speed, and as such we will henceforth refer to 1-step MBPO as simply "MBPO."
3.3. Beyond MBPO: Another Dyna-based Algorithm
In this section, we show that the conclusions in Section 3.2 extend beyond MBPO to another member of the Dyna fam- ily, Aligned Latent Models (ALM) (Ghugare et al. 2022 We choose ALM for two reasons. First, unlike MBPO
Training curves for ALM in all fifteen challenging DMC envi- ronments are provided in Appendix G in Figure
4

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforvement Learning
quedruped-wwlk
hepper-hop
humanold
Key insights:
150
150
500
MBPO consistently matches or outperforms the sample efficiency and final episodic return of its "no Dyna" variant (i.e. SAC), across OpenAl Gym environments.
0.0
81
02
83 64
02 83 3.4
Environmnn Stops (x10%)
Envronment Steps (x30%)
Envnon Shass (x10%)
hogper-stand
bumannidd-stand
quadruped-on
-30
Gooo 2401
Elooo
502
MBPO consistently underperforms the sam- ple efficiency and final episodic return of its "no Dyna" variant (i.e. SAC), across DMC environments.
0.2
Environment Sheps (x10%)
Stepe (x10')
Envronment Ox30f)
Dyne)
Figure 5. Performance of ALM with and without Dyna-style hancements on six DMC benchmarks. Like MBPO, ALM shows no policy improvement with Dyna-style enhancements, but per- forms strongly without them (6 trials, + std).
This same performance gap can be observed for ALM, another Dyna-based RL algorithm, suggesting these issues may be endemic to a larger subclass of the Dyna algorithm family.
4. The Effect of the Predictive Model on Algorithm Performance
4.1. Is High Model Error the Problem?
In Section 3 we saw that Dyna-style "enhancements" caused both MBPO and ALM to dramatically underper- form their "no Dyna" counterparts in DMC, whereas the same enhancements led performance gains in OpenAl Gym. Since the predictive model ensemble is central to MBPO's sample efficiency gains over model-free methods, we focus on this first as a source of MBPO's issues. Studies have shown that high model error causes significant per- formance degradation (Gu et al. 2016 Rajeswaran et al.) 2017). and models are most effective over short to moderate planning horizons (Sikchi et al.. 2022 Holland al. 2019) if compounding model errors can be mitigated (Tal 2014 Buckman et al. 2018). We find that severe model er- ror in many DMC environments makes even 1-step rollouts unrealistic, rendering multi-step methods unproductive.
ALM is built on top of DDPG, which allows us to test Dyna in deterministic policy framework, in contrast to SAC's stochastic approach. Second, DDPG's reliance on Gaussian action noising offers a distinct exploration strategy com- pared to SAC's entropy-based exploration.
To evaluate whether Dyna-based "enhancements" nega- tively impact ALM, we repeated the same experiment for MBPO across six challengigg DMC tasks. We train ALM with and without Dyna-style enhancements using identi- cal hyperparameters, and compare their performance. This "no-Dyna" variant of ALM is inspired by (Ghugare et al. 2022) where the authors replaced actor training via synthetic model rollouts with TD3 (Fujimoto et al. 2018) actor loss that relies only on real transitions.
To quantify this effect, we introduce the percent model error, defined as:
The results of this experiment are shown in Figure and, as with MBPO, the base off-policy algorithm performs quite well, while its style counterpart (i.c. ALM) fails to make any policy improvements when training from scratch. This result, when combined with those of Section 3.2, pro- vides clear evidence that Dyna-style enhancements can pre- vent improvement beyond the performance of randomly initialized policy across many environments.
1 vll2
100,
where ý denotes the model's predicted next observation and reward, and y is the corresponding ground-truth target. This normalized metric allows for fair comparison across tasks with differing scales.
The relationship between percent model error and perfor- mance degradation in MBPO is challenging to assess, par- ticularly within DMC environments. We investigated this via comparisons of percent model errors across the training distribution (i.e., the replay buffer) for six tasks in both Ope- nAI Gym and DMC (cf. Figures 12 and 13 in Appendix G instead of relying on computationally expensive on- cn-policy rollouts. We observed that in all six OpenAl Gym tasks the model error converges significantly below 25%. In con-
Throughout the remainder of this paper, we analyze the un- derlying causes of these performance discrepancies through the lens of MBPO, and explore potential solutions. By doing so, we aim to identify whether these challenges are intrinsic limitations of Dyna-style algorithms in certain settings or if they can be mitigated in a general manner.
Training curves for ALM in all fifteen challenging DMC envi- ronments are provided Appendix G in Figure
5

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
trast, for the six DMC tasks where MBPO fails, the tasks fall into two categories. The hopper tasks exhibit model errors that converge above 100%. while the remaining tasks have model errors converging above 25% as training pro- gresses. Notably, all of these environments fail to improve policy performance beyond that of a random policy (recall
humanold-wal
hopper-hop
hopperstand
750
150
Toscon Betumm
750
500
500
150
250
250
0.0
63
02
63
64
85
Endicommnt Steos (x10%)
End cemont Steps (x209)
Endirorment Steas (x10
humaarddd-
cheetab-uun
welleesun
Elaude 24trr
750
Cravode Saburs
150
Eoauot Saturs
750
Figure 4
500
100
300
While modeling errors in MBPO for DMC environments not universally larger than those in OpenAI Gym, their impact is difficult to contextualize without further analy- sis. To investigate this impact, and inspired by previous work (Kalweit & Boedecker, 2017), we vary the synthetic- to-real data ratio, S. which represents the proportion of synthetic rollouts in each training batch. By varying S, we can evaluate spectrum between MBPO (S + 0.95) and SAC (S + 0), thereby systematically studying the ef- fect of synthetic data on policy performance and assessing the significance of modeling errors in degrading MBPO's s performance (Gu et al. 2016).
Envicommnn Steas (x20f)
Eniiiomment (x:1f)
Endiceemnt Steas (x10f)
MBPO (perfect model)
MBPO
SAC
Figure 6. Comparing MBPO with perfect predictive model to SAC and MBPO with a learned model. Solid curves correspond to the mean and shaded regions represent the standard deviation across six trials.
Focusing on hopper-st and and humanoid-stande which represent the two categories of model error magni- tude and highlight where MBPO consistently fails with learned model, we find clear trend: increasing S reduces episodic returns (cf. Figure 14 in Appendix G.. This effect is particularly evident for
the default hyperparameters and access to perfect rollouts can improve upon its base off-policy method (i.e., SAC) across multiple environments in DMC.
where even minimal synthetic data severely degrades performance, high- lighting the detrimental effect of model errors on MBPO.
To answer this question, we modified the reset procedure of the DMC simulator to process arbitrary initial states pro- vided by the user. Using this augmented simulator, we gener- ated real trajectories starting from states sampled randomly from the replay buffer during training. All other hyperpa- rameters for MBPO were left unchanged. The results for six DMC environments are presented in Figure
To rule out tuning issues, we performed comprehen- sive hyperparameter sweeps for sumanoid-stane (cf. Appendix [F) since introducing any synthetic data can induce failures in MBPO with the default hyperparameters. We swept over the key hyperpa- rameters related to the training and utilization of the predic- tive model, including the size of the model's hidden layers, the interval at which the model retrained, the learning rate, and the number of training steps. None provided any improvement policy performance across training despite reductions in model-error, which suggests that deeper archi- tectural and algorithmic changes are required. Reduction of model bias remains an open issue in the broader MBRL lit- erature (Wang et al. 2019 Deisenroth & Rasmussen 2011] Luo et a 2024) and we aim to address this in future work as part of a broader solution to Dyna-style algorithm issues identified here.
and
Access to a perfect predictive model produces compelling yet unsatisfying results. In particular, across these six envi- ronments - which include mix of cases settings in which MBPO previously had either promising results or no policy improvements- - even
eveen with a perfect model, MBPO cannot outperform its base off-policy method consistently.
In OpenAI Gym, even MBPO with a learned model achieves sample efficiency comparable to or exceeding SAC across all environments (ef. Figure 3) However, in DMC, while perfect model enables policy improvement in environments where MBPO with a learned ensemble fails completely, it does not consistently surpass SAC's sample efficiency in four out of the six environments shown in Figure 6
4.2. What If We Had Perfect Model?
It becomes clear that even with an idealized model, model- ing errors alone cannot fully explain the failings of MBPO in DMC. The hopper r tasks exhibit significantly larger model errors compared to humano i d a-dtandd and other tasks (Figure 13), yet even with the perfect model, MBPO does not achieve the same level of success for hopper (Figurelo) as SAC. This observation indicates that further analysis is needed to investigate other sources of these shortcomings.
These results raise fundamental question: if we had ac- cess to perfect model of the environment, would MBPO outperform SAC in DMC, as it does in OpenAl Gym? Ad- dressing this question allows us to bypass the long-standing challenge of training a reliable predictive model (Atkeson & Schaal 1997), and directly evaluate whether MBPO with
6

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
humnnool stand
stand
quadruped-run
quadruped-wwlk
critic Q-values throughout training for SAC and MBPO in DMC environments where MBPO fails, and report the results in Figure MBPO's Q-values exhibited significantly greater divergence despite identical hyperparameters. This suggests that inaccurate synthetic transitions drive the Q- value estimation issues and, by extension, hinder learning.
Volees
Womo vokes
Meat vitues
0.5
02
0.0
02
5.4
0.0
02
24
Enviroment Stace (x10))
Endronment StADS (K)0))
Envirommmnt Stega (K20%)
humanoid-eoa walk
happerhop
hogperstand
Noin O
Nesn Owiiees
10P
Since the MBPO model predicts both next states and re- wards inaccurately even on its training distribution, and because synthetic data dominates the training set by default in MBPO, the actor and critic repeatedly encounter infeasi- ble transitions and unrealistic returns. We hypothesize that these synthetic transitions conflict with actual replay data, causing severe critic target non stationarity and substantial Q-value estimation errors.
Envfronment Steos (x30")
Endrrommnnt Steas (x10f)
Envfrommen Steps (x20f)
Figure 7. Mcan Q-values (6 seeds) with shaded regions showing min-max range. MBPO's s critic either massively underestimates or predicts return when compared to SAC. Overestimation is magnified in quadruped tasks with Dyna-style enhancements. Results across all 15 DMC tasks are provided in Figure [6
To validate this hypothesis, we compared the Q-values ob- tained using both learned model and a perfect model throughout training With a perfect model, massive un- derestimation is much less pronounced, implicating mod- eling errors as the primary driver of this issue. However, per the discussion of Section 4.2 even without massive underestimation MBPO fails to match SAC's performance.
4.3. Investigating Underlying Failure Mechanisms Induced By Model Bias
Since even access to the true environment dynamics fails to resolve MBPO's poor performance in DMC, we explored additional possible failure modes beyond model bias that could hinder MBPO's success. These additional failure modes relate to the concept of replay ratio (RR).
Since critic divergence stems from learned functions extrap- olating in an unconstrained manner, we employed Layer Normalization (Ba et al. (016a) as regularization tech- nique as in (Ball et al., 2023) to bound the critic output. This technique has proven quite successful in previous work Nauman et al. 2024), but even after partly mitigating critic divergence, MBPO still cannot outperform SAC consistently as shown in Appendix G in Figure 20
In off policolly reinforcement learning, the replay ratio refers to the frequency of RL training updates relative to the num- ber of environment interactions. For example, SAC per- forms one update per environment interaction (RR=1), while MBPO performs twenty updates (RR=20), resulting in a much higher replay ratio. MBPO's high replay ratio, com- bined with its lack of performance improvement beyond the randomly initialized policy, suggests the presence of critic divergence a well-known challenge in off- off--poliyy RL (Thrun & Schwartz, 1993) which is often exacerbated by overly-replaying stale data during training. Recent studies in high replay ratio off-policy RL (Nauman et al. 2024
Key Insights:
MBPO's learned model struggles to make ac- curate predictions in DMC, even on the train- ing distribution.
A perfect I-step model improves MBPO in DMC but still fails to match or outperform SAC, as was previously achieved in Gym.
Hussing et al. 2024) have highlighted two forms of critic divergence: overestimation and underestimation.
Overestimation, common in offline and online off-policy methods with high replay ratios, arises when a learned Q- function is queried for state-action pairs that are out-of- distribution for the training data (Thrun & Schwartz, 1993). Online methods typically mitigate this by using real experi- ences to explore high-reward regions, naturally correcting overestimation through interaction. Conversely, underesti- mation occurs in unseen regions where the Q-function is overly pessimistic. Unlike overestimation, underestimation is harder to address, as it requires purely exploratory actions to visit these regions, which policies inherently avoid due to their focus on maximizing expected return.
Predictive model errors and non-stationary critic targets induce critic divergence.
Layer Normalization mitigates critic diver- gence, but does not allow MBPO to succeed.
5. If It's Not Just The Model Could It Be The Learning Dynamics?
Section 4 suggests that errors in the predictive model con- tribute to MBPO's poor performance DMC, but cannot fully explain the issues identified in Section This obser-
To investigate critic divergence, we measured the average
*These results are Appendix in Figure 18
7

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
vation leads investigate how learning dynamics and network plasticity in MBPO might impact the predictive model or the base off-policy method's s abilities.
quadrupped walk
hopper-hop
humanold-wok
750
150
Toscon Betum
150
500
500
150
250
250
Plasticity loss - where prolonged training diminishes a network's 's capacity to learn new tasks - has been studied in off-policy RL (Lyle et al. 2022) and model-based RL
63
82
63
64
85
Endiiommnt Steas (xl0
End cemontt Steps (K209)
Endiroemant Steps (x10%)
hepper-saadd
hurannidd
Edaust saturs
750
150
Boasets Saturs
750
QQia et al. 2023), ). with successful applications in DMC (Nikishin et al. 2022 D D'Oro et al. (023). High replay ratios in off-policy RL are known to exacerbate plasticity issues, as both the predictive model and the learned Q-function face continually changing data distributions and using a high replay ratio forces them to learn to solve a sequence of similar, but distinct, tasks (Dabney et al. 2021).
500
100
100
:50
Ervironment Steas (x20f)
Endiromment Steps (x10f)
Endirrmmnt Steas (x10f)
Figure 8. Comparison of MBPO and SAC with and without peri- odic resets of the actor, critic, target critic, and automatically tuned temperature. While periodic resets help MBPO improve policy performance beyond the random initial policy in some environ- ments, SAC with resets consistently outperforms MBPO across all tasks, Full results are in Appendix G and Figure 15
Qiao et al. 2023) demonstrate that periodic reinitialization of the learned model parameters can mitigate the loss of plasticity in model-based RL and enhance model accuracy. Therefore, to determine if loss of plasticity is contributing to MBPO's failures, we completely reset all parameters of the predictive model every 2 x 10 environment steps, which is a frequency aligned with recommendations from previous work (Qiao et al. 2023 D'Oro et al. 2023) for replay ratio of 128. We also experimented with both more conservative intervals (e.g., every 1.28 x 10s environment steps) and intermediate values, but found that results were indistinguishable from those in Figure + regardless of reset interval. These findings suggest that model plasticity is not the primary cause of low model accuracy in MBPO 5
that Dyna-style enhancements can degrade the performance of their off off-policy base algorithm in DMC.
Key Insights:
Loss of plasticity in the predictive model does not contribute to MBPO's failures.
Loss of plasticity in the off-policy base (actor, critic, target critic, and temperature) only partially responsible for MBPO's limitations.
Next, we investigated whether plasticity issues were affect- ing the networks that comprise the off- off-policy base of MBPO the actor, critic, target critic, and the automatically tuned temperature in SAC. Unlike the investigation of the model losses due to plasticity, we reset all of the aforementioned networks, except the predictive model, every 2 x 10 in- teractions. In Figure g. we see that in both quadruped tasks performing periodic resets allows MBPO to not only improve its policy from the initial random policy, but out- see that
MBPO's Dyna-style enhancements exacerbate plasticity loss in the off-policy base, leading to worse performance compared to SAC.
SAC continues to significantly outperform MBPO in DMC when periodic resets are ap- plied to both algorithms.
perform SAC by a large margin. Additionally in the humanoid d wa wa task performing periodic resets allows MBPO to improve from the initial random policy, but it still underperforms SAC. In all other tasks MBPO with periodic resets still cannot improve its policy beyond the initial random policy. We can conclude that plasticity issues related these base off-policy components do, at least partially, account for some failings of MBPO.
6. Conclusion
In this work, we have illustrated surprising and consistent performance gap of Dyna-style model-based reinforcement learning algorithms when applied across diverse benchmark environments. While these algorithms can excel in Ope- nAI Gym benchmark tasks, their performance degrades significantly in the DeepMind Control Suite (DMC), even though tasks in these benchmarks share broadly similar characteristics and the same underlying physies engine. Our extensive experiments, reinforced by modern techniques designed to bolster both model-free and model- meth- ods, reveal that synthetic rollouts - central to Dyna's gains in OpenAl Gym - can arrest policy improvement rather than enhance it when deployed across more diverse envi- ronments. Concretely, for all 15 DMC tasks we examined, adding gener-eg samples consistently undermined
Nevertheless, alleviating plasticity loss is not universal solution. Of even more significance, the periodic resets ap- plied in MBPO are equally applicable to SAC. As shown in Figure g SAC with periodic resets significantly outperforms MBPO with the same resets on most tasks. This highlights
'Training curves for model resets every 2 x 10 environment steps may be found in Figure 20 Appendix G
8

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
both sample efficiency and wall-clock performance relative simpler, model-free off-policy algorithms with identical hyperparameters. In short, there is no free lunch.
These findings suggest that our community's 's reliance on a limited set of benchmarks may have contributed to an inflated view of the generality and robustness of Dyna-style approaches. Just as importantly our results highlight sys- temic issues within the research community and publica- tion ecosystem: there are few incentives monetary otherwise - for conferences, journals, and researchers to invest in the critical re-examination of widely cited methods. Model-Based Policy Optimization (MBPO), in particular, an instructive example. Although is frequently repro- duced, highly cited, and even featured in a reproducibility study accepted a major conference (Liu et al. 2020), our analysis shows that its touted advantages fail to carry to an equally conventional testbed (DMC). This underscores the need for cultural shift towards more rigorous, critical, and comprehensive evaluation practices, including active self- self- policing of influential algorithms and promoting rigor- ous follow-up studies that challenge established claims.
Looking ahead, have not solved the pervasive issues underlying the subclass of Dyna-style methods investigated in this work, but we have made progress towards facilitating such work by identifying that there is problem, investi- gating potential causes and solutions, and providing code that significantly speeds up evaluation procedures. We hope that this work provides a building block from which our community can dissect, diagnose, and ultimately address the shortcomings that currently limit Dyna's utility.
Recognizing that our work is not exhaustive, we also have included frequently asked questions section in Appendix to address common questions, provide context, and foster collaboration. We intend to update it continually, incor- porating insights and feedback from the broader research community as we refine our methods.

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
Acknowledgements
the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34:29304-29320, 2021.
This work was supported by the National Science Founda- tion under Grant No. 2409535. We would like thank Harshit Sikchi, Eugene Vinitsky, Siddhant Agarwal, and Max Rudolph for insightful discussions and their feedback on this paper.
Ahmed, N., Wahed, M., and Thompson, N. C. The growing influence of industry in ai research. Sci- ence, 379(6635):884- ade2420. URL https: http:::::/wwww.a..
2023, doi: 10. 126sscience org/
doi ab/ 10 11 http://o.112//ccceeeeeeoo ade222
Impact Statement
Amos, B., Stanton, S., Yarats, D., and Wilson, A. G. On stochastic value gradient for continuous reinforcement learning. In Proceedings of the 3rd Con- ference on Learning for Dynamics and Control, volume 144 of Proceedings of Machine Learning Research, pages 6-20. PMLR, 2021.
This paper underscores the importance of robust and di- verse benchmarking in the Machine Learning community ensure that reported progress translates to meaningful advances. By pointing to the limitations of Dyna-style meth- ods in broader contexts, we encourage the community to critically re-evaluate widely adopted approaches and the systems that reward them. Addressing systemic issues, such as the incentives for replicating influential results without deeper serutiny, is essential fostering meaningful progress in the field.
the
Atkeson, C. and Schaal, S. Learning tasks from a single demonstration. In Proceedings of International Confer- ence Robotics and Automation, volume 2, pp. 1706- 1712 vol.2, 1997. doi: 10.110//0BB0...99...1110
Furthermore, our work highlights the barriers to accessibility in reproducibility and evaluation. Many influential methods, including those analyzed in this study, remain difficult to reproduce due to the significant computational resources required to replicate results or run new experiments. Such challenges prevent broader participation in the refinement and critical evaluation of algorithms,, especially from re- searchers with limited access to cutting-edge hardware. To help address these concerns for Dyna-style methods, we pro- vide efficient, open-source code that significantly reduces evaluation costs and facilitates more inclusive collaboration within the research community.
Ba, L., Kiros, J. R., and Hinton, G. E. Layer normal- ization, 2016a. URL httpss://aras org/ abs/
(607.66450)
Ball, J., Smith, L.,, Kostrikov, I,, and Levine, Effi- cient online reinforcement learning with offline data. In Proceedings of the International Conference on Machine Learning, pages 1577-1594. PMLR, 2023.
Bender, E. M., Gebru, T., SiMiilln-Majon A., and Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference Fairness, Account- ability, and Transparency, FAccT '21, pp. 610-623, New York, NY, USA, 2021. Association for Comput- ing Machinery. ISBN 9781450383097. doi: 10.1145/ 3442188.3445922. URL https://doi.orq// I145/3442188 3445922
From an ethical standpoint, this work does not present immediate societal risks, but rather focuses on improving the methodological rigor within reinforcement learning re- search. By advocating for more comprehensive evaluation practices, reducing resource requirements, and fostering transparency, we hope to encourage accountability and ac- cessibility research practices.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. JAX: composable transformations of Python+NumPy programs, 2018 URI. http::/githul: htw.:hhhb.ccmm/ceemome ml/jax
We believe this work holds value for the community by drawing attention challenges in generalization, overfitting to benchmarks, accessibility, and the broader need for cul- ture that prioritizes the evaluation of actual progress. While our study is not exhaustive, we provide a framework for fur- ther investigation and invite collaboration to refine and build upon our findings. By enabling the community to address these shortcomings, we aim to facilitate research that leads to more robust, impactful, and accessible al algorithms.
Brockman, G., Cheung. V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Ope- nai gym, 2016. URL
1606. 01540
References
Buckman, J., Hafner, D., Tucker, G., Brevdo, E., and Lee, H. Sample-efficient reinforcement learning with stochastic ensemble value expansion. Advances in Neural Informa- tion Processing Systems, 31, 2018.
Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. Deep reinforcement learning at
10

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforvement Learning
Dabney, W., Barreto, A., Rowland, M., Dadashi, R., Quan, J., Bellemare, M. G., and Silver, D. The value- improvement path: Towards better representations for reinforcement learning, In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, number 8, pages 7160-7168, 2021.
Hafner, D., Pasukonis, J., Ba, J., et al. Mastering diverse control tasks through world models. Nature, 640:647- 653, 2025. doi: 10. 00.1088/441556-025-0874444
Haibe- B., Adam, G. A., Hosny, A., Khodakarami, F., Waldron, L.,, Wang. B., McIntosh, C., Goldenberg, A., Kundaje, A., Greene, S., et al. Transparency and re- producibility in artificial intelligence. Nature, 586(7829): E14-E16, 2020. doi: 10. (0.1038/41138-002-2277
Deisenroth, M. P. and Rasmussen, C. E. Pilco: model- based and data-efficient approach to policy search. In Pro- ceedings of the 28th International Conference on Inter- national Conference on Machine Learning. ICML'11, pp. 465-472, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306195.
Hansen, N. A., Su, H.,, and Wang, X. Temporal difference learning for model predictive control. In Chaudhuri, K., Jegelka, S., Song, L.., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Confer- ence Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 8387-8406. PMLR, 17-23 Jul 2022. press v1 62 http///annenna...2 html
Dong, K., Luo, Y., Wang, Y., Liu, Y., Qu, C., Zhang, Q. Cheng, E., Sun, Z., and Song, B. Dyna- style model-based reinforcement learning with model-free policy optimization. Knowledge-Bon Based Systems, 287:111428, 2024. ISSN 0950-7051. doi:
.mlr
Holland, G. Z., Talvitie, E. J., and Bowling, M. The effect of planning shape on dyna-style planning in high-dimensional state spaces, 2019. URL https
URL
com/ science http:///aaaaaee111//0/0001222220
//arxiv.org/abs//1 01825
D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-L.., Belle- mare, M. G., and Courville, A. Sample-efficient rein- forcement learning by breaking the replay ratio barrier. In Proceedings of the Eleventh International Conference on Learning Representations, 2023.
Hussing, M., Voelcker, C. A., Gilitschenski, I., Farahmand, A.-m., and Eaton, E. Dissecting deep RL with high update ratios: Combatting value divergence. In Proceedings of the Reinforcement Learning Conference, 2024.
Janner, M. Mbpo: Model-based policy optimization, 2019. URL https: http::://qihhuut com/ jammernnne mbpo Accessed: 2024-12-11.
Fujimoto, S., Hoof, H., and Meger, D. Addressing func- tion approximation error in actor-critic methods. In Pro- ceedings of the International Conference on Machine Learning, pages 1587-1596. PMIR, 2018
Janner, M., Fu, J., Zhang, M., and Levine, S. When to trust your model: Model-based policy optimization. In Proceedings of the 33rd International Conference on Neu- ral Information Processing Systems, pp. 1122-1133, Red Hook, NY, USA, 2019. Curran Associates Inc.
Ghugare, R., Bharadhwaj. H., Eysenbach, B., Levine, S., and Salakhutdinov, R. Simplifying model-based rl: leam- ing representations, latent-space models, and policies with one objective. In International Conference on Learn- ing Representations, 2022.
Jordan, S. M., White, A., da Silva, B. C., White, M., and
Thomas, P. S. Position: Benchmarking is limited in reinforcement learning research, 2024. URL https
Gu, S., Lillicrap, T., Sutskever, I., and Levine, S. Continu- ous deep q-learning with model-based acceleration. In Proceedings of the International Conference on Machine Learning, pages 2829-2838. PMLR, 2016.
7/arxiv. //axxiv.org/ab//20000 wwt.alssrrggab////o//ao/
/arzg//bb/2406.. 16241
Kalweit, G. and Boedecker, J. Uncertainty- imagi- nation for continuous deep reinforcement learning. In Levine, S., Vanhoucke, V., and Goldberg, K. (eds.), Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learn- ing Research, pp. 195-206. PMLR, 13-15 Nov 2017. URL httlltttt//proeeeetc ka wewwit17al html
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., and Levine, S. Soft actor-critic algorithms and appli- cations, 2019a. URL 1812.05905
/
mlr. wwt.mmmr..r....... prwss/v78/
Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. Dream to control: learning behaviors by latent imagination. In Proceedings of the International Conference on Learning Representations, 2020.
Lai, H., Shen, J., Zhang, W., and Yu, Y. Bidirectional basel-based policy optimization. In Proceedings of the International Conference on Machine Learning, pages 5618-5627. PMLR, 2020

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforvement Learning
Lai, H., Shen, J., Zhang, W., Huang, Y., Zhang, X., Tang, R., Yu, Y., and Li, Z. On effective scheduling ofmModel- based reinforcement Learning. Advances in Neural Infor- mation Processing Systems, 34:3694-3700 2021.
Qiao, Z., Lyu, J., and Li, X. Mind the model, not the agent: the primacy bias in model-based RI.. In Proceedings of the European Conference Artificial Intelligence, 2023.
Rajeswaran, A., Ghotra, S., Levine, S., and Ravindran, B. EPOpt: learning robust neural network policies using model ensembles. In Proceedings of the International Conference on Learning Representations, 2017.
Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, A. Reinforcement learning with augmented data. Advances in Neural Information Processing Systems, 33: 19884-19895, 2020.
Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. figh-diiensiioal continuous control using generalized advantage estimation, 2015. URL https://arxiv:
Li, Y. Dong, Z., Luo, E., Wu, Y. Wu, S., and Han, S. When to trust your data: Enhancing dyna-style model- based reinforcement learning with data filter, 2024. URL https: arxiv. org/ abs/ 2410 12160
org www.aa/...... 1506.024388 02438
Sikehi, H., Zhou, W., and Held, D. Learning off-policy with online planning. In Proceedings of the Conference on Robot Learning, pages 1622-1633, PMLR, 2022.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning, 2019. URL https: /tas:///rxiii http://wwwwworrg..oa/ooo haggg/bbs.. www.bb//100 02971
Laskin, M., Srinivas, A., and Abbeel, P. CURL: contrastive unsupervised representations for reinforcement learning. Proceedings of the International Conference on Ma- chine Learning, pages (639-5650. PMLR, 2020.
Liu, Y., Xu, J., and Pan, Y. [Re] When to trust your model: modell-asee policy optimization. ReScience C, 6(2), 2020. URL httpt::t////tttt /ttp::///peerviiieeeeeee
Sutton, R. S. Dyna, an integrated architecture for learn- ing, planning, and reacting. SIGART Bull., 2(4): 160-163, July 1991. ISSN 0163-5719. doi: 10. 1 1145/122344 122377. URL http::://oi.org//////
net. forum? id-rkezvI9f6r Accepted NeurIPS 2019 Repro- ducibility Challenge.
1145122344.122377
Luo, F.-M., Xu, T., Lai, H., Chen, X.-H., Zhang, W., and Yu, Y. A survey model-based reinforcement learning. Science China Information Sciences, 67(2):121101, 2024.
Talvitie, E. Model regularization for stable sample rollouts. In Proceedings of the Thirtieth Conference on Uncer- tainty in Artificial Intelligence, UAI' 14, pp. 780-789, Arlington, Virginia, USA, 2014. AUAI Press. ISBN 9780974903910.
Lyle, C., Rowland, M., and Dabney, W. Understanding and preventing capacity loss in reinforcement learning. In Proceedings of the International Conference on Learning Representations, 2022.
Tassa, Y., Tunyasuvunakool, S., Muldal, A., Doron, Y., Trochim, P., Liu, S., Bohez, S., Merel, J., Erez, T., Lillicrap, T., and Heess, N. dm control: Software and Tasks for Continuous Control. Software Impacts, 6: 100022 November 2020. ISSN 26659638. doi:
Nauman, M., Bortkiewicz, M.. Mitoš, P., Trzcinski, T., Ostaszewski, M., and Cygan, M. Overestima- tion, overfitting, and plasticity in actor-critic: the Bitter lesson of Reinforeement learning. In Pro- ceedings of the 41st International Conference Machine Learning, pp. 37342-37366. PMLR, July 2024. URL URL/tttss www..mrrrr..s...... v235/nauman24,,
httL:http//aaate 10. 10.1016/jsimmpa.200..00000 arXiv:2006.1298: [cs].
org abs.2006.1292.
Thrun, S. and Schwartz, A. Issues in using function ap- proximation for reinforcement learning. In Proceedings of 4th Connectionist Models Summer School. Erlbaum Associates, June 1993.
ISSN: 2640-3498.
Nikishin, E., Schwarzer, M., D' Oro,,, P., Bacon, P.-.., and Courville, A. The primacy bias in deep reinforcement learning. In Proceedings of the International Conference on Machine Learning, pages 16828-16333 16847. PMLR, 2022.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model- control. 2012 IEEE/RSJ In- ternational Conference on Intelligent Robots and Sys- lems, pp. 5026-5033, 2012. URL https: /api semanticscholat org/ wttp://ooooooooooso005505.
Paszke, A., Gross, S. Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, F., Vetoto, Z., Rai- son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, perf-peefor deep learning library, 2019. URL http://ww..ooo/b////////////////
Voelcker, C. A., Hussing, M., and Faton, E. Can we hop in general? A discussion of benchmark selection and design using the Hopper environment. In Finding the Frame: An RLC Workshop for Examining Conceptual Frameworks, 2024.
12

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
Wang, C., Chen, Y., and Murphy, K. Model-based policy optimization under approximate Bayesian inference. In Proceedings of the 27th International Conference on Ar- tificial Intelligence and Statistics (AISTATS), volume 238. PMLR, 2024.
Wang. T., Bao, X., Clavera, I., Hoang, J., Wen, Y., Langlois, E., Zhang, S., Zhang, G., Abbeel, P., and Ba, J. Bench- marking model-based reinforcement learning, 2019. URL https://arxiv.oot http://wwxwv..oaao////////o/a///
abs 907. 02057
Wang, X.,
Wongkamjan, W., Jia, R., and Huang, Live in the moment: learning dynamics model adapted to evoly- ing policy. In Proceedings of the International Confer- ence on Machine Learning, pages 36470-36493. PMLR, 2023.
Xu, J., Makoviychuk, V., Narang, Y., Ramos, F., Matusik, W., Garg, A., and Macklin, M. Accelerated policy learn ing with parallel differentiable simulation. In Proceedings of the International Conference on Learning Representa- tions, 2022.
Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Master- ing visual continuous control: improved data-angmented reinforcement learning. Proceedings of the Tenth Inter- national Conference on Learning Representations, 2022.
Zheng, R., Wang, X., Xu, H., and Huang, F. Is model en- semble necessary? ftodel-based RL via single model with Lipschitz regularized value function. In Proceedings of the International Conference on Learning Representa- tions, 2023.
13

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforvement Learning
A. Frequently Asked Questions
Are you saying that no Dyna-style model-based RL algorithm can succeed in DMC?
No. As mentioned at the end of Section 2.T] DreamerV3 (Hainer al. (025) achieves ofth-of-the-art performance in DMC environments. This evidence suggests that there are two subclasses of DMBRL algorithms: those that succeed in DMC and those that do not. This paper highlights two algorithms in the latter subclass, with the aim of inspiring future work to identify algorithmic improvements that enable all DMBRL algorithms to perform successfully not only in OpenAl Gym or DMC, but across a broader range of tasks.
Isn't the poor performance of the models evidence that model learning, not Dyna, is the real problem? No. Even if we ignore the results of Section 4.2] the difficulty of learning an accurate model is central to the viability of Dyna-style methods (Sutton 1991). While the learning of the model itself may lie outside the scope of methods based on the Dyna architecture, these algorithms fundamentally rely the model to generate useful synthetic experience. If the model struggles, so does the entire method. The success of Dyna-style approaches, especially MBPO and ALM, hinges on their ability to make progress even with imperfect models, making model learning an inseparable and integral part of their evaluation.
Notably, the model learning architectures proposed alongside MBPO and ALM have demonstrated strong performance in OpenAI Gym environments (ef. Section 3.T), D. yet they struggle in DeepMind Control (DMC) (cf. Section 3.2 and Section 3.3). Given the similarities between these environments (cf. Appendix C this discrepancy highlights the need to further investigate where and why these models falter in DMC, how to improve their robustness, and how to improve the underlying Dyna algorithms' ability tolerate inevitable model errors.
Did you consider overfitting of the model as a potential source of error? Yes. We did not find any significant overfitting of the model in either the OpenAI Gym environments or in DMC. Further, we performed intermittent model resets in Section as a means to mitigate this issue overfitting was occurring throughout resets do not resolve the performance gap.
training QQao al. 2023). As discussed in Section
Did you consider running- <insert experiment here>?
This paper does not (and cannot) exhaustively consider all potential of sources of the performance gap we observed. If you can think of additional experiments to try, we would love to hear from you.
B. Timing Claim for MBPO Compared to Our Implementation
Based on previous work (Ghugare et al. 2022 MBPO's average time per environment step was approximately 0.6 seconds across Gym environments. Even if we optimistically reduce this to 0.5 seconds per step (to account for our usage of slightly more modern computational hardware) the time required to run 6 seeds across the 6 DMC environments shown in Figure for 500k environment steps is:
0.5x500000xxx6 = 104 days.
(3600 24)
In contrast, our code was directly timed during experimentation. The total runtime for six seeds across six environments and 500k steps came out to approximately 4 days on an NVIDIA RTX A5000 GPU.
C. Extended Discussion on Benchmark Modeling Differences
As discussed in Section 2.2 OpenAl Gym and DM Control (DMC) use different reward structures, physical parameters, and termination conditions. These differences certainly could contribute Dyna-style algorithms failing in DMC but not Gym. However, both are equally conventional and widely accepted testbeds for reinforcement learning. This is evidenced by the
fact that DMC has been successfully used for demonstrating RL with not only proprioceptive observations [D'Oro et al.
2023; Nikishin et al., 2022), but for visual observations too (Laskin et al.) 2020 Hafner et al. 2020 Laskin et al. 2020
Yarats et al. 2022).
is important to emphasize that the goal of this work is not to pinpoint specific environmental factors that might cause DMBRL fail. Instead, we aim to demonstrate that while DMBRL performs well on one benchmark, it largely fails on another that, at first glance, appears quite similar. This observation raises concerns about the robustness of DMBRL algorithms. Rather than delving into the algorithmic causes of this issue we have for the majority of this paper, this section highlights and discusses key differences between the two benchmarks, providing a foundation for future work to
14

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforvement Learning
investigate environment-speccffc failures of DMBRL algorithms.
Gym environments typically feature unnormalized rewards and explicit termination conditions, while DMC provides normalized rewards and lacks termination conditions. Despite these differences, our discussion below shows that reward and termination conditions alone do not fully explain performance disparities. Given the substantial similarity between many tasks across the two frameworks, we can compare their MDP specifications side-by-side, focusing on termination conditions first.
In a unique experiment in the hopper environment, previous work (Voelcker et al. 2024 removed only termination conditions from Gym and ran MBPO, ALM, and SAC. While removing termination conditions caused MBPO fail, ALM was still able to train a successful policy. However, DMC, which also has no termination conditions, ALM fails. This evidence shows that at least for environments that are broadly similar to hopper termination alone does not explain DMBRL failing in DMC.
show in Section 2.2 soon as MBPO ALM are deployed in
Conversely, MBPO performed relatively well on walker tasks in both Gym and DMC environments, as shown in Figures and despite DMC lacking termination conditions. Interestingly, ALM exhibited a different pattern, failing on wa ker in DMC but succeeding in Gym. Additionally, switching frameworks reversed the relative performance of SAC and MBPO for walker. In the humanoid d environment, both MBPO and ALM succeeded in Gym but failed entirely in DMC. These results indicate that termination conditions alone do not account for DMBRL method failures across multiple environments that are broadly similar, as performance varies across frameworks and tasks.
The physical differences in robot models between the two frameworks are also notable, particularly in the configuration of joints, actuators, and contact dynamics. For instance, at a high level. OpenAI Gym's 's humanoid is built to be more stable and controlled, using higher stiffness and damping to keep its movements steady and balanced. On the other hand, DMC's humanoid d is designed for smoother and more natural motion, with lower stiffness and damping in peripheral joints (i.e. the arms, legs), but still adding extra stiffness where needed for stability (i.e. in the hips and torso). The Gym humanoid also has armature (rotational inertia added by actuators) values that vary by joint, which helps it move quickly and adjust better to different tasks. DMC's humanoid keeps the same armature value everywhere, making it simpler but less flexible for specific challenges.
Similar trends are observed in the hopper environment. Gym's model emphasizes precision and stability with higher damping and armature values, while DMC's design favors fluidity with lower damping and moderate, uniform armature values. However, this trend does not extend to the wa 1 ker model, where both frameworks use identical damping and armature values.
These differences may very well contribute to the performance gap identified in Section 3 However, the analysis of Section 4 wherein we find that MBPO's learned world model tends to exhibit substantially higher error in DMC than in Gym, but that even with access to a perfect model MBPO cannot reliably outperform SAC - indicates that this is only part of the story.
There are numerous other differences across the various environments in Gym and DMC that we cannot enumerate here, but one final distinction lies in the numerical integration schemes used by the two frameworks. Across the environments we have discussed in this section, Gym employs smaller timestep and the 4th-order Runge- Kutta method for numerical integration, while DMC uses MuJoCo's 's default semi- ammiiimpiicit Euler method (Tassa et al.) 2020). These differences result in Gym having a shorter effective horizon for tasks that take identical numbers of environment steps and higher accuracy in simulating dynamics. Such variations could induce significant discrepancies even in environments that are otherwise identical in their physical specifications.
In summary, while termination conditions, physical parameters, and numerical integration schemes differ between OpenAl Gym and DM Control, no single factor fully explains the performance variations observed in DMBRL methods. Instead, the interplay between these elements, and likely ones we have not discussed, contributes to the disparities observed when deploying DMBRL between different benchmarks.
If you have other ideas about how the differences between the Gym and DMC benchmarks may be contributing to the performance gap we identify in this work, we would love to hear about them. Please reach out to the corresponding author.
15

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforvement Learning
D. Implementation Details
ALM
The code to run ALM was used with only minimal modifications introduced to allow interfacing with DMC environments. No hyperparameters were changed in either our Gym experiments or our DMC experiments from what was provided the original ALM repository. When applying ALM without Dyna-style enhancements loss with the td loss that is provided in the experimental branch of the ALM repository and kept all hyperparameters fixed. The specific implementation details below are reproduced with minor modification from (Ghugare et al. 2022):
simply replaced the lambda svg
We implement ALM using DDPG (Liliccrap et al. 2019) as the base algorithm. Following prior SVG methods (Amos et al., 2021). we parameterize the encoder, model, policy, reward, classifier and Q-function as 2-layer neural networks, all with 512 hidden units except the model which has 1024 hidden units. The model and the encoder output multivariate Gaussian distribution over the latent-space with diagonal covariance. Like prior work (Hansen al.
2022; Yarats al., 2022), we apply layer normalization (Ba al., 20166) to the value function and rewards. Similar to prior work (Schulman et al. 2015 Hafner al. 2020),
reduce variance of the policy objective by computing exponentially weeonenttalll--wiihheeedd average of the hyperparameter. To train the policy, reward, classifier and Q-function we use the representation sampled from the target encoder. For exploration, we use added normal noise with a linear schedule for the standard deviation Yarats et al. 2022). All hyperparameters are listed Table 3
objective for rollouts of length to an integer K' chosen
For additional details see (Ghugare al. 2022
D.2. MBPO
The code for our MBPO implementation is available at https///ghhua.coomm/c////eeassas
E. Nominal Hyperparameters for SAC, MBPO, and ALM
Table 1. Hyperparameters used SAC and the SAC component MBPO.
Hyperparameters
Value
Discount (?)
0.99
Warmup steps
10000
Minibatch size
256
Optimizer
Adam
Learning rate (a)
0.0003
Optimizer Bi
0.9
Optimizer Ba
0.999
Optimizer
0.00015
Networks activation Number of hidden layers
ReLU
2
Hidden units per layer Initial temperature (on)
256
Replay buffer size Updates per step
105
20
Target network update period
Soft update rate (r)
0.995
16

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
Table Hyperparameters used for training and deployment of Dyna-style enhancements in MBPO.
Hyperparameters
Value
Ensemble retrain interval
250
Minibatch size
256
Optimizer
Adam
Ensemble learning rate
0,0003
Optimizer 31
0,9
Optimizer $2
0.999
Optimizer
0,00015
Networks activation
Swish
Synthetic ratio
0.95
Model rollouts per environment step
400
Number of ensemble layers
2
Hidden units per layer
200
Number of elite models
5
Number of models in ensemble
7
Model horizon
1
Table Hyperparameters used ALM with and without Dyna-style enhancements and reproduced from (Ghugare et al. 2022 Hyperparameters
Value
Discount (?)
0.99
Warmup steps
5000
Soft update rate (T)
0,005
Weighted target parameter (A)
0.95
Replay buffer
10 for humanoid
10 otherwise
Batch size
512
Learning rate
lc-4
Max grad norm
100.0
Latent dimension Coefficient classifier rewards
50
0.1
Exploration stddev. clip
0.3
Exploration stddev. schedule
linear(1.08 1.0, 0.1, 100000)
F. Hyperparameter Sweeps
We conducted large-scale hyperparameter sweep evaluate MBPO's sensitivity in hopper-stand. This sweep comprised 93 hyperparameter configurations, with each configuration evaluated across three seeds to capture variation due to initialization and training dynamics, resulting in total of 279 runs. The full range of values is listed in Table 4 All runs used synthetic-to-re tel data ratio of 0.95 and were trained online for 50k environment steps. No combination of hyperparameters in this sweep yielded consistent performance.
To test whether the findings central to our claims generalized across environments and longer training durations, we conducted an additional sweep using single-seed runs on both hopper-st stand and humanoiin woweroodddddddddddo For each environment, we evaluated 81 hyperparameter configurations for 200k environment steps (see Table These experiments targeted learning dynamics across higher number of environment interactions and helped ensure that the lack of improvement was not simply due to undertraining insufficient data. No combination of hyperparameters yielded consistent performance.
Since none of the 174 hyperparameter variations across 2 environments and 441 runs led to meaningful performance gains, we omit per-parameter plots for brevity. Instead, Figuregaand and Figure show unlabeled return curves for each configuration. Despite substantial hyperparameter sweeps, return performance remained poor, reinforcing our claim that model- based policy optimization fails to benefit from tuning in this setting.
17

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforvement Learning
Table 4. Hyperparameter sweep ranges for MBPO hopper-stand.
Hyperparameter
Values
ansemble.hidden
{50, 100, 200, 400]
ensemble.1z
[1.5e-4, 3e-4, 6e-4]
ensemble_inteevall
[125, 250, 500)
model.train.stees
[1250, 2500, 5000, 10000)
model_rolllotts.err..tttt
{100, 200, 400, 800, 1600)
num.elites
{1.3,5}
Table Hyperparameter sweep ranges used single-seed runs on both hopper-stand and humanof d-stand
Hyperparameter
Values
ensemble.hidden
{200, 400, 800} { [1.5e-4, 3e-4, 6e-4] { {125, 250, 500)
ensemble.1r
ensemble..interval model rollouts per. step
(200, 400, 800)
hopper-stand
Episode Return
750
500
250
0.00
0.01
0.02
0.03
0.04
0.05
Environment Steps (x10%)
Figure 9. Hyperparameter sweep for MBPO trained on the hopper-st and environment across 93 hyperparameter configurations (3 trials each, std), As none of these configurations yielded meaningful or consistent performance, omit per-paramete plots for brevity. Despite one seed of one configuration temporarily reaching return of 93, the policy immediately collapses to ncar-randon performance after further training. For sake comparison, after 50k environment steps a SAC policy this same an average return of 300.
hopper-stand
humanoid-siee
Episode Return
750
Episode Return
750
500
500
250
250
0.15
0.20
0.05
Environment Steps (x10*)
Environment Steps (x10*)
(a)
(b)
Figure Hyperparameter sweep for MBPO trained on the hopper-stand and bumanoid-ster environment across hyperpa- rameter configurations each (1 trials each).
18

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
G. Full Experimental Results
finger-turn hard
sendulum-swingup
humanoid-stee
Episode Return
Episode Retur
Episode Return
750
750
750
500
500
500
250
250
250
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x10%)
Environment Steps
(x109)
Environment Steps (x10%)
quadruped-run
humanoid-walk
cheetah-run
Episode Return
Episode Return
Episode Return
750
750
500
500
500
250
250
250
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x10*)
Environment Steps (x10%)
Environment Steps (x10%)
swimmer-swimmer6
hopper-stand
reacher-hard
Episode Return
Episode Return
Episode Return
750
750
500
500
250
250
250
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x109)
Environment Steps (x109)
Environment Steps (x109)
humanoid-run
walker-run
quadruped-walk
Episode Return
Episode Return
Episode Return
750
750
500
500
500
250
250
250
0.0
0.1
0.2
0.0
0.1
0.5
0.2
0.3
0.4
0.5
Environment Steps (x109)
Environment Steps (x109)
Environment Steps
(x109)
hopper-hop
fish-swim
acrobot-swingup
Episode Return
Episode Return
Episode Return
750
750
500
500
250
250
250
0.0
0.1
0.2
0.0
0.1
0.5
0.4
0.5
Environment Steps (x10%)
Environment Steps (x10%)
Environment Steps (x109)
MBPO
SAC
Figure 11. Full performance of MBPO and SAC for 15 challenging DMC benchmark tasks referenced in Section 3.2
19

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
OpenAl Gym Environments
175
Ant-v2
150
HalfCheetah-v3
Model Percent Error
Hopper-v3
Humanoid-v3
100
InvertedPendulum-22
75
Walker2d-v3
50
25
0-
0.00
0.05
0.10
0.15
0.20
0.25
0.30
Environment Steps (x10%)
Figure Illustration the change percent model error during training OpenAl Gym, as measured on the training distribution. (6 trials, +: std).
Deepmind Control Environments
250
hopper-hop
hopper-stand
Model Percen: Enor
200
humanold-stard
humanoid-wall
quadruped run
quadruped-wa walk
1000
50
0-
0.08
0.10
Environment Steps (x10%)
Figure 13. Illustration of the change in percent model error during RL training for DMC, as measured on the training distribution. (6 trials, std).
hopper-stand
humanoid-stand
Final Returns of MBPO
Final Returns of MBPO
500
800
400
600
300
400
200
200
100
0.0
0.4
0.6
0.0
0.2
0.8
1.0
Synthetic Ratio
Synthetic Ratio
a
bb)
Figure Illustration of the relationship between final episodic return and synthetic-to-rea ratio used for MBPO training for (a) hopper-ste standstnnnd for 6 seeds and 100k steps each, and (b) humanoid-stand for 3 seeds and 400k steps each. These results show that more synthetic data in training batch leads worse final policy performance MBPO.
20

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
finger-turn hard
pendulum-swingup
humanoid-stand
Episode Return
Episode Retum
Episode Return
750
750
750
500
500
500
250
250
250
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x10%)
Environment Steps (x 10%)
Environment Steps (x109)
quadruped-run
humanoid-walk
cheetah-run
Episode Return
Episode Return
Episode Return
750
750
750
500
500
500
Why W
250
250
250
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x10%)
Environment Steps (x10%)
Environment Steps (x109)
swimmer-swimmer6
hopper-stand
reacher-hard
Episode Return
Episode Return
Episode Return
750
750
750
500
500
MW
250
250
250
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x109)
Environment Steps (x10%)
Environment Steps
(x109)
humanoid-run
walker-run
quadruped-walk
Episode Return
Episode Return
Episode Return
750
750
750
500
500
500
250
250
250
0.0
0.1
0.2
0.3
0.4
0.5
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x10%)
Environment Steps (x10%
Environment Steps (x10%)
hopper-hop
fish-swim
acrobot-swingup
Episode Return
Episode Return
Episode Return
750
750
750
500
500
500
250
250
250
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x109)
Environment Steps (x10%)
Environment Steps (x10s)
MBPO with Resets
MBPO
SAC
Figure Comparing full performance MBPO, SAC, MBPO with actor critic resets challenging DMC benchmark tasks referenced in 5
21

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
swimmer-swimmer6
finger-turn hard
fish-swim
Mean Q-Values
100
Mean Q-Values
Mean Q-Values
2000
75
20
50
25
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
Environment Steps (x10%)
Environment Steps (x109)
Environment Steps (x10%)
pendulum-swingup
acrobot-swingup
humanoid-star
Mean Q-Values
Mean Q-Values
10
Mean Q-Values
200000
50-
5
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
Environment Steps (x10%)
Environment Steps (x10%)
Environment Steps (x10%)
quadruped-run
quadruped-waz
1e10
1e10
cheetah-run
Mean Q-Values
5
Mean O-Values
Mean Q-Values
5000
1.0
2500
0
0.0
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
Environment Steps (x10%)
Environment Steps (x10*)
Environment Steps (x10*)
humanoid-walk
reacher-hard
le6
hopper-hop
Mean Q-Values
Mean Q-Values
200
Mean O-Values
5000
100
-2
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
Environment Steps (x10%)
Environment Steps
(x10*)
Environment Steps (x10*)
walker-run
1e6
hopper-stand
humanoid-run
Mean Q-Values
Mean Q-Values
Mean Q-Values
200000
100000
-2
0.0
0.2
0.4
0.0
0.2
Environment Steps (x10%)
Environment Steps (x10*)
Environment Steps (x10%)
MBPO
SAC
Figure 16. Mean Q-values for 15 challenging DMC benchmark tasks as referenced in Section 4.3
22

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
reacher-hard
quadruped-walk
hopper-hop
Episode Return
Episode Return
Episode Return
750
750
750
500
500
500
M
250
M
250
250
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x109)
Environment Steps (x10%)
Environment Steps (x109)
humanoid-walk
hopper-stand
fish-swim
Episode Return
Episode Return
Episode Return
750
750
750
500
500
500
250
250
250
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x10%)
Environment Steps (x10%)
Environment Steps (x106)
walker-run
humanoid-stand
humanoid-run
Episode Return
Episode Return
Episode Return
750
750
750
500
500
500
250
250
250
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x10%)
Environment Steps (x109)
Environment Steps (x10%)
cheetah-run
finger-turn hard
quadruped-run
Episode Return
Episode Return
Episode Return
750
750
750
500
500
250
250
250
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x10%)
Environment Steps
(x10%)
Environment Steps
(x10%)
acrobot-swingup
swimmer-swimmer6
pendulum-swingup
Episode Return
Episode Return
Episode Return
750
750
750
500
500
500
250
250
250
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x10%)
Environment Steps
(x10%)
Environment Steps (x109)
ALM
ALM (no Dyna)
MBPO
SAC
Figure 17. Comparing performance of MBPO, SAC, ALM, and ALM without Dyna for 15 challenging DMC benchmark tasks.
23

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforvement Learning
cheetah-run
humanoid-stand
humanoid-walk
10a
10a
10a
Mean Q-Values
Mean Q-Values
Mean Q-Values
10
10*
10*
100
100
100
0
0
100
100
100
104
104
10
-108
-10
100
0.0
0.2
0.4
0.0
0.4
0.0
0.2
0.4
Environment Steps (x10%)
Environment Steps (x10f)
Environment Steps (x10%)
walker-run
hopper-hop
hopper-stand
10a
10a
10a
Mean O-Values
Mean Q-Values
Mean Q-Values
104
104
104
100
100
100
-10
-10
-10
-104
10
10
-108
-108
-00
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.4 Environment Steps (x10%)
0.2
Environment Steps (x10%)
Environment Steps (x10%)
MBPO
MBPO (perfect model)
SAC
Figure 18. Mean Q-values across six seeds each for challenging DMC benchmark tasks. Comparisons are of MBPO and SAC to MBPO with a perfect predictive model as discussed Section 4.3 Shaded regions correspond to minimum and maximum Q values across trials and solid lines are the mean.
quadruped-run
quadruped-walk
hopper-stand
Mean O-Values
10a
Mean Q-Values
10a
Mean Q-Values
10a
10a
104
10a
100
100
100
0
-10
100
100
-00
10
-10
-108
-10
-10
0.0
0.2
0.4
0.4
0.0
0.2
0.4
Environment Steps (x10%)
Environment Steps (x10*)
Environment Steps (x10s)
humanoid-stan
humanoid-wale
hopper-hop
10a
10a
10
Mean Q-Values
Mean Q-Values
Mean Q-Values
10a
104
10a
100
100
100
-10
-00
-10
-
-
-104
-108
-10
-10
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.4 Environment Steps (x109)
0.2
Environment Steps (x10%)
Environment Steps (x10*)
MBPO
MBPO (w/ Layer norm)
SAC
Figure 19. Mcan Q-values for 6 challenging DMC benchmark tasks across six seeds each. Comparisons are of MBPO and SAC to MBPO with layer norm applied discussed Section 4.3 Shaded regions correspond to minimum and maximum values across trials and solid lines are the mean.
24

---

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
numanoid-wale
quadruped
hopper-hop
Episode Return
Episode Return
Episode Return
750
750
750
500
500
500
250
250
250
0.0
0.1
0.2
0.3
0.4
0.5
0.1
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x10%)
Environment Steps (x10%)
Environment Steps
(x106)
hopper-stand
quadruped-run
humanoid-stane
Episode Return
Episode Return
Episode Return
750
750
750
500
500
500
250
250
250
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment Steps (x10%)
Environment Steps (x10%)
Environment Steps (x10%)
MBPO with Layer Norm
MBPO with Model Resets
MBPO
SAC
Figure 20. Comparing performance of MBPO, SAC, MBPO with layer norm applied the critic, and MBPO with periodic model resets for 6 challenging DMC benchmark tasks as discussed in Section
25


Reviews:
[{"review_id": "t4t6EmigZ1", "reviewer": "Reviewer_gMX1", "summary": "The paper investigates why popular Dyna‐style model‐based reinforcement learning (RL) methods, such as MBPO and ALM, perform well on OpenAI Gym tasks but struggle on the DeepMind Control Suite (DMC), despite both benchmarks having similar physics and task structures. The authors document a consistent performance gap: when synthetic rollouts—core to the Dyna approach—are used, these methods can even underperform a random policy in DMC, while in Gym they improve sample efficiency over their model‐free counterparts like SAC. The key contributions include:\n- Demonstrating that Dyna-style enhancements, which generate synthetic transitions to supplement real experience, can impede policy improvement in DMC environments even when the underlying off-policy algorithm works well on its own.\n- Conducting extensive experiments to analyze potential causes such as high model error, critic divergence (stemming from over- or underestimation due to unrealistic synthetic data), and issues with network plasticity. The study shows that even using a perfect predictive model or applying techniques to stabilize critic updates does not fully close the performance gap.\n- Introducing a new JAX-based implementation that accelerates experimentation by up to 40× compared to previous PyTorch-based approaches, thereby reducing the computational barrier for thorough evaluation of model-based RL algorithms.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and positive review. We appreciate your recognition of our contributions and the strengths of our experimental design and analysis.\n\nRegarding your mention of DreamerV3, we agree that its success in DMC highlights the diversity of model-based RL approaches. For further detail, we refer the reviewer to the section titled \"On comparisons with other model-based RL algorithms\" in our response to Reviewer TyLB, clarifying why DreamerV3 falls outside the scope of our evaluation of Dyna-style methods.\n\nRegarding the code, we consider the code an integral part of our contributions and are actively refining it. If the paper is accepted, we will certainly release the code alongside the camera-ready version.", "claims_and_evidence": "The assertion that Dyna-style enhancements consistently hinder performance in DMC is well illustrated by experiments with MBPO and ALM, yet the evidence is drawn from a specific set of environments and methods. While the paper argues these issues are inherent to the Dyna family, the existence of approaches like DreamerV3 (which succeed in DMC) suggests that further investigation is needed to determine if the observed limitations are truly universal across all Dyna-style methods.", "methods_and_evaluation_criteria": "The paper’s methodology is well-aligned with its goal of probing the limits of Dyna-style RL. The authors use two widely recognized benchmarks—OpenAI Gym and DeepMind Control Suite—to highlight that improvements seen with synthetic rollouts in one setting do not necessarily carry over to another with similar physics but subtle differences in task dynamics. The paper further dissects key factors (e.g., model error, critic divergence, and network plasticity) through controlled experiments and ablation studies.", "theoretical_claims": "The paper invokes the “no free lunch” concept to argue that no single RL algorithm can be optimal across all environments, but it does not provide a formal proof of this claim—it leans on established theoretical principles and empirical observations.", "experimental_designs_or_analyses": "The experimental design and analyses are sound, with thoughtful controls and comparisons.\nThe paper rigorously compares Dyna-style methods (MBPO and ALM) against their base off-policy counterparts (e.g., SAC) across two distinct benchmark suites—OpenAI Gym and the DeepMind Control Suite (DMC). This dual-benchmark evaluation is a strong point because it highlights the performance discrepancies in different settings. The use of controlled ablation studies (e.g., varying the synthetic-to-real data ratio, using a perfect predictive model, and applying periodic resets) provides granular insights into potential sources of failure such as critic divergence and network plasticity. This detailed analysis helps isolate the factors contributing to the performance gap.", "relation_to_broader_scientific_literature": "The paper revisits and challenges the celebrated gains of MBPO (Janner et al., 2019) and similar Dyna‐style methods, which had previously been shown to dramatically improve sample efficiency in OpenAI Gym. By exposing a performance gap when these techniques are applied to the DeepMind Control Suite (DMC), the paper questions the generality of prior successes.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The work creatively challenges established successes of Dyna-style methods by systematically comparing performance across benchmarks that, on the surface, appear similar. This cross-benchmark analysis exposes hidden limitations, an angle that is relatively novel in the literature.  By demonstrating that enhancements such as synthetic rollouts can actually impede performance in certain environments, the study raises important questions about the generality and robustness of widely adopted model-based RL methods.", "other_comments_or_suggestions": "It would be great if code is available for this work.", "questions_for_authors": "I have no questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y4QEsYYUHX", "reviewer": "Reviewer_TyLB", "summary": "This paper shows that Dyna-style off-policy model-based reinforcement learning (DMBRL) algorithms perform well in OpenAI Gym, while their performance can drop significantly in DeepMind Control Suite (DMC). And the paper analyzes potential causes (model error, lay normalization, etc) for this discrepancy, which ultimately fail to consistently resolve these problems.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**On comparisons with other model-based RL algorithms:**\nWe appreciate the reviewer’s suggestion regarding DreamerV3, PlaNet, and PETS. Our study focuses on Dyna-style algorithms MBPO and ALM, which use synthetic rollouts in proprioceptive, state-based settings. Pixel-based methods such as DreamerV1-V3 and PlaNet abstract away low-level issues central to our study. From the perspective of pixels, two hopper environments that move and look similar are effectively indistinguishable, whether simulated in Gym or DMC. In contrast, state-based methods must engage with environment-specific dynamics and state representations, which our results show can vary significantly despite a shared MuJoCo backend. For this reason, pixel-based approaches fall outside the scope of our analysis.\n\nDreamerV3, although impressive on proprioceptive tasks in addition to pixel-based, significantly differs from simpler Dyna-style methods in complexity and design philosophy. Understanding its success relative to MBPO and ALM, as noted in Section 2.1 and our FAQ, is valuable but warrants a dedicated analysis. Additionally, DreamerV3 has not been evaluated on OpenAI Gym tasks, complicating comparisons due to benchmarking inconsistency (as discussed in our paper). \n\nSimilarly, PETS and TD-MPC diverge from the Dyna paradigm by employing learned models for online trajectory optimization (e.g., model predictive control), rather than using synthetic rollouts for policy/value training. Our work targets hallucinated experience in Dyna-style pipelines.\n\nUltimately, while broader comparisons are important, evaluating methods that differ in observation modality or depart from the Dyna structure falls outside our intended scope. Our goal is to identify previously overlooked failure modes in a prominent subclass of model-based RL - Dyna-style methods - under realistic, proprioceptive conditions.\n\n**On the possibility that more sophisticated models could close the performance gap:**\nWe thank the reviewer for the insightful question regarding whether more sophisticated model architectures or training techniques (e.g., meta-learning) could close the performance gap. While such advances may indeed improve results, they would only reinforce our central claim: standard Dyna-style methods like MBPO and ALM exhibit fundamental limitations when used “out of the box,” even with extensive tuning. These methods were originally presented as simple, sample-efficient augmentations to off-policy RL, relying on short synthetic rollouts and standard neural network dynamics models. Yet across 15 diverse DMC environments, they consistently underperform their base off-policy RL learners - and sometimes even random policies - revealing deeper algorithmic issues.\n\n**On environmental differences between DMC and Gym:**\nWe appreciate the reviewer’s point that differences in reward structures, termination conditions, and dynamics between OpenAI Gym and DMC may influence performance. While we address these factors (Section 2.2, Appendix C), our analysis in the paper is deliberately algorithm-focused. We show that the base algorithm, SAC, performs well in both testbeds, while its Dyna-style counterpart (e.g., MBPO) fails in DMC. This contrast isolates synthetic rollouts - not environment differences - as the primary source of degradation, challenging the assumption that Dyna-style methods reliably enhance strong off-policy learners across domains. We agree that further work should explore how environment-side factors interact with Dyna-style methods. These are important questions, but beyond the current paper’s scope.\n\n**On hyperparameter sensitivity:**\nAs described at the end of Section 4.1, we performed extensive hyperparameter sweeps on key environments (e.g., hopper-stand, humanoid-stand), varying model size, retraining frequency, learning rates, and gradient steps. While these reduced model error in some cases, they did not produce consistent or meaningful gains in policy performance beyond a random policy in many cases. Sweeps over the off-policy algorithm's hyperparameters yielded similarly negligible effects.\nThis insensitivity supports our central claim: the failure of Dyna-style methods arises from fundamental challenges in leveraging synthetic rollouts, not suboptimal hyperparameter choices. We therefore did not emphasize sensitivity analyses, as tuning alone does not address the core limitations of these approaches. We are happy to include additional details in the camera-ready manuscript.\n\n**Clarification on the “percent model error” metric:**\nWe will revise the text to include an explicit formula and explanation of our percent model error metric.\nSpecifically, let $\\hat{y}$ denote the model's prediction (next observation and reward) and let $y$ be the corresponding ground-truth target. We define:\n\n$$\n\\text{Percent Model Error} =  \\frac{ \\left\\| \\hat{y} - y \\right\\|_2 }{ \\left\\| y \\right\\|_2 } \\times 100\n$$\n\n**Reproducibility:**\n\nSee response to Reviewer gMX1.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The paper does not contain proofs or theoretical developments.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The paper shows a significant gap in the literature regarding the performance of Dyna-style model-based reinforcement learning (DMBRL) algorithms across different benchmarks, specifically OpenAI Gym and DeepMind Control Suite (DMC). The findings challenge the prevailing assumption that DMBRL algorithms, such as Model-Based Policy Optimization (MBPO), generalize well across similar environments.", "essential_references_not_discussed": "The paper could benefit from discussing PlaNet (Hafner et al., 2019) and PETS (Chua et al., 2018), which are other model-based RL algorithms that have been evaluated in both Gym and DMC. Additionally, the paper could discuss more recent advancements in model-based RL. This could provide a broader perspective on the limitations of purely Dyna-style methods.", "other_strengths_and_weaknesses": "**Strengths:** \n\n1. The paper presents a observation that Dyna-style algorithms, which perform well in OpenAI Gym, significantly underperform in DeepMind Control Suite (DMC). This finding challenges the assumption that these algorithms generalize well across similar benchmarks and raises important questions about the robustness of DMBRL methods.\n\n2. The paper provides a thorough empirical analysis, including extensive experiments across multiple environments and benchmarks. The authors systematically investigate potential causes for the performance gap, such as model error, overestimation bias, and neural network plasticity, and provide detailed results to support their conclusions.\n\n**Weaknesses:**\n\n1. While the paper focuses on MBPO and ALM, it does not sufficiently explore other model-based RL algorithms that have shown success in DMC, such as DreamerV3. A more comprehensive comparison with these algorithms would provide a clearer picture of why certain model-based methods succeed where others fail.\n\n2. The paper briefly mentions differences between OpenAI Gym and DMC in terms of reward structures, termination conditions, and physical parameters, but it does not delve deeply into how these differences might contribute to the performance gap. A more detailed analysis of these environmental factors could provide additional insights into the limitations of Dyna-style algorithms.\n\n3. The paper mentions hyperparameter sweeps but does not provide a detailed analysis of how sensitive the results are to different hyperparameter settings. Given that hyperparameters can significantly impact the performance of RL algorithms, a more thorough exploration of this aspect would strengthen the paper's conclusions.", "other_comments_or_suggestions": "1. **Clarification on Model Error Metrics:** The paper uses \"percent model error\" as a metric, but it is not clearly defined how this metric is calculated. Providing a precise definition or formula for this metric would improve the clarity of the results.\n\n2. **Reproducibility:** While the authors claim a JAX-based implementation, it would be helpful to provide the code details.", "questions_for_authors": "1. The paper suggests that the performance gap is \"fundamental,\" but have the authors considered the possibility that it could be mitigated through more sophisticated model architectures or training techniques, such as meta-learning or transfer learning?\n\n2. Given that DreamerV3 performs well, what specific design choices or architectural differences might explain its success compared to MBPO and ALM?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6rWCMnONhl", "reviewer": "Reviewer_ffFz", "summary": "This paper investigates why two MBRL algorithms (MBPO & ALM) perform well on OpenAI Gym but poorly on DMC.\nThe authors show that apart from Dyna model prediction error, the synthetic rollouts themself could arrest policy improvement rather than enhance it when deployed across more diverse environments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful suggestions to improve the clarity and presentation of our paper. We will incorporate the suggested changes to:\n- More precisely describe the subset of OpenAI Gym environments used in our experiments.\n- Cite ALM upon first mention and reduce repeated citations later on to avoid redundancy.\n\n**What would be the result of Figure 12 if the model is perfect?**\nThis is an excellent question and one we are also curious about. As depicted in Figure 6, we generated a single data point for both humanoid-stand and hopper-stand at the 0.05 synthetic ratio using perfect transitions (i.e., querying the ground truth environment). This process required approximately 4 GPU-days and substantial CPU usage, due to the environment transitions becoming the bottleneck. Unlike synthetic rollouts from learned models (which are efficiently parallelizable via JAX), real environment rollouts suffer from parallelism limitations that drastically slow down runtime.\n\nExtrapolating from our timing, generating the full version of Figure 12 with perfect transitions would require roughly 40 GPU-days across multiple machines. Given the resource cost and limited additional insight expected beyond what is already shown in the paper, we opted not to pursue the full experiment at this time. However, if we succeed in optimizing environment rollouts via vectorized environments in the future, we agree this would be a worthwhile avenue to explore.\n\nThat said, our preliminary results suggest that even with perfect transitions, performance does not recover to the level of pure real-data training, regardless of synthetic-to-real ratio. This aligns with our central claim that the degradation arises not solely from model bias but from algorithmic limitations in how synthetic data is incorporated. \n\n**For MBPO/ALM on OpenAI Gym that surpass SAC (or methods like DreamerV3/EfficientZeroV2 on DMC), do you also observe the underestimation issue?**\nThank you for this insightful question. We interpret it as: \"Do value underestimation issues persist in model-based methods when they outperform SAC in Gym or state-of-the-art model-free methods in DMC?\" Please let us know if we have misunderstood.\n\nIn our experiments, we did not observe value underestimation in ALM or MBPO when evaluated on OpenAI Gym. This suggests that underestimation is largely a DMC-specific phenomenon in our setting. As discussed in the paper, even with a perfect model, underestimation is much less pronounced in DMC compared to a learned model, implicating model error as a contributor to Dyna failing, but not the only issue. However, Section 4.2 further shows that MBPO still fails to match SAC in DMC even when underestimation is not prominent, indicating that underestimation, while influential, is not the sole factor - nor a \"smoking gun\" - behind the performance gap.\n\n**For the left column of line 425 - could you clarify the update count after model reset?**\nCertainly. During our investigation of model plasticity, we explored multiple reset strategies inspired by prior work [1]. We found that increasing the reset frequency up to a point improved performance. The strategy we ultimately adopted - resetting every 20k environment steps - aligns with the reset frequency recommended in [1] for a replay ratio of 128. Empirically, this approach worked better in our setting than the more conservative reset frequency from [1] prescribed for a replay ratio of 20 (every 2.56 × 10⁶ gradient steps, equivalent to ~128k env steps), or intermediate strategies between the two.\n\nWe will clarify this detail in the revision to aid reproducibility and interpretability.\n\nCitations:\n[1] P. D’Oro, M. Schwarzer, E. Nikishin, P.-L. Bacon, M. G. Bellemare, and A. Courville, “Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier,” 2023.", "claims_and_evidence": "The claims made in the submission are mostly supported by clear and convincing evidence", "methods_and_evaluation_criteria": "The methods used mostly fit the problem and the authors' claims.", "theoretical_claims": "Few theoretical proofs are involved in this paper.", "experimental_designs_or_analyses": "The soundness/validity of the experimental designs or analyses is good if the implementation is correct.", "relation_to_broader_scientific_literature": "This paper studies the problem that the performance of MBRL algorithms varies depending on the environment and problem characteristics. Some previous work pointed out the inefficiency of MBRL on DMC tasks, and some previous work studied the impact of model error.", "essential_references_not_discussed": "No missing key references found yet.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. There are many papers investigating new MBRL models, but few studies the underlying reason why modern MBRL works, or why a learned model could improve the sample efficiency of model-free RL algorithms. Thus the community lacks relevant theories for design. This paper is a good start in bridging this gap.\n\n2. The authors' claims are clear and the evidence is supportive. They studied the impact of model error, underestimation, learning dynamics and so on, so that the limitation of the Dyna style's learning is clear.\n\n**Weaknesses:**\n\nSee comments and questions.", "other_comments_or_suggestions": "1. At the abstract/intro of this paper, I think it would be better to say something like \"$\\underline{\\text{control tasks}}$ in OpenAI Gym\", since it contains environments other than control tasks.\n\n2. On the left column of the line 057, I suggest cite ALM when it first appears and reduce other repeated citations later.", "questions_for_authors": "1. What would be the result of Figure 12 if the model is perfect? \n\n2. For MBPO/ALM on OpenAI Gym that surpasses SAC (or like DreamerV3/EfficientZeroV2 on DMC), could you also observe the underestimation issue? This may provide insights for further improving the SOTA MBRL methods.\n\n3. For the left column of the line 425. When resetting the predictive model, it may take more updates than the original train ratio to optimize the model. If the number of updates is too low, then the model may fail to converge and provide little information. If the number of updates is sufficient, then the key insights 1 may still hold. Could you clarify the details on this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W9gEwnbM9z", "reviewer": "Reviewer_n69W", "summary": "The paper investigates the performance discrepancies of Dyna-style model-based reinforcement learning (DMBRL) algorithms across different benchmark environments, specifically OpenAI Gym and DeepMind Control Suite (DMC). The authors highlight a significant performance gap, where DMBRL algorithms like Model-Based Policy Optimization (MBPO) and Aligned Latent Models (ALM) perform well in OpenAI Gym but struggle in DMC, despite the environments sharing similar tasks and physics backends. \n\nMain Findings: \n\nPerformance Gap: DMBRL algorithms show a performance drop in DMC compared to OpenAI Gym. \n\nModel Error Impact: High model error in DMC environments leads to significant performance degradation, even with short synthetic rollouts. \n\nCritic Divergence: Inaccurate synthetic transitions cause critic divergence, hindering learning in DMC. \n\nMain Results: \n\nEmpirical Evaluations: MBPO and ALM were evaluated across six OpenAI Gym tasks and fifteen DMC tasks, showing that Dyna-style enhancements can prevent policy improvements in DMC. \n\nModel Error Analysis: Even with a perfect model, MBPO cannot consistently outperform its base off-policy method (SAC) in DMC. \n\nCritic Divergence Mitigation: Layer Normalization mitigates critic divergence but does not enable MBPO to outperform SAC consistently. \n\nPlasticity Loss Mitigation: Periodic resets of off-policy base components improve MBPO's performance in some DMC tasks but do not match SAC's performance. \n\nContributions: \n\nIdentifying Performance Gap: The paper demonstrates a significant performance gap of DMBRL methods across different benchmarks. \n\nAnalyzing Causes: The authors analyze potential causes for the performance discrepancy and apply modern mitigation approaches. \n\nAccelerating Experimentation: The paper introduces a new JAX-based implementation of MBPO, achieving up to a 40x decrease in wall-clock time, enabling more accessible DMBRL studies.\n\n## update after rebuttal\n\nThank you for the author's reply. It answered almost all of my questions. I have no further questions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**How do you ensure synthetic data doesn’t introduce bias into the critic’s learning process? Are there methods to filter or correct this data?**\n\nWe performed sweeps over the synthetic-to-real data ratio (Figure 12) and found that introducing any amount of synthetic data into off-policy learning in DMC leads to a substantial degradation in policy performance. Moreover, in our Dyna-style experiments using perfect synthetic data (i.e., no model error), we showed in Figure 6 that learning still fails to outperform the base off-policy methods on which these algorithms are built. These findings suggest that the failure is not due to model inaccuracies or data quality, but rather due to algorithmic issues intrinsic to how synthetic data is integrated.\n\nWhile methods such as those proposed in [1,2] attempt to filter or adapt synthetic rollouts, our results imply that such corrections are insufficient to fix the core algorithmic shortcomings. Even with perfect data, the performance gap remains, underscoring the limits of current Dyna-style designs.\n\nAddressing this challenge remains an active area of research for us, and we welcome further discussion on promising directions for improving synthetic data integration in off-policy learning.\n\n**Besides periodic resets, have you considered other methods to address plasticity loss, such as regularization or dynamic architectures?**\nYes. While we did not frame them explicitly in terms of plasticity, we explored several techniques targeting related issues. Layer normalization was employed in prior work [3] as a method that simultaneously mitigates overestimation and stabilizes learning dynamics - both of which relate to plasticity loss. Additionally, we experimented with unit ball normalization [4],  which was proposed to combat overestimation and thereby also helps address plasticity indirectly. We did not include these results in the main text as they did not substantially outperform the more commonly used layer norm baseline.\n\nWe did not pursue further architectural or regularization strategies because our literature review found no alternatives that clearly outperform these methods. Periodic resetting, layer norm, and unit ball normalization appear to represent the state-of-the-art approaches for maintaining plasticity in high-update-ratio settings. If we have overlooked a promising direction, we would sincerely appreciate any further insight from the reviewer.\n\n**Have you tested these algorithms on other benchmarks or real-world applications to see if the performance gap persists?**\nWe did not. Our motivation was to critically re-examine claims of sample efficiency made by prior dyna-style methods (e.g., ALM, MBPO), which were accepted largely based on Gym benchmarks. These methods were widely adopted under the assumption that they outperformed their model-free counterparts in sample efficiency. Our work revisits that conclusion using a comparable benchmark - DeepMind Control Suite (DMC) - which shares the same MuJoCo physics backend.\n\nThis choice minimizes the likelihood that differences in performance are due to changes in dynamics simulation, and therefore emphasizes the impact of the Dyna-based algorithmic changes introduced in ALM and MBPO. Our results demonstrate that the original claims made about those methods do not generalize even within this extremely similar experimental setting. While evaluating additional domains might provide further insight into generalization challenges, we chose to focus on a clean and controlled comparison to highlight foundational flaws in current Dyna-style designs. Introducing more benchmarks could risk diluting the clarity of this central message.\n\n**Citations:**\n\n[1] B. Frauenknecht, A. Eisele, D. Subhasish, F. Solowjow, and S. Trimpe, “Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption,” Jun. 21, 2024, arXiv: arXiv:2405.19014. doi: 10.48550/arXiv.2405.19014. \n\n[2] Y. Li, Z. Dong, E. Luo, Y. Wu, S. Wu, and S. Han, “When to Trust Your Data: Enhancing Dyna-Style Model-Based Reinforcement Learning With Data Filter,” Oct. 16, 2024, arXiv: arXiv:2410.12160. doi: 10.48550/arXiv.2410.12160. \n\n[3] C. Lyle, Z. Zheng, E. Nikishin, B. A. Pires, R. Pascanu, and W. Dabney, “Understanding plasticity in neural networks,” Nov. 27, 2023, arXiv: arXiv:2303.01486. Accessed: Jun. 29, 2024. [Online]. Available: http://arxiv.org/abs/2303.01486 \n\n[4] M. Hussing, C. Voelcker, I. Gilitschenski, A. Farahmand, and E. Eaton, “Dissecting Deep RL with High Update Ratios: Combatting Value Overestimation and Divergence,” Mar. 09, 2024, arXiv: arXiv:2403.05996. Accessed: Apr. 08, 2024. [Online]. Available: http://arxiv.org/abs/2403.05996", "claims_and_evidence": "Yes, Well-Supported Claims: \n\nPerformance Gap: DMBRL algorithms like MBPO and ALM perform well in OpenAI Gym but struggle in DMC. This is backed by empirical evaluations across multiple tasks. \n\nModel Error Impact: High model error in DMC leads to performance degradation, supported by detailed measurements and analysis. \n\nCritic Divergence: Inaccurate synthetic transitions cause critic divergence, hindering learning in DMC. Evidence includes Q-value measurements and analysis of synthetic data impact. \n\nPlasticity Loss: Periodic resets of off-policy base components improve MBPO's performance in some DMC tasks, though not universally. \n\nAreas for Further Clarification: \n\nDyna-Style Enhancements and Plasticity Loss: More detailed analysis could strengthen the claim that Dyna-style enhancements exacerbate plasticity loss. \n\nIntrinsic Performance Gap: The claim that the performance gap is intrinsic to Dyna-style algorithms could benefit from more direct evidence or theoretical analysis.", "methods_and_evaluation_criteria": "Proposed Methods: \n\nEmpirical Evaluations: The paper evaluates MBPO and ALM across multiple tasks in OpenAI Gym and DeepMind Control Suite (DMC). This comparative approach is appropriate for identifying performance gaps and understanding the robustness of DMBRL algorithms. \n\nModel Error Analysis: The authors measure model error across training distributions in both OpenAI Gym and DMC. This method is relevant for understanding how model accuracy impacts performance. \n\nCritic Divergence Analysis: The paper measures average critic Q-values during training to assess critic divergence. This method helps identify how inaccurate synthetic transitions affect learning. \n\nPlasticity Loss Mitigation: The authors perform periodic resets of the off-policy base components to mitigate plasticity loss. This method is sensible for evaluating the impact of plasticity loss on performance. \n\nPerfect Model Experiments: The paper modifies the DMC simulator to use a perfect model to evaluate the impact of model accuracy on performance. This method provides insights into the intrinsic limitations of Dyna-style algorithms. \n\nEvaluation Criteria: \n\nBenchmark Datasets: The use of OpenAI Gym and DMC as benchmark datasets is appropriate. Both are widely accepted testbeds for reinforcement learning, providing a variety of continuous control tasks with similar physics backends. \n\nPerformance Metrics: The paper uses normalized final return and sample efficiency as performance metrics. These metrics are standard in reinforcement learning and provide a clear basis for comparison. \n\nModel Error Metrics: The authors measure model error as a percentage of the training distribution. This metric is relevant for assessing the accuracy of the predictive model. \n\nCritic Divergence Metrics: The paper measures average critic Q-values to assess critic divergence. This metric is appropriate for evaluating the impact of synthetic data on learning.", "theoretical_claims": "This paper focuses on empirical evaluations rather than theoretical claims, so there are no formal proofs to verify. However, the key conceptual claims are well-supported by empirical evidence: \n\nDyna-style enhancements can prevent policy improvements in DMC: Supported by performance metrics across multiple tasks. \n\nHigh model error in DMC leads to performance degradation: Backed by detailed measurements of model error. \n\nCritic divergence due to inaccurate synthetic transitions hinders learning: Evidenced by Q-value measurements and analysis. \n\nPlasticity loss contributes to MBPO's failures: Demonstrated through periodic resets and their impact on performance. \n\nIssues: \n\nDyna-Style Enhancements and Plasticity Loss: Could use more detailed theoretical analysis. \n\nIntrinsic Performance Gap: Would benefit from additional theoretical insights.", "experimental_designs_or_analyses": "The paper uses sound experimental designs to support its claims: \n\nComparative Performance Evaluation: Valid comparison of MBPO and ALM across OpenAI Gym and DMC tasks. \n\nModel Error Analysis: Measures model error effectively to show its impact on performance. \n\nCritic Divergence Analysis: Tracks Q-values to demonstrate the effects of synthetic data on learning. \n\nPlasticity Loss Mitigation: Uses periodic resets to assess the impact of plasticity loss on performance. \n\nPerfect Model Experiments: Modifies DMC to use a perfect model, isolating the effect of model accuracy.", "relation_to_broader_scientific_literature": "Key Relations to Prior Work: \n\nDyna-Style Algorithms: \n\nPrior Work: Based on Sutton (1991), with extensions like MBPO (Janner et al., 2019). \n\nContribution: Shows performance gaps across OpenAI Gym and DMC, highlighting robustness issues. \n\nMBRL Algorithms: \n\nPrior Work: MBPO and ALM (Ghugare et al., 2023) demonstrate success in specific benchmarks. \n\nContribution: Reveals degraded performance in DMC, emphasizing the need for diverse benchmarking.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Here are key questions for the authors that could clarify important points: \n\nSynthetic Data Bias: How do you ensure synthetic data doesn’t introduce bias into the critic’s learning process? Are there methods to filter or correct this data? \n\nPlasticity Loss Solutions: Besides periodic resets, have you considered other methods to address plasticity loss, such as regularization or dynamic architectures? \n\nGeneralizability: Have you tested these algorithms on other benchmarks or real-world applications to see if the performance gap persists?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates Dyna-like reinforcement learning algorithms and, especially, addresses the differences in performance on standard benchmarks (OpenAI Gym and DeepMind Control Suite). 

Their is a consensus that the problem is important and is worth studying. Most of the reviewers appreciated the experimental setup and the rigorous methodology that challenges the different dimensions of Dyna-like RL methods. Especially, the generation of synthetic roll outs that seems to harm performance in some contexts while it enhances performance in others. 

Among the raised concerns, the choice of the tested algorithms was pointed as a weakness of the paper. Especially, no comparisons to sophisticated methods such as DreamerV3 was made. It makes the paper somewhat lacking of completeness and the authors were encouraged to study such methods.
Decision: Accept (poster)