Question: What happens if we don't apply [the logarithmic transformation] in practice?
Evidence: Our first comparison uses three locomotion tasks from the OpenAI Gym benchmark, which has become the standard benchmark for model-based RL algorithms. Thes tasks have dense rewards and pose no significant exploration challenge, so we do not expect MnM to outperform prior methods. The results (Fig. (top left)) show that MnM performs roughly on par with MBPO.
Tasks with sparse rewards, complex contact dynamics, and those requiring hard exploration often present a challenge for model-based RL algorithms, which are liable to exploit inaccuracies in the learned dynamics model. Our next experiment evaluates MnM on control tasks used in prior work that demonstrate these properties. These tasks the sparse-reward cartpole task from the DM Control benchmark, four manipulation tasks from the Metaworld benchmark, and four dexterous manipulation tasks from the ROBEL benchmark. See Appendix C for experiment details. The results shown in Fig. indicate that the complex environment dynamics of these tasks can cause prior model-based algorithms (MBPO) to perform worse than model-free algorithms (SAC), both in terms of asymptotic performance and sample complexity. Nonetheless, we observe that MnM frequently outperforms all prior methods and, more importantly, it consistently does well across all tasks. We include ablation experiments in Appendix B.1.
Real Dynamics:  Finally, we visualize the dynamics learned by MnM on two robotic control tasks. These tasks have deterministic dynamics, so our theory would predict that an idealized version of MnM would learn a dynamics model exactly equal to the deterministic dynamics. However, our implementation relies on function approximation (neural networks) to learn the dynamics, and the limited capacity of function approximators makes otherwisedeterministic dynamics appear stochastic. On the Pusher-v2 task, the MnM dynamics cause the puck to move towards the robot arm even before the arm has come in contact with the puck. While this movement is not physically realistic, it may make the exploration problem easier. On the HalfChetah-v2 task, the MnM dynamics increase the probability that the agent remains upright after tripping, likely making it easier for the agent to learn how to run. We expect that the implicit stochasticity caused by function approximation to be especially important for real-world tasks, where the complexity of the real dynamics often dwarfs the capacity of the learned dynamics model.
Our next experiments use continuous-control robotic tasks to answer two questions. We first investigate whether MnM performs at least comparably with prior work. We then study tasks with sparse rewards and more challenging exploration, where we suspect the optimistic dynamics learned by MnM may be beneficial. We illustrate a subset of the environments in Fig. 4 and include implementation details in Appendix C.2. One detail of note is that we omit the reward augmentation (Eq. 3) for MnM during these experiments, as it hinders exploration leading to lower returns. We use MBPO (Janner et al., 2019) as a baseline for modelbased RL because it achieves state-of-the-art results and is a prototypical example of model-based RL algorithms that use maximum likelihood models.
Answer: As noted in Section 5.2 (first paragraph), the continuous control experiments already don't apply this logarithmic transformation; we found that it hurt exploration on these domains. This result raises an interesting direction for future work: how can we design model-based RL algorithms for complex domains that explore efficiently while still maximizing a provable lower bound on expected return?