SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this work, we applied stability tricks introduced by DreamerV3 to PPO and demonstrated that they do not result in a general performance improvement, but can be valuable in specific cases. The fact that these tricks do not benefit PPO raises the question of how exactly they impact DreamerV3. These tricks are directly applicable to PPO and required little to no modification to implement. It is possible that they may work for the entropy-regularized actor-critic that DreamerV3 uses, which is likely a weaker baseline than PPO on its own. It's also possible that symlog predictions, twohot encoding, or unimix categoricals are specifically beneficial to world model learning, or that the world model-specific tricks not studied in this paper are the source of its improved performance. We note
that their use of SiLU activation functions $^{21}$ , layer normalization $^{22}$ , and a 200 million parameter architecture may also contribute to its state-of-the-art results. We include one experiment using a similar architecture in the appendix, but find that it does not work well for PPO.
Hafner et al. make few direct comparisons to the previous DreamerV2 method and include limited ablations on only six environments, so it is difficult to confirm exactly how each trick contributes to performance. The ablation studies in this paper should serve as a valuable reference for studying these tricks and developing new methods that promote robustness to varying reward scales in reinforcement learning. Solving this problem would allow RL to more easily be applied to new problems without extensive engineering efforts. DreamerV3 achieves state-of-the-art results, but our results show that the relationship between these implementation tricks and those results may not be as straightforward as it would seem otherwise. Due to the importance of this problem and the impressive results of DreamerV3, we believe these tricks deserve further investigation both using DreamerV3 and in new contexts.
Due to the many differences between PPO and DreamerV3, we cannot say whether the findings in this paper transfer to DreamerV3 or other similar algorithms. Our experiments also do not fully cover the range of tasks evaluated in the original DreamerV3 paper. We have focused on a more limited set of environments in order to provide thorough, high-quality ablations. We experiment on Atari environments without reward clipping to study the effects of each trick on poorly normalized reward scales, and the DeepMind Control Suite for environments with well-normalized returns, but it's possible that we would see different results on more complex environments.
We have presented a thorough study of the stability techniques introduced in DreamerV3 to the widely used PPO algorithm. These experiments allow us to identify key areas where tricks improve performance and demonstrate the broader applicability and potential benefits of these techniques for the reinforcement learning community. In the spirit of openness and reproducibility, we have released our complete code base and experiment data at https://github.com/RyanNavillus/PPO-v3, further promoting the adoption and study of these techniques in the reinforcement learning community.
We would like to thank James MacGlashan for his helpful comments and suggestions, as well as CarperAI for providing the majority of the compute used for our experiments.
1. Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination, 2020.
2. Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. CoRR, abs/2010.02193, 2020. URL https://arxiv.org/abs/2010.02193.
3. Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models, 2023.
4. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
5. Karl Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient, 2020.
6. Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreview.net/forum?id=H1JJJnR5Ym.
19. Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
20. Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI'18/IAAI'18/EAAI'18. AAAI Press, 2018. ISBN 978-1-57735-800-8.
21. Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning, 2017.
22. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.