Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Tricks
Authors: Ryan Sullivan
Akarsh Kumar
Shengyi Huang
John P Dickerson
Joseph Suarez
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Proximal Policy Optimization
Reward Normalization
TL;DR: Studying the impacts of DreamerV3 Tricks on Proximal Policy Optimization

Abstract: Most reinforcement learning methods rely heavily on dense, well-normalized environment rewards. DreamerV3 recently introduced a model-based method with a number of tricks that mitigate these limitations, achieving state-of-the-art on a wide range of benchmarks with a single set of hyperparameters. This result sparked discussion about the generality of the tricks, since they appear to be applicable to other reinforcement learning algorithms. Our work applies DreamerV3's tricks to PPO and is the first such empirical study outside of the original work. Surprisingly, we find that the tricks presented do not transfer as general improvements to PPO. We use a high quality PPO reference implementation and present extensive ablation studies totaling over 10,000 A100 hours on the Arcade Learning Environment and the DeepMind Control Suite. Though our experiments demonstrate that these tricks do not generally outperform PPO, we identify cases where they succeed and offer insight into the relationship between the implementation tricks. In particular, PPO with these tricks performs comparably to PPO on Atari games with reward clipping and significantly outperforms PPO without reward clipping.

Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Tricks
Ryan Sullivan rsulli Gumd. edu University of Maryland College Park, MD, USA
Akarsh Kumar akarshkumarO wtt.kkammmrr000001111 010@mmaa1.....mmmmmmmmmmm1111111 com Massachusetts Institute of Technology Cambridge, MA, USA
Shengyi Huang costa. huang@outlook. com Drexel University Philadelphia, PA, USA
John P. Dickerson johnd@umd edu University of Maryland College Park, MD, USA
Joseph Suarez jsuarezêmit edu Massachusetts Institute of Technology Cambridge, MA, USA
Abstract
Most reinforcement learning methods rely heavily on dense, well-normalized environment rewards, DreamerV3 recently introduced model-based method with a number of tricks that mitigate these limitations, achieving state-of cote-of-theerrr on a wide range of benchmarks with a single of hyperparameters. This result sparked discussion about the generality of the tricks, since they appear to be applicable to other reinforcement learning algorithms. Our work applies DreamerV3's tricks PPO and is the first such empirical study outside of the original work. Surprisingly, we find that the tricks presented do not transfer as general improvements PPO. We use high quality PPO reference implementation and present extensive ablation studies totaling over 10,000 100 hours on the Arcade Learning Environment and the DeepMind Control Suite. Though our experiments demonstrate that these tricks do not generally outperform PPO, we identify cases where they succeed and offer insight into the relationship between the implementation tricks. In particular, PPO with these tricks performs comparably to PPO on Atari games with reward clipping and significantly outperforms PPO without reward clipping.
Introduction
Reinforcement learning (RL) has shown great promise in addressing complex tasks across various do- mains. However, each new application typically requires environment-specife tuning and significant engineering effort. The Dreamer 1.2.3 line of work has focused on world-modeling to achieve high performance across wide range of tasks. The most recent version, DreamerV3, outperforms the previous version with single set of hyperparameters shared across all tasks. DreamerV3 introduces several stability and performance-enhanciin technique accomplish this, but also includes several orthogonal changes such as larger architecture. Additionally, most of these new techniques are inherently tied to the world-modeling formulation and could potentially be applied other rein- forcement learning settings. Table categorizes the tricks according to which part of the DreamerV3 algorithm they apply to. Specifically, the following techniques are directly applicable model-free actor-critic methods:
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Symlog Predictions: A transformation applied to the targets of neural network, helping to smooth predictions different magnitudes
Twohot Encoding: A discrete regression objective representing continuous values as a weighting of two adjacent buckets
Critic EMA Regularizer: Regularizes the critic outputs towards its own weight Exponential Moving Average (EMA) to improve stability during training
Percentile Scaling: Scales returns by an exponentially decaying average of the range between their 5th and 95th batch percentile to improve robustness to outliers and varying reward scales
Unimix Categoricals: Combines 99% actor network outputs with 1% uniform random sampling to inject entropy into action selection
Proximal Policy Optimization (PPO)* is a popular RL algorithm due its simplicity and widespread success variety of domains. PPO an actor-critic algorithm that approximates trust-region optimization strategy to update the policy. The objective function for PPO is given by:
L( = Ex min
where r0)
A, is the estimated advantage function, and e is a hyperparameter controlling the size of the trust region.
(a ls.)
Tuu (ax]s:)'
Our work applies the stability techniques introduced DreamerV3 to PPO, aiming to improve the algorithm's ' stability and performance. We adapt these techniques for the slightly different algorithmic setting of model-free learning and provide working implementation tested on various environments. We provide comprehensive ablations demonstrating that these techniques generally do not enhance the performance of PPO. We believe this negative result merits sharing because the widespread interest in D3eamerV3 and serves temper expectations of the generality of the tricks presented.
The contributions of our work include:
Applying and adapting DreamerV3's stability techniques to PPO
Demonstrating the effects of these techniques on PPO's stability and performance across diverse
environments
Analysis of the strengths and weaknesses of each trick, with a more detailed exploration of the two most promising tricks, twohot encoding and symlog predictions
A high-quality implementation in CleanRL. to enable further research in this direction
2
Related Work
The Dreamer line of work focuses on learning from imagined experi- ences using world models. Each ver- sion improves upon the previous, with DreamerV3 achieving state-of-the-ar results on variety of tasks. While these algorithms primarily focus on world-modeling, we are interested in model-free algorithms because of their simplicity and widespread appli- cability. This work focuses PPO, which has become the primary algo- rithm for applying RL to new domains in recent years. Numerous extensions and improvements
Table 1: Application of DreamerV3 Tricks
Implementation Trick
Actor
Critic
World Model
Symlog Predictions Twohot Encoding
X
X
X
X
X
Percentile Scaling
X
Critic EMA Regularizer
X
Unimix Categoricals
X
X
to PPO have been proposed, such as incorporating multiple optimization phases or exploration bonuses
Several works have focused on improving the stability of reinforcement learning algorithms. For instance, Pop-Art? introduced adaptive normalization of value targets, while R2D2 and Ape-X address the challenges of off-policy learning. However, these approaches often require extensive modifications the base algorithms or focus specific challenges, whereas our work studies

---

set of general stability techniques that are designed to apply more broadly to various reinforcement learning settings.
Previous work has studied the impact of the implementation details of PPO on its performance. Huang et al. 10 described the implementation details of PPO and explored how each impacts its performance. Engstrom et al. !I explores how much of PPO's s performance over Trust Region Policy Optimization 12 can be attributed to algorithmic improvements versus implementation tricks. None of these discuss the methods explored in our work, though some of the tricks they discuss serve similar purpose.
In this work, we build on the stability techniques introduced in DreamerV3, exploring their applica- bility beyond worddmmodlingg Our approach enhances the performance and stability of PPO when rewards are unbounded. However, as we further show, reward clipping is simple and strong baseline that is hard to beat. Our work highlights the strengths and weaknesses of these techniques.
3 Methods
In this section, we describe the application of the stability techniques introduced in DreamerV3 to the PPO algorithm. We detail each technique and discuss the necessary adaptations for incorporating them into PPO.
We implement these tricks as minimal extensions to CleanRL's extensively validated and bench- marked 15,14 PPO implementation, We used both DreamerV3's open source code base and the paper as references. We found the implementations of all tricks to be consistent with their descriptions in the paper, and we checked several small ambiguities with the authors. We performed manual hyperparameter tuning for each of the implementation tricks, as well as automatic hyperparameter tuning of the learning rate, entropy coefficient, and value loss coefficient using Optuna 15 for the full algorithm with all tricks enabled. We found improvement in performance and therefore chose to keep CleanRL's default hy hyperparameters. We also compared our code with all tricks disabled to the original unaltered scripts to ensure that there were performance regressions a result of improperly implemented controls on the tricks. Finally, we have open-sourced the implementation of our tricks and experiments https://github.. http://github.commRyaan www..//yaaaaaaalll/rr......
3.1 Symlog Predictions
In D3eamerV3, symlog predictions are used to compress the magnitudes of target values. Symlog approximates the identity function near the origin so that it has little impact on returns that are already small. The symlog transform and inverse symexp transform are defined Hafner al. as:
(exn(z)(exp((x)) - We also apply the same transformation to the observations in environments with continuous vector observations and call this method symlog observations.
symlog(2>) x) = sign(r) In(aa) + 1)
symexp(x))
(2)
To use symlog predictions in PPO, we calculate the value loss using the symlog transformation, and we apply the symexp operation to outputs from the critic network exactly the same in Hafner et al. 3, When twohot encoding is disabled, for a critic f(x, 0) with inputs æ and parameters 0 we use MSE loss to predict the symlog transformed returns y.
(())= = (f(x symlog(y)))
(3)
3.2 Twohot Encoding
Twohot Encoding is a generalization of one-hot encoding that represents continuous values as weighting between two equal buckets. For integer values, this is identical to the one-hot encoding of the value. This discrete regression technique allows the critic to predict distribution over buckets instead of a single value. The definition of twohot encoding from Hafner et al. 3 is shown in Equation 4:
|bR+1 x//
/bb|1b++1 byl
if k
B
twohot(s);;
1
x|/ |b++1 1 byl
if = k:
k= Σ
8(b;
x)
(4)
0
else
/=1
3

---

We implement two-hot encoding by replacing the critic value output with 255 bins as in Dreamer V3 and calculating the critic value output as the expected value the softmax distribution of those logits. The critic learns via categorical cross entropy between the critic logits and the twohot encoding of bootstrapped returns from the environment.
As in Hafner et al. ², when using two-hot encoding we initialize the critic logits to zero to prevent from predicting extremely large values at the start of training. In ablations with symlog enabled, we follow DreamerV3 and set the range of the twohot bins to [-20, 20], allowing the critic to represent values from |- cxp(2)),xx((00) When symlog disabled, we instead choose range [-15000, 15000] for Atari environments without reward clipping enabled, and |-1000) 1000] for Atari environments with reward clipping enabled well as the DeepMind Control Suite. These ranges were chosen to allow the critic to represent the maximum and minimum values seen during training. We find that the choice of range for two-hot encodings has a significant impact on performance. We include limited experiments demonstrating this relationship in subsection 4.5
3.3 Critic EMA Regularizer
Hafner et al. 3 regularize the critic towards an Exponential Moving Average (EMA) its own outputs, improving stability during training.
We the critic EMA to regularize the critic loss using the same decay rate (0.98) and regularizer coefficient (1.0) as D3eamerV3. We tried tuning these values but found little difference in performance. The EMA is updated once during each optimization step. For our ablation studies, when two-hot encoding is enabled, we use categorical cross-entropy loss between the critic logits and EMA logits. When two-hot disabled, we calculate the Mean Squared Error (MSE) between the critic and EMA value predictions.
3.4 Percentile Scaling
As discussed in Hafner et 3, previous work PPO has typically scaled returns by dividing them by their exponentially decaying standard deviation. However, they point out that when rewards are sparse and the standard deviation is small, this method amplifies the noise in small returns, preventing the policy from exploring sufficiently. Percentile scaling instead tracks an exponentially decaying average of the 5th and 95th percentile of each batch of returns. Scaling advantages by the difference between these percentiles promotes exploration while allowing DreamerV3 to use a fixed entropy scale. Instead of directly scaling returns as in DreamerV3. we scale the advantages predicted by Generalized Advantage Estimation. As in Hafner et al. 3 we only scale advantages when the scaling factor (the difference between the 5th and 95th percentile) greater than I to avoid amplifying errors in small advantages.
To add this method to PPO, we compute bootstrapped returns in the same way DreamerV3, then scale the advantages used to calculate the policy loss. Note that we do not modify the values or returns used calculate the critic loss. We also found that when the EMA updated too quickly, could lead returns to smoothly drop to in the DeepMind Control Suite. To counteract this, we change the percentile EMA decay rate from 0.99 0.995 for the DeepMind Control Suite and 0.999 for the Arcade Learning Environment. We also found that in practice this method works best in combination with the standard advantage normalization used in PPO.
3.5 Unimix Categoricals
Unimix categoricals are a mixture of 99% neural network outputs and 1 % random uniform sampling used for action selection. This prevents the probability of selecting any action from being zero, and encourages exploration similar to an entropy bonus. Unimix categoricals are only used in the Atari experiments because the DeepMind Control Suite does use categorical actions. We do not experiment with changing the unimix ratio in this paper and use 1% as in Hafner et al.
4 Experiments
In the following sections, we explain the environments we use to evaluate the implementation tricks. We test each trick on the entire environment suite across multiple seeds. We used approximately

---

PPO
PPO
All Tricks
PPO
PPD Reward Clipping
PPO
Tricks
PPO Reward Clipping All
Median Human Nermalized Score
Median Episodic Return
0.03
0.4
0.02
0.01
0.0
0M
10M
20M
30M
40M
50M
0M
5M
10M
15M
20M
25M
Steps
Stops
(a) DeepMind Control Suite
(b) Arcade Learning Environment
Median
IQM
Mean
Optimality Gap
PPO
0.05
0.10
0.15
0.04
0,06
0.08
0.10
0.15
0.18
0.21
0.24
0.78
0.81 0.84
(c) DeepMind Control Suite
Median PPO Reward Clipping tricks
Optimallty
Reward Clipping
(d) Arcade Learning Environment
Figure 1: PPO with all tricks enabled compared to base PPO, using the DeepMind Control Suite in a) and c) and Atari environments (with and without reward clipping) b) and d).
8000 GPUs hours for the experiments in this paper as well as 4000 more for testing and development, most of which were run Nvidia A100s. We report results using standard metries well those provided by the RLiable library as recommended Agarwal et 16 when results require uncertainty measures. They describe methodology and metries for creating reproducible results with only handful runs by reporting uncertainty. These metrics include 95% stratified bootstrap confidence intervals for the mean, median, interquartile mean (IQM), and optimality gap (the amount by which an algorithm fails to meet a minimum normalized score of 1).
4.1 Environments
4.1.1
Atari 100M
We train agents on each of the 57 environments in the Arcade Learning Environment 17 for multiple seeds each of our ablations. In this paper, we are examining whether these tricks improve PPO's robustness to different reward scales. The high scores, or maximum possible returns, for Atari games range from 10 for Surround to 89 million for VideoPinball. They also have drastically different reward densities, where some games guarantee reward at each step while others require the agent learn complex behavior experience any positive reward. Typically, previous work on the Arcade Learning Environment has used reward clipping to limit individual step returns to This significantly reduces the scale of returns and increases the effective density of rewards by weighting all nonzero rewards equally. To better study different reward scales, we perform ablations with and without reward clipping enabled. Aside from reward clipping, we use the standard wrappers recommended by Machado et al. For Atari experiments, standard benchmark of median human normalized scores 20 instead episodic returns.
4.1.2 DeepMind Control Suite
We also test our method on 35 proprioceptive control environments from the DeepMind Control Suite. These physics-basee environments with well-normalized reward functions. Each environment has
5

---

minimum return of O and maximum return of 1000, allowing us to test how these tricks perform on environments with returns that are already well-normalized. Rewards can also take on decimal values unlike in Atari environments. In our plots, we divide returns by 1000 to limit them to the range [0, I] and ensure that optimality gap has a consistent definition across environments.
4.2 Enabling All Tricks
We first compare our version of PPO with all of the stability tricks enabled the PPO baseline in each set of environments in Figure 1.
When reward clipping is enabled, we see that PPO achieves similar performance with and without the tricks. When reward clipping is disabled, the stability tricks allow PPO to recover most of the performance of PPO with reward clipping. This suggests that the tricks make PPO significantly more robust to varying reward scales, though they slightly underperform a simple reward clipping baseline.
4.3 Add-One Ablations
We perform add-one ablations where we enable one trick at a time and evaluate on both the DeepMind Control Suite and Atari environments (with and without reward clipping) to determine if any tricks provide a general improvement to the PPO baseline. The results are shown in Figure 2.
We find that all of the tricks perform comparably or slightly worse than the PPO baseline the DeepMind Control Suite and Atari with reward clipping. In particular, symlog predictions and twohot encoding underperform compared to the baseline. However, dramatically improve performance when reward clipping is disabled, and all of the remaining tricks perform comparably better most metries. This indicates that the tricks are effective return normalization tools, but underperform when returns are already normalized to reasonable ranges.
that symlog predictions
4.4 Drop-One Ablations
We perform drop-one ablations where we enable all tricks and disable one at a time to see the interactions between tricks and identify combinations that might outperform PPO. The results are shown in Figure 3. In these Atari experiments, reward clipping is disabled when symlog is enabled. Our drop-one ablations focus on symlog predictions, twohot encoding, percentile scaling, and the critic EMA regularizer. We exclude unimix categoricals from the Atari drop-one ablations because they had little impact in the add-one experiments and do not interact with the other tricks. Likewise, we exclude symlog observations from the DeepMind Control Suite drop-one ablations because they do not interact with the other tricks.
We find that for the DeepMind Control Suite, removing twohot encoding and the critic EMA regularizer actually improves performance. Removing percentile scaling and symlog predictions seems to harm performance while in the Arcade Learning Environment, only removing symlog predictions harms performance. In our add-one ablations for both environment suites, adding symlog predictions and twohot encodings to PPO also performed worse than the baseline. Our twohot encoding experiments require us to define a single bin value range across each entire environment suite which may be causing its lackluster performance. To diagnose this issue, we explore the interaction between symlog predictions and twohot encoding in the following section.
4.5 Symlog Predictions and Twohot Encoding
Twohot encoding can only represent bounded range represented by its bins. In Atari environments with widely-ranging possible scores, it can be difficult or impossible to choose tight bounds, which is why it is always paired with symlog transform in Hafner et al. 3, This section provides additional context to the interactions between these tricks and the results are displayed in Figure 4. We first examine the effects that the range and number of bins have on the performance of PPO with twohot encoding. Large bounds also seem to have detrimental effect on learning, possibly by reducing the effective number of bins used to predict values. We see that mean episodic return increases with the number of bins, then falls off at a much slower rate as we increase past the optimal number. Surprisingly, the agent only suffers a small drop performance even when the twohot range is set to 1. It's possible that the performance loss would be more significant in environment with larger

---

tone
croccaa symbbs
nore semiigg
twonce
cREcoma unimii
techot
drdcoma unimix
symbo
secencle
percerble
Symbigg
percentle
Modeen Humen Nomm STLTE
Westun Eassed heture
2.66
0.04
0.6
0.03
6-4
0.02
0.1
0.0
0k
1:M
20M
30M
40M1
50M
5W
13M
200
1.M
23H
25M
Steps
Stags
Steps
(a) DeepMind Control Suite
(b) Atari with reward clipping
(c) Atari without reward clipping
Median
Mean
Optima lity Gap
none symlog twohot percentile crilicemna symobs
(d) DeepMind Control Suite
Median
IQM
Mean
Optima lity Gap
none symlog twahot percentill
criticema
unimix
0.45
0.60
0.75
0.5
0.6
0.7
0.8
2.4
2.7
3.0
3.3
0.42
0.48
0.54
0.60
(e) Atari with reward clipping
Optimality Gap
symlog twohot perce ntile critic unimix
0.2
0.4
0.6
0.30
0.45
0.60
1.5
2.0
0.5
0.6
0.7
(f) Atari without reward clipping
Figure 2: Median scores averaged across multiple seeds for the add-one ablations in each environment set. 5 seeds were used for Deepmind Control Suite and 3 seeds for Atari environments.
returns than Breakout where the critic value prediction is much farther from the true value. We also see Figure that even when combined with symlog predictions, twhot encoding underperforms compared to base PPO across the entire Arcade Learning Environment.
Discussion
The tricks tested in this paper allow PPO achieve strong performance in environments with drastically different reward scales using a single set of hyperparameters, while the original PPO algorithm is unable do so. Our modified version of PPO performs comparably to the original algorithm in Atari games with and without requiring reward clipping, but we see that our agents perform worse the DeepMind Control Suite. This suggests that the tricks may be poorly suited to environments with normalized, bounded, continuous rewards. Symlog predictions are by far the most impactful trick in all experiments. Conversely, all of our experiments seem to suggest that twohot encoding is detrimental addition to PPO even in combination with other tricks. Percentile scaling, critic EMA regularizer, and unimix categoricals all slightly improve performance when we disable reward clipping, and slightly harm performance when we enable it, again suggesting that they are most useful environments without normalized returns. Symlog observations underperform in the DeepMind Control Suite, but could be useful in environments with larger unbounded observations.

---

all
twohot
criticema
all
twohot
criticema
symlog
percentile
symlog
percentile
Med Median Human Normalized Score
Median Episocic Return
0.03
0.6
0.02
0.2
0.01
0.0
0M
10M
20M
30M
40M
50M
0M
10M
15M
20M
25M
Steps
Steps
Deep Control Suite
(b) Atari with reward clipping
Median
IQM
Mean
Optimality Gap
symlog twohot
percentile
criticema
0.08 0.12
0,060
0.20
0.80
0.82
0.86
(c) DeepMind Control Suite
Median
IQM
Mean
Optimality Gap
symlog twohat percentlee criticemna
0.72
(d) Atari with reward clipping
Figure Median scores for the drop-one ablations in each environment set. seeds were used for DeepMind Control Suite and seeds for the Arcade Learning Environment
18
86
256
1024
twchot
100 15002
32
512
4696
tschot
Mhadan Humoa Nommiteea Store
0.8
2.00
Modan Hum
0.4
0.4
0.50
0.37
0.2
0.0
25M
314
(a) Twohot range (Breakout only)
(b) Twohot hins (Breakout only)
(c) Symlog and Twohot
Figure 4: Experiments varying the a) range (bins=256) and b) number of bins (range=100) used for twohot encoding in Atari Breakout, averaged across 3 seeds. c) shows symlog predictions with twohot encoding across all 57 games in the Arcade Learning Environment.
In this work, we applied stability tricks introduced by DreamerV3 to PPO and demonstrated that they do not result in general performance improvement, but can be valuable in specific cases. The fact that these tricks do not benefit PPO raises the question how exactly they impact DreamerV3. These tricks are directly applicable to PPO and required little to no modification to implement. It possible that they may work for the entropy-regularize actor-critic that DreamerV3 uses, which is likely weaker baseline than PPO its own. It's also possible that symlog predictions, twohot encoding, or unimix categoricals are specifically beneficial world model learning, or that the world model-specific tricks not studied in this paper the source of its improved performance. We note

---

that their use of SiLU activation functions 21 layer normalization ?2 and 200 million parameter architecture may also contribute to state-of-the-ar results. We include one experiment using similar architecture in the appendix, but find that it does not work well for PPO.
Hafner al." make few direct comparisons to the previous DreamerV2 method and include limited ablations on only six environments, so is difficult to confirm exactly how each trick contributes to performance. The ablation studies in this paper should serve as valuable reference for studying these tricks and developing new methods that promote robustness to varying reward scales in reinforcement learning. Solving this problem would allow RL to more easily be applied new problems without extensive engineering efforts. DreamerV3 achieves state-of tite--o--hheaare art results, but our results show that the relationship between these implementation tricks and those results may not be as straightforward as it would seem otherwise. Due to the importance of this problem and the impressive results of DreamerV3, we believe these tricks deserve further investigation both using DreamerV3 and in new contexts.
Limitations
Due to the many differences between PPO and DreamerV3, we cannot say whether the findings in this paper transfer to DreamerV3 or other similar algorithms. Our experiments also do not fully cover the range of tasks evaluated in the original DreamerV3 paper. We have focused a more limited set of environments in order to provide thorough, high-quality ablations. We experiment on Atari environments without reward clipping study the effects of each trick on poorly normalized reward scales, and the DeepMind Control Suite for environments with well-normalized returns, but it's possible that we would see different results on more complex environments.
7 Conclusion
We have presented a thorough study of the stability techniques introduced in DreamerV3 to the widely used PPO algorithm. These experiments allow us identify key areas where tricks improve performance and demonstrate the broader applicability and potential benefits of these techniques for the reinforcement learning community. In the spirit of openness and reproducibility, have released our complete code base and experiment data at promoting the adoption and study of these techniques in the reinforcement learning community.
further
8 Acknowledgments
We would like to thank James MacGlashan for his helpful comments and suggestions, as well as CarperAl for providing the majority of the compute used for our experiments.
References
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination, 2020.
2. Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. CoRR, abs/2010.02193, 2020. URL www.hhttss//atttee htt///axxvv...aaabee arg./////0.b........ 02193.
3. Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models, 2023.
4. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
5. Karl Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient, 2020.
6. Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL hwt.h:tt///eeet h/tp::s///ppereviee
net

---

7. Hado P van Hasselt, Arthur Guez, Matteo Hessel, Volodymyr Mnih, and David Silver. Learning values across many orders of magnitude. Advances neural information processing systems, 29, 2016.
8. Steven Kapturowski, Georg Ostrovski, John Quan, Rémi Munos, and Will Dabney. Recurrent experience replay in distributed reinforeement learning. In International Conference on Learning Representations, 2018.
9. Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. International Conference on Learning Representations.
10. Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimiza- tion. In ICLR Blog Track, 2022. URL https://iclr- track track.ggihuu................ 1 http:://drr-log--ckkggghhbhbboou mtc.kiitthubooo22200///////
implemmeniaiion-detalll..
11. Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras. Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters deep rl: A case study on ppo and trpo. In International Conference on Learning Representations, 2020. URL https:// openreview wttp://www........eeeeee wet./foruunddrrettlrr..
12. John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optmization. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning. volume 37 of Proceedings of Ma- chine Learning Research, pages 1889-1897, Lille, France, 07-09 Jul 2015. PMLR. URL https://proceedings mlr / html.
13. Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and JoAfo G.M. AraA"jo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022. URL http:://jml org/ papers/v23, 12//222112.1111.111 13m2.html.
14. Shengyi Huang, Quentin Gallouédec, Florian Felten, Antonin Raffin, Rousslan Fernand Julien Dossa, Yanxiao Zhao, Ryan Sullivan, Victor Makoviychuk, Cyril Roumégous, Chu- fan Chen, Masudur Rahman, João João G.M. Araújo, Guorui Quan, Daniel Tan, Timo K. Rujikorn Charakorn, Mark Towers, Yann Berthelot, Kinal Mehta, Dipam Chakraborty, Ar- jun KG, Valentin N/A, Chang Ye, and Zichen Liu. openrlbenchmark, May 2023. URL https://gt
15. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Op- tuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD) International Conference on Knowledge Discovery & Data Mining, KDD '19, page 2623-2631, New York, NY, USA, 2019, Association for Computing Machinery. ISBN 9781450362016. doi: 10.1145/3292500.33300 URL https:///dot org/10..... 1145/ 3292500 3330701.
16. Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
17. Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, Vol. 47:253-279, 2012. 1207. cite 1eriv:1207..47
doi: 10. 1/jain3/.1. URL http: h/tp///xxxv/////////////// org/ abs/
18. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529-33. February 2015. ISSN 00280836. URL http://dx. doi. 1tg/100.1088/mtture142888
10

---

19. Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness. Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61 61:52556,, 2018.
20. Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improve- ments in deep reinforcement learning. In Proceedings of the Thirty-Second AAAI Confer- ence on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium Educational Advances in Artificial Intelligence. 18. AAAI Press, 2018. ISBN 978-1-57735-800--
21. Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Weigoid-weighted linear units for neural network function approximation in reinforcement learning, 2017.
22. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.


Reviews:
[{"review_id": "nXcwZVbERD", "reviewer": "Reviewer_75kB", "summary": "The authors investigate the architectural and algorithmic improvements proposed in the DreamerV3 paper, but applied to PPO. The tricks are each compared on the DMC and Atari suites.", "questions": "During the course of writing this paper, did you try or manage to reproduce any results of DreamerV3? \n\nWhy the choice of PPO instead of e.g., SAC?", "limitations": "Seems clear.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Thorough benchmark papers are a valuable asset to the RL community due to the difficulty in accurately comparing them and typically large error margins.\n\nThe paper is well-written and easy to read, and cites related work\n\nDreamerV3 and PPO are both very relevant.", "weaknesses": "The conclusions of the experiments were not really surprising. While I think a benchmark / reproducibility paper like this still has value, I have some concerns with that:\n\n* First, it misses the obvious opportunity to first reproduce the DreamerV3 tricks on the Dreamer RL algorithm itself. It would not have to be every benchmark, but getting similar results on e.g. Atari would have helped support the claim that these tricks were correctly implemented, as well as reproduced a recent method of interest.\n\n* Second, why PPO? Is it even a good target for these tricks? According to the introduction, a big motivation for these were to enhance stability of actor-critic approaches, which typically struggle with stability. While PPO has some similarity to actor-critic approaches due to taking multiple steps and value function/GAE estimation, it was my impression that it behaved more like a policy gradient approach (which are typically more stable but on-policy and less efficient in theory). As such, the DreamerV3 tricks might have been a more natural fit for e.g., SAC.\n\n---\n\nAfter rebuttal: I think the authors gave adequate answers. Although they did not remedy all weaknesses, empirical studies like this are very important for RL research and it would be prohibitively expensive to test more algorithms. As such I think it would be useful for the community even though it has some limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and suggestions. First, we would like to clarify a bit of nuance surrounding the paper. The five algorithmic tricks were presented by DreamerV3 as techniques to improve performance and decrease/eliminate the need for hyperparameter tuning. This sparked a tremendous amount of interest in the academic community. From their mathematical definitions, very few of the tricks appeared limited to world-modeling, and could seemingly be added to any Actor-Critic algorithm. Every other RL researcher we spoke to expected them to generalize broadly to other algorithms. \n\nIn this work, we are specifically interested in transferring these tricks to other actor-critic methods, so we would also not consider it a reproduction of the DreamerV3 work. The authors open-sourced DreamerV3 shortly after publication, so we could have added PPO to their implementation in place of REINFORCE. However, it would have taken much longer to implement and been harder to verify correctness. Our code for this project is around 10x shorter than the DreamerV3 codebase and builds on CleanRL, one of the simplest and most widely used learning libraries. Our experiments consumed around 10,000 A100 hours and well over 100,000 CPU hours just to test on PPO. We had to only pick one algorithm, and we explain why we used PPO specifically in our overall response.\n\nAs for the implementation quality, we believe that by cross-referencing our implementations with both the DreamerV3 paper and open-source code, individually testing each trick with regular regression checks against the original PPO algorithm, and open-sourcing our own code for community use, we have sufficiently safeguarded our implementations of these fairly simple tricks from mistakes.\n\nIf this changes your perspective on our work, please consider changing your score. Publication makes it possible to continue securing funding for this type of large-scale empirical work."}, {"review_id": "UxQMWsSIhU", "reviewer": "Reviewer_DsH5", "summary": "This paper empirically tests the effectiveness of the DreamerV3 tricks on PPO. The authors conduct a comprehensive study on each trick, especially, extensive ablation studies, including add-one and drop-one ablations. Based on these experiments, the authors give some insights into the effect of the DreamerV3 tricks.", "questions": "- Do the observations of this paper applicable to other RL algorithms?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Extensive empirical study on each of the DreamerV3 tricks;\n- Experiments are conducted on various environments, including the environments from Atari 100M and DeepMind Control Suite, and using the rliable metrics to present robust performance comparisons;\n- Based on the experiments, the authors give some insights.", "weaknesses": "- The largest weakness of this paper is that they test the effectiveness of DreamerV3 tricks on only a single RL algorithm, i.e., PPO. Given the diversity of RL algorithms, the conclusions derived on PPO, at a large possibility, cannot generalize to other RL algorithms. Indeed, the observations given by this paper are valuable for the community as a technical report, but I don't think the contributions of this paper are enough for a NIPS paper.\n- The authors claim that \"our method performs comparably to PPO on Atari games with reward clipping and significantly outperforms PPO without reward clipping\". However, it can be seen from Figure 2 (e) and (f) that, the algorithms with reward clipping generally outperform the ones without reward clipping, which suggests that reward clipping is a useful technique. Therefore, I don't think it is valuable to achieve a SOTA performance under the condition without reward clipping. The vanilla PPO with reward clipping is still the SOTA method on Atari.\n- Furthermore, on DeepMind Control Suite, the vanilla PPO is also the SOTA algorithm. So it can be concluded that almost all the DreamerV3 tricks do not have a positive effect on PPO, leading the conclusions of this paper to be much less meaningful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in reviewing our work. We certainly agree that there are areas where our focus on PPO limits generalization to other RL algorithms, but we believe it is important to emphasize the significance of PPO as an algorithm. There is a large body of work aimed at studying and improving PPO specifically because it is the most popular RL algorithm [1] [2]. Our work transferring techniques from a very well known model-based RL algorithm to the most popular model-free algorithm will be of significant interest to many RL practitioners, especially since every RL researcher we talked to was sure that they would work. We explain our choice to use PPO further in our overall response.\n\nThe purpose of experimenting on Atari without reward clipping was to study a case with widely varying reward scales, which is where these tricks were expected to help most. We see in this case that symlog significantly improves PPO’s base performance here, as well as percentile scaling to a lesser degree. We see in our experiments with Atari and DMC that symlog is actually valuable in environments with large reward ranges, but less effective when rewards are already well normalized. These tricks do not provide a general improvement to PPO over standard methods, as we state in our paper, but we identify specific cases where they are valuable.\n\nTo answer your question about other algorithms, we think it is likely that our results transfer to similar actor-critic algorithms such as A2C, in particular, due to the similarity between A2C and PPO [3]. Certain tricks like percentile scaling could be applied to other algorithms (Q-Learning, Vanilla Policy Gradients, etc), but it’s unclear how the tricks would affect them.\n\nIf this changes your perspective on our work, please consider changing your score. Large empirical studies like this require publication to continue securing compute funding.\n\n[1] Engstrom, Logan, et al. \"Implementation matters in deep policy gradients: A case study on ppo and trpo.\" arXiv preprint arXiv:2005.12729 (2020).\n\n[2] Andrychowicz, Marcin, et al. \"What matters in on-policy reinforcement learning? a large-scale empirical study.\" arXiv preprint arXiv:2006.05990 (2020).\n\n[3] Huang, Shengyi, et al. \"A2C is a special case of PPO.\" arXiv preprint arXiv:2205.09123 (2022)."}, {"review_id": "AcTGd1HtCM", "reviewer": "Reviewer_WmdD", "summary": "This paper aims to assess the generality of DreamerV3's stabilization techniques by adapting them to the widely-used model-free algorithm PPO. The authors then proceed to conduct a thorough ablation study across Atari games and the DeepMind Control Suite to evaluate their effectiveness.", "questions": "The paper is coherent and does not present any confusion, so I don't have any specific questions. I am open to discussing the impact further, as it is undoubtedly challenging to evaluate in advance.", "limitations": "The authors discussed the scientific limitations of their work. The societal impact is not discussed, but not very relevant in this case I believe.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- The paper is well written and easy to follow.\n- The obtained findings are intriguing and highlight the need for a more comprehensive ablation study of these tricks in the model-based context of DreamerV3.\n- Demonstrating a negative result poses a challenge as poor outcomes can be attributed to implementation errors. In this regard, the authors have done a rather good job:\n    - They chose PPO, an extensively studied algorithm, and conducted experiments based on the widely-accepted CleanRL implementation.\n    - They perform add-one and drop-one ablations across the complete set of DMC environments and Atari games, while adhering to the recommended Atari wrappers. Multiple seeds were used, and the results were reported using the recommended metrics from RLiable.\n    - They rely on the open source implementation of DreamerV3 and communicated with DreamerV3’s authors to implement theses tricks. The few differences with DreamerV3 are well-explained and justified.\n    - The code is open source, which enables the community to further validate their study and potentially build upon it.", "weaknesses": "- Originality: The sole innovation of the paper lies in the adaptation of these tricks to PPO, which, on its own, may not be considered a significant contribution.\n- Impact: While the experiments demonstrate technical rigor and the publication of negative results is valuable, the overall impact of the paper appears to be relatively limited.\n- Minor writing details:\n    - There are instances in the bibliography where names with accents have been incorrectly rendered.\n    - Some references should probably be included (L120, relevant papers on Twohot encoding for distributional RL; L148, explicit the previous work, etc).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review of our work. We have no objections and will fix the minor writing details, thank you for pointing those out. As a point of discussion, we would like to add some context as to the impact of our work. The DreamerV3 paper was received with significant and sustained academic interest. Every other RL researcher we spoke to believed that the algorithmic tricks introduced in the paper would be immediately applicable to other algorithms. Our work indicates that this is not the case. As you mentioned, negative results require extreme care in implementation, and this study required a combination of extensive testing, code review by all of the authors, and correspondence with the DreamerV3 authors. Finally, we’d like to emphasize that a major contribution of this work is the extensive ablation studies, as well as a detailed study of two-hot encoding and symlog predictions specifically, not only their adaption to PPO.\n\nIf the above changes your impression of the importance of this analysis and the difficulty and expense of conducting it, please consider updating your score."}, {"review_id": "QhJbd9VOvo", "reviewer": "Reviewer_QvZz", "summary": "This paper explores implementation details from Dreamer-v3 in the context of PPO to examine the generality of these implementation tricks to DeepRL. They identify 5 unique implementation details from Dreamer-v3 and do a thorough examination of its effects on both continuous and discrete control environments (i.e. Deepmind Control Suite and Atari respectively). They offer solid evidence for and against certain implementation tricks in the context of PPO and find that environments with poorly normalized, unbounded rewards were where these identified tricks helped most.", "questions": "My questions align with the points raised in the \"weaknesses\" section:\n* Other than the reward clipping ranges, did you find these implementation tricks very sensitive to hyperparameter?\n* Where there other insights about types of environments within a benchmark suite and its relationship to the implementation tricks?\n* What was the main rationale for choosing PPO over A2C when applying Dreamerv3 tricks? How many of implementation details do you think were specific to World Model training?", "limitations": "Yes, they have adequately addressed the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "### Originality\n---\nThis paper follows a line of literature that investigates implementation tricks for DeepRL algorithms. I believe its valuable and necessary research given the large amount of implementation tricks in DeepRL. It was interesting and original for the authors to specifically study tricks from an on-policy model-based DeepRL algorithm to the model-free DeepRL algorithm. \n\n### Quality and Clarity\n---\nI believe they did a very thorough evaluation on very large benchmark domains. It is impressive that they were able to do both all of Atari and DMC control. The writing was clear and I believe the authors were very clear about what they were investigating and presenting from the very beginning. \n\n### Significance\n---\nThe per-environment learning curves from the ablations and experiments for both DMC and Atari (in the supplementary) is a significant contribution that democratizes this information to many researchers who do not have great compute access. Although there wasn't a strong clear implementation trick that drastically improved performance, across DMC and Atari, I commend the authors for reporting all both what worked well and not well which is very useful for practitioners moving forward. ", "weaknesses": "* It would be great to see hyperparameter ranges tested for each implementation detail. Perhaps this is already done through the code-release but would be nice to have in writing with some insight from the authors\n* Perhaps diving a bit deeper into subcategories of tasks within the benchmark suites and doing some analysis there. For example, some environments are long horizon, sparse/dense reward, small/large action space, etc. Given all of the per-environment data already documented and available, it would be great if the authors could provide some insight on why some implementation tricks worked/didn't work based upon features of the task. \n* Some more insight on which implementation tricks were more tailored towards World Model learning and specifically Dreamer's actor-critic learning. It would be interesting to implement these implementation tricks with A2C rather than PPO to test whether these tricks were indeed for the world model or for the RL optimization. (I would suggest PPO for Dreamer-V3 but that also sounds very non-trivial).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to thoroughly consider our work. Based on your feedback, we will make the following clarifications and additions to the paper:\n\nHyperparameters: Symlog does not have any hyperparameters and we don’t experiment with changing the unimix ratio from 0.01. The Critic EMA did not seem particularly sensitive to its hyperparameters in our tests.\n\nIn our paper, we already explore the hyperparameters for two-hot (number of bins and the range of values represented by those bins) in section 4.5. We also mentioned in section 3.4 that percentile scaling was very sensitive to its EMA decay rate, and we had to increase it slightly from the default in DreamerV3 to avoid some learning regressions. We will explain that this may be related to the fact that DreamerV3 and PPO calculate returns at different points, so they likely update this EMA at different rates. \n\nTask Subcategories: We did not observe any performance correlations with the well-known taxonomies of Atari games (easy/hard exploration, dense/sparse rewards). There also does not appear to be a correlation with reward sparsity in individual DMC environments. It seems that in general symlog predictions perform better on environments with relatively large reward ranges, but not as consistently as one might expect. While there is no clear property that predicts the performance of the implementation tricks, we hope others will find the plots to be valuable references for their own work.\n\nWorld Models: Free bits and the world model regularizer introduced by DreamerV3 are only applicable to model-based methods. The remaining tricks apply to some components of actor-critic learning. It is also possible that some of the actor-critic tricks perform better in a model-based system. In particular, the percentile scaling and critic EMAs might have a greater effect when returns are generated by the model instead of the environment. We have no reason to suspect that two-hot encoding, symlog predictions/observations, or unimix categoricals would perform better in a model-based context. Unfortunately testing this explicitly would require extensive changes to the DreamerV3 codebase or more likely a full reimplementation.\n\nQuestion: We do not fully understand your suggestion of using A2C, given that it is just a special case of PPO with no sample reuse and some minor implementation detail differences. Is your intent that we should run on a simpler RL algorithm without the bells and whistles of PPO? Or did you perhaps mean to suggest a different algorithm that includes world modeling?"}]

Author rebuttal:
Thank you to all of the reviewers for their comments and suggestions about our work. We’d like to address our choice of PPO because several reviewers suggested other potential starting algorithms for this work (DreamerV3, A2C, SAC).

To start, this work is not a reproduction of DreamerV3 but instead an attempt to transfer them to PPO and a careful study of their interactions. The tricks introduced by DreamerV3 were designed to stabilize actor-critic learning so we chose PPO, the most popular actor-critic algorithm. A2C is another viable choice, but it is much less popular than PPO and can be implemented as a special case of PPO [1], so PPO was the clear choice between the two. 

SAC is another viable option. DreamerV3 borrows some ideas from SAC, such as its entropy regularized objective. However, SAC trains two Q functions in place of a typical V function critic, which makes the implementation of many tricks less straightforward. SAC and DreamerV3 also require a forward and backward pass for each environment step, causing them to train slower (in terms of wall-clock time) than other actor-critic algorithms.

PPO is the most popular RL algorithm because of its simplicity and efficacy in a broad range of environments. Unlike other algorithms, it has been used at scale in Dota2, silicon chip design, and ChatGPT. PPO is an improvement to REINFORCE, the policy gradient algorithm that DreamerV3 uses for policy updates, and uses the same standard V function critic as DreamerV3. PPO also trains faster than DreamerV3 and SAC. Ultimately we felt that transferring the tricks to PPO would require the least modifications to the tricks introduced by DreamerV3 and have the highest impact.

[1] Huang, Shengyi, et al. "A2C is a special case of PPO." arXiv preprint arXiv:2205.09123 (2022).

Meta-review: A detailed empirical investigation on a relevant algorithm (PPO with DreamerV3 tricks), executed decently, should be broadly useful to the community even if the results are not surprising. Its main caveat is not reproducing the actual DreamerV3 baseline within the same setting, hence leaving some reviewers with residual doubts even after the rebuttal.
Decision: Accept (poster)