Beyond The Rainbow: High Performance Deep Reinforcement Learning On A Desktop PC

## Abstract

Abstract Rainbow Deep Q-Network (DQN) demonstrated combining multiple independent enhancements could significantly boost a reinforcement learning (RL) agent’s performance. In this paper, we present “Beyond The Rainbow” (BTR), a novel algorithm that integrates six improvements from across the RL literature to Rainbow DQN, establishing a new state-of-the-art for RL using a desktop PC, with a human-normalized interquartile mean (IQM) of 7.4 on Atari-60. Beyond Atari, we demonstrate BTR’s capability to handle complex 3D games, successfully training agents to play Super Mario Galaxy, Mario Kart, and Mortal Kombat with minimal algorithmic changes. Designing BTR with computational efficiency in mind, agents can be trained using a desktop PC on 200 million Atari frames within 12 hours. Additionally, we conduct detailed ablation studies of each component, analyzing the performance and impact using numerous measures.

## 1 Introduction

Deep Reinforcement Learning (RL) has achieved numerous successes in complex sequential decision-making tasks, most rapidly since Mnih et al. () proposed Deep Q-Learning (DQN). With this success, RL has become increasingly popular among smaller research labs, the hobbyist community, and even the general public. However, recent state-of-the-art approaches (Schrittwieser et al., ; Badia et al., ; Hessel et al., ; Kapturowski et al., ) are increasingly out of reach for those with more limited compute resources, either in terms of the required hardware or the walltime necessary to train a single agent. This is a unique issue in RL compared to natural language processing or image recognition which have foundation models that can be efficiently fine-tuned for a new task or problem (Lv et al., ). Meanwhile, RL agents must be trained afresh for each environment. Therefore, the development of powerful RL algorithms that can be trained quickly on inexpensive hardware is crucial for smaller research labs and the hobbyist community.

These concerns are not new. Ceron & Castro () highlighted that Rainbow DQN (Hessel et al., ) required 34,200 GPU hours (equivalent to 1435 days) of training, making the research impossible for anyone except a few research labs, with more recent algorithms exacerbating this problem. Recurrent network architectures (Horgan et al., ), high update to sample ratio (D’Oro et al., ), and the use of world-models and search-based techniques (Schrittwieser et al., ) all increase the computational resources necessary to train agents, many using distributed approaches requiring multiple CPUs and GPUs (or TPUs), or requiring numerous days and weeks to train a single agent. These features have dramatically decreased RL’s accessibility.

For this purpose, we develop “Beyond the Rainbow” (BTR), taking the same principle as Rainbow DQN Hessel et al. (), selecting 6 previously independently evaluated improvements and combining them into a singular algorithm (Section 3). These components were chosen for their performance qualities or to reduce the computational requirements for training an agent. As a result, BTR sets a new state-of-the-art score for Atari-60 (Bellemare et al., ) (excluding recurrent approaches) with an Interquartile Mean (IQM) of 7.4(^1^11All reported IQM scores use the best single evaluation for each environment throughout training as is standard, rather than the agent’s score at 200 million, hence the discrepancy between the overall score and Figure 1.) using a single desktop machine in less than 12 hours, and outperforms Rainbow DQN on Procgen (Cobbe et al., ) in less than a fifth of the walltime (Section 4.1). Further, we demonstrate BTR’s potential by training agents to solve three modern 3D games for the first time, Mario Kart Wii, Super Mario Galaxy and Mortal Combat, that each contain complex mechanics and graphics (Section 4.2). To verify the effectiveness and effect of the six improvements to BTR, in Section 5.1, we conduct a thorough ablation of each component, plotting their impact on the Atari-5 environments and in Section 5.2, we utilise seven different measures to analyse the component’s impact on the agent’s policy and network weights. This allows us to more precisely understand how the components impact BTR beyond performance or walltime.

## 2 Background

Before describing BTR’s extensions, we outline standard RL mathematics, how DQN is implemented, and Rainbow DQN’s extensions.

### 2.1 RL Problem Formulation

We adopt the standard formulation of RL (Sutton & Barto, ), described as a Markov Decision Process (MDP) defined by the tuple $\mathcal{(S,A,P,R)}$, where $\mathcal{S}$ is the set of states, $\mathcal{A}$ is the set of actions, $\mathcal{P:S\times A\rightarrow}\Delta\mathcal{(S)}$ is the stochastic transition function, and $\mathcal{R:S\times A\rightarrow}\mathbb{R}$ is the reward function. The agent’s objective is to learn a policy $\pi:S\rightarrow\Delta\mathcal{(A)}$ that maximizes the expected sum of discounted rewards $\mathbb{E}_{\pi}[\sum^{\infty}_{t=0}\gamma^{t}r(s_{t},a_{t})]$, where $\gamma\in[0,1)$ is the discount rate.

### 2.2 Deep Q-Learning (DQN)

One popular method for solving MDPs is Q-Learning (Watkins & Dayan, ) where an agent learns to predict the expected sum of discounted future rewards for a given state-action pair. To allow agents to generalize over states and thus be applied to problems with larger state spaces, Mnih et al. () successfully combined Q-Learning with neural networks. To do this, training minimizes the error between the predictions from a parameterized network $\mathrm{Q}_{\theta}$ and a target defined by

$$ $r_{t}+\gamma\max_{a\in A}\mathrm{Q_{\theta^{{}^{\prime}}}}(s_{t+1},a)\;,$ (1) $$

where $Q_{\theta^{{}^{\prime}}}$ is an earlier version of the network referred to as the target network, which is periodically updated from the online network $\mathrm{Q}_{\theta}$. The data used to perform updates is gathered by sampling from an Experience Replay Buffer (Lin, ), which stores states, actions, rewards, and next states experienced by the agent while interacting with the environment.

### 2.3 Rainbow DQN and Improvements to DQN

In collecting 6 different improvements to DQN, Rainbow DQN (Hessel et al., ) proved cumulatively that these improvements could achieve a greater performance than any individually. We briefly explain the individual improvements, ordered by performance impact, most of which are preserved within BTR (see Table 1), for more detail, we refer readers to the extension’s respective papers:

- 1.
Prioritized Experience Replay - To select training examples, DQN sampled uniformly from an Experience Replay Buffer, assuming that all examples are equally important to train with. Schaul et al. () proposed sampling training examples proportionally to their last seen absolute temporal difference error, encouraging more training on samples for which the network most inaccurately predicts their future rewards.
- 2.
N-Step - Q-learning utilizes bootstrapping to minimize the difference between the predicted value and the resultant reward plus the maximum value of the next state (Eq. 1). N-step (Sutton et al., ) reduces the reliance on this bootstrapped next value by considering the next $n$ rewards and observation in $n$ timesteps (Rainbow DQN used $n=3$).
- 3.
Distributional RL - Due to the stochastic nature of RL environments and agent’s policies, Bellemare et al. () proposed learning the return distribution rather than scalar expectation; this was done through modelling the return distributions using probability masses and the Kullbeck-Leibler divergence loss function.
- 4.
Noisy Networks - Agents can often insufficiently explore their environment resulting in sub-optimal policies. Fortunato et al. () added parametric noise to the network weights, causing the model’s outputs to be randomly perturbed, increasing exploration during training, particularly for states where the agent has less confidence.
- 5.
Dueling DQN - The agent’s Q-value can be rewritten as the sum of state-value and advantage ($Q(s,a)=V(s)+A(s,a)$). Looking to improve action generalisation, Wang et al. () split the hidden layers into two separate streams for the value and advantage, recombining them with $Q(s,a)=V(s)+(A(s,a)-\frac{1}{|\mathcal{A}|}\sum_{a^{\prime}}A(s,a^{\prime}))$.
- 6.
Double DQN - In selecting the next observation’s maximum Q-value (Eq. 1), this can frequently overestimate the target’s Q-value, negatively affecting the agent’s performance. To reduce this overestimation, Van Hasselt et al. () propose utilising the online network rather than the target network to select the next action when forming targets, defined as:
$r_{t}+\gamma\mathrm{Q_{\theta^{{}^{\prime}}}}(s_{t+1},\operatorname*{arg\,max}
_{a\in A}\mathrm{Q_{\theta}}(s_{t+1},a))\;.$
(2)

## 3 Beyond the Rainbow - Extensions and Improvements

Building on Rainbow DQN (Hessel et al., ), BTR includes 6 more improvements undiscovered in 2018.(^2^22After the completion of our work, we additionally found Layer Normalization applied after the stem of each residual block and between dense layers to be beneficial (see Appendix I for a discussion)) Additionally, as hyperparameters are critical to agent performance, Section 3.2 discusses key hyperparameters and our choices. In the appendices, we include a table of hyperparameters, a figure of the network architecture and the agent’s loss function (Appendices C.2, D and D.2). Finally, the source code using Gymnasium (Towers et al., ) is included within the supplementary material to help future work build upon or utilise BTR.

**Table 1: A comparison of components between Rainbow DQN (Hessel et al., ) and BTR.**
| Added To Rainbow DQN | Same As Rainbow DQN | Removed From Rainbow DQN |
| --- | --- | --- |
| Impala (Scale=2) | N-Step TD Learning | Double (N/A with Munchausen) |
| Adaptive Maxpooling (6x6) | Prioritized Experience Replay | C51 (Upgraded to IQN) |
| Spectral Normalisation | Dueling |  |
| Implicit Quantile Networks | Noisy Networks |  |
| Munchausen |  |  |
| Vectorized Environments |  |  |

### 3.1 Extensions

Impala Architecture + Adaptive Maxpooling - Espeholt et al. () proposed a convolutional residual neural network architecture based on He et al. () featuring three residual blocks(^3^33The network architecture is referred to as Impala due to the accompanying training algorithm IMPALA proposed in Espeholt et al. ()), substantially increasing performance over DQN’s three-layer convolutional network. Following Cobbe et al. (), we
scale the width of the convolutional layers by 2 to enhance its capabilities. We include an additional 6x6 adaptive max pooling layer after the convolutional layers (Schmidt & Schmied, ) found to speed up learning and support different input resolutions.

Spectral Normalisation (SN) - To help stabilize the training of discriminators in Generative Adversarial Networks (GANs), Miyato et al. () proposed Spectral Normalisation to help control the Lipschitz constant of convolutional layers. SN works to normalize the weight matrices of each layer in the network by their largest singular value, ensuring that the transformation applied by the weights does not distort the input data excessively, which can lead to instability during training. Bjorck et al. () found that SN could improve performance in RL, especially for larger networks and Schmidt & Schmied () found SN reduced the number of updates required before initial progress is made.

Implicit Quantile Networks (IQN) - Dabney et al. () improved upon Bellemare et al. (), learning the return distribution over the probability space rather than probability distribution over return values. This removes the limit on the range of Q-values that can be expressed, and enables learning the expected return at every probability.

Munchausen RL - Boostrapping is a core aspect of RL; used to calculate target values (Eq. 1) with most algorithms using the reward, $r_{t}$, and the optimal Q-value of the next state, $Q^{*}$. However, since in practice the optimal policy is not known, the current policy $\pi$ is used. Munchausen RL (Vieillard et al., ) looks to leverage an additional estimate in the bootstrapping process by adding the scaled-log policy to the loss function (Eq. 3 where $\alpha\in[0,1]$ is a scaling factor, $\sigma$ is the $\operatorname{softmax}$ function, and $\tau$ is the softmax temperature). This assumes a stochastic policy, therefore DQN is converted to Soft-DQN with $\text{with }{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}
\pi_{\theta^{\prime}}=\sigma(\frac{Q_{\theta^{\prime}}}{\tau})}$. As Munchausen does not use $\operatorname{argmax}$ over the next state, Double DQN is obsolete. Munchausen RL’s update rule is

$$ $Q_{\theta}(s_{t},a_{t})=r_{t}+{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}\alpha\tau\ln\pi_{\theta^{\prime}}(a_{t}|s_{t})}+ \gamma\sum_{a^{\prime}\in A}{\pi_{\theta^{\prime}}(a^{\prime}|s_{t+1})(Q_{ \theta^{\prime}}(s_{t+1},a^{\prime})-{\color[rgb]{0,0,1}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,1}\tau\ln(\pi_{\theta^{\prime}}(a^{\prime}|s_{t+1}))} }\;.$ (3) $$

Vectorization - RL agents typically take multiple steps in a single environment, followed by a gradient update with a small batch size (Rainbow DQN took 4 environment steps, followed by a batch of 32). However, taking multiple steps in parallel and performing updates on larger batches can significantly reduce walltime. We follow Schmidt & Schmied (), taking 1 step in 64 parallel environments with one gradient update with batch size 256 (Schmidt & Schmied () took two gradient updates rather than the one we take). This results in a replay ratio (ratio of gradient updates to environment steps) of $\frac{1}{64}$. Higher replay ratios have been shown to improve performance (D’Oro et al., ), however we opt to keep this value low to reduce walltime.

### 3.2 Hyperparameters

How frequently the target network is updated is closely intertwined with batch size and replay ratio. We found that updating the target network every 500 gradient steps(^4^44This equates to 32,000 environment steps (128,000 frames), compared to Rainbow DQN’s 8,000 steps.) performed best. Given our high batch size, we additionally performed minor hyperparameter tests using different learning rates finding that a slightly higher learning rate of $1\text{\times}{10}^{-4}$ performed best, compared to $6.25\text{\times}{10}^{-5}$ in Rainbow DQN. In Appendix C.2, we clarify the meaning of the terms frames, steps and transitions.

For many years, RL algorithms have used a discount rate of $0.99$, however, when reaching high performance, lower discount rates alter the optimal policy, causing even optimally performing agents to not collect the maximum cumulative rewards. To prevent this, we follow MuZero Reanalyse (Schrittwieser et al., ) using $\gamma=0.997$. For our Prioritized Experience Replay, we use the lower value of $\alpha=0.2$, the parameter used to determine sample priority, recommended by Toromanoff et al. () when using IQN. Lastly, many previous experiments used only noisy networks or $\epsilon$-greedy exploration, however, we opt to use both until 100M frames, then set $\epsilon$ to zero, effectively disabling it. We elaborate on this decision in Appendix G.

## 4 Evaluation

To assess BTR, we test it on two standard RL benchmarks, Atari (Bellemare et al., ) and Procgen (Cobbe et al., ) in Section 4.1. Secondly, we train BTR agents for three modern games (Super Mario Galaxy, Mario Kart Wii, and Mortal Combat) with complex 3D graphics and physics in Section 4.2, never shown to be trainable with RL previously.

### 4.1 Atari and Procgen Performance

We evaluate BTR on the Atari-60 benchmark following (Machado et al., ) and without life information (see Appendix J for the impact), evaluating every million frames on 100 episodes. Figure 1 plots BTR against Rainbow DQN and DQN, achieving an IQM of 7.4 compared to Rainbow DQN’s 2.7 and DQN’s 0.9. In comparison to human expert performance, BTR equals or exceeds them in 52 of 60 and Rainbow DQN in 57. Importantly, we find that BTR appears to continue increasing performance beyond 200 million frames, indicating that higher performance is still possible with more time and data. Results tables and graphs can be found in Appendices A and B respectively.

Figure: Figure 2: IQM min-max Performance of BTR compared to Rainbow DQN + Impala (width x4) Cobbe et al. () after 200M frames on all 16 Procgen Environments.
Refer to caption: x2.png

To further confirm BTR’s performance, we benchmark on Procgen (Cobbe et al., ), a procedurally generated set of environments aiming to prevent overfitting to specific tasks, a prevalent problem in RL (Justesen et al., ; Juliani et al., ). The results are shown in Figure 2 with individual games in Appendix B. BTR is able to exceed Rainbow DQN + Impala’s performance, despite using significantly fewer convolutional filters (which Cobbe et al. () found to significantly improve performance) and using 8 hours of walltime compared to 41. These results demonstrate BTR’s general learning capability across a wide range of standard RL benchmarks.

### 4.2 Applying BTR to Modern Games

To demonstrate BTR’s capabilities beyond standard RL benchmarks, we utilised Dolphin (Dolphin-Emulator, ), a Nintendo Wii emulator, to train agents for a range of modern 3D games: Super Mario Galaxy, Mario Kart Wii and Mortal Combat. Using a desktop PC, we were able to train the agent to complete some of the most difficult tasks within each game. Namely, the final level in Super Mario Galaxy, Rainbow Road (a notoriously difficult track in Mario Kart Wii) and defeating all opponents in Mortal Kombat Endurance mode (for details about the environments and setup, see Appendix K). For this, BTR required minimal adjustments: first, to input image resolution, 140x114 (from Atari’s 84x84) due to the game’s higher resolution and aspect ratio, and second, to reduce the number of vectorized environments to 4 as a result of the games’ memory and CPU requirements.

Figure: Figure 3: BTR being used to play Super Mario Galaxy (final level), Mario Kart Wii (Rainbow Road) and Mortal Kombat: Armageddon (Endurance Mode) respectively.
Refer to caption: extracted/5939675/GamesFigureNoNFS.png

For all the games, BTR was able to solve the game level, including consistently finishing in first place in Mario Kart. We provide videos of our agent playing all three Wii games, in addition to the games in the Atari-5 benchmark (^5^55[https://www.youtube.com/playlist?list=PL4geUsKi0NN-sjbuZP_fU28AmAPQunLoI](https://www.youtube.com/playlist?list=PL4geUsKi0NN-sjbuZP_fU28AmAPQunLoI)).

## 5 Analysis

Given BTR’s performance demonstrated in Section 4, in this Section, we ablate each component to evaluate their performance impact (Section 5.1). Using the ablated agents, we then measure numerous attributes during and after training to assess each component’s impact (Section 5.2).

### 5.1 Ablations Studies

BTR amalgamates independently evaluated components into a single algorithm. To understand and verify each’s contribution, Figures 4 and B1 plot BTR’s performance without each component on the Atari-5 benchmark.(^6^66Due to computational resources required to evaluate each component on 60 environments, Aitchison et al. () proposes a subset of 5 games that closely correlate with the performance across all 60.)

Figure: Figure 4: Individual game performance of BTR on Atari-5 with individual components removed averaged over 3 seeds. Shaded areas show 1 standard deviation.
Refer to caption: x3.png

We find that Impala had the largest effect on performance, with the other components generally causing a less significant effect on final performance. However, when BTR’s performance is compared before 200 million frames, we find Munchausen and Spectral Normalisation provide significant performance improvements ($+24\%$ and $+25\%$ at 40M frames, and $+13\%$ and $+35\%$ at 120M frames). We compare the performance of components at different stages of training in Appendix E.

For vectorization and maxpooling, while their inclusion reduces performance, we find their secondary effects crucial to keep BTR computationally accessible. Omitting vectorization increases walltime by 328% (Figure 5) by processing environment steps in parallel and taking fewer gradient steps (781,000 compared to Rainbow DQN’s 12.5 million).(^7^77Another consequence of removing vectorization is using smaller batches, which Obando Ceron et al. () finds improves exploration, possibly explaining our results found in Qbert.) We find maxpooling makes the agent more robust to noise as discussed in Section 5.2, and decreases the model’s parameters by 77%.

Figure: Figure 5: Walltime of BTR on a desktop PC with components removed, compared with Hessel et al. () and Schmidt & Schmied (). For hardware details, see Appendix H.
Refer to caption: x4.png

### 5.2 What are the effects of BTR’s components?

To help interpret the results in Section 5.1, we measure seven different attributes of the agent either during or after training: action gaps and action swaps, linked to causing approximation errors (Bellemare et al., ); policy churn, which can cause excessive off-policyness (Schaul et al., ); score with additional noise indicating robustness and weight norm and SRank (Sokar et al., ), correlated with loss of plasticity.

For why Impala contributes to performance so strongly, Figure 6 shows that with Impala, BTR has fewer dormant neurons and higher weight matrix norms. Despite this, without BTR’s other components, Impala exhibits a notable drawback, learning a highly noisy and unstable policy. Table 2, demonstrates that without IQN and Munchausen the agent experiences very low action gaps (absolute Q-value difference between the highest two valued actions), causing the agent to swap its argmax action almost every other step. This is likely to result in approximation errors altering the policy and causing a high degree of off-policyness in the replay buffer. This is particularly detrimental in games requiring fine-grained control, such as Phoenix where the agent needs to narrowly dodge many projectiles, reflected in BTR’s performance without these components.

Furthermore, we find that spectral normalization and maxpooling are useful in dealing with noisy environments. To test this, we evaluate the performance of BTR’s ablations when taking different quantities of $\epsilon$-actions, and find these two components prevent performance from dropping substantially 2. Lastly, we find Munchausen and IQN to have a significant impact on Policy Churn (Schaul et al., ), with Munchausen reducing it by 5.6% and IQN increasing it by 2.4%. As a result, when these components are used together, they appear to reach a level of policy churn which does not harm learning and potentially provides some exploratory benefits.

**Table 2: Comparison of policy churn, action gaps, actions swaps and evaluation performance with different quantities of $\epsilon$-actions and color jitter (both only applied for evaluation). All measurements use the final agent, trained on 200 million frames, for Atari Phoenix. Action Gap is the average absolute Q-value difference between the highest two valued actions. % Actions Swap is the percentage of times the agent’s argmax action has changed from the last timestep. Policy churn is the percentage of states which the agent’s argmax action has changed on after a single gradient step. Color jitter applies a random 10% change to the brightness, saturation and hue of each frame.**
| Category | BTR | w/o Munchausen | w/o IQN | w/o SN | w/o Impala | w/o Maxpool |
| --- | --- | --- | --- | --- | --- | --- |
| Action Gap | 0.276 | 0.060 | 0.154 | 0.280 | 0.403 | 0.282 |
| % Action Swaps | 33.9% | 47.0% | 45.2% | 42.2% | 30.9% | 39.7% |
| Policy Churn | 2.9% | 8.5% | 0.5% | 2.1% | 3.5% | 2.8% |
| Score ColorJitter | 196k | 90k | 97k | 167k | 5k | 149k |
| Score $\epsilon=0.03$ | 96k | 53k | 49k | 86k | 5k | 72k |
| Score $\epsilon=0.01$ | 211k | 88k | 89k | 173k | 5k | 138k |
| Score $\epsilon=0$ | 351k | 211k | 196k | 366k | 5k | 500k |

## 6 Related Work

The most similar work to BTR, developing a computationally-limited non-distributed RL algorithm, is “Fast and Efficient Rainbow” (Schmidt & Schmied, ). They optimised Rainbow DQN to maximise performance for 10 million frames through parallelizing the environments and dropping C51 along with hyperparameter optimisations. This differs from our goals of producing an algorithm that scales across training regimes (up to 200 million frames) and domains (Atari, Procgen, Super Mario Galaxy, Mario Kart and Mortal Combat), resulting in different design decisions.

For less computation-limited approaches, Ape-X (Horgan et al., ) was the first to explore highly distributed training, allowing agents to be trained on a billion frames in 120 hours through using   100 CPUs. Following this, Kapturowski et al. () proposed R2D2 using a recurrent neural network, increasing sample efficiency but slowing down gradient updates by 38%. Agent57 (Badia et al., ) was the first RL agent to achieve superhuman performance across 57 Atari games, though required 90 billion frames. MEME (Kapturowski et al., ), Agent57’s successor, focused on achieving superhuman performance within the standard 200 million frames limit, achieved by used a significantly higher replay ratio and larger network architecture. Most recently, Dreamer-v3 (Hafner et al., ) used a 200 million parameter model requiring over a week of training, achieving similar results as MEME. We detail some of the key differences between BTR, MEME and Dreamer-v3 in Table 3. While these approaches perform equally to or better than BTR, all are inaccessible to smaller research labs or hobbyists due to their required computational resources and walltime. Therefore, while these algorithms have important research value demonstrating the possible performance of RL agents, performative algorithms with a lower cost of entry, like BTR, are a necessary component for RL to become widely applicable and accessible.

Figure: Figure 6: Plot showing L2 norm of network weights, SRank $\delta=0.01$ (Kumar et al., ) and % of dormant neurons (dormant defined as $<$ 0.1 (Sokar et al., ), for details see Appendix F).
Refer to caption: x5.png

**Table 3: Comparison of performance, walltime, observations and complexity of different algorithms.**
| Category | BTR | MEME | Dreamer-v3 |
| --- | --- | --- | --- |
| A100 GPU Days | 0.9 | Not Reported | 7.7 |
| Recurrent? | No (4 stacked frames) | Yes | Yes |
| Learns from? | Single Transitions | Trajectories (length 160) | Trajectories (length 64) |
| World Model | No | No | Yes |
| Parameters | 2.9M | Not Reported ($\approx>$20M) | 200M |
| Observation Shape | 84x84 | 210x160 | 64x64 |
| Gradient Steps | 781K | 3.75M | 1.5M |
| Atari-60 IQM | 7.4 | 9.6 | 9.6 |

## 7 Conclusion and Future work

We have demonstrated that, once again, independent improvements from across Deep Reinforcement Learning can be combined into a single algorithm capable of pushing the state-of-the-art far beyond what any single improvement is capable of. Importantly, we find that this can be accomplished on desktop PCs, increasing the accessibility of RL for smaller research labs and hobbyists.

We acknowledge that there are many more promising improvements we were not able to include in BTR, leaving room for more future work in a few years to create even stronger integrated agents. For example, BTR does not add an explicitly exploration component, resulting in it struggling in hard-exploration tasks such as Montezuma’s Revenge; therefore, mechanisms used in Never Give Up (Badia et al., ) or other components may prove useful. Section 5.1 found that the neural network’s core architecture, Impala, had the largest impact on performance, an area we believe is generally underappreciated in RL. Previous work (Kapturowski et al., ) has incorporated recurrent models enhancing performance, though we are uncertain how this can be incorporated into BTR without affecting its computational accessibility.

## 8 Ethics and Reproducibility Statements

Our work does not involve human subjects or methodologies with direct ethical concerns such as discrimination, bias, or privacy violations. Additionally, we have no conflicts of interest, sponsorship issues, or violations of legal or research integrity were present during the development of this research. However, we acknowledge that by improving the accessibility and performance of Reinforcement Learning (RL), our contributions may inadvertently provide more powerful tools to malicious actors, thus, we urge the research community to remain vigilant regarding these issues.

To ensure reproducibility, we provide a detailed background of the work we build upon and clearly explain all changes made to the base algorithm. Furthermore, Appendix C.2 provides all relevant environmental and algorithmic hyperparameters needed to reproduce our work. Additionally, we provide clarity about often misunderstood terms (Appendix C.3), a detailed architecture diagram (Appendix D) and the exact hardware we tested our algorithms on (Appendix H). Most importantly, we provide BTR’s code within the supplementary material. Lastly, we provide many details regarding the Wii games tested BTR on, including the minor changes from BTR, how the environment was setup and the reward functions used.

## Appendix A Full Results Tables

**Table A1: Maximum scores obtained during training (averaged over 100 episodes and all performed using random seeds) after 200M Frames on the Atari-60 benchmark. We compared against other non-recurrent algorithms where FE-Rainbow DQN refers to Fast and Efficient Rainbow DQN (Schmidt & Schmied, ), and M-IQN refers to Munchausen-IQN (Vieillard et al., ).**
| Game | Random | Human | Rainbow DQN | Rainbow DQN | M-IQN | FE-Rainbow DQN(^8^88FE-Rainbow DQN uses Life Information, but only 10M frames.) | BTR |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | (Dopamine) | (Full) |  |  |  |
| AirRaid | 400 | 3000 | 14056 |  | 19111 |  | 53544 |
| Alien | 228 | 7128 | 3587 | 9492 | 4492 | 12508 | 19149 |
| Amidar | 6 | 1720 | 2630 | 5131 | 1875 | 2071 | 17808 |
| Assault | 222 | 742 | 3511 | 14198 | 7504 | 10710 | 19385 |
| Asterix | 210 | 8503 | 18367 | 428200 | 49865 | 346758 | 593650 |
| Asteroids | 719 | 47389 | 1489 | 2713 | 1685 | 12346 | 169272 |
| Atlantis | 12850 | 29028 | 838590 | 826660 | 918183 | 812826 | 899104 |
| BankHeist | 14 | 753 | 1148 | 1358 | 1292 | 1411 | 1598 |
| BattleZone | 2360 | 37188 | 40895 | 62010 | 52517 | 112652 | 168340 |
| BeamRider | 364 | 16926 | 6529 | 16850 | 12775 | 26399 | 138102 |
| Berzerk | 124 | 2630 | 842 | 2546 | 736 | 3388 | 6704 |
| Bowling | 23 | 161 | 49 | 30 | 32 | 41 | 48 |
| Boxing | 0 | 12 | 99 | 100 | 99 | 100 | 100 |
| Breakout | 2 | 30 | 120 | 418 | 320 | 538 | 677 |
| Carnival | 380 | 4000 | 5069 |  | 5588 |  | 6031 |
| Centipede | 2091 | 12017 | 6618 | 8167 | 4371 | 8368 | 76243 |
| ChopperCommand | 811 | 7388 | 12844 | 16654 | 4573 | 4208 | 980233 |
| CrazyClimber | 10780 | 35829 | 147743 | 168788 | 150783 | 140712 | 140723 |
| DemonAttack | 152 | 1971 | 17802 | 111185 | 68825 | 131657 | 135448 |
| DoubleDunk | -19 | -16 | 22 | 0 | 22 | -1 | 23 |
| ElevatorAction | 0 | 3000 | 79968 |  | 89237 |  | 82669 |
| Enduro | 0 | 860 | 2230 | 2126 | 2332 | 2266 | 2352 |
| FishingDerby | -92 | -39 | 43 | 31 | 55 | 42 | 56 |
| Freeway | 0 | 30 | 34 | 34 | 34 | 34 | 34 |
| Frostbite | 65 | 4335 | 8572 | 9590 | 9538 | 5282 | 19332 |
| Gopher | 258 | 2412 | 10641 | 70355 | 27469 | 25607 | 99739 |
| Gravitar | 173 | 3351 | 1272 | 1419 | 1134 | 2107 | 5284 |
| Hero | 1027 | 30826 | 46764 | 55887 | 26037 | 15377 | 21560 |
| IceHockey | -11 | 1 | 2 | 1 | 12 | 7 | 38 |
| Jamesbond | 29 | 303 | 1106 |  | 1637 |  | 29828 |
| JourneyEscape | -18000 | -1000 | -959 |  | -806 |  | 5166 |
| Kangaroo | 52 | 3035 | 13460 | 14638 | 10939 | 11499 | 13849 |
| Krull | 1598 | 2666 | 6229 | 8742 | 10703 | 10324 | 11124 |
| KungFuMaster | 258 | 22736 | 27900 | 52181 | 27119 | 27444 | 54330 |
| MontezumaRevenge | 0 | 4753 | 500 | 384 | 0 | 0 | 0 |
| MsPacman | 307 | 6952 | 4027 | 5380 | 6029 | 5982 | 11494 |
| NameThisGame | 2292 | 8049 | 9229 | 13136 | 12761 | 19819 | 28360 |
| Phoenix | 761 | 7243 | 8605 | 108529 | 5327 | 60954 | 350722 |
| Pitfall | -229 | 6464 | -1 | 0 | 0 | -2 | 0 |
| Pong | -21 | 15 | 20 | 21 | 19 | 21 | 21 |
| Pooyan | 500 | 1000 | 5640 |  | 13096 |  | 24280 |
| PrivateEye | 25 | 69571 | 21532 | 4234 | 100 | 254 | 100 |
| Qbert | 164 | 13455 | 18503 | 33818 | 14739 | 25712 | 39484 |
| Riverraid | 1338 | 17118 | 21091 |  | 16271 |  | 24585 |
| RoadRunner | 12 | 7845 | 55300 | 62041 | 61269 | 81832 | 590236 |
| Robotank | 2 | 12 | 66 | 61 | 73 | 71 | 83 |
| Seaquest | 68 | 42055 | 11362 | 15899 | 23885 | 63724 | 409991 |
| Skiing | -17098 | -4337 | -20518 | -12958 | -10336 | -22077 | -9132 |
| Solaris | 1236 | 12327 | 2438 | 3560 | 5765 | 2878 | 8198 |
| SpaceInvaders | 148 | 1669 | 4420 | 18789 | 13871 | 28099 | 53863 |
| StarGunner | 664 | 10250 | 57909 | 127029 | 65757 | 310404 | 574106 |
| Tennis | -24 | -8 | 0 | 0 | 0 | 16 | 24 |
| TimePilot | 3568 | 5229 | 12283 | 12926 | 15155 | 31333 | 110981 |
| Tutankham | 11 | 168 | 245 | 241 | 207 | 167 | 315 |
| UpNDown | 533 | 11693 | 39065 |  | 216080 |  | 397876 |
| Venture | 0 | 1188 | 1579 | 6 | 1101 | 437 | 0 |
| VideoPinball | 0 | 17668 | 513484 | 533936 | 625118 | 269619 | 589066 |
| WizardOfWor | 564 | 4756 | 8201 | 17862 | 13644 | 15519 | 50828 |
| YarsRevenge | 3093 | 54577 | 45567 | 102557 | 111583 | 98908 | 177431 |
| Zaxxon | 32 | 9173 | 15089 | 22210 | 19080 | 18833 | 47096 |
| IQM | 0.000 | 1.000 | 1.994 | 2.718 | 2.609 | 2.687 | 7.419 |
| Median | 0.000 | 1.000 | 1.533 | 2.270 | 1.774 | 1.906 | 4.695 |
| Mean | 0.000 | 1.000 | 4.319 | 7.828 | 6.319 | 7.670 | 19.697 |
| Best | - | - | 1 | 7 | 5 | 3 | 50 |
| >Human | - | - | 45 | 37 | 41 | 38 | 52 |
| Surround | 7 | -10 |  |  |  |  | 10 |
| Defender | 2875 | 18689 |  | 55105 |  | 169930 | 461380 |

**Table A2: Maximum scores obtained during training (averaged over 100 episodes and all performed random seeds) after 200M Frames on the Atari-5 Environment, compared against other non-recurrent non-distributed algorithms. FE-Rainbow refers to Fast and Efficient Rainbow DQN (Schmidt & Schmied, ), and M-IQN refers to Munchausen-IQN (Vieillard et al., ).**
| Game | Random | Human | Rainbow DQN | Rainbow DQN | M-IQN | FE-Rainbow | BTR |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | (Dopamine) | (Full) |  |  |  |
| BattleZone | 2360 | 37188 | 40895 | 62010 | 52517 | 112652 | 151877 |
| DoubleDunk | -19 | -16 | 22 | 0 | 22 | -1 | 23 |
| NameThisGame | 2292 | 8049 | 9229 | 13136 | 12761 | 19819 | 28710 |
| Phoenix | 761 | 7243 | 8605 | 108529 | 5327 | 60955 | 367284 |
| QBert | 164 | 13455 | 18503 | 33818 | 14739 | 25712 | 45034 |
| IQM | 0.000 | 1.000 | 1.265 | 3.583 | 1.452 | 4.070 | 7.627 |
| Median | 0.000 | 1.000 | 1.21 | 2.532 | 1.44 | 3.167 | 4.589 |
| Mean | 0.000 | 1.000 | 3.714 | 5.817 | 3.745 | 4.684 | 16.561 |

## Appendix B Full Results Graph

Figure: Figure B1: Performance of BTR on Atari-5 with individual components removed using Human-Normalized Interquartile Mean. Averaged over 3 seeds.
Refer to caption: x6.png

Figure: Figure B2: Performance of BTR on each individual game in the Atari-60 benchmark.
Refer to caption: x7.png

Figure: Figure B3: Performance of BTR on each individual game in the Procgen benchmark. Shaded areas show one standard deviation of the performed evaluations. The red dotted line shows the performance of Rainbow DQN + Impala with 4x scaled Impala blocks (Cobbe et al., ), after 200M frames.
Refer to caption: x8.png

## Appendix C Hyperparameters

### C.1 Environment Details

**Table C3: Environment Details for Atari Experiments.**
| Hyperparameter | Value |
| --- | --- |
| Grey-Scaling | True |
| Observation down-sampling | 84x84 |
| Frames Stacked | 4 |
| Reward Clipping | [-1, 1] |
| Terminal on loss of life | False |
| Life Information | False |
| Max frames per episode | 108K |
| Sticky Actions | True |

**Table C4: Environment Details for Procgen Experiments.**
| Hyperparameter | Value |
| --- | --- |
| Grey-Scaling | True |
| Observation Size | 64x64 |
| Frames Stacked | 4 |
| Reward Clipping | False |
| Max frames per episode | 108K |
| Distribution Mode | Hard |
| Number of Unique Levels (Train & Test) | Unlimited |

### C.2 Algorithm Hyperparameters

**Table C5: Table showing the hyperparameters used in the BTR algorithm.**
| Hyperparameter | Value |
| --- | --- |
| Learning Rate | 1e-4 |
| Discount Rate | 0.997 |
| N-Step | 3 |
| IQN Taus | 8 |
| IQN Number Cos’ | 64 |
| Huber Loss $\kappa$ | 1.0 |
| Gradient Clipping Max Norm | 10 |
| Parallel Environments | 64 |
| Gradient Step Every | 64 Environment Steps (1 Vectorized Environment Step) |
| Replace Target Network Frequency (C) | 500 Gradient Steps (32K Environment Steps) |
| Batch Size | 256 |
| Total Replay Ratio | $\frac{1}{64}$ |
| Impala Width Scale | 2 |
| Spectral Normalization | All Convolutional Residual Layers |
| Adaptive Maxpooling Size | 6x6 |
| Linear Size (Per Dueling Layer) | 512 |
| Noisy Networks $\sigma$ | 0.5 |
| Activation Function | ReLu |
| $\epsilon$-greedy start | 1.0 |
| $\epsilon$-greedy decay | 2M Frames |
| $\epsilon$-greedy end | 0.01 |
| $\epsilon$-greedy disabled | 100M Frames |
| Replay Buffer Size | 1,048,576 Transitions ($2^{20}$) |
| Minimum Replay Size for Sampling | 200K Transitions |
| PER Alpha | 0.2 |
| Optimizer | Adam |
| Adam Epsilon Parameter | 1.95e-5 (equal to $\frac{0.05}{batchsize}$) |
| Adam $\beta 1$ | 0.9 |
| Adam $\beta 2$ | 0.999 |
| Munchausen Temperature $\tau$ | 0.03 |
| Munchausen Scaling Term $\alpha$ | 0.9 |
| Munchausen Clipping Value ($l_{0}$) | -1.0 |
| Evaluation Epsilon | 0.01 until 125M frames, then 0 |
| Evaluation Episodes | 100 |
| Evaluation Every | 1M Environment Frames (250K Environment Steps) |

### C.3 Clarity of the terms Frames, Steps and Transitions

Throughout the Arcade Learning Environment’s history (ALE) (Bellemare et al., ; Machado et al., ), there have been many ambiguities around the terms: frames, steps and transitions, which are sometimes used interchangeably. Frames refer to the number of individual frames the agent plays, including those within repeated actions (also called frame skipping). This is notably different from the number of steps the agent takes, which does not include these skipped frames. When using the standard Atari wrapper, training for 200M frames is equivalent to training for 50M steps. Lastly, transitions refer to the standard tuple $(s_{t},a_{t},r_{t},s_{t+1})$, where the timestep $t$ refers to a steps, not frames. We encourage researchers to make this clear when publishing work, including when mentioning values of different hyperparameters.

## Appendix D Beyond The Rainbow Architecture & Loss Function

### D.1 Architecture

Figure D4 shows the the neural network architecture of the BTR algorithm. The architecture is highly similar to the Impala architecture (Espeholt et al., ), with notable exceptions:

Figure: Figure D4: Architectural diagram of the BTR algorithm’s neural network. The model contains a total of 2.91 million parameters, 2.52 million of which are within linear layers.
Refer to caption: x9.png

Within each Impala CNN blocks, each residual layer (containing two Conv 3x3 + ReLu) has spectral normalization applied, as discussed in Section 3.1.

Following the CNN blocks, a 6x6 adaptive maxpooling layer is added.

In order to use IQN, it is required to draw Tau samples which are multiplied by the output of the CNN layers, as shown by the section ‘IQN Samples’ in figure D4.

Dueling (as included in the original Rainbow DQN) splits the fully connected layers into value and advantage streams, where the advantage stream output has a mean of 0, and is then added to the value stream.

As included in Rainbow DQN, Noisy Networks replace the linear layers with noisy layers.

Lastly, the sizes of many of the layers given in Figure D4 are dependant upon the Impala width scale, of which we use the value 2. For example, the Impala CNN blocks have [16$\times$width, 32$\times$width, 32$\times$width] channels respectively. The output size of the convolutional layers (including the maxpooling layer) is 6$\times$6$\times$32$\times$width, as a 6x6 maxpooling layer is used. Lastly, the cos embedding layer after generating IQN samples requires the same size as the output of the convolutional layers, hence the size is selected accordingly. Another benefit of the 6x6 maxpooling layer is following the product of the convolutional layers and IQN samples, the number of parameters is fixed, regardless of the input size. Figure D5 shows the numbers of parameters the ablated versions of BTR have.

Figure: Figure D5: Total number of parameters in BTR with different components removed. Those not included in the graph (Munchausen and Spectral Normalisation) used the same number of parameters as BTR.
Refer to caption: x10.png

### D.2 Loss Function

The resulting loss function for the BTR algorithm remains the same as that defined in the appendix of the Munchausen paper, which gave a loss function for Munchausen-IQN. As the other components in BTR do not affect the loss, the resulting temporal-difference loss function is the same. For self-containment, we include this loss function below:

$$ $TD_{BTR}=r_{t}+\alpha[\tau\ln\pi(a_{t}|s_{t})]_{l_{0}}^{0}+\gamma\sum_{a\in A} \pi(a|s_{t+1})(z_{\sigma^{{}^{\prime}}}(s_{t+1},a)-\tau\ln\pi(a|s_{t+1}))-z_{ \sigma}(s_{t},a_{t})$ (D1) $$

with $\pi(\cdot|s)=sm(\frac{\tilde{q}(s,\cdot)}{\tau})$ (that is, the policy is softmax with q˜, the quantity with respect to which the original policy of IQN is greedy). It is also worth noting here that due to the character conflict of both Munchausen and IQN using $\tau$ (Munchausen as a temperature parameter, and IQN for drawing samples), we replace IQN’s $\tau$ with $\sigma$. $l_{0},\tau$ and $\alpha$ are hyperparameters set by Munchausen. We use the same values in BTR, also shown in our hyperparameter table in Appendix C.2.

## Appendix E BTR with Fewer Training Frames

Some of BTR’s improvements provide a relatively small improvement after 200M frames, however we want to point out their importance using fewer samples. Table E6 shows that many improvements provide large benefits earlier in training.

**Table E6: A comparison of BTR’s ablations when using less than 200M frames on the Atari-5 benchmark. Percentages are given relative to BTR’s score.**
| Algorithm | 40M Frames | 80M Frames | 120M Frames |
| --- | --- | --- | --- |
| BTR w/o Maxpooling | 5.399 (+90%) | 6.895 (+4%) | 7.443 (+2%) |
| BTR | 2.837 ($\pm$ 0%) | 6.613 ($\pm$0%) | 7.297 ($\pm$0%) |
| BTR w/o IQN | 4.030 (+42%) | 3.888 (-41%) | 6.644 (-9%) |
| BTR w/o Spectral Normalisation | 2.145 (-24%) | 5.752 (-13%) | 6.351 (-13%) |
| BTR w/o Munchausen | 2.114 (-25%) | 3.341 (-49%) | 4.755 (-35%) |
| BTR w/o Impala | 0.535 (-81%) | 1.183 (-82%) | 1.372 (-81%) |

## Appendix F Analysis of BTR’s Extensions

In recent years, several measures have been devised to understand the impact of extensions on a model’s behaviour, one of the most popular of which is dormant neurons (Kumar et al., ; Sokar et al., ). Figure F6 plots their prevalence for each of the ablations. We observe three key changes from the ablations; within the CNN layers, the Nature CNN (as used in Rainbow DQN) has a higher number of dormant/low activation neurons, indicating that the Impala network is significantly better for reducing dormant neurons. As for the fully connected layers, dormant neurons were very high across the board, with IQN lowering the number of dormant neurons.

Figure: (a) Convolutional Layers.
Refer to caption: x11.png

## Appendix G BTR with and without Epsilon Greedy

One of the first observations we made early in the testing process was that the inclusion of using $\epsilon$-greedy in addition to NoisyNetworks benefited some environments but not others. Specifically, performance was reduced on BattleZone and Phoenix, both games where the agent reached very high levels of performance with extremely precise control. However, DoubleDunk performed significantly worse, only reaching a score of 0, rather than the score of 23 the final BTR algorithm achieved. Similar findings were also found in the full version of Rainbow DQN which used only NoisyNetworks, which achieved a best score of -0.3 (Dopamine’s “compact” Rainbow DQN however which did not use NoisyNetworks achieved 22). From this, we conclude that NoisyNetworks alone failed to sufficiently explore the environment, whereas $\epsilon$-greedy did not. From these results, we eventually decided to use both methods, but disable $\epsilon$-greedy halfway through training to reap the best of both techniques.

## Appendix H Experiment Compute Resources

### H.1 Our Compute Resources

For running our experiments, we used a mixture of desktop computers and internal clusters. The desktop PCs used an GPU Nvidia RTX4090, CPU intel i9-14900k and 64GB of DDR5 6000mhz RAM. When using internal clusters, we used a mixture of GPUs, including Nvidia A100s, Nvidia Volta V100 and Nvidia Quadro RTX 8000. As for CPUs, we used 2 x 2.4 GHz Intel(R) Xeon(R) Gold 6336Y, 48 Cores. Lastly, we saved the models used to produce our analysis, totalling around 300gb across all of our ablations on the Atari-5 benchmark, saving a model every 1 million frames.

As most of our experiments were performed on desktop PC, in the main body of our paper we reference these speeds. We found that desktop PCs actually outperformed internal clusters, likely due to desktop CPUs being more suited to performing environment steps, outlined in the next subsection.

When testing ideas originally (those mentioned in Appendix I), we only tested them using a single run of the games BattleZone, NameThisGame and Phoenix unless otherwise stated. Whilst this method of evaluation is not statistically significant, for preliminary purposes with computational restrictions, we deemed this the best option.

### H.2 BTR with Different Hardware

In this work, we look to make high-performance RL more accessible to those with less compute resources, especially those only with access to desktop computers. Most of our experiments were performed with an RTX4090, we also provide some walltimes for 200M Atari frames for lower-end machines, and provide a brief comparison of desktop PCs against internal clusters:

Desktops:

Original: RTX 4090, Intel i9-13900k (2023), 64GB RAM - 11.5 Hours

RTX 3070, Ryzen 9 3900X (2019), 64GB RAM - 52 Hours

RTX 2080 ti, Intel(R) Xeon(R) Silver 4112 CPU @ 2.60GHz (2018), 128GB RAM - 32 Hours

Internal Clusters:

Nvidia H100, 48 Core Intel(R) Xeon(R) Platinum 8468 (2023), 2TB RAM - 15 Hours

Nvidia A100, 24 Core Intel(R) Xeon(R) Gold 6336Y (2021), 512GB RAM - 22 Hours

We note that there is significant variability in hardware (processors, memory bus speeds, etc), but the results still show reasonable times compared to not using BTR. Overall, we found that training BTR was very capable of running on lower end machines, with the agent (excluding the environments) using around 15GB of RAM. The main performance bottleneck was running the environment in parallel, making the number of CPU cores and processor speed most important. BTR also provides strong performance long before 200M frames, thus providing practical utility for lower-end machines.

## Appendix I Other Things We Tried

Throughout the development of the BTR algorithm, we experimented with many different components and hyperparameters. A brief list of ideas we tried that performed worse or equivalent to the final algorithm includes:

Using Exponential Moving Average networks rather than using fixed target networks (this was both computationally slower and performed worse), varying the frequency of updating the target network, changing the size of maxpool layer following the convolutional layers (we tried 4 and 8, however 6 performed significantly better) and decaying the learning rate over the course of training. Using a linearly decaying learning rate from $1\text{\times}{10}^{-4}$ to 0 0 over the course of training gave showed no significant difference. We also experimented with some different learning rates (with and without decay), and found $1\text{\times}{10}^{-4}$ to perform best, however $5\text{\times}{10}^{-5}$ also performed similarly as was used in Implicit Quantile Networks (IQN). We also tried replacing all ReLu activation units with GeLu, however this lead to dramatically worse performance. Some other ideas which we performed a single-game analysis of included annealing the discount rate from 0.97 to 0.997 (no significant difference on performance), applying spectral normaliation to the linear layers (dramatically worse performance), increasing the number of cos’ from IQN (no significant difference on performance) and using Dopamine’s Prioritized Experience Replay buffer which doesn’t include a $\alpha$ value (moderately worse performance). As discussed in G, we also tried not using $\epsilon$-greedy when using noisy nets.

Recently in RL, it has been shown that Neural Networks have a severe problem with under-parameterisation (Kumar et al., ) and neurons becoming dormant (Sokar et al., ), preventing larger models from seeing the success evident in other areas of Deep Learning. One method used to remedy this is weight decay through the use of the AdamW optimizer (Loshchilov & Hutter, ). As this is far simpler than many other the other techniques to prevent under-parameterisation we decided to test this approach, however no significant differences in performance were observed. We tested this approach using the decay parameter $1e-4$, however potentially using a higher value may results in significant changes in performance.

Lastly we also tried removing some of the original components from Rainbow DQN on Atari BattleZone, including Dueling, Prioritized Experience Replay and Noisy Networks. Prioritized Experience Replay and Noisy Networks both proved beneficial, so were kept in the algorithm. Dueling did not seem to make any significant difference, however we did not choose to remove it for a clearer continuation of Rainbow DQN, in addition to potentially being useful in other Atari environments.

Shortly after the submission of this work, we tested BTR with addition of Layer Normalization, and found positive results. Layer Normalization can improve the robustness to a variety of
pathologies that cause loss of plasticity (Lyle et al., ), and helps to improve the conditioning of the network’s gradients in RL (Ball et al., ). Below in Figures I7, I8 and Table I7, we show the results of this addition into BTR.

Figure: Figure I7: Graph shows individual game performance of BTR with and without Layer Normalization. Layer Normalization makes a notable improvement on multiple games, including NameThisGame and BattleZone.
Refer to caption: x13.png

Figure: Figure I8: Graph shows IQM human-normalized performance of BTR with and without Layer Normalization on the Atari-5 Benchmark.
Refer to caption: x14.png

**Table I7: Maximum scores obtained during training (averaged over 100 episodes and all performed random seeds) after 200M Frames on the Atari-5 Environment, compared to BTR with Layer Normalization.**
| Game | Random | Human | BTR + Layer Normalization | BTR |
| --- | --- | --- | --- | --- |
| BattleZone | 2360 | 37188 | 204380 | 151877 |
| DoubleDunk | -19 | -16 | 23 | 23 |
| NameThisGame | 2292 | 8049 | 32834 | 28710 |
| Phoenix | 761 | 7243 | 498264 | 367284 |
| QBert | 164 | 13455 | 48485 | 45034 |
| IQM | 0.000 | 1.000 | 8.369 | 7.627 |
| Median | 0.000 | 1.000 | 5.801 | 4.589 |
| Mean | 0.000 | 1.000 | 21.099 | 16.561 |

## Appendix J Altered Atari Environment Settings

In order to investigate the impact of the environmental sticky actions parameter and to compare against other works, we include results for it on the Atari-5 benchmark in Figure J9.

Figure: Figure J9: Graph shows individual game performance of BTR with and without Sticky Actions. NameThisGame and Phoenix see small improvements. When using sticky actions, the impact of disabling $\epsilon$-actions is far more noticeable ($\epsilon$-actions are disabled after 125M frames). This indicates using sticky actions produces a policy more robust to noise, as would be expected.
Refer to caption: x15.png

Some prior works choose to pass life information to the agent (Schmidt & Schmied, ). To clarify, this is different to terminal on loss of life. Life information does not reset the episode upon losing a life, but does pass a terminal to the buffer, allowing the agent to experience further into episodes while also giving the agent a negative signal for losing a life. This setting is not recommended in Machado et al. (), and works which use it are not comparable to those which don’t. To emphasize this point, we take the three games from the Atari-5 benchmark which use lives (as not all Atari games do), and perform a comparison.

Figure: Figure J10: BTR with and without life information.
Refer to caption: x16.png

## Appendix K BTR for Wii Games

BTR interfaces with different Wii Games via the Dolphin Emulator. Specifically, we use a forked repository of to allow Python scripts to interact with the emulator. This includes loading savestates (used to reset episodes), grabbing the screen as a PIL (Clark, ) image at the Wii’s internal resolution of 480p (downsampled to 140x114 and grey-scaled, used for all observations), reading the Wii’s RAM (used for reward functions and termination conditions) and allows programmatic input into the emulator (used for actions). Using Dolphin’s portable setting, we are able to run multiple Dolphin Emulators simultaneously on the same machine. Each instance runs as a unique process, and communicates with the agent via Python’s multiprocessing library. Similarly to the Atari benchmark, for all games we used a frameskip of 4.

### K.1 Super Mario Galaxy

This environment used Super Mario Galaxy’s final level, The Center of the Universe, and had to make it to the final fight at the end of the game. The agent had 6 actions, including None, moving in each direction and jumping. Additionally, if the jump action was performed following a movement, the agent would continue to move in that direction.

Rewards were given via finding many values in the Wii’s memory that resembled progress in the level. The agent was then rewarded for this progress value increasing from the last frame. If the agent’s position entered a set region, the progress variable would be moved. Additionally, the game uses a life system, where the player has a maximum of 3 lives and can lose or gain lives in many different ways. The agent was given a reward of +1 for gaining a life, and -1 for losing a life. Lastly, episode termination occurred if the agent reached 0 lives, or if the agent made it to the end of the level. For this task, we also allowed the agent to start episodes at many points throughout the level, which rapidly sped up training since the agent could easily experience different areas of the level.

Whilst a difficult task, once the agent first completed the level, it did not take long to start consistently completing it due to the deterministic nature of the game.

### K.2 Mario Kart Wii

The Mario Kart Wii environment had the agent play against the game’s internal opponents (on hard mode, with 12 racers including the agent), on the course Rainbow Road (with items on the 150cc speed setting). The agent had to complete 3 laps of the course to finish the race. The agent had just 4 action, including accelerate, drifting left or right, and using its item. While this limited the agent’s potential actions substantially, we found using fewer actions to dramatically accelerate training.

Rewards of +1 were given via reaching checkpoints that were scattered throughout the course (100 in total per lap). Additionally, if the agent’s speed dropped below a set threshold (65 km/h), the agent would receive a reward of -0.01 per frame. The agent would be terminated with a reward of -10 if its speed dropped below the threshold for over 80 frames, or with a reward of +10 for finishing the race, with a bonus based on the position the agent finished in. Lastly, the agent was rewarded with a +1 for using its item. Without this reward, we found the agent to often neglect using its item, likely due to many of the items only providing rewards in the long term, such as slowing down other racers or blocking incoming items far in the future. Similarly to Super Mario Galaxy, we had the agent start the episode in multiple positions around the first lap, allowing it to experience the whole track early in training.

This agent took the longest to train, taking around 160M frames to reach consistent completion. In particular, the agent took a long time to consistently complete the race due to the other racers and randomized items making the environment highly stochastic, with many rare scenarios which could cause the episode to terminate.

### K.3 Mortal Kombat

The Mortal Kombat environment put the agent in the game’s endurance mode, where the agent would sequentially fight 15 different opponents, but keep retain its health between fights, and only gain health after defeating every 3 opponents. We provided the agent with 14 actions, including: None, Left, Right, Up, Down, Axe Kick, Punch, Snap Kick, Grab, Block, Toggle Weapon, Jump Left, Jump Right, and Crouch. These actions were far from the game’s total action space, and limited the agent’s ability to perform some of the combos within the game. We limited the agent’s actions as the full action space is extremely large.

The agent was positively rewarded for damaging the opponent, and negatively rewarded for taking damage, with one taking one tenth of the health bar equating to +1 reward respectively. The episode was terminated with a reward of -10 for reaching 0 health, and +10 for defeating the 15th and final enemy.

The Mortal Kombat agent learned considerably faster than Super Mario Galaxy and Mario Kart Wii, first completing the environment in 50M frames, and getting progressively more consistent until training was stopped at 90M frames. The agent quickly learned how to dodge enemy hits, and relied heavily upon this strategy.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: 1. This paper is well-written and well-organized. The ideas are clear and could be easily understood
2. The experiments are comprehensive and the results are strong. As shown in Section 4, the proposed BTR algorithm could greatly outperform state-of-the-art baselines in two classic benchmarks and handle three hard and complex modern games with a desktop PC.
3. The paper includes extensive ablation studies and experimental data. Section 5 presents a detailed analysis of the performance and impact of each component of the BTR algorithm, providing readers with insights into the sources of the algorithm's performance gains. Additionally, the authors include complete experimental results and settings in the appendix, helping to clarify any potential confusion or misunderstanding for readers.
summary: This paper introduces Beyond The Rainbow (BTR), a novel reinforcement learning (RL) algorithm that enhances Rainbow DQN by integrating six key improvements. The BTR algorithm is computationally efficient, capable of training powerful agents on a standard desktop computer within a short time. Experimental results show that BTR outperforms state-of-the-art RL algorithms on both the Atari-60 and Procgen benchmarks. Additionally, BTR can handle training agents for challenging levels in complex, modern games. Finally, this paper includes a comprehensive ablation study to analyze the performance and impact of each component within the BTR algorithm.
weaknesses: 1. The BTR integrates six improvements from existing RL literature to Rainbow DQN. While the algorithm demonstrates strong performance, its novelty might appear limited.  Could you further clarify the novelty of this work? Or specifically, could you briefly discuss if there is any challenges in integrating these existing improvements into the BTR algorithm?

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. What exactly is adaptive maxpooling? Would it be possible to add a description of it with either an equation, pseudo-code, or diagram?
2. Where did the formula 0.05/batch_size for Adam's epsilon come from?
3. The final algorithm has a considerable number of hyperparameters, would it be possible to discuss a bit which ones are the most important to tune should someone try to apply this algorithm to a new domain?
soundness: 2
strengths: The presentation is overall clear, the methodology is sound, and the results are compelling. Both extensive use of ablations, and the connection to other important metrics related to pathologies in Deep RL algorithms are an example that more papers should follow.
The appendices are also data rich, showing ablations' performances on each of ALE's 60 games, and even having one appendix about things that were tried but did not lead to improvements in performance, which may help others not repeat the experiments.
summary: The paper presents a variant of Rainbow that adds further architectural and algorithmic improvements to improve not only the agent's score but also to increase its training speed to around 3x what has been previously reported, while running it on top-notch consumer hardware. Finally the authors also show that their improved version of rainbow can deal with modern games with complex graphics and physics.
weaknesses: 1. Adaptive Maxpooling is never defined. It's not a common layer in reinforcement learning and it's never defined in the paper, in fact skimming (Schmidt and Schmied, 2021) that layer is also not defined, I believe this is the only seious weakness in the paper's presentation, but still I believe it is a serious weakness (though hopefully the authors can fix it and so I can increase their grade).
2. There are at least 2 relevant citations missing, "Spectral Normalisation for Deep Reinforcement Learning: An Optimisation Perspective" when talking about Spectral Normalisation, and "On the consistency of hyper-parameter selection in value-based deep reinforcement learning" when talking about the need for tuning Deep RL hyperparameters and the benefits of using layer norm between dense layers.
3. I believe it's slightly misleading to not specify "a high-end PC" when talking about the kind of machine that can run the algorithm in 12 hours (4090 RTXs are quite expensive, and i9s are Intel's high-end consumer line)
4. I believe a more direct comparison with Schmidt and Schmied, 2021 is warranted, given its foundational importance to the paper.
5. Using only 3 seeds while having a large increase in the number of tuned hyperparameters weakens the validity of the results as explained in "Empirical Design in Reinforcement Learning", though at the same time the analysis of metrics beyond simply the score and the extensive use of ablations help.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - The 12 hour number/other timings, is that the total time it takes to train BTR on a single game or on all 57 games?
- It seems like you made quite a few hyperparameter choices (e.g. how often to update, etc.) Do you use the same values for each domain?
- What is the shaded area in the plots? If it is standard deviation it seems like the proposed BTR algorithm is very inconsistent across seeds. Could you elaborate please/maybe provide results for individual seeds?
- Figure 3 does show that you can apply your approach to other games, which is great. I would really like to see some point of comparison, however, to act as a reference point. For instance, run vanilla PPO or DQN or Rainbow as a baseline.
- Why is fig 4 using raw score as the y-axis, as opposed to e.g. normalised?
- Figure 4 is somewhat hard to follow as there are so many lines and it seems like most of them overlap quite a lot.
- Is it feasible to run rainbow with vectorisation? This is not that crucial, it just seems like something obvious to run given figure 5, where vectorisation is the main speedup factor.
- Table 2: Would be nice to have another method, e.g. rainbow or DQN to act as a reference point.
- One recent work that seems to have a similar purpose is "Simplifying Deep Temporal Difference Learning" (https://arxiv.org/pdf/2407.04811). It seems like they use vectorisation as well to achieve large speedups. More importantly, however, is that they primarily use JAX---which is becoming increasingly common in RL, and is reducing computational load significantly/making RL more accessible to compute-poor labs/institutions. Could you please comment on a few things
	- How does this paper's score compare to yours?
	- How does the walltime compare to yours?
	- What do you see as the benefits/disadvantages of this hardware accelerated paradigm compared to the more classic approach you are taking?
- I know it is not usual in these types of papers but I would really appreciate a PPO comparison, both in terms of walltime and performance.
soundness: 3
strengths: - This paper gives RL researchers a way to do pretty well in atari without expending too significant computational resources.
- They perform ablations on their individual changes to identify what helps and what has the most effect on performance/walltime. This is quite useful.

I am not giving this a lower score because I think making RL more accessible is worthwhile, and this paper takes a step towards this, and further analyses many of these independent components to see what their effect is. I am not giving a higher score because I think the paper's significance does not warrant it.
summary: This paper combines several different RL improvements to a single algorithm, with a focus on high performance with reasonable computational requirements. In doing so, they find that their approach achieves a new SoTA (not including recurrent methods), while being able to be run on a desktop machine in under a day.

They analyse the factors that led to this performance in detail through several ablations.

Overall, this paper makes rainbow/dqn-type methods more accessible to non-industry labs
weaknesses: - To me it is unclear if 12 hours is for all games or just 1.
- I wonder how this fits in with the recent trend of hardware-accelerated RL (see e.g., Stoix/PureJaxRL/Gymnax and all of the hardware-accelerated environments). Does that line of work better achieve the goal of making RL more accessible? In that setting, the environment is often run entirely on the GPU, leading to hundreds or thousands of times speedups.

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 2
questions: See weaknesses.


Overall, this work is just not good enough in its current format. I recommend that the authors fix the presentation of the results, especially adding error bars and effective aggregation using a tool like rliable. Given the significant problems with every figure and table in the main body of the paper, this work is not good enough for this venue in its current form and would require wholesale changes to fix that.

[1] Deep Reinforcement Learning at the Edge of the Statistical Precipice. Agarwal et al. Neurips 2021.

[2] Aitchison, Matthew, Penny Sweetser, and Marcus Hutter. "Atari-5: Distilling the arcade learning environment down to five games." International Conference on Machine Learning. PMLR, 2023.

[3] Sokar, Ghada, et al. "The dormant neuron phenomenon in deep reinforcement learning." International Conference on Machine Learning. PMLR, 2023.
soundness: 1
strengths: The paper has a number of positive points:
- The core idea of trying to achieve strong performance using Q-learning on a desktop PC has significant merit and would constitute an interesting contribution
- The introduction of new games to evaluate on is interesting and the games chosen would make good potential benchmarks.
- The paper is easy to follow and clearly written
summary: The authors present Beyond the Rainbow (BTR), an algorithm combining advances in Q-learning based approaches to Atari developed since Rainbow. 

The authors train their agent on Atari, Procgen and 3 games which aren't well-established benchmarks in RL (Super Mario Galaxy, Mario Kart and Mortal Kombat). They run ablations on their method and demonstrate that the Impala architecture contributes the most to their method's performance. They also demonstrate that vectorization of the environment is key to the faster runtime of their algorithm.
weaknesses: I slightly feel for the authors of this paper. I would like to be able to commend the paper on its empirical results, or the performance on the new environments, but the results are not presented scientifically enough for me to do that, and so I can't recommend acceptance at this venue.

To make my objections more concrete:
- In Figure 2, the authors claim that their method outperforms the red baseline, but this is plainly not the case from the plot. The error bars so significantly overlap the red line there is no way this result is significant. 
- The authors do not aggregate their results in accordance with recommended practice [1]. Although they use the inter-quartile mean, they do not provide bootstrapped confidence intervals to estimate errors and do not seem to provide errors in their baseline results. This issue appears in Figures 1 and 2. As far as I know, the authors do not state what the error bars in Figures 1 and 2. If the plotted error bars are standard 
- While the evaluation of their method on new games is nice, I can't take any information away from this without even a semblance of a baseline. Training an RL policy on Wii games has no intrinsic scientific value -- it is only by contextualisation of a baseline that this would be a compelling result. Similarly, the authors provide no error bars in this domain.
- Figure 4 again because of the way the results were processed provides almost no information. Atari-5 [2] provides a way to estimate the median given performance on those 5 games. But this is only after the application of a regression procedure. Without the application of this summary metric, it is just not clear what to take away from these results. This figure does not even present human normalised scores, as is standard. This Figure should therefore be replaced by a plot of the regressed median for Atari-5 with bootstrapped confidence intervals. The authors can use rliable [1] for this.
- Again, the analysis in Section 5.2 *should* be compelling and interesting reading, but it's just not done thoroughly enough. Figure 6 is presented without error bars and so are the results in Table 2 and the IQM in Table 3. It's just not possible to believe the authors' conclusions on their work without any estimates of error. 
- Additionally, the authors use dormancy [3], but set a threshold of 0.1. Although resetting dormant neurons was shown to improve performance, neurons with a small activation are not in themselves a problem! A neuron followed by a ReLU activation that always outputs 0 is not learning, which clearly constitutes a problem, but a neuron that outputs a small value is still perfectly plastic. The dormancy results therefore also aren't a proxy for any form of plasticity. 
- The authors make multiple claims about their method being "state-of-the-art for a desktop PC" (or similar). These should be removed from the paper as they are just impossible to verify. Even as an expert, I do not know the hardware that every paper ran experiments on and whether it would be possible to run it on a desktop PC, and it is not a claim that can be clearly backed-up. I note that the authors did not do all their experimentation on a desktop PC, but only claim that their method can run on one effectively.

Meta-review: This paper proposes Beyond The Rainbow (BTR) an RL algorithm combining multiple recent advancements into a single method with the single aim of outperforming on the Atari benchmark. There is a fairly high bar for this type of work to be accepted, given that the Atari environments are no longer near the forefront of our field and it has been reported multiple times that experimental results in this domain do not necessarily translate. Indeed, the lack of experimental rigor is ultimately the issue that inclines me to recommend rejection for this paper, in particular the lack of seeds.

Decision: Reject