Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC
Authors: Tyler Clark
Mark Towers
Christine Evers
Jonathon Hare
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
Deep Learning
Atari
Desktop PC
Rainbow DQN
DQN
Computationally Accessible
Primary area: reinforcement_learning
TL;DR: A high-performance Reinforcement Learning algorithm capable of running on desktop computers.

Abstract: Rainbow Deep Q-Network (DQN) demonstrated combining multiple independent enhancements could significantly boost a reinforcement learning (RL) agent’s performance. In this paper, we present “Beyond The Rainbow” (BTR), a novel algorithm that integrates six improvements from across the RL literature to Rainbow DQN, establishing a new state-of-the-art for RL using a desktop PC, with a human-normalized interquartile mean (IQM) of 7.6 on Atari-60. Beyond Atari, we demonstrate BTR’s capability to handle complex 3D games, successfully training agents to play Super Mario Galaxy, Mario Kart, and Mortal Kombat with minimal algorithmic changes. Designing BTR with computational efficiency in mind, agents can be trained using a high-end desktop PC on 200 million Atari frames within 12 hours. Additionally, we conduct detailed ablation studies of each component, analyzing the performance and impact using numerous measures.

Beyond The Rainbow: High Performance Deep Reinforcement Learning on a
Desktop PC
Tyler Clark
Mark Towers'
Christine Evers'
Jonathon Hare'
Abstract
IQM Humaan Normalized Score
10
Rainbow Deep Q-Network (DQN) demonstrated combining multiple independent enhancements could significantly boost a reinforcement learn- ing (RI.) agent's performance. In this paper, we present "Beyond The Rainbow" (BTR), a novel algorithm that integrates six improvements from across the RL literature to Rainbow DQN, establishing new state-of-the-art for RL us- ing desktop PC, with a human-normalized in- terquartile mean (IQM) of 7.4 on Atari-60. Be- yond Atari, we demonstrate BTR's capability to handle complex 3D games, successfully train- ing agents to play Super Mario Galaxy, Mario Kart, and Mortal Kombat with minimal algo- rithmic changes. Designing BTR with compu- tational efficiency in mind, agents can be trained using high-end desktop PC on 200 million Atari frames within 12 hours. Additionally, duct detailed ablation studies of each compo- nent, analyzing the performance and impact us- ing numerous measures. Code is available at http:://gthubccom///////////
25
50
75
100
125
150
175
A100 Walltime (Hours)
Beyond The Rainbow (BTR) Dreamer-v3
Reinbow
(Nature)
perfor- mance for BTR against other RL algorithms on the Atari benchmark in terms of walltime performance (all results use 200M frames). The results for DQN and Rainbow DQN are those reported in RLiable (Agarwal et al. 2021), and Dreamer-v3 refers to Hafner et al. (2023). Shaded ar- eas show 95% bootstrapped confidence intervals, with BTR using seeds.
Figure 1: Interquartile mean
con-
ited compute resources, either in terms of the required hard- ware or the walltime necessary train a single agent. This is unique issue in RL compared natural language pro- cessing image recognition which have foundation mod- els that can be efficiently fine-tuned for new task prob- lem (Lv et al. 2024). Meanwhile, RL agents must be trained afresh for each environment. Therefore, the de- velopment of powerful RL algorithms that can be trained quickly on inexpensive hardware is crucial for smaller re- search labs and the hobbyist community.
1. Introduction
Deep Reinforcement Learning (RL) has achieved nu- merous successes in complex sequential decision- decision-making making tasks, most rapidly Mnih et al. (2015) proposed Deep Q-Learning (DQN). With this success, RL has become in- creasingly popular among smaller research labs, the hobby- ist community, and even the general public. However, re- cent state-of-the-art approaches (Schrittwieser et al.) 2020 Badia et al. 2020a Hessel et al. 2021 Kapturowski et al. 2023 are increasingly out of reach for those with more lim-
These concerns are not new. Ceron & Castro 2021 high- lighted that Rainbow DQN (Hessel et al. 2018) required 34,200 GPU hours (cquivalent to 1435 days) of training, making the research impossible for anyone except a few re- search labs, with more recent algorithms exacerbating this problem. Recurrent network architectures (Horgan et al., 2018), high update sample ratio (D'Oro al. (0022). and the of world-models and search-based techniques (Schrittwieser et al.) 2020), all increase the computational resources necessary to train agents. Many of these use dis- tributed approaches requiring multiple CPUs and GPUs (or TPUs), or requiring numerous days and weeks to train a single agent, dramatically decreasing RL's accessibility.
'School of Electronics and Computer Science, University Of Southampton, Southampton, UK. Correspondence Tyler Clark <tjc2g19 19 @oton..c.oo...
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC
For this purpose, we develop "Beyond the Rainbow" (BTR), taking the same principle as Rainbow DQN (Hes- sel et al., 2018), selecting 6 previously independently eval- uated improvements and combining them into a singular algorithm (Section 3 These components were chosen for their performance qualities or to reduce the computational requirements for training an agent. As a result, BTR sets new state-of-the-art score for Atari-60 (Bellemare et al., 2013) (excluding recurrent approaches) with an Interquar- tile Mean (IQM) of 7.40 using a single desktop machine in less than 12 hours, and outperforms Rainbow DQN on Procgen (Cobbe et al. (020) in less than a fifth of the wall- time (Section 4.Ip. Further, we demonstrate BTR's s poten- tial by training agents to solve three modern 3D games for the first time, Mario Kart Wii, Super Mario Galaxy and Mortal Kombat, that each contain complex mechanies and graphics (Section 4.2). To verify the effectiveness and ef- fect of the six improvements to BTR, in Section 5.1 we conduct a thorough ablation of each component, plotting their impact on the Atari-5 environments and in Section 5.2 we utilize seven different measures to analyse the com- ponent's impact on the agent's policy and network weights. This allows us to more precisely understand how the com- ponents impact BTR beyond performance walltime.
Component Impact Analysis (Section We con- duct thorough ablations of BTR without each compo- nent, investigating performance and other measures, We discover that BTR widens action gaps (reduc- ing the effects of approximation errors), is robust to observation noise, and reduces neuron dormancy and weight matrix norm (shown to improve plasticity throughout training).
2. Background
Before describing BTR's extensions, outline standard RL mathematics, how DQN is implemented, and Rainbow DQN's extensions.
2.1. RL Problem Formulation
We adopt the standard formulation of RL. (Sutton & Barto 2018), described as Markov Decision Process (MDP) defined by the tuple AS,A, R),, where S is the set of states, A is the set of actions, P ::S. S x A (S) is the stochastic transition function, and R: S x A +R is the reward function. The agent's objective is learn a policy π S A(A) that maximizes the expected sum of dis- counted rewards E,, 1E (r(ss, where Y E [0, is the discount rate.
In summary, we make the following contributions to state- oh-the-art RL.
2.2. Deep Q-Learning (DQN)
High Performance (Section 4.1 BTR outper- forms the state-of-the-art for non-recurrent RL on the Atari-60 benchmark, with an IQM of 7.4 (compared to Rainbow DQN's 1.9), outperforming humans on 52/60 games. Furthermore, BTR outperforms Rain- bow DQN with Impala the Procgen benchmark de- spite using smaller model and 80% less walltime.
One popular method for solving MDPs is Q-Learning (Watkins & Dayan, 1992) where an agent learns to predict the expected sum of discounted future rewards for a given state-action pair. To allow agents to generalize over states and thus be applied to problems with larger state spaces, Mnih et al. (2013) successfully combined Q-Learning with neural networks. To do this, training minimizes the error between the predictions from parameterized network Qo and a target defined by
Modern Environments (Section 4.2 Testing be- yond Atari, we demonstrate BTR can train agents for three modern games: Super Mario Galaxy (final stage), Mario Kart Wii (Rainbow Road), and Mortal Kombat (Endurance mode). These environments con- tain 3D graphics and complex physies and have never been solved using RL.
It y max Qø' ()+ aEA
(1)
where Qg' is an earlier version of the network referred to as the target network, which periodically updated from the online network Qo. The data used to perform updates is gathered by sampling from an Experience Replay Buffer Lin 1992), which stores states, actions, rewards, and next states experienced by the agent while interacting with the environment. To effectively explore the environment, e- greedy exploration is used, where each observation has a probability of choosing a random action.
Computationally Accessible (Figure Using high-end desktop PC, BTR trains Atari agents for 200 million frames in under 12 hours, significantly faster than Rainbow DQN's s 35 hours. This increases RL research's 's accessibility for smaller research labs and hobbyists without the need for GPU clusters or exces- sive walltime.
2.3. Rainbow DQN and Improvements to DQN
All reported IQM scores use the best single evaluation for each environment throughout training as is standard, rather than the agent's score 200 million, hence the discrepancy between the overall score and Figure]] T
In collecting different improvements to DQN, Rain- bow DQN (Hessel et al., 2018) proved cumulatively that these improvements could achieve a greater performance
2

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC
than any individually. We briefty explain the individual improvements, ordered by performance impact, most of which are preserved within BTR (see Table D). For more detail, we refer readers to the extension's respective papers:
3. Beyond the Rainbow Exxensions and Improvements
Building on Rainbow DQN (Hessel et al. 2018), BTR in- cludes 6 more improvements undiscovered in 2018 Ad- ditionally, as hyperparameters are critical to agent perfor- mance, Section 3.2 discusses key hyperparameters and our choices. In the appendices, we include a table of hyper- parameters. a figure of the network architecture and the agent's loss function (Appendices 1D.2 R and H.2).. Finally, the source code using Gymnasium (Towers et al. 2024) is included within the supplementary material to help future work build upon or utilize BTR.
1. Prioritized Experience Replay- To select training examples, DQN sampled uniformly from an Experi- ence Replay Buffer, assuming that all examples are equally important to train with. [Schaul et al. (2015) proposed sampling training examples proportionally to their last seen absolute temporal difference error, increasing training on samples for which the network most inaccurately predicts their future rewards.
3.1. Extensions
2. N-Step Q-learning utilizes bootstrapping to mini- mize the difference between the predicted value and the resultant reward plus the maximum value of the next state (Eq. D N-step (Sutton et al. 1998) reduces the reliance on this bootstrapped next value by con- sidering the next n rewards and the observation in n timesteps (Rainbow DQN used n = 3).
Impala Architecture + Adaptive Maxpooling Espe- holt et al. (2018) proposed a convolutional residual neu- ral network architecture based on He et al. (2016), featur- ing three residual block substantially increasing perfor- mance over DQN's three- three-layer convolutional network. Fol- lowing Cobbe et al. (2020), we scale the width of the con- volutional layers by 2 to improve performance. We include an additional 6x6 adaptive max pooling layer after the con- volutional layers (Schmidt & Schmied, [202]), which was found to speed up learning and support different input reso- lutions. The adaptive maxpooling is identical to a standard 2D maxpooling layer, but can used with any input reso- lution it automatically adjusts the stride and kernel size to fit specified output size.
3. Distributional RL
Due to the stochastic nature of RL environments and agent policies. |Bellemare et al. (2117) proposed learning the return distribution rather than scalar expectation. This was done through mod- cling the return distributions using probability masses and the Kullbeck-Leiibee divergence loss function.
4. Noisy Networks - Agents can often insufficiently ex- plore their environment resulting in sub-optimal poli- cies. Fortunato et al. (2018) added parametric noise to the network weights, causing the model's outputs to be randomly perturbed, increasing exploration dur- ing training, particularly for states where the agent has less confidence.
Spectral Normalization (SN)- To help stabilize the train- ing of discriminators in Generative Adversarial Networks (GANs), MMyato et al. (2018) proposed Spectral Normal- ization to help control the Lipschitz constant of convolu- tional layers. SN works to normalize the weight matrices of each layer in the network by their largest singular value, en- suring that the transformation applied by the weights does not distort the input data excessively, which can lead to in- stability during training. Bjorck et al.] (202T) and Gogianu et al. (202T) found that SN could improve performance in RL, especially for larger networks and [Schmidt & Schmied (20TT) found SN reduced the number of updates required before initial progress is made.
5. Dueling DQN- The agent's Q-value can be rewritten the sum of state and advantage (Q(s, a) V(x)= + A(s, a)). Looking to improve action general- ization, Wang et al. (2016) split the hidden layers into two separate streams for the value and advantage, re- combining them with Q(s, = V(s) + a)
[A Ear A(s, d))).
Implicit Quantile Networks (IQN) HOON--aace Dabney et al. (2018) improved upon Bellemare et al. (2017), used in Rainbow DQN, learning the return distribution over the probability space rather than probability distribution over return val-
6. Double DQN Selecting a target Q-value with the maximum Q-value from the next observation (Eq. [ can frequently cause overestimation, negatively af- fecting agent performance. To reduce this overesti- mation, Van Hasselt et al. (2016) propose utilizing the online network rather than the target network to select the next action when forming targets, defined as:
After the completion of our work, we additionally found Layer Normalization applied after the stem of each residual block and between dense layers to be beneficial (scc Appendix H for a discussion)
'The network architecture is referred to as Impala duc to the accompanying training algorithm IMPALA proposed in Espcholt
T3 + "Og' (St+1, arg max Qo Qo(5t41,a))
al. 20118)
aEA
3

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC
Table 1: A comparison of components between Rainbow DQN
(Hessel et al. 2018 and BTR.
Added To Rainbow DQN
Same As Rainbow DQN
Removed From Rainbow DQN
Impala (Scale=2)
N-Step TD Learning
Double (N/A with Munchausen)
Adaptive Maxpooling (6x6)
Prioritized Experience Replay
C51 (Upgraded to IQN)
Spectral Normalization
Dueling
Implicit Quantile Networks
Noisy Networks
Munchausen Vectorized Environments
ues. This removes the limit on the range of Q-values that can be expressed, and enables learning the expected return at every probability.
formance. Firstly, how frequently the target network is up- dated is closely intertwined with batch size and replay ra- tio. We found that updating the target network every 500 gradient
best. Given our high batch size, we additionally performed minor hyperparameter tests us- ing different learning rates finding that a slightly higher learning rate of x 10 performed best, compared to 6.25 x 10-5 in Rainbow DQN. In Appendix [D.2 we clarify the meaning of the terms frames, steps and transitions.
Munchausen RL - Bootstrapping is a core aspect of RL; used to calculate target values (Eq. M with most algorithms using the reward, rt. and the optimal Q-value of the next state, Q" However, since in practice the optimal policy is not known, the current policy π is used. Munchausen RL (Vieillard et al. 2020) leverages an additional estimate in the bootstrapping process by adding the scaled-log policy to the loss function (Eq. 3 where E [0, 1] is a scaling factor, o is the softmax function, and F is the softmax tem- perature). This assumes a stochastic policy, therefore DQN is converted to Soft-DQN with मe. = a ).).. As Mun- chausen does not use argmax over the next state, Double DQN is obsolete. Munchausen RL's update rule is
For many years, RL algorithms have used a discount rate of 0.99, however, when reaching high performance, lower discount rates alter the optimal policy, causing even opti- mally performing agents to not collect the maximum cu- mulative rewards. To prevent this, we follow MuZero Re- analyse (Schrittwieser et al., 2021) using 7 = 0.997. For our Prioritized Experience Replay, we use the lower value of 0.2, the parameter used to determine sample prior- ity, recommended by TToromaofff et al. (2019) when using IQN. Lastly, many previous experiments used only noisy networks or e-greedy exploration, however, we opt to use both until 100M frames, then set e to zero, effectively dis- abling it. We elaborate on this decision in Appendix
Qo(s,, r. + or In (alls))) Σ To (42))( In( (a' ((42)). ''EA
Y
4. Evaluation
Vectorization RL agents typically take multiple steps in single environment, followed by gradient update with small batch size (Rainbow DQN took 4 environment steps, followed by a batch of 32). However, taking multiple steps in parallel and performing updates on larger batches can significantly reduce walltime. We follow [Schmidt & Schmied (20T)) taking 1 step in 64 parallel environments with one gradient update with batch size 256 (Schmidt & Schmied (202T) took two gradient updates). This results in replay ratio (ratio of gradient updates to environment steps) of 04 Higher replay ratios have been shown to im- prove performance (D'Oro et al. 2022), however we opt to keep this value low to reduce walltime.
To assess BTR, we test on two standard RL benchmarks, Atari (Bellemare et al. 2013) and Procgen (Cobbe et al. 2020 in Section 4.1 Secondly, we train agents for three modern games (Super Mario Galaxy, Mario Kart Wii, and Mortal Kombat) with complex 3D graphics and physics in never before shown to be trainable with RL.
Section 4.2
4.1. Atari and Procgen Performance
We evaluate BTR on the Atari-60 benchmark following (Machado et al., 2018) and without life information (see Appendix[[for the impact), evaluating every million frames on 100 episodes. Figure [ plots BTR against DQN, Rain- bow DQN and Dreamer-v3, showing BTR's R competitive performance despite using significantly less walltime. Fig-
3.2. Hyperparameters
Hyperparameters have repeatedly shown to have a very large impact on performance in RL (Ceron et al. 2024). thus we perform a small amount of tuning to improve per-
"This equates to 32,000 environment steps (128,000 frames), compared to Rainbow DQN's 8,000 steps.
4

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
Median
IQM
Mean
Optimality Gap
MEME (200M, 274 Hours, 2xTPU-v4)
Dreamer-v3 (200M, 185 Hours, A100)
BTR (200M, 22 Hours, A100)
Rainbow (200M, 70 Hours, A100)
DQN (200M, 48 Hours, A100)
2
2.5
7.5
10.0
24 32
0.08 0.16
Human Normalized Score
Median
IQM
Mean
Optimality Gap
MEME (200M, 274 Hours, 2xTPU-v4)
Dreamer-v3 (200M, 185 Hours, A100)
=
BTR (200M, 22 Hours, A100)
Rainbow (200M, 70 Hours, A100)
DQN (200M, 48 Hours, A100)
EZV2 (100K, 2.7 Hours, 4xRTX3090)
BBF (100K, Hours, A100)
4 8 12
5
10
15
20
15 30
45
0.1 0.2 0.3
Human Normalized Score
Figure 2: Box plot performance of BTR (4 seeds) against other popular algorithms such as MEME (Kapturowski et al. 2023), Dreamer v3 (Hffner et al. 2023). Bigger, Better, Faster (BBF) (Schwarzer et al. 2023) and EfficientZero-v2 (EZV2) [Wan et al. 2024). Brackets show the number of frames the algorithms use, the number of walltime hours and the hardware used respectively. Shaded areas show 95% confidence intervals. Top: Atari game benchmark we used the overlapping
games 55 between the popular Atari-57 benchmark, and the 60 games used in RLiable (Agarwal al. 2021 Bottom:
Atari-26 benchmark, commonly used for testing sample-efficient algorithms.
nificantly fewer convolutional filters (which Cobbe et al. (220)) found to sigmificantly improve performance) and us- ing 8 hours of walltime compared to 41. While BTR pro- vides an improvement over Rainbow DQN in Procgen, we did not target procedurally generated environments thus does currently compete with the state-of-the-at (Cobbe et al. 2021 Hafner al., 2023). There numerous ways performance can be improved (Jesson & Jiang 2024
Min Min-Max Normalized lized IOM
0.4
0.2
0.0
Beyand The Rainbow (8 Hours)
Rainbow DQN Impala x4 Hours)
Cobbe et al. 2020) which we leave to future work.
100 150 Number of Frames (Millions)
50
200
4.2. Applying BTR to Modern Games
Figure BTR compared to Rainbow DQN Impala (width x4) (Cobbe et al. 2020) after 200M frames on the Procgen benchmark. Shaded areas show 95% Cls, with re- sults averaged over 5 seeds.
To demonstrate BTR's capabilities beyond standard RL benchmarks, we utilized Dolphin (Dolphin EeabinnEmulatorrj 2024). Nintendo Wii emulator, to train agents for a range of modern 3D games: Super Mario Galaxy, Mario Kart Wii and Mortal Kombat. Using desktop PC, we were able to train the agent complete some of the most difficult tasks within each game. Namely, the final level in Super Mario Galaxy, Rainbow Road (a notoriously difficult track in Mario Kart Wii), and defeating all opponents in Mortal Kombat Endurance mode. For details about the environ- ments and setup. see Appendix[] To achieve this, BTR re- quired minimal adjustments: changing the input image res- olution to 140x 14 (from Atari's 84x84) due the game's higher resolution and aspect ratio, and to reduce the number of vectorized environments to 4 due to the games' memory and CPU requirements.
shows box plot comparison of final performance. In comparison human expert performance, BTR equals or exceeds them in 52 of 60. Importantly, we find that BTR appears to continue increasing performance beyond 200 million frames, indicating that higher performance is still possible with more time and data. Results tables and graphs can found Appendices and respectively.
To further confirm BTR's performance, we benchmark on Procgen (Cobbe al. 2020), procedurally generated set of environments aiming prevent overfitting to specific tasks, a prevalent problem in RL (Justesen et al. 2018 Ju- liani et al.] 2019). The results are shown in Figure with individual games in Appendix B BTR able to exceed Rainbow DQN- Impala's performance, despite using sig-
BTR was able to solve all three games, including consis- tently finishing first place in Mario Kart. In contrast, Rain- how DQN's performance plateaued before completing any of the games. We provide videos of our agent playing all
5

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
Super Mario Galaxy. Frii Level
Mario Kart Wi: Rainbo Road
Kombat: Endurance
My A
Seyond kainhon
MMA Mo
130
Ra nloe DON
Framec;
Consstort Completion
177M Frames))
Rist Bace Completion
Average
Ramboo
Beyontt Rarbow BGN
100
150
250
50
100
Fra M.llrdd
355
200
366
24
THE
120
MAP
HA HA.Pkon
Figure 4: BTR being used to play Super Mario Galaxy (final level), Mario Kart Wii (Rainbow Road) and Mortal Kombat: Armageddon (Endurance Mode) respectively. Consistent completion is defined as over 90%.
three Wii games and all games in the Atari-5 benchmark
5.1. Ablations Studies
BTR amalgamates independently evaluated components into single algorithm. To understand and verify each com- ponent's 's contribution, Figure 5 plots BTR's performance without each component on the Atari-5 benchmark 6
5. Analysis
Given BTR's performance demonstrated Section in this section, we ablate each component to evaluate their performance impact (Section (.T). Using the ablated agents, we measure numerous attributes during and after training assess each component's impact (Section 5.2
We find that Impala had the largest effect on performance (+ (+142% IQM), with the other components generally caus- ing a less significant effect. Despite this, simply using Rainbow with Impala does not produce similar results (6.3 IQM compared to 7.7 Atari-5). Munchausen and IQN have a strong impact on environments requiring fine-
Shttp:: http:://wwwwwouuuuueee//ooo//o/p fu2
P14geJsKiONN ONN htw.::0wwoSu.ua...a.a..aaaaaaaa.
"Duc to the resources required evaluate all environments, Aitchison et al. (2025) proposes subset of games that closely correlate with the performance across all of them.
Atari-57 Predicted Median From Atari-5
8
7
6
Atari-5 1QM
5
4
4
3
1
0
0
50
100
150
200
0
50
100
150
200
Number of Frames (in millions)
Number of Frames (in millions)
Beyond Rainbow Impala
na IQN
no Munchausen
Impala
Maxpool
Spectral
Figure 5: BTR's human- normaliied scores without different components, with shaded areas showing 95% bootstrapped confidence intervals averaged over seeds. Left: Predicted Atari-57 median score using regression procedure defined in Aitchison et al. (202)). However, we find the prediction does not match the true median (see Appendix K Right: Interquartile mean across the games. For individual game graphs and additional ablations, Appendices Badd
6

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
grained control such Phoenix, as explored in Section 5.2
20
heurnns
For vectorization and maxpooling, while their inclusion re- duces performance, we find their secondary effects crucial to keep BTR computationally accessible. Omitting vector- ization increases walltime by 328% ( by process- ing environment steps in parallel and taking fewer gradi- ent steps (781,000 compared to Rainbow DQN's 12.5 mil- We find that maxpooling decreases the model's pa-
25
25
50
75
100
125
150
175
200
lion) | rameters by 77%, and makes using wider convolutional lay- ers possible without causing the total number of parameters to increase drastically.
Shank
100
0
75
150
175
25
Laver Norm
120
BTR w/o Vectorization
+328%
15
Rainbow
+215%
FERainbow
+499
BTR w/o Maxpooling
+7%
0
75
150
BTR
11
+-0%
Beyend The Rambbon (BTR)
IGN
Murchauzen IM Speciall
BTR w/o Spectral
11
-0%
19 Ieseela
Marcoo
BTR w/o Munchausen
10
-8%
BTR w/o IQN
10
-8%
Figure 7: Plot showing % of dormant neurons (Sokar al. 2023), SRank with 5 0.01 (Kumar et al. 2021) and 1.2 norm of network weights, for details see Appendix E.3 Re- sults are averaged over 3 seeds and 5 tasks (Atari-5), Shad- ing shows 95% confidence intervals.
BTR IMPALA
-50%
30 Training Hours Single Desktop PC (Atari 200M)
Figure 6: Walltime of BTR on a desktop PC with com-
ponents removed, compared with Hessel et al. (2018) and Schmidt & Schmied (2021) For hardware details, see Ap- pendix
Furthermore, we find that maxpooling produces more robust policy. To test this, we evaluate the performance of BTR's ablations when taking different quantities of e- actions and with altered observations and find maxpooling alleviates some of the performance loss (Table Dp Lastly, we find Munchausen and IQN to have a significant impact Policy Churn (Schaul et al., 2022), with Munchausen reducing it by 6.4% and IQN increasing it by 3.3%. As result, when these components are used together, they ap- pear to reach a level of churn which does not harm learning and potentially provides some exploratory benefits.
5.2. What are the effects of BTR's components?
To help interpret the results in Section 5.1 Table 2 mea- sures seven different attributes of the agent either during or after training: action gaps and action swaps (linked to caus- ing approximation errors (Bellemare et al. 2016);; policy churn (which can cause excessive off-policyness and insta- bility (Schaul et al.) )) and score with additional noise (indicating robustness of the policies).
Lastly, Figure shows an analysis of the trained model weights across the Atari-5 benchmark. We find little dif- ference between trained models other than when removing Impala, which decreases dormant neurons and increases the L2 Norm of different layers, which have been linked with plasticity loss (Lyle et al. 22024)
While is clear that Impala strongly contributes to per- formance, we find that without BTR's other components the learned policy is highly noisy and unstable. Table [ demonstrates that without IQN and Munchausen the agent experiences very low action gaps (absolute Q-value differ- ence between the highest two valued actions), causing the agent to swap its argmax action almost every other step. This is likely to result in approximation errors altering the policy and causing a high degree of off-policyness in the replay buffer. This is particularly detrimental in games requiring Aie-grainee control, such as Phoenix where the agent needs to narrowly dodge many projectiles, reflected in BTR's 's performance without these components.
6. Related Work
The most similar work BTR, developing computationally-Imiine non-distributed RL algorithm, is "Fast and Efficient Rainbow" (Schmidt & Schmied, 2021). They optimized Rainbow DQN to maximize performance for 10 million frames through parallelizing the envi- ronments and dropping C51 along with hyperparameter optimizations. This differs from our goals of producing
A result of removing vectorization is using smaller batches, which Obando Ceron el al. (2024) finds improves exploration.

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
Table 2: Comparison of policy churn, action gaps, actions swaps and evaluation performance with different quantities of e-actions and color jitter (both only applied for evaluation). All measurements use the final agent, trained on 200 million frames, for Atari Phoenix, averaged over seeds. Action Gap is the average absolute Q-value difference between the highest two valued actions. Actions Swap is the percentage of times the agent's 's argmax action has changed from the last timestep. Policy churn is the percentage of states in which the agent's argmax action has changed after single gradient step. Color jitter applies random 10% change to the brightness, saturation and hue of cach frame. For associated errors with these values, please see Appendix E.3
Category
BTR
w/o Munchausen
w/o IQN
w/o SN
w/o Impala
w/o Maxpool
Action Gap
0,282
0.055
0.180
0.274
0,215
0.264
% Action Swaps
36.6%
47.7%
42.2%
40.3%
41.1% 1%
39.3%
Policy Churn
3.8%
11.0%
0.5%
3.3%
4.5%
4.2%
Score ColorJitter
212k
85k
110k
187k
19k
187k
Score 0.03
94k
42k
62k
75k
10k
86k
Score 0.01
194k
70k
110k
132k
13k
171k
Score E = 0
330k
184k
187k
296k
21k
406k
an algorithm that scales across training regimes (up to 200 million frames) and domains (Atari, Procgen, Super Mario Galaxy, Mario Kart and Mortal Kombat), resulting in different design decisions.
better than BTR, all are inaccessible smaller research labs or hobbyists due to their required computational re- sources and walltime. Therefore, while these algorithms have important research value demonstrating the possible performance of RL agents, performative algorithms with a lower cost of entry, like BTR, are necessary for RI. to be- come widely applicable and accessible.
For less computation limited approaches, Ape-X X (Horgan et al. 2018) was the first to explore highly distributed train- ing, allowing agents to be trained on a billion frames in 120 hours through using > 100 CPUs. Following this, Kaptur- ooski et al. 2018) proposed R2D2 using a recurrent neu- ral network, increasing sample efficiency but slowing down gradient updates by 38%. Agent57 (Badia et al., 2020a) was the first RL agent to achieve superhuman performance across 57 Atari games, though required 90 billion frames. MEME (Kapturowski et al., 2023), Agent57's successor, focused on achieving superhuman performance within the standard 200 million frames limit, achieved by using significantly higher replay ratio and larger network archi- tecture. Most recently, Dreamer-v3 (Hafner et al. 2023 used a 200 million parameter model requiring over a week of training, achieving similar results as MEME. We detail some key differences between BTR, MEME and Dreamer- v3 in Table g
7. Conclusions
We have demonstrated that, once again, independent im- provements from across Deep Reinforcement Learning can be combined into single algorithm capable of pushing the state-of-the-ar far beyond what any single improvement is capable of. Importantly, we find that this can be accom- plished on desktop PCs, increasing the accessibility of RI. for smaller research labs and hobbyists.
We acknowledge there exists many more promising im- provements we could not include in BTR, leaving room for more future work to create stronger integrated agents in a few years. For example, BTR does not add an explicit exploration component, resulting in it struggling in hard-
While these approaches perform equally or
Table 3: Comparison of performance, walltime, observations and complexity of different algorithms.
Category
BTR
MEME
Dreamer-v3
A100 GPU Days
0.9
Not Reported
7.7
Recurrent?
No (4 stacked frames) Single Transitions No
Yes
Yes
Learns from?
Trajectories (length 160)
Trajectories (length 64)
World Model
No
Yes
Parameters
2.9M
Not Reported (=>>20M
200M
Observation Shape
84x84
210x
64x64
Gradient Steps
781K
3.75M
1.5M
Atari-60 IQM
7.4
9.6
9.6
8

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
exploration tasks such as Montezuma's Revenge; therefore, mechanisms used in Never Give Up (Badia et al. 2020b), etc may prove useful. Section 5.1 found that the neural network's 's core architecture, Impala, had the largest impact on performance, an area we believe is generally underap- preciated in RL. Previous work (Kapturowski et al. 2018) has incorporated recurrent models enhancing performance, though we are uncertain how this can be incorporated into BTR without affecting its computational accessibility, question which warrants future research.
Impact Statement
This paper presents work whose goal is to advance the field of Reinforeement Learning, particularly to improve acces- sibility to those with limited computational resources. As with any work increasing accessibility, this has potential for misuse by bad actors. However, we believe these con- cerns are offset by the field's potential to tackle key socictal problems.
Acknowledgments
This work supported by the UK Research and In- novation (UKRI) Centre for Doctoral Training in Ma- chine Intelligence for Nano-electronic Devices and Sys- tems [EP/S024298/1] and the Engineering and Physical Sciences Research Council (EPSRC) ActivATOR project [EP/W017466/1]. The authors acknowledge the use of the IRIDIS High Performance Computing Facility, and the Southampton-Wolfe Wolfson AI Research Machine (SWARM) GPU cluster generously funded by the Wolfson Founda- tion, together with the associated support services at the University of Southampton in the completion of this work. The authors dedicate this work to the memory of George Patton-Fallows, whose passion for computer science in- spired this research.

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC
References
Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. Leverag- ing procedural generation benchmark reinforcement learning. In International conference machine learn- PMLR, 2020.
Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C.., and Bellemare, M. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural in- formation processing systems, 34:29304-29933 2021.
ing. pp.
Cobbe, K. W., Hilton, J., Klimov, O., and Schulman, J. Phasic policy gradient. In International Conference on Machine Learning, pp. 2020-2024 PMLR, 2021.
Aitchison, M., Sweetser, P., and Hutter, M. Atari-5: Distill- ing the arcade learning environment down to five games, In International Conference on Machine Learning, Pp. 421-438. PMLR, 2023.
Dabney, W., Ostrovski, G., Silver, D., and Munos, R. Im-
plicit quantile networks for distributional reinforcement learning. In International conference on machine learn- ing. pp. 1096-1105, PML............MM PMLR. 2018.
Badia, A. P., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, Z. D., and Blundell, C. Agent57: Outperforming the atari human benchmark. In Inter- national conference on machine learning, pp. 507-517. PMLR, 2020a.
Dolphin-Emulator: Dolphin emulator.
https:
gi www.c///ooooommoeocco cessed: 2024-09-30.
phin 2024. Ac-
Badia, A. P., Sprechmann, P., Vitvitskyi, A., Guo, D.,, Piot, B., Kapturowski, S., Tieleman, O., Arjovsky, M., Pritzel, A., Bolt, A. et al. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2020b.
D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-..,, Belle- mare, M. G., and Courville, A. Sample-efficient rein- forcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022.
Ball, J., Smith, L.., Kostrikov, I,, and Levine, S. Efficient online reinforcement learning with offline data. In Inter- national Conference on Machine Learning, pp. 1577- 1594. PMLR, 2023.
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., al. Impala: Scalable distributed deep-rl with impor- tance weighted actor-learner architectures. In Interna- tional conference on machine learning, pp. 1407-1416. PMLR, 2018
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M.
The arcade learning environment: An evaluation plat- form for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Fortunato, M., Azar, M. G., Piot, B., Menick. J., Hessel, M., Osband, I., Graves, A., Mnih, V., Munos, R., Hass- abis, D., et al. Noisy networks for exploration. In Inter- national Conference Learning Representations, 2018.
Bellemare, M. G., Ostrovski, G., Guez, A., Thomas, P., and Munos, R. Increasing the action gap: New operators for reinforcement learning In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 30, 2016.
Gallici, M., Fellows, M., Ellis, B., Pou, B., Mas- mitja, I., Foerster, N., and Martin, M. Simplify- ing deep temporal difference learning. arXiv preprint WwXiiv2407.044 2024.
Bellemare, M. G., Dabney, W., and Munos, R. A distri- butional perspective on reinforcement learning. In Inter- national conference on machine learning, pp. 449-458. PMLR, 2017.
Gogianu, F., Berariu, T., Rosca, M. C., Clopath, C., Bu- soniu, L., and Pascanu, R. Spectral normalisation for deep reinforcement learning: optimisation perspec- tive. In International Conference Machine Learning. pp. 3734-3744, PMLR, 2021.
Bjorck, N., Gomes, C. P., and Weinberger, K. Q. Towards deeper deep reinforcement learning with spectral nor- malization. Advances in neural information processing systems, 34:8242 2225, 2021.
Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Master- ing diverse domains through world models, 2023. URL https://arxiv. arg/abs/2301.04104 2023.
Ceron, J. S. O. and Castro, P. S. Revisiting rainbow: Pro- moting more insightful and inclusive deep reinforcement learning research. In International Conference on Ma- chine Learning, pp. 1373-1383, PMLR, 2021.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 770-778, 2016.
Ceron, J. S. O., Araújo, J. G. M., Courville, A., and Castro, S. On the consistency of syper-parameter selection in value-based deep reinforcement learning. In Reinforce- ment Learning Conference, 2024.
Hessel, M., Modayil, J., Van Hasselt, H.,, Schaul, T., Ostro- vski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and
10

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC
Combining improvements in deep reinforcement learning. In Proceedings of the AAAI con- ference artificial intelligence, volume 32, 2018.
Silver, D. Rainbow:
Lyle, C., Zheng, Z., Khetarpal, K., van Hasselt, H., Pas- canu, R., Martens, J., and Dabney, W. Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402. 18762, 2024.
Hessel, M., Danihelka, I., Viola, F., Guez, A., Schmitt, S., Sifre, L., Weber, T., Silver, D., and Van Hasselt, Muesli: Combining improvemetss in policy optimiza- tion. In International conference machine learning, pp. 4214 4226. PMLR, 2021.
Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., and Bowling, M. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61 01:3255662, 2018.
Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hes- sel, M., van Hasselt, H., and Silver, D. Distributed prior- itized experience replay. In International Conference Learning Representations, 2018.
Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. Spectral normalization for generative adversarial net- works. In International Conference on Learning Rep- resentations, 2018.
Jesson, A. and Jiang, Y. Improving generalization on the procgen benchmark with simple architectural changes and scale. arXiv preprint PrXiv:2410.10900 2024.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning, arXiv preprint 0rXiv:/312.5602. 2013.
Juliani, A., Khalifa, A., Berges, V.-P., Harper, J., Teng, E., Henry, H., Crespi, A., Togelius, J., and Lange, D. Ob- stacle tower: generalization challenge in vision, con- trol, and planning. In Proceedings of the Twenty-Eighth International Join! Conference Artificial Intelligence, 2019.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Ve- ness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level con- trol through deep reinforcement learning. nature, 518 (7540):529-533, 2015.
Justesen, N., Rodriguez Torrado, R., Bontrager, P., Khalifa, A., Togelius, J., and Risi, S. Illuminating generalization in deep reinforcement learning through procedural level generation. In NeurlPS Workshop on Deep Reinforce- ment Learning, 2018.
Obando Ceron, J., Bellemare, M., and Castro, P. S. Small batch deep reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay. arXiv preprint arXiv:1511.0592) 2015.
Kapturowski, S., Ostrovski, G., Quan, J., Munos, R., and Dabney, W. Recurrent experience replay in distributed reinforcement learning. In International conference learning representations, 2018.
Schaul, T., Barreto, A., Quan, J., and Ostrovski, G. The phenomenon of policy churn. Advances in Neural Infor- mation Processing Systems, 35:2537-2549, 2022.
Kapturowski, S., Campos, V., Jiang, R., Rakicevic, N., van Hasselt, H., Blundell, C.,, and Badia, A. P. Human-level atari 200x faster. In The Eleventh International Confer- ence on Learning Representations, 2023.
Schmidt, D. and Schmied, T. Fast and data-efficient train- ing rainbow: an experimental study on atari. Deep RL Workshop NeurIPS 2021, 2021.
Kumar, A., Agarwal, R., Ghosh, D.. and Levine, S. Im- plicit under- parametittaa inhibits data-efficient deep reinforcement learning. In International Conference Learning Represeniations, 2021.
Schritwiesser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588 (7839):604 609, 2020.
Lin, L.-J. Self-improving reactive agents based on rein- forcement learning, planning and teaching. Machine learning, 8:293-321, 1992.
Schrittwieser, J., Hubert, T., Mandhane, A., Barekatain, M., Antonoglou, I., and Silver, D. Online and offline re- inforcement learning by planning with learned model. Advances Neural Information Processing Systems, 34: 27580-27591, 2021.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. In International Conference on Learning Rep- resentations, 2019.
Lv, K., Yang, Y., Liu, T., Guo, Q., and Qiu, X. Full param- eter fine-tuning for large language models with limited resources. In ACL. (1), 2024.
Schwarzer, M., Ceron, S. O., Courville, A., Bellemare, M. G., Agarwal, R., and Castro, P. S. Bigger, better, faster: Human-level atari with human-level efficiency.

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
In International Conference on Machine Learning, pp. 00865-30380. PMLR, 2023.
Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. The dor- mant neuron phenomenon in deep reinforeement learn- ing. In International Conference on Machine Learning, pp. 3212114 32145-32166. PMLR, 2023.
Sutton, R. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
Sutton, R. S., Barto, A. G., et al. Introduction to rein- forcement learning, volume 135. MIT press Cambridge, 1998.
Toromanoff, M., Wirbel, E., and Moutarde, F. Is deep rein- forcement learning really superhuman on atari? In Deep Reinforcement Learning Workshop of 39th Conference on Neural Information Processing Systems (Neurips' 2019), 2019.
Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., et al. Gymnasium: A standard in- terface for reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024.
Van Hasselt, H., Guez, A., and Silver, D. Deep reinforce- ment learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 30, 2016.
Vieillard, N., Pietquin, O., and Geist, M. Munchausen re- inforcement learning. Advances in Neural Information Processing Systems, 33:4235 4246, 2020.
Wang, S., Liu, S., Ye, W., You, J., and Gao, Y. Effi- cientzero v2: Mastering discrete and continuous control with limited data. In International Conference on Ma- chine Learning, pp- 51041-1110000 PMLR, 2024.
Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M.. and Freitas, N. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1993-2003. PMLR, 2016.
Watkins, C. J. and Dayan, P. Q-learning Machine learning, 8:279-292, 1992.
12

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
A. Full Results Tables
Table A1: Maximum scores obtained during training (averaged over 100 episodes and all performed using random seeds)
after 200M Frames on the Atari-60 benchmark. Fast & Efficient Rainbow DQN and Munchausen-IQN refer to Schmidt & Schmied (02T)) and (Vieillard et 2020p respectively. FE-Rainbow uses Life Information (See Appendix only 10M frames, and has missing games, metrics are based existing games.
Game
Random
Humman
DQN (Nature)
Rainbow
M-IQN
FE-Rainbow
BTR
AirRaid
400
1000
7523
12472
19111
51719
Alien
227
7127
2354
3610
4249
12508
18999
Amidar
1719
1268
2390
1653
2071
14027
Assuult
222
742
1526
3490
6014
10709
19064
Asterix
210
8503
2803
16547
42615
346758
608829
Asteroids
719
47388
846
1494
1666
12345
153589
Atlantis
12850
29028
843372
791393
866810
812825
891773
BankHeist
753
560
1070
1305
1411
1482
BattleZone
37187
18425
40316
50501
112652
BeamRider
363
16926
5203
6084
12322
26398
110415
Berzerk
2630
832
719
11417
Bowling
23
160
30
43
23
40
63
Boxing
12
98
99
Breakout
30
92
109
241
537
682
Curnival
4000
4523
5588
Centipede
2090
12017
2378
6595
4425
8368
64242
ChapperCommund
Chopper
7387
TrazyClimer
10780
35829
1035549
146262
146419
140712
140927
DemonAttack
1971
17411
63143
131657
135626
DoubleDuukk
-18
-16
-5
22
21
23
ElevatorAction
3000
79372
89237
76941
Enduro
860
642
2165
2247
2266
2358
FerbyngDerby
-38
42
Freeway
29
26
33
33
34
34
Frostbite
4334
482
8309
9419
5282
15158
Gopher
257
2412
5440
9987
23310
25606
97879
Gruvitar
173
3351
1249
1105
2107
4253
Hero
1027
30826
15766
46290
25555
15377
25371
IceHockey
-11
0
0
11
44
Jameshond
302
671
995
1526
59991
JourneyEscape
-18000
-1000
-3300
1096
-806
2841
Kangaroo
3035
13005
10704
11498
Krull
1598
2665
6029
4368
10309
10324
11268
KungFuMaster
22736
22397
27066
25588
27444
53302
MontezumaRevenge
4753
500
0
0
MsPacman
6951
3431
3989
5630
5981
NumeThisGame
2292
8049
7549
8900
12440
19819
27917
Phoenix
8800
5315
427481
Pitfall
-229
6463
-55
-27
-32
Pong
20
19
Pooyan
500
1000
3452
4344
13096
22003
PrivatelEye
Qbert
163
13455
9801
18332
131.99
25712
42927
Riverraid
1338
17118
9725
20675
16143
25192
RoadRunner
11
7845
38430
55104
60370
81831
579800
Robotank
11
67
71
83
Seaquest
68
42054
2416
9590
23885
63724
428263
Skiing
-17098
-4336
-16281
-29268
-10404
-22076
-7938
Solaris
1236
12326
1478
1686
1835
2877
6301
SpuceInvaders
148
1668
1797
4455
10810
28098
54262
Star
664
10250
48498
57255
64875
310403
577547
Tennis
-23
-8
0
15
24
TimePilot
3568
$229
3704
11959
14600
31333
113801
Tutankham
167
103
244
205
167
297
UpNDown
533
11693
8797
37936
197043
391439
Venture
1187
13
1537
978
437
VideoPinball
17667
38720
460245
508012
269619
573774
WizardOrWor
563
4756
1473
7952
11352
18518
47314
YarsRevenge
3092
54576
46456
106929
98908
209499
Zaxxon
32
9173
4471
14983
14286
18832
52619
IQM (f)
0.000
1,000
0.771
1.852
2,181
2.709
7.361
Median (t)
0.000
1. 1,000
0.731
1,506
1,559
1.906
4.690
Mean (
0.000
1,000
2.261
4.152
5,260
7.700
21.574
Optimality Gap (1)
1.000
0.407
0.200
0.224
0.180
0.098
Best
3
3
54
Human
Surround
-10
Defender
2875
18689
169929
461.80
13

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
Table A2: Maximum scores obtained during training (averaged over 100 episodes and all performed random seeds) after 200M Frames on the Atari-5 Environment, compared against other non-recurrent non-distributed algorithms. FE-Rainbow refers to Fast and Efficient Rainbow DQN (Schmidt Schmied, 2021], and M-IQN refers to Munchausen-IQN (Vieillard
al. 2020 Metrics do use the recommended regression procedure, as explained Appendix K
Game
Random
Human
Rainhow DQN (Dopamine)
Rainhow DQN (Full)
M-IQN
FF-Rainbow
BTR
BattleZone
2360
37188
40895
62010
52517
112652
168340
DoubleDunk
-19
-16
22
22
NameThisGame
2292
8049
9229
13136
12761
19819
27917
Phoenix
761
7243
8605
108529
5327
60955
427481
QBert
164
13455
18503
33818
14739
25712
42927
IQM
0,000
1,000
1.265
3.583
1,452
4.070
7.739
Median
0.000
1.000
1.21
2.532
1.44
3.167
4.766
Mean
0.000
1.000
3.714
5.817
3.745
4.684
18.453
Table A3: Comparison of performance and walltime against PQN (Gallici et al. 2024). PQN only reports results at 400M frames and includes life information, which greatly affects performance (see Appendix [p. To provide fairer comparison, we also report our results using life information but only use 200M frames. Below are Atari-5 IQM and per-game Scores, with BTR averaged over 3 seeds. Human-Normalized scores are reported for individual games, with the raw score in brackets.
Game
BTR (with life info, 200M frames)
PQN (with life info, 400M frames)
Inter-Quartile Mean
12.18
3.86
BattleZone
12.73 (445,827)
1.51 (54,791)
DoubleDunk
14 (23.0)
6.03 (-0.92)
NameThisGame
4.51 (28,834)
3.18 (20,603)
Phoenix
90.85 (589,662)
38.79 (252,173)
QBert
9.79 (130,348)
2.37 (31,716)
Walltime (A100)
22 Hours
2 Hours
Backend
(PyTorch (non-compiled) + gymnasium async)
(JAX envpool)
14

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
B. Full Results Graphs
AirRaid
Alien
Amidar
Assault
Asterix
Asteroids
10
5
20
50
2
Atlantis
BankHeist
BattleZone
BeamRider
Berzerk
Bowling
50
5
Boxing
Breakout
Carnival
Centipede
ChopperCommand
CrazyClimber
20
100
0
DemonAttack
DoubleDunk
ElevatorAction
Enduro
FishingDerby
Freeway
20
50
20
2
0
0
Human Normalized Score
Frostbite
Gopher
Gravitar
Hero
IceHockey
Jamesbond
5
200
2.5
25
0.0
JourneyEscape
Kangaroo
Krull
KungFuMaster MontezumaRevenge
MsPacman
5
0.2
0.0
NameThisGame
Phoenix
Pitfall
Pong
Pooyan
PrivateEye
0.5
50
0,025
2.5
0.0
0,000
0
0.0
Qbert
Riverraid
RoadRunner
Robotank
Seaquest
Skiing
10
50
5
Solaris
Spacelnvaders
StarGunner
Tennis
TimePilot
Tutankham
2
0.5
50
20
0.0
0
UpNDown
Venture
VideoPinball
WizardOfWor
YarsRevenge
Zaxxon
5
10
20
20
Number of Frames (0 200 Million)
Beyond The Rainbow (BTR)
Rainbow
DQN (Nature)
Figure B1: Performance of BTR (4 Seeds) on each individual game in all 60 Atari games. Shaded areas show 95% confidence intervals over different seeds.
15

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
Score Distributions: ALE
Score Distributions with Non Linear Scaling
1.00
1.00
Fraction of runs with score > T
Fraction of runs with score
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
0
2
6
8
0.0
0.5
1.0
2.0
4.0
8.0
Human Normalized Score (T)
Human Normalized Score (T)
DQN (Nature)
C51
IQN
M-IQN
BTR
DQN (Adam)
REM
Rainbow
DreamerV2
Figure B2: Final performance of BTR on Atari-60 (as used in RLiable (Agarwal et al. 2021)), against other popular algorithms. The plot displays performance profiles, with 95% confidence intervals and 4 seeds for BTR.
Atari-5 IQM
BattleZone
DoubleDunk
8
Human Normalized Score
6
10
5
2
0
100
150
50
100
150
200
50
100
150
NameThisGame
Phoenix
Qbert
60
3
Human Normalized
40
2
2
20
1
0
0
50
100
150
200
50
100
150
200
0
50
100
150
200
Number Frames (millions)
Number of Frames (millions)
Number Frames (millions)
Beyond The Rainbow (BTR) Impala
no IQN
Munchausen
Rainbow Impala
no Maxpool
Spectral
Figure B3: Performance of BTR with different components removed on individual games, and interquartile mean in the top left. Results are averaged over 4 seeds, with shaded areas showing 95% bootstrapped confidence intervals.
16

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
IQM Human Normalized Score
8
Beyond The Rainbow (BTR)
PPO + IMPALA
6
4
2
0
0
25
50
75
100
125 150
175
200
Number of Frames (in millions)
Figure B4: Figure shows BTR against PPO Atari 57 (PPO's Atari 60 scores have not been reported). PPO uses the results provided by DreamerV3 (Hatner et al.) 2023), which additionally uses vectorization and the impala algorithm. Shaded areas show 95% confidence intervals with BTR using 4 seeds.
bigfish
bossfight
caveflyer
chaser
1.0
0.4
0.5
0.5
0.5
0.2
0.0
0.0
0.0
0.0
100
200
100 coinrun
200
100 dodgeball
200
100 fruitbot
200
climber
0.0
Min-Max Normalized Score
0.5
0.5
0.5
0.5
0.0
0.0
-1.0
0.0
0
100
200
0
100 jumper
200
100
200
0
100 maze
200
heist
leaper
1.0
0.5
0.0
0.5
0.5
0.0
0.2
0.0
0.0
0.5
0
100
200
0
100
200
0
100
200
0
100
200
miner
ninja
plunder
starpilot
0.50
0.50
1.0
0.5
0.25
0.25
0.5
0.00
0.00
0.0
0.0
0
100
200
0
100
200
100
200
0
100
200
Number of Frames (millions)
Beyond The Rainbow (8 Hours)
Rainbow DQN + Impala x4 (41 Hours)
Figure B5: Performance of BTR on each individual game in the Procgen benchmark. Shaded areas show 95% confidence intervals. The red dotted line shows the performance of Rainbow DQN Impala with 4x scaled Impala blocks (Cobbe) al. 2020 after 200M frames.
17

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
C. Additional Ablations
Atari-57 Predicted Median From Atari-5
5
Atari-5 1QM
2
0
25
50
75
100
125
150
175
200
0
25
50
75
100
125
150
175
200
Number of Frames (in millions)
Number of Frames (in millions)
Beyond The Rainbow (BTR)
Rainbow Vectorization
BTR Vectorization
Rainbow Impala
Figure C6: Figures show BTR's human-normalized scores without different components, with shaded areas showing 95% bootstrapped confidence intervals averaged over 4 seeds. Left: Predicted Atari-60 median score using the regression procedure defined in Aitchison et al. (2023). However, we find the predicted median does not match the true median (see Appendix KJ. Right: Interquartile mean across the games.
D. Hyperparameters
D.I. Environment Details
Table D4: Environment Details for Atari Experiments.
Hyperparameter
Value
Grey-Scaling
True
Observation down-sampling
84x84
Frames Stacked
4
Reward Clipping
[-1, I]
Terminal on loss of life
False
Life Information
False
Max frames per episode
108K
Sticky Actions
True
18

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC
Table D5: Environment Details for Procgen Experiments.
Hyperparameter
Value
Grey-Scaling
False
Observation Size
64x64
Frames Stacked
1
Reward Clipping
False
Max frames per episode
108K
Distribution Mode
Hard
Number of Unique Levels (Train & Test)
Unlimited
19

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
D.2. Algorithm Hyperparameters
Table D6: Table showing the hyperparameters used the BTR algorithm.
Hyperparameter
Value
Learning Rate
le-4
Discount Rate
0.997
N-Step
3
IQN Taus
8
IQN Number Cos"
64
Huber Loss K
1.0
Gradient Clipping Max Norm
10
Parallel Environments
64
Gradient Step Every
64 Environment Steps (1 Vectorized Environment Step)
Replace Target Network Frequency (C)
500 Gradient Steps (32K Environment Steps)
Batch Size
256
Total Replay Ratio
64
Impala Width Scale
2
Spectral Normalization
All Convolutional Residual Layers
Adaptive Maxpooling Size
6x6
Linear Size (Per Dueling Layer)
512
Noisy Networks o
0.5
Activation Function
ReLu
<-greedy Start
1.0
e-greedy Decay
8M Frames
e-greedy End
0.01
e-greedy Disabled
100M Frames
Replay Buffer Size
1,048,576 Transitions (220)
Minimum Replay Size for Sampling
200K Transitions 0.2
PER Alpha
Optimizer
Adam
Adam Epsilon Parameter
1.95e-5 (equal to batchsize 0.9
0.005
Adam ,31
Adam $2
0.999
Munchausen Temperature
0.03
Munchausen Scaling Term
0.9
Munchausen Clipping Value (l))
-1.0
Evaluation Epsilon
0.01 until 125M frames, then
Evaluation Episodes
100
Evaluation Every
1M Environment Frames (250K Environment Steps)
D.3. Clarity of the terms Frames, Steps and Transitions
Throughout the Arcade Learning Environment's history (ALE) (Bellemare et al. 2013 Machado et al. 2018), there have been many ambiguities around the terms: frames, steps and transitions, which are sometimes used interchangeably. Frames refer to the number of individual frames the agent plays, including those within repeated actions (also called frame skip- ping). This is notably different from the number of steps the agent takes, which does not include these skipped frames. When using the standard Atari wrapper, training for 200M frames is equivalent to training for 50M steps. Lastly, transitions refer to the standard tuple (st, ae Ft, St41), where the timestep refers to a steps, not frames. We encourage researchers to make this clear when publishing work, including when mentioning the values of different hyperparameters.
20

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
E. Beyond The Rainbow Architecture & Loss Function
E.1. Architecture
Figure E7 shows the neural network architecture of the BTR algorithm. The architecture is highly similar to the Impala architecture (Espeholt et al. 2018), with notable exceptions:
Q-Values
layer layer.mean
Spectral Normalization
Conv 3x3, Stride
Noisy Linear (1)
Noisy Linear
Advantage Stream
(Num Actions)
Value Stream
ReLu
Relu
Relu
Spectral Normalization
Conv 3x3, Stride
Linear (512)
NoisyLinear(511)
ReLu
Hadamard Product
Adaptive Maxpooling 6x6
ReLu
Spectral Normalization
Conv 3x3, Stride
Linear Layer 6x6x64
Impala CNN Block (64 Channels)
IQN Samples
Embedding)
ReLu
Impala CNN Block (64 Channels)
Cosine(Taus Taus
Spectral Normalization
Conv 3x3, Stride
Generate 8 Random Taus [o. 1]
Impala CNN Block (32 Channels)
ReLu
/ 255
MaxPool, Stride 2
Conv 3x3, Stride
(0-255), 4x84x84, Greyscale
Figure E7: Architectural diagram of the BTR algorithm's neural network. The model contains total of 2.91 million parameters, 2.52 million of which are within linear layers.
Spectral Normalization Within each Impala CNN block, each residual layer (containing two Conv 3x3 + ReLu) has spectral normalization applied, as discussed Section 3.1
Maxpooling Following the CNN blocks, a 6x6 adaptive maxpooling layer is added.
21

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
IQN
In order to IQN, it is required draw Tau samples, which multiplied by the output of the CNN layers, as shown by the section 'IQN Samples' in figure E7
Dueling Dueling (as included in the original Rainbow DQN) splits the fully connected layers into value and advantage
streams, where the advantage stream output has mean of and is then added the value stream.
Noisy Networks As included in Rainbow DQN, Noisy Networks replace the linear layers with noisy layers.
Lastly, the sizes of many of the layers given in Figure E7 are dependent upon the Impala width scale, of which we use the value 2. For example, the Impala CNN blocks have [16x width, 32x width, 32 width] channels respectively. The output size of the convolutional layers (including the maxpooling layer) is 6x6x32 width, as 6x6 maxpooling layer is used, Lastly, the cos embedding layer, after generating IQN samples, requires the same size as the output of the convolutional layers. Hence, the size is selected accordingly. Another benefit of the 6x6 maxpooling layer is following the product of the convolutional layers and IQN samples, the number of parameters is fixed, regardless of the input size. Figure E8 shows the number of parameters the ablated versions of BTR have.
no Maxpooling
8.8M
BTR
2.9M
+0%
+203%
no IQN
2.8M
-3%
no IMPALA
1.9M
-34%
0
2
4
6
8
Total Parameters (Millions)
Figure ES: Total number of parameters in BTR with different components removed. Those not included in the graph (Munchausen and Spectral Normalization) used the same number of parameters as BTR.
E.2. Loss Function
The resulting loss function for the BTR algorithm remains the same that defined in the appendix of the Munchausen paper, which gave loss function for Munchausen-IQN. As the other components in BTR do affect the loss, the resulting lemporal-differenee loss function is the same. For self-containment, we include this loss function below:
TDBTR a/7 In (a, s MR + Y m(als(4 (8441, In (alss 25 az)
(E1)
with T-|ss) sm ((a.) (that is, the policy is softmax with q the quantity with respect to which the original policy of IQN is greedy). It is also worth noting here that due to the character conflict of both Munchausen and IQN using (Munchausen as a temperature parameter, and IQN for drawing samples), we replace IQN's with a. lo. T and a are hyperparameters set by Munchausen. We use the same values BTR, also shown in our hyperparameter table Appendix D.2
E.3. Analysis Confidence Intervals
Due to space constraints, we was unable to include confidence intervals for Table paper Table can be found in Table E7 with the associated confidence intervals in Table ES
2 in the main paper. A repeat of the main
F. BTR with and without Epsilon Greedy
One of the first observations we made early in the testing process was that the inclusion of using e-greedy in addition to NoisyNetworks benefited some environments but not others. Specifically, performance was reduced on BattleZone and Phoenix, both games where the agent reached very high levels of performance with extremely precise control. However, DoubleDunk performed significantly worse, only reaching a score of 0, rather than the score of 23 the final BTR algorithm achieved. Similar findings were also found in the full version of Rainbow DQN, which used only NoisyNetworks, which
22

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC
Table E7: Repeat of the main paper's Table 2 for reference against the table of 95% confidence intervals below. Comparison of policy churn, action gaps, actions swaps and evaluation performance with different quantities of e-actions and color jitter (both only applied for evaluation). All measurements use the final agent, trained on 200 million frames, for Atari Phoenix, averaged over 3 seeds. Action Gap is the average absolute Q-value difference between the highest two valued actions. % Actions Swap is the percentage of times the agent's argmax action has changed from the last timestep. Policy churn is the percentage of states which the agent's argmax action has changed after single gradient step. Color jitter applies random 10% change to the brightness, saturation and hue of each frame. For associated error with these values, please see Appendix
Category
BTR
w/o Munchausen
w/o IQN
w/o SN
w/o Impala
w/o Maxpool
Action Gap
0.282
0.055
0.180
0.274
0.215
0.264
% Action Swaps
36.6%
47.7%
42.2%
40.3%
41.1%
39.3%
Policy Churn
3.8%
11.0%
0.5%
3.3%
4.5%
4.2%
Score ColorJitter
212k
85k
110k
187k
19k
187k
Score 0.03
94k
42k
62k
75k
10k
86k
Score e 0.01 194k
70k
110k
132k
13k
171k
Score 0
330k
184k
187k
296k
21k
406k
Table E8: 95% confidence intervals for the main paper Table calculated using 6 seeds. A repeat of the original table is
shown above in Table E7
Category
BTR
w/o Munchausen
w/o IQN
w/o SN
w/o Impala
w/o Maxpool
Action Gap
[0.25, 0.31]
[0.05, 0.06]
[0.17, 0.19]
[0.25, 0.3]
[0.1, 0.33]
[0.23, 0.3]
% Action Swaps
[33.6, 39.6]
[45.8, 49.7]
[41.0, 43.3]
[38.5, 42.2]
[31.4, 50.8]
[38.2, 40.4]
Policy Churn
[3.3, 4.2)
[10.2, 11.7]
[0.5, 0.5]
[2.9. 3.6]
[3.9, 5.0]
[3.7, 4.7]
Score ColorJitter
[204k. 218k]
[77k, 92k]
[90k, 128k]
[173k. 201k]
[Ok, 40k]
[158k, 215k]
Score 0.03
[88k, 99k]
[38k, 46k1
[52k, 72k]
[68k, 82k]
[2k, 18k]
[74k, 97k1
Score = 0.01
[177k, 211k]
[62k, 77k]
197k, 122k]
[139k, 176k]
[Ok, 52k]
[143k, 199k]
Score = 0
[282k, 377k]
[116k, 251k]
[163k, 209k]
[244k, 348k]
[Ok, 43k]
[332k, 479k]
achieved a best score of -0.3 (Dopamine's 's "compact" Rainbow DQN, however, which did not use NoisyNetworks achieved 22). From this, we conclude that NoisyNetworks alone failed to sufficiently explore the environment, whereas <-greddy did not. From these results, we eventually decided to use both methods, but disable e-greedy halfway through training to reap the best of both techniques.
G. Experiment Compute Resources
G.1. Our Compute Resources
For running our experiments, we used mixture of desktop computers and internal clusters. The desktop PCs used an GPU Nvidia RTX4090, CPU intel 19-14900k and 64GB of DDRS 6000mhz RAM. When using internal clusters, we used a mixture of GPUs, including Nvidia A A100s, Nvidia Volta V100 and Nvidia Quadro RTX 8000. As for CPUs, we used 2 x 2.4 GHz Intel(R) Xeon(R) Gold 6336Y, Cores. Lastly, we saved the models used to produce our analysis, totalling around 300gb across all of our ablations on the Atari-5 benchmark, saving model every million frames.
As most of our experiments were performed on desktop PC, in the main body of our paper we reference these speeds. We found that desktop PCs actually outperformed internal clusters, likely due to desktop CPUs being more suited to performing environment steps, outlined in the next subsection.
When testing ideas originally (those mentioned Appendix H we only tested them using a single run of the games BattleZone, NameThisGant and Phoenix unless otherwise stated. Whilst this method of evaluation is not statistically significant, for preliminary purposes with computational restrictions, we deemed this the best option.
23

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
G.2. BTR with Different Hardware
In this work, we look make nigh-performance RL more accessible those with fewer computing resources, especially those only with access to desktop computers. Most of our experiments were performed with RTX4090, we also provide some walltimes for 200M Atari frames for lower-end machines and provide brief comparison of desktop PCs against internal clusters:
Desktops:
Original: RTX 4090, Intel 19-13900k (2023), 64GB RAM- 11.5 Hours
RTX 3070, Ryzen 3900X (2019), 64GB RAM 52 Hours
RTX 2080 ti, Intel(R) Xeon(R) Silver 4112 CPU @ 2.60GHz (2018), 128GB RAM 32 Hours
Internal Clusters:
Nvidia H100, 48 Core Intel(R) Xeon(R) Platinum 8468 (2023), 2TB RAM 15 Hours
Nvidia A 100, 24 Core Intel(R) Xeon(R) Gold 6336Y (2021), 512GB RAM 22 Hours
We note that there is significant variability in hardware (processors, memory bus speeds, etc), but the results still show reasonable times compared to not using BTR. Overall, found that training BTR was very capable of running lower end machines, with the agent (excluding the environments) using around 15GB of RAM. The main performance bottleneck was running the environment in parallel, making the number of CPU cores and processor speed most important. BTR also provides strong performance long before 200M frames, thus providing practical utility for lower-end machines.
H. Other Things We Tried
Throughout the development of the BTR algorithm, experimented with many different components and hyperparame- ters. A brief list of ideas tried that performed worse equivalent to the final algorithm includes:
Using Exponential Moving Average networks rather than using fixed target networks (this was both computationally slower and performed worse).
Varying the frequency of updating the target network (we tested 250, 500 and 1000, finding 500 to perform best).
Changing the size of maxpool layer following the convolutional layers (we tested 4 and 8, however 6 performed significantly better).
Decaying the learning rate from 10-4 to 0 over the course of training (this made no significant difference).
Different learning rates, finding 1 10-4 to perform best, however 5 15 also performed similarly as was used in Implicit Quantile Networks (IQN).
Using the AdamW optimizer Loshchiiov & Hutter (019) which uses weight decay with the decay parameter le 4, however found this made no significant difference.
Using the GeLu activation instead of ReLu, which drastically reduced performance.
Only testing single environment (BattleZone), also tried:
Annealing the discount rate from 0.97 to 0.997 throughout training, but found significant difference.
Applying spectral normalization to the linear layers (dramatically worse performance).
Increasing the number of cos' from IQN (no significant difference performance).
Using Dopamine's Prioritized Experience Replay buffer which doesn't include value (moderately worse perfor- mance).
As discussed we also tried not using 6-greedy when using noisy nets.
24

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
Lastly also tried removing some of the original components from Rainbow DQN Atari BattleZone, including Duel- ing, Prioritized Experience Replay and Noisy Networks. Prioritized Experience Replay and Noisy Networks both proved beneficial, were kept in the algorithm. Dueling did not seem make any significant difference, however did not choose to remove it for clearer continuation of Rainbow DQN, in addition to potentially being useful in other Atari environments.
Shortly after the submission of this work, we tested BTR with addition of Layer Normalization, and found positive results. Layer Normalization can improve the robustness to a variety pathologies that cause loss of plasticity (Lyle al. 2024). and helps to improve the conditioning of the network's gradients in RL (Ball al. 2023). Below Figure H9 we show the impact of including layer normalization into BTR.
7
Atari-57 Predicted Median From Atari-5
6
Atari-5 IQM
4
2
0
0
50
100
150
200
0
50
100
150
200
Number of Frames (in millions)
Number of Frames (in millions)
Beyond The Rainbow (BTR)
BTR LLayerNorm
Figure H9: Graph shows Atari-5 performance with and without layer normalization using Inter-quartile mean and Atari- 60 predicted median from Aitchison et al. (2023). Layer normalization uses 3 seeds, with shaded areas showing 95% confidence intervals.
Table H9: Maximum scores obtained during training (averaged 100 episodes and all performed random seeds) after 200M Frames on the Atari-5 Environment, compared to BTR with Layer Normalization.
Game
Random
Human
BTR + Layer Normalization
BTR
BattleZone
2360
37188
183240
168340
DoubleDunk
-19
-16
23
23
NameThiGGame
2292
8049
33258
27917
Phoenix
761
7243
493762
427481
QBert
164
13455
47384
42927
IQM
0.000
1.000
8.191
7.739
Median
0.000
1.000
5,379
4,766
Mean
0.000
1.000
20.836
18.453
I. Altered Atari Environment Settings
In order to investigate the impact of the environmental sticky actions parameter and to compare against other works, we include results for on the Atari-5 benchmark in Figure I10
25

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
Atari-57 Predicted Median From Atari-5
5
Atari-5 1QM
4
3
2
1
0
0
25
50
75
100
125
150
175
25
50
75
100
125
150
175
Number of Frames (in millions)
Number Frames (in millions)
Beyond The Rainbow (BTR)
BTR no Sticky Actions
Figure 110: Graph shows Atari-5 performance with and without sticky actions (sticky actions the default) using Inter- quartile mean and Atari-60 predicted median from Aitchison et al. (2023). No Sticky Actions uses a single seed, so this result should be used with caution.
Some prior works choose to pass life information to the agent (Schmidt Schmied 2021). To clarify, this is different to terminal on loss of life. Life information does not reset the episode upon losing a life, but does pass terminal the buffer, allowing the agent experience further into episodes while also giving the agent negative signal for losing life. This setting is not recommended in Machado et al. (2018), and works which use not comparable those which don't. To emphasize this point, take the three games from the Atari-5 and perform comparison.
Atari-57 Predicted Median From Atari-5
10
12
8
10
Atari-5 IQM
8
6
6
4
4
2
0
50
100
150
200
50 Number of Frames (in millions)
100
150
Number of Frames (in millions)
Beyond The Rainbow (BTR) BTR + Life Info
Figure 111: Graph shows Atari-5 performance with and without life information using Inter-quartile mean and Atari-60 predicted median from Aitchison et al. (2023). Life Information uses 3 seeds, with shaded areas showing 95% confidence intervals. From this we conclude results using life information are invalid for comparison.
J. BTR for Wii Games
BTR interfaces with different Wii Games via the Dolphin Emulator. Specifically, we use a forked repository of Dolphin Emulator allow Python scripts interact with the emulator. This includes loading savestates (used reset episodes),
26

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC
grabbing the screen PIL image the Wii's internal resolution of 480p (downsampled 140x I 14 and grey-scaled, used for all observations), reading the Wii's RAM (used for reward functions and termination conditions) and allowed programmatic input into the emulator (for setting actions), Using Dolphin's is portable setting, we are able to run multiple Dolphin Emulators simultaneously on the same machine. Each instance runs as unique process, and communicates with the agent via Python's 's multiprocessing library. Similarly to the Atari benchmark, for all games we used a frameskip of 4.
J.1. Super Mario Galaxy
This environment used Super Mario Galaxy's 's final level, The Center of the Universe, and had to make it to the final fight at the end of the game. The agent had 6 actions, including None, moving in each direction and jumping. Additionally, if the jump action was performed following a movement, the agent would continue to move in that direction.
Rewards were given via finding many values in the Wii's memory that resembled progress in the level. The agent was then rewarded for this progress value increasing from the last frame. If the agent's position entered set region, the progress variable would be moved. Additionally, the game uses a life system, where the player has a maximum of 3 lives and can lose or gain lives in many different ways. The agent was given reward of +1 for gaining life, and -1 for losing life. Lastly, episode termination occurred if the agent reached 0 lives, or if the agent made it to the end of the level. For this task, we also allowed the agent to start episodes at many points throughout the level, which rapidly sped up training since the agent could easily experience different areas of the level.
Whilst a difficult task, once the agent first completed the level, it did not take long to start consistently completing it due to the deterministic nature of the game.
J.2. Mario Kart Wii
The Mario Kart Wii environment had the agent play against the game's 's internal opponents (on hard mode, with 12 racers including the agent), on the course Rainbow Road (with items on the 150ce speed setting). The agent had to complete 4 laps of the course to finish the race. The agent had just 4 action, including accelerate, drifting left or right, and using its item. While this limited the agent's potential actions substantially, we found using fewer actions to dramatically accelerate training.
Rewards of +1 were given via reaching checkpoints that were scattered throughout the course (100 in total per lap). Additionally, if the agent's speed dropped below a set threshold (65 km/h), the agent would receive a reward of -0.01 per frame. The agent would be terminated with a reward of -10 if its speed dropped below the threshold for over 80 frames, or with reward of +10 for finishing the race, with a bonus based on the position the agent finished in. Lastly, the agent was rewarded with +1 for using its item. Without this reward, we found the agent to often neglect using its item, likely due to many of the items only providing rewards in the long term, such as slowing down other racers or blocking incoming items far in the future. Similarly to Super Mario Galaxy, we had the agent start the episode in multiple positions around the first lap, allowing it to experience the whole track early in training.
This agent took the longest to train, taking around 160M frames to reach consistent completion. In particular, the agent took a long time to consistently complete the race due to the other racers and randomized items making the environment highly stochastie, with many rare scenarios which could cause the episode to terminate.
J.3. Mortal Kombat
The Mortal Kombat environment put the agent in the game's endurance mode, where the agent would sequentially fight 15 different opponents, but keep retain its health between fights, and only gain health after defeating every 3 opponents. We provided the agent with 14 actions, including: None, Left, Right, Up, Down, Axe Kick, Punch, Snap Kick, Grab, Block, Toggle Weapon, Jump Left, Jump Right, and Crouch. These actions were far from the game's total action space, and limited the agent's ability to perform some of the combos within the game. We limited the agent's 's actions as the full action space is extremely large.
The agent was positively rewarded for damaging the opponent, and negatively rewarded for taking damage, with one taking one tenth of the health bar equating to +1 reward respectively. The episode was terminated with a reward - for reaching 0 health, and +10 for defeating the 15th and final enemy.
The Mortal Kombat agent learned considerably faster than Super Mario Galaxy and Mario Kart Wii, first completing the
27

---

Beyond The Rainbow: High Performance Deep Reinforcement Learning on Desktop PC
environment in 50M frames, and getting progressively consistent until training was stopped 90M frames. The agent quickly learned how dodge enemy hits, and relied heavily upon this strategy.
K. Atari-5 Regression Procedure
In our main paper ablation figure (Figure 5 we used the regression procedure recommended Atari-5 (Aitchison et al, 2023). This procedure is typically used to predict the Median score across entire game Atari suite, while only needing to use games. While we believe this procedure produces valid and useful plot, we find that BTR did differ significantly from the predicted value. We opted to use both the predicted median and the IQM across the 5 games to give two easy to interpret averages. Figure K12 shows the 57 game suite's true median, compared to the median predicted by Atari-5.
Median Human Normalized Score
BTR 57 Games
Rainbow DQN
5
DQN
BTR-Regressed
4
3
2
1
0
0
25
50
75
100
125
150
175
200
Number of Frames (Millions)
Figure K12: BTR's 's 60 game median against that predicted by Atari-5 using the regression procedure from Aitchison et al. 2023) Shaded areas show 95% confidence intervals using 4 seeds.
28


Reviews:
[{"review_id": "JtUU70HVg1", "reviewer": "Reviewer_bNh2", "summary": "The paper presents BTR, which integrates several well-established techniques to achieve strong performance on Atari and Procgen with limited computational resources. Through detailed ablation studies, the authors demonstrate how each component contributes to their method, showcasing the trade-offs between computational efficiency and RL performance. While falling short of DreamerV3 and MEME, BTR significantly surpasses Rainbow DQN on the Atari-60 benchmark in under 12 hours of training for 200 million frames on a Desktop PC.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their very thorough and constructive review, which clearly took a great deal of time and effort. We appreciate that the reviewer clearly understands the value of producing an accessible and high-performance algorithm.\n\n**Complexity of Wii Games** - We would like to argue that in particular, Mario Kart Wii is a very challenging environment, more so than it has been given credit for. This environment contains 11 other racers and the use of randomized items, introducing a very high degree of stochasticity into the environment, far beyond the stochasticity introduced in Atari. Furthermore, we tested the agent on the track “Rainbow Road”, commonly known as the game’s hardest track, taking over a minute even for a single lap, and featuring a very complex and noisy observation space (fully 3D, with multicoloured roads and constantly changing 3D backgrounds that resemble the track). Although Super Mario Galaxy uses a single fixed level, we would argue this is unique and more challenging than typical benchmarks due to complex movement mechanics and high-resolution images.\n\n**Claim of SoTa for non-recurrent RL** - While non-recurrent RL may not be a category of interest, we chose this phrasing as it is indicative of walltime efficiency, since off-policy recurrent algorithms (particularly image-based ones) are very expensive [1]. We thought this better than stating “SoTa for walltime efficiency”, or “SoTa for Desktop PCs”, as these are somewhat tricky to formally verify (though we believe it is important to the field). As you mention, strong baselines heavily rely on expensive recurrent models.\n\n**Dreamer v3 and MEME baselines** - As you mention, we already included these baselines in Table 3; we did not include these results in Figures 1 and 2  as it would be misleading to compare BTR against algorithms which are inaccessible to our targeted use case (academics, hobbyists, etc). However, to be more transparent about BTR’s place among these algorithms, we would be happy to add Dreamer v3 and MEME to Figure 2 (the box plot), while also stating the walltime required for these algorithms to emphasize the difference. As for Figure 1, we can replace this with a walltime efficiency curve that includes Dreamer v3 as we feel this better aligns with the paper’s contributions (MEME, however, did not release sample efficiency curves to our knowledge). We didn’t include PQN in any main paper figures/tables as they used Life Information, which we discuss at length in Appendix I, and leads to completely non-comparable results and is not standard (Table 3 also uses Atari-60, not Atari-5, and we didn’t perform experiments on Atari-60 using life information). \n\n**Wii Game Results** – As you mention, the setting is somewhat informal, and thus we find it difficult to make direct comparisons to other benchmarks. We did not intend to use these environments as formal benchmarks (we already rely on Atari and Procgen for that purpose), but rather as a demonstration of what BTR is capable of. Furthermore, we have been unable to find any other work on Super Mario 64 that uses a reinforcement learning approach, and we would be interested if you could reference such work. As for Super Mario Kart, we would like to highlight the difference in complexity between it and Mario Kart Wii, as the latter includes far more complex graphics, tracks, and driving mechanics (e.g., mini-turbos, wheelies, tricks, etc.).\n\n\n**Missing References** – We are happy to add the references you suggested to our section on Procgen to provide a more comprehensive discussion. We also found some of the work you referenced [2] very interesting, and it may help improve BTR’s performance on the Procgen benchmark.\n\n**Comments and Suggestions** – We are happy to address the minor adjustments you suggested in points 1, 3, and 5 of this section. As for point 2, we also considered adding LayerNorm, though it may introduce inconsistencies across the ablations and other figures, and could complicate the narrative of the paper. Regarding point 4, we are currently running additional experiments for the VizDoom environment, which we will add to the appendix.\n\n**Questions** – We define consistent completion as achieving over a 90% success rate, and we will add this definition to the caption of Figure 3. We believe your remaining questions have been addressed throughout our response.\n\n[1] Kapturowski, Steven, et al. \"Recurrent experience replay in distributed reinforcement learning.\" International conference on learning representations. 2018.\n[2] Jesson, Andrew, and Yiding Jiang. \"Improving Generalization on the ProcGen Benchmark with Simple Architectural Changes and Scale.\" arXiv preprint arXiv:2410.10905 (2024).", "claims_and_evidence": "1. The authors state that BTR outperforms the state-of-the-art performance for non-recurrent RL. However, the distinction of *non-recurrent* alone is not particularly meaningful or impressive, as employing RNN encoders is typically a deliberate design choice rather than a limitation. Although, in principle, this is a valid claim, due to the stronger baselines employing RNNs by default, I don’t think it is very noteworthy.\n2. The authors claim that BTR can solve modern games by demonstrating performance on three Wii games. However, these games do not present significantly greater challenges than typical Atari or ProcGen tasks. They feature limited discrete action spaces, dense rewards, and lack the complexity, variety, and stochasticity characteristic of modern games:\n- *Mortal Kombat: Armageddon*, despite its 3D physics engine, barely utilizes the 3rd dimension and effectively functions as a 2D side-scroller.\n- *Mario Kart Wii*, with only 4 discrete actions and repetitive lap-based gameplay, offers limited complexity and variety.\n- *Super Mario Galaxy* is fully deterministic, allowing an agent to directly overfit to a single level.\n    \n    To better substantiate the claim that BTR can handle modern games, the authors should evaluate it on more open-ended, truly 3D environments [1, 2].\n    \n\n[1] Raad, Maria Abi, et al. \"Scaling instructable agents across many simulated worlds.\" *arXiv preprint arXiv:2404.10179* (2024).\n\n[2] Fan, Linxi, et al. \"Minedojo: Building open-ended embodied agents with internet-scale knowledge.\" *Advances in Neural Information Processing Systems* 35 (2022): 18343-18362.", "methods_and_evaluation_criteria": "Although the authors did not introduce novel algorithmic enhancements, their research is valuable as it effectively combines existing RL components into a cohesive and efficient algorithm. By integrating several independently validated improvements, the authors successfully demonstrate a method that achieves high performance while remaining computationally accessible, targeting smaller research labs and hobbyists with limited computing resources. The authors attempt to bridge the gap between low-cost simple algorithms and resource-intensive, cutting-edge RL methods, thoroughly justifying their design choices for the trade-offs in computational efficiency versus performance.", "theoretical_claims": "This paper does not contain theoretical claims or formal proofs; its primary contributions are experimental and methodological.", "experimental_designs_or_analyses": "1. The component impact analysis is well executed, clearly showing how each part of BTR affects performance and training hours on Atari. The authors convincingly demonstrate how each design choice contributes not only to BTR's performance improvements but also to other metrics such as the percentage of dormant neurons, network weights, SRank, action gaps, and policy churn. Detailing the network architecture choices, loss components, and hyperparameters is particularly helpful. Explicitly discussing which attempts did not work, further adds transparency and makes the results more credible.\n2. The authors selectively omit evaluations of other baselines from key results without justification, artificially inflating the apparent superiority of BTR. **DreamerV3** and **MEME** achieve higher IQM scores on Atari-60 (9.6 vs. BTR’s 7.4, Table 3), yet they are excluded from Figures 1 and 2, Tables A1 and A2. Meanwhile, **PQN**, which runs ~20x faster but performs worse (Table A3), is omitted from Table 3, where BTR's lower walltime and network size are emphasized. This cherry-picking distorts the comparison and inflates BTR’s perceived advantage.\n3. While demonstrating BTR on three Wii games demonstrates the algorithm's versatility, these games are not established benchmarks, making it difficult to determine how notable BTR’s performance is relative to other methods. Without baseline comparisons or prior evaluations, it is unclear whether BTR’s results represent a unique achievement or if comparable performance could be attained by existing methods. To strengthen the evaluation, the authors should include comparisons with other RL baselines. Notably, basic RL algorithms have managed to beat similar games such as *Super Mario Kart*, *Mortal Kombat 3*, and *Super Mario 64*, although in informal, non-academic contexts, such as blog posts and YouTube videos.", "relation_to_broader_scientific_literature": "The contributions of this paper clearly build on existing RL literature. The authors integrate 6 previously established methods and algorithmic tricks to work with Rainbow DQN to achieve strong performance with limited computational resources.", "essential_references_not_discussed": "Prior works have achieved higher performance on ProcGen with similar or lower data budgets [1, 2, 3].\n\n[1] Jesson, Andrew, and Yiding Jiang. \"Improving Generalization on the ProcGen Benchmark with Simple Architectural Changes and Scale.\" *arXiv preprint arXiv:2410.10905* (2024).\n\n[2] Cobbe, Karl W., et al. \"Phasic policy gradient.\" *International Conference on Machine Learning*. PMLR, 2021.\n\n[3] Hafner, Danijar, et al. \"Mastering diverse domains through world models.\" *arXiv preprint arXiv:2301.04104* (2023).", "other_strengths_and_weaknesses": "I’ve outlined the strengths and weaknesses above.", "other_comments_or_suggestions": "1. Section E.3 consists only of Tables E7 and E8, which are placed on the following page, making it difficult to follow. Explicit in-text references to these tables would improve clarity and readability.\n2. For the camera-ready version, the authors could update the BTR results with LayerNorm, as Appendix H suggests it further improves performance.\n3. Figures 2 and B2 include results for the baseline *REM*, but this method is never introduced or referenced in the paper. This baseline likely refers to Random Ensemble Mixture [1],\n4. The authors could evaluate BTR on well-established image-based 3D benchmarks with discrete action spaces like DMLab [2] and ViZDoom [3].\n5. Typos\n    1. Lines 131-134 missing punctuation: *the convolutional layers,* ***which***\n    2. Line 180 *Boo**t**strapping*\n    3. Line 190 *with ~~with~~*\n    4. Line 320 *we find **that** maxpooling*\n    5. Line 1245 *many other ~~the other~~ techniques*\n    6. Line 1376 *we use a forked repository ~~of~~ to allow*\n    7. Mortal Combat —> Mortal **K**ombat\n\n[1] Agarwal, Rishabh, Dale Schuurmans, and Mohammad Norouzi. \"An optimistic perspective on offline reinforcement learning.\" *International conference on machine learning*. PMLR, 2020.\n\n[2] Beattie, Charles, et al. \"Deepmind lab.\" *arXiv preprint arXiv:1612.03801* (2016).\n\n[3] Kempka, Michał, et al. \"Vizdoom: A doom-based ai research platform for visual reinforcement learning.\" *2016 IEEE conference on computational intelligence and games (CIG)*. IEEE, 2016.", "questions_for_authors": "1. How is *consistent completion* established in the Wii games (Figure 3)?\n2. Why aren’t the Atari results of MEME and DreamerV3 not included in Figure 1 and Figure 2?\n3. How do other baselines perform on The Wii games?\n4. What is noteworthy about non-recurrent RL?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ggWlgJ9e22", "reviewer": "Reviewer_eCYf", "summary": "Rainbow now for few years has been a SoTA DQN based RL method. In this paper, authors redo the basic idea behind the Rainbow and collect a new, since Rainbow, a set of tips and tricks and include them into Rainbow, thus obtaining BTR, Beyond-the-Rainbow. They mostly evaluate performance of the completed system with ALE, but also run some experiments with three games from WII and some Procgen experiments.\n\n## update after rebuttal\n\nI appreciate authors rebuttal, but taking into account rebuttal and other reviewers opinions I will keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review and their appreciation of how our work will be useful for both researchers and practitioners.\n\n**Steps vs Walltime** – We feel the reviewer has somewhat misunderstood the main point of this paper. As stated in the title, abstract, and contributions, the purpose of our work is to create a high-performance algorithm that is computationally accessible—something that is poorly measured by environment steps. For example, while BTR takes approximately 12 hours to process 200 million frames, Dreamer v3 takes almost 8 days to do the same and requires considerably more RAM and GPU memory, illustrating that environment steps are a poor metric for computational accessibility.\n\n**Use of the ALE** – The ALE remains the most widely used RL benchmark for good reason, and we feel it is being unfairly dismissed here. The standard evaluation protocol for the ALE [2] (which we follow) includes both no-op starts and sticky actions, which are explicitly intended to prevent agents from exploiting determinism in the environment. Moreover, the ALE gained its popularity due to its extremely diverse set of tasks, a strength we believe is being overlooked. Other SoTa algorithms, such as MEME [3], benchmark exclusively on Atari for this very reason. The ALE also contains numerous sparse-reward environments, which can be found in our Atari-60 results.\n \n**Bump in Figure 1** – As discussed in Appendix F, this bump is due to epsilon-greedy exploration being disabled partway through training. While we found exploration beneficial early in training, disabling it halfway through led to improved performance.\n\n[1] Schwarzer, Max, et al. \"Bigger, better, faster: Human-level atari with human-level efficiency.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Machado, Marlos C., et al. \"Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents.\" Journal of Artificial Intelligence Research 61 (2018): 523-562.\n\n[3] Kapturowski, Steven, et al. \"Human-level atari 200x faster.\" arXiv preprint arXiv:2209.07550 (2022).", "claims_and_evidence": "Claims are convincingly supported by evidence.", "methods_and_evaluation_criteria": "Mostly makes sense, but the focus on wallclock time instead of environment steps can be misleading. Wallclock time measurement favors the use of vectorization, which is useful if the goal is to engineer a system that is able to get more bang out of the hardware, but I would argue that most RL researchers are interested in sample efficiency and there wallclock time is misleading as is the number of gradient steps. This point obviously does not discount the usefulness of vectorization that is amply demonstrated by this paper.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes I did check and no issues.", "relation_to_broader_scientific_literature": "-", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "It is clear that this paper is very useful for the RL researchers and practitioners. It contains an ample evidence on what tricks together are the most useful. I am sure the emplrical results in this paper will be greatly studies and scrutinized. Authors have taken their time to look at the results from many different points of view, I find Table 2 to be very instructive in terms of really evaluating the effectiveness of respective tricks. \n\nThis paper is styled from the benchmarking point of view, where idea seems to be to get the most out of ALE. This itself is not necessarily a bad thing as it is useful know what works and does not work with respect to experimenting with ALE. But it is known to have serious issues, such as non-stochasticity that is fixed in Procgen. But in this paper authors run only cursory experiments with Procgen, I would have liked to see maybe the roles reversed where Procgen is the focus and ALE is shown for benchmarking reasons. Finally, three games using Delphi, WII games, are just a curioisity. So, to strenghen the scientific message of this paper I would suggest to refocus this paper for Procgen. \n\nContinuing from the previous point, sparse reward scenarious would benefit from better sample efficiency, so environment that emphasizes it would then be most useful in this respect. One option is to use some scenarious in the MiniWorld.", "other_comments_or_suggestions": "-", "questions_for_authors": "- I am curious about the Fig 1, why there is a bump in the BTR curve?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LXzUNSEhUX", "reviewer": "Reviewer_sM3r", "summary": "This paper proposes Beyond the Rainbow (BTR), an algorithmic successor to Rainbow DQN that improves asymptotic performance, data-efficiency, and wall-time efficiency through a series of algorithmic and architectural modifications informed by existing but recent literature. Key changes include using an Impala backbone rather than the traditional 3-layer Nature CNN, adaptive maxpooling, spectralnorm, IQN value estimation instead of C51, Munchausen RL for TD-learning instead of Double DQN, and finally use of vectorized environments and hyperparameter-tuning for improved wall-time efficiency. Experiments are conducted on Atari as well as a number of 3D video game environments, and the authors demonstrate that BTR performs significantly better than Rainbow DQN and a number of other model-free baselines.\n\n## Post-rebuttal assessment\n\nAs mentioned in my rebuttal reply, I will maintain my score of weak accept but appreciate the authors' response to my comments as well as those of my fellow reviewers.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and constructive review, and for your appreciation of the evaluation on unique environments.\n\n**Comparison against baselines** - We already provided a discussion on SoTa algorithms for Atari 200M in Table 3, and two of the listed algorithms (BBF and EfficientZero V2) are sample sample efficient algorithms which only used 100K frames rather than 200M. Furthermore, they were never benchmarked on the full Atari set, but rather the reduced 26 game sample efficient benchmark. Therefore, we cannot include these in Figures 1 and 2. We did not include MEME and Dreamer v3 in Figures 1 and 2 as we did not want to present an ‘apples to oranges’ comparison, since these SoTa algorithms are inaccessible to our targeted users due to heavy compute requirements. However, to avoid any confusion around BTR’s performance, we are willing to replace Figure 1 with a walltime efficiency curve that includes Dreamer v3, as we feel this better conveys the message of the paper. We can also add Dreamer v3 and MEME to Figure 2, with a note indicating their walltime to allow for a fairer comparison as we believe that with the appropriate context this will improve the paper. If this is your main critique of the paper, given that this is simple to remedy, we hope it may be grounds for improving your score.\n\n| Algorithm                             | Frames | A100 Walltime        | IQM (26-Game Subset) |\n|--------------------------------------|---------------|----------------|----------------------|\n| MEME                                | 200M | Not Reported*            | 18.491               |\n| Dreamer v3                           | 200M | 7.7 Days                | 14.305               |\n| Beyond The Rainbow (BTR)            | 200M | 22 Hours                | 11.202               |\n| PQN [1]                                  | 400M | 1 Hour                | 5.014**                |\n| EfficientZero v2                     | 100K | 2.7 Hours***            | 1.305                |\n| BBF                                  | 100K | 7.8 Hours               | 1.045                |\n| Dreamer v3                           | 100K | 2.4 Hours               | 0.543                |\n\n\n*MEME used a shared server with a TPUv4.\n\n**PQN used life information, making its results appear significantly higher.\n\n***EfficientZero was tested on a server with 8 RTX 3090s, not an A100.\n\n**More Baselines for Wii Games** - As you mention, unfortunately this may not be possible during the rebuttal period. Out of interest, what algorithm would you like to see? We found other SoTa algorithms such as Dreamer v3 to be too computationally expensive to run on more demanding environments.\n\n[1] Gallici, Matteo, et al. \"Simplifying deep temporal difference learning.\" arXiv preprint arXiv:2407.04811 (2024).", "claims_and_evidence": "The main claim of the paper is that the algorithmic and architectural improvements upon Rainbow DQN lead to a new state-of-the-art for model-free (non-recurrent) RL on the chosen domains. The proposed changes are well motivated and validated through a series of ablation studies that provide empirical evidence for the claims. Experiments establish that BTR outperforms Rainbow DQN by a large margin on each of the chosen domains.  I appreciate the inclusion of a number of unconventional domains for evaluation (the 3D environments); they all look fairly challenging and represent real use cases for RL in the context of video games. My main issue with the experimental results is that all of the baselines are considered rather old and outdated. For example, the authors mainly compare against Rainbow DQN (2018), as well as methods for which numbers are reported in RLiable (2021). There has been substantial progress in the area of RL in the past 4 years, which is largely ignored in the experimental evaluations. This makes the claim that BTR achieves a new state-of-the-art somewhat shaky. Given that most RL algorithms for discrete action spaces report benchmark numbers for Atari (either 100k or 200M) I see little reason not to include more recent results, such as DreamerV3 (2023) as opposed to DreamerV2 (2020). If I read the results correctly, it appears that DreamerV3 achieves 8.30 human-normalized median score on Atari 60 vs. 4.69 for the proposed method, and 6.93 for MuZero [1]. While I undoubtedly find the proposed method appealing due to its wall-time efficiency, I feel that not positioning the results wrt any RL results more recent than 2021 is highly problematic. I include two model-based RL results here to prove my point, but I'm sure that there are more recent model-free results readily available for Atari 200M as well.\n\n[1] Hafner et al., DreamerV3: https://arxiv.org/abs/2301.04104 (2023)", "methods_and_evaluation_criteria": "Yes, I believe that the chosen domains are appropriate for benchmarking. I have concerns about the choice of baselines, as discussed above.", "theoretical_claims": "Not relevant. Paper is empirical in nature.", "experimental_designs_or_analyses": "Experimental design appears solid. I have concerns regarding baselines, as previously discussed.", "relation_to_broader_scientific_literature": "The contributions are well motivated and positioned wrt. previous work. There is a serious lack of discussion and comparison to more recent literature; the chosen baselines are 3-12 years old.", "essential_references_not_discussed": "I am not very familiar with recent literature on model-free RL algorithms for Atari 200M, but there has been substantial work on model-based algorithms for Atari 200M (MuZero and DreamerV3 come to mind [1]), as well as both model-free (DrQ, SPR, BBF) [2] and model-based (EfficientZero, EfficientZero-V2) [3] algorithms for Atari 100k. The paper would benefit from discussion and empirical comparison to more recent literature.\n\n[1] Hafner et al., DreamerV3: https://arxiv.org/abs/2301.04104 (2023)\n\n[2] Schwarzer et al., BBF: https://arxiv.org/abs/2305.19452 (2023)\n\n[3] Wang et al., EfficientZero-V2: https://arxiv.org/abs/2403.00564 (2024)", "other_strengths_and_weaknesses": "The paper is generally well written, the proposed algorithmic and architectural changes are well motivated, and ablations provide insights into the relative importance of each design choice. I appreciate the use of RLiable metrics.", "other_comments_or_suggestions": "The paper has a few minor typos but they do not detract from my understanding in any meaningful way.", "questions_for_authors": "I would like the authors to provide justification for their choice of baselines (in particular the wrt omission of more recent literature) as well as a general lack of discussion of newer literature. I have provided some references but hope that the authors can conduct a more thorough literature study as well given their focus on model-free methods for Atari 200M in particular.\n\nIt would be helpful to provide additional model-free baselines for the 3D video game domains but I understand that this may not be possible within the strict time frame of a rebuttal. I highly recommend comparison to a more recent algorithm for a future revision.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No notable concerns"}}, {"review_id": "rgP9FHUKLU", "reviewer": "Reviewer_SiM3", "summary": "Similar to Rainbow DQN, this paper integrates several improvements from existing RL literature to improve the performance of reinforcement learning agents. The authors show that their ensemble method BTR achieves state-of-the-art performance on Atari tasks using 200M frames on a desktop PC. The authors also show performance on other complex 3D games such as Super Mario Galaxy, Mario Kart, and Mortal Kombat. The ablation experiments in the paper show the impact of each improvement proposed.\n\n\n## Update after rebuttal\nDuring the rebuttal session, since the authors have added more recent baseline results that greatly improve the persuasiveness of the paper, I now tend to weak accept this paper. Good work!", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their concise review.\n\n**Life Information** - We do not use life information in any main paper results, as we specify in Section 4.1. Furthermore,  we provide a detailed comparison of BTR with and without life information in Appendix I. If your low score was due to doubts about our empirical performance stemming from this matter, we hope you are willing to reconsider now that this has been clarified.\n\n**Comparison against baselines** - We already compare against SoTa algorithms for Atari 200M in Table 3. We did not include such baselines in Figures 1 and 2 as we wanted to limit this comparison to those algorithms which can be run with widely accessible compute power. However, we are willing to replace Figure 1 with a walltime efficiency curve against Dreamer v3, as we feel this better conveys the message of the paper (MEME did not release this data). We can also add Dreamer v3 and MEME to Figure 2, with a note indicating their walltime to allow for a fairer comparison as we believe that with the appropriate context this will improve the paper. We do, however, argue against the inclusion of sample efficient algorithms (Atari-100K), as these algorithms use drastically different resources (100K frames vs 200M), and critically are benchmarked on a 26-game subset, rather than the full suite, making for an unfair comparison. If you believe this to be pivotal, we can include a figure in the appendix for the 26-game subset. Below is a comparison against SoTa algorithms on the mentioned 26-game subset:\n\n| Algorithm                             | Frames | A100 Walltime        | IQM (26-Game Subset) |\n|--------------------------------------|---------------|----------------|----------------------|\n| MEME                                | 200M | Not Reported*            | 18.491               |\n| Dreamer v3                           | 200M | 7.7 Days                | 14.305               |\n| Beyond The Rainbow (BTR)            | 200M | 22 Hours                | 11.202               |\n| PQN [1]                                  | 400M | 1 Hour                | 5.014**                |\n| EfficientZero v2                     | 100K | 2.7 Hours***            | 1.305                |\n| BBF                                  | 100K | 7.8 Hours               | 1.045                |\n| Dreamer v3                           | 100K | 2.4 Hours               | 0.543                |\n\n\n*MEME used a shared server with a TPUv4.\n\n**PQN used life information, making its results appear significantly higher.\n\n***EfficientZero was tested on a server with 8 RTX 3090s, not an A100.\n\n\n**Practical Implications of RL on Desktop PCs** - Could you further clarify this point? We are unsure exactly what you are asking.\n\n[1] Gallici, Matteo, et al. \"Simplifying deep temporal difference learning.\" arXiv preprint arXiv:2407.04811 (2024).", "claims_and_evidence": "As the title and experiments show, this paper mainly proposes an integrated method that integrates multiple existing reinforcement learning technologies, and proves through experiments that it has achieved SOTA performance on 200M Atari (mainly) and can run on personal PCs.\n\nHowever, the methods mainly compared in the paper, such as Rainbow, DQN, etc., may be far from the current SOTA algorithms. Even if BTR can surpass them, it is difficult to prove its SOTA performance, and thus it is difficult to prove the main contribution of this paper (i.e. the advantages of integrating multiple RL technologies). If the author can give a comparison with more recent papers, it may greatly enhance the persuasiveness of the paper, such as MEME [1], EfficientZero [2], etc.\n\n[1] Human-level Atari 200x faster\n\n[2] Mastering Atari Games with Limited Data", "methods_and_evaluation_criteria": "The proposed method is an integration of several existing improvements (at the algorithm level). The main experiments were conducted on the 200M Atari task. The evaluation metrics used in the experiment, such as IQM, are reasonable. However, it is worth noting that the baseline method compared in this paper may be far from representing the current SOTA method on the 200M Atari task. As mentioned in (Claims And Evidence), there are at least more recent baselines like MEME and EfficientZero. If the author can add corresponding comparisons, it may be more convincing.\n\nFurthermore, the ability to run RL algorithms on personal PCs is certainly exciting, but its practical implications may still require extensive discussion and clarification.", "theoretical_claims": "This paper is mainly an experimental paper, that is, to verify the effectiveness of the proposed algorithm, and there are no outstanding theoretical results in the main body of the paper.", "experimental_designs_or_analyses": "The experiment in this paper mainly focuses on using RL algorithms to train multiple video game agents, and mainly compares evaluation metrics  such as IQM, Mean HNS, and Median HNS. However, the main problem may be the lack of more recent and more convincing baselines.\n\nIn addition, the method in the paper seems to use life info from Atari, so what is the performance without using life info?", "relation_to_broader_scientific_literature": "Since the paper mainly conducts experiments on Atari tasks, it mainly compares a series of other existing RL algorithms, such as Rainbow, Impala, DQN, etc. However, there is still a lack of more recent paper comparisons, such as MEME, EfficientZero, etc.", "essential_references_not_discussed": "Since the main contribution of the paper is to propose a set of practical and efficient RL algorithms, this paper lacks comparison with some recent existing efficient RL algorithms, which at least include MEME, EfficientZero, etc.", "other_strengths_and_weaknesses": "The main advantage of this paper is that it proposes a set of efficient RL algorithms that can run on high-performance personal PCs, but this paper lacks comparison with the key latest related literature, and also lacks more convincing applications of BTR method in PC.", "other_comments_or_suggestions": "If the author can compare it with some more recent RL algorithms, it will greatly enhance the persuasiveness of the paper.", "questions_for_authors": "1. How does BTR compare with newer RL algorithms on the 200M Atari task? For example, MEME, EfficientZero, etc.\n2. In addition to video games, can the author provide other more convincing application scenarios for BTR?\n3. On the Atari task, how does BTR perform without using life info?\n4. In addition to the integration of multiple methods, can the author prove its original contribution?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents BTR, an approach that integrates several well-established techniques to achieve strong performance on Atari and Procgen with limited computational resources. Although the algorithmic novelty is limited, the paper provides value by effectively combining existing reinforcement learning components into a cohesive and efficient framework. Additionally, the authors addressed the reviewers’ concerns regarding baselines thoroughly during the rebuttal. Therefore, I recommend acceptance.
Decision: Accept (poster)