Title: SLM LAB: A COMPREHENSIVE BENCHMARK AND MODULAR SOFTWARE FRAMEWORK FOR REPRO-DUCIBLE DEEP REINFORCEMENT LEARNING

Abstract: We introduce SLM Lab, a software framework for reproducible reinforcement learning (RL) research. SLM Lab implements a number of popular RL algorithms, provides synchronous and asynchronous parallel experiment execution, hyperparameter search, and result analysis. RL algorithms in SLM Lab are implemented in a modular way such that differences in algorithm performance can be confidently ascribed to differences between algorithms, not between implementations. In this work we present the design choices behind SLM Lab and use it to produce a comprehensive single-codebase RL algorithm benchmark. In addition, as a consequence of SLM Lab's modular design, we introduce and evaluate a discrete-action variant of the Soft Actor-Critic algorithm (Haarnoja et al., 2018) and a hybrid synchronous/asynchronous training method for RL agents.

1. INTRODUCTION:

Progress in reinforcement learning (RL) research proceeds only as quickly as researchers can implement new algorithms and publish reproducible empirical results. But it is no secret that modern RL algorithms are hard to implement correctly (Tucker et al., 2018) , and many empirical results are challenging to reproduce (Henderson et al., 2017; Islam et al., 2017; Machado et al., 2017) . Addressing these problems is aided by providing better software tools to the RL research community.

In this work we introduce SLM Lab,1 a software framework for reinforcement learning research designed for reproducibility and extensibility. SLM Lab is the first open source library that includes algorithm implementations, parallelization, hyperparameter search, and experiment analysis in one framework.

After presenting the design and organization of SLM Lab, we demonstrate the correctness of its implementations by producing a comprehensive performance benchmark of RL algorithms across 77 environments. To our knowledge this is the largest single-codebase RL algorithm comparison in the literature. In addition, we leverage the modular design of SLM Lab to introduce a variant of the SAC algorithm for use in discrete-action-space environments and a hybrid synchronous/asynchronous training scheme for RL algorithms.

2. SLM LAB:

2.1 LIBRARY ORGANIZATION Modularity is the central design choice in SLM Lab, as depicted in Figure 1 . Reinforcement learning algorithms in SLM Lab are built around three base classes:

• Algorithm: Handles interaction with the environment, implements an action policy, computes the algorithm-specific loss functions, and runs the training step. • Net: Implements the deep networks that serve as the function approximators for an Algorithm.

• Memory: Provides the data storage and retrieval necessary for training.

The deep learning components in SLM Lab are implemented using PyTorch Paszke et al. (2017) .

The Net and Memory classes abstract network training details, data storage, and data retrieval, simplifying algorithm implementations. Furthermore, many Algorithm classes are natural extensions of each other.

Figure 1 : SLM Lab classes and their inheritance structure.

Modular code is critical for deep RL research because many RL algorithms are extensions of other RL algorithms. If two RL algorithms differ in only a small way, but a researcher compares their performance by running a standalone implementation of each algorithm, they cannot know whether differences in algorithm performance are due to meaningful differences between the algorithms or merely due to quirks in the two implementations. Henderson et al. (2017) showcase this, demonstrating significant performance differences between different implementations of the same algorithm.

Modular code is also important for research progress. It makes it as simple as possible for a researcher to implement -and reliably evaluate -new RL algorithms. And for the student of RL, modular code is easier to read and learn from due to its brevity and organization into digestible components.

Proximal Policy Optimization (PPO) (Schulman et al., 2017) is a good example. When considered as a stand alone algorithm, PPO has a number of different components. However, it differs from the Actor-Critic algorithm only in how it computes the policy loss, runs the training loop, and by needing to maintain an additional actor network during training. Figure 2 shows how this similarity is reflected in the SLM Lab implementation of PPO.

The result is that the PPO class in SLM Lab has five overridden methods and contains only about 140 lines of code. Implementing it was straightforward once the ActorCritic class was implemented and thoroughly tested. More importantly, we can be sure that the performance difference between Actor-Critic and PPO observed in experiments using SLM Lab, shown below in Section 3, are due to something in the 140 lines of code that differentiate ActorCritic and PPO, and not to other implementation differences.

Another example of modularity in SLM Lab is that, thanks to the consistent API shared between Algorithm, Net, and Memory subclasses, synchronous parallelization using vector environments (Dhariwal et al., 2017 ) can be combined with asynchronous parallelization of an individual RL agent's learning algorithm. This multi-level parallelization is further discussed in Section 3.3, where we demonstrate its performance benefits. 

2.2. EXPERIMENT ORGANIZATION:

Reinforcement learning algorithms vary greatly in their performance across different environments, hyperparameter settings, and even within a single environment due to inherent randomness. SLM Lab is designed to easily allow users to study all these types of variability.

SLM Lab organizes experiments in the following hierarchy, shown in Figure 3 : The metrics are also visualized using TensorBoard (Abadi et al., 2015) . Hyperparameter search is implemented using Ray Tune (Liaw et al., 2018) , and the results are automatically analyzed and presented hierarchically in increasingly granular detail.

2.3. REPRODUCIBILITY:

The complexity of RL algorithms makes reproducing RL results challenging (Henderson et al., 2017; Machado et al., 2017) . Every RL researcher knows the difficulty of trying to reproduce an algorithm from its description in a paper alone. Even if code is published along with RL research results, the key algorithm design choices (or mistakes (Tucker et al., 2018) ) are often buried in the algorithm implementation and not exposed naturally to the user.

In SLM Lab, every configurable hyperparameter for an algorithm is specified in a spec file. The spec file is a JSON file containing a git SHA and all the information required to reproduce a Session, Trial, or Experiment as per Section 2.2. Reproducing the entirety of an RL experiment merely requires checking out the code at the git SHA and running the saved spec file.

The main entries in a spec file are given below. Examples of spec files are given in full in Supplementary Section A.4.

1. agent -A list (to allow for multi-agent settings in the future), each element of which contains the spec for a single agent. Each agent spec contains the details for its components as described in Section 2.1:

(a) algorithm. The main parameters specific to the algorithm, such as the policy type, exploration strategy (e.g. -greedy, Boltzmann), algorithm coefficients, rate decays, and training schedules. (b) memory. Specifies which memory to use as appropriate to the algorithm along with any specific memory hyperparameters such as the batch size and the memory size. (c) net. The type of neural network, its hidden layer architecture, activations, gradient clipping, loss function, optimizer, rate decays, update method, and CUDA usage.

2. env -A list (to allow for multi-environment settings in the future), each element of which specifies an environment. Each environment spec includes an optional maximum time step per episode, the total time steps (frames) in a Session, the state and reward preprocessing methods, and the number of environments in a vector environment.

3. body -Specifies how (multi-)agents connect to (multi-)environments.

4. meta -The high-level configuration of how the experiment is to be run. It gives the number of Trials and Sessions to run, the evaluation and logging frequency, and a toggle to activate asynchronous training.

5. search -The hyperparameters to search over and the methods used to sample them. Any variables in the spec file can be searched over, including environment variables.

3. RESULTS:

Reporting benchmark results is essential for validating algorithm implementations. SLM Lab maintains a benchmark page and a public directory containing all of the experiment data, models, plots, and spec files associated with the reported results. 2 We welcome contributions to this benchmark page via a Pull Request.

We tested the algorithms implemented in SLM Lab on 77 environments: 62 Atari games and 11 Roboschool environments available through OpenAI gym (Brockman et al., 2016) and 4 Unity environments (Juliani et al., 2018) . These environments span discrete (Table 1 ) and continuous (Table 2 ) control problems with high-and low-dimensional state spaces.

The results we report in each of the tables are the score per episode at the end of training averaged over the previous 100 training checkpoints. Agents were checkpointed every 10k (Atari) or 1k (Roboschool, Unity) training frames. This measure is less sensitive to rapid increases or decreases in performance, instead reflecting average performance over a substantial number of training frames.

To our knowledge, the results we present below are a more comprehensive performance comparison than has been previously published for a single codebase. A full set of learning curves as well as a full table of results for the Atari environments are provided in the supplementary materials.

3.1. EXPERIMENT DETAILS:

A complete set of spec files listing all hyperparameters for all algorithms and experiments are included in the slm_lab/spec/benchmark directory of SLM Lab as well in the experiment data released along with the results. Example spec files listing all of the hyperparameters for PPO and DDQN + PER on the Atari and Roboschool environments are included in Supplementary Section A.4.

For the Atari environments, agents were trained for 10M frames (40M accounting for skipped frames). For the Roboschool environments, agents were trained for 2M frames except for RoboschoolHumanoid (50M frames), RoboschoolHumanoidFlagrun (100M) and RoboschoolHu-manoidFlagrunHarder (100M). For the Unity environments, agents were trained for 2M frames.

Training was parallelized either synchronously or with a hybrid of synchronous and asynchronous methods.

Our results for PPO and A2C on Atari games are comparable the results published by Schulman et al. (2017) . The results on DQN and DDQN + PER on Atari games are mixed: at the same number of training frames3 we sometimes match or exceed the reported results and sometimes perform worse. This is likely due to two hyperparameter differences. We used a replay memory of size 200,000 compared to 1M in Mnih et al. (2015) , van Hasselt et al. (2015) , and Schaul et al. (2015) .

The final output layer of the network is smaller fully-connected layer with 256 instead of 512 units. Finally, our results for SAC confirm the strength of this algorithm compared to PPO for continuous control problems. However the absolute performance is typically worse than the published results from Haarnoja et al. (2018b) . Due to computational constraints, SAC was trained with a replay buffer of 0.2M elements and combined experience replay (Zhang & Sutton, 2017) compared with 1M elements in Haarnoja et al. (2018b) and this is potentially a significant difference. All published results for the Soft Actor-Critic (SAC) algorithm (Haarnoja et al., 2018a; b) are for continuous control environments. However, nothing in its algorithmic description makes it unsuitable in principle for use in discrete action-space environments. As a consequence of the modular structure of SLM Lab, it was straightforward to design and implement a discrete variant of SAC. We did so by using policy Nets that produced parameters of a Gumbel-Softmax distribution (Jang et al., 2016; Maddison et al., 2016) from which discrete actions were sampled. The results of this discrete variant of SAC are in Table 1 .

On environments for which its training converged, we found SAC to be of comparable or better sample efficiency than all of the other algorithms that we tested. For example, SAC achieves an average score over the previous 100 checkpoints of around 20 after 1M frames on Atari Pong, whereas all other algorithms that we tested require at least 2M frames to achieve the same result (plot shown in Supplementary Section A.3). However, even though SAC trained successfully on Pong and Lunar Lander, we were not able to successfully train it on all other Atari environments. We also note that while SAC is sample efficient it is more computationally expensive than the other algorithms, which presents an obstacle for extensive performance tuning.

3.3. HYBRID SYNCHRONOUS AND ASYNCHRONOUS TRAINING:

Synchronous and asynchronous parallelization can be combined in SLM Lab to accelerate training, as shown in Figure 4 . SLM Lab implements synchronous parallelization within Sessions (Section 2.2) using vector environments (Dhariwal et al., 2017) and asynchronous parallelization within Trials (Section 2.2) using multiple workers, one per Session. There are two available methods for Trial level parallelization; Hogwild! (Recht et al., 2011) or a server-worker model in which workers periodically push gradients to a central network and pull copies of the updated parameters.

If training is constrained by data sampling from the environment, then increasing the number of vector environments (synchronous parallelization) speeds up training. But this speed-up saturates as the training step becomes a bottleneck and the environment waits for the agent to train. In the example shown in Figure 4 the frames per seconds (fps) increases from around 200 for a single worker and 1 environment to around 360 for 1 worker and 16 environments, and saturates thereafter.

Once fps becomes constrained by the training step, it is beneficial to add workers (asynchronous parallelization) to effectively parallelize the parameter updates. A hybrid of 16 workers each with 4 environments resulted in the maximum fps of around 3800.

4. RELATED WORK:



4.1. REPRODUCIBILITY IN REINFORCEMENT LEARNING:

The instability of RL algorithms (Haarnoja et al., 2018a) , randomness in agent policies and the environment (Henderson et al., 2017; Islam et al., 2017) , as well as differences in hyperparameter tuning (Islam et al., 2017) and implementations (Henderson et al., 2017; Tucker et al., 2018) all contribute to the challenge of reproducing RL results. Consequently, the importance of comprehensively documenting all hyperparameters along with published results and software is well recognized within the research community (Machado et al., 2017; Castro et al., 2018) .

4.2. SOFTWARE FOR REINFORCEMENT LEARNING:

To date more than twenty reinforcement-learning-themed open source software libraries have been released. They can be organized into two categories: those implementing RL algorithms and those implement RL environments.4 SLM Lab is an algorithm-focused library with built-in integration with the OpenAI gym (Brockman et al., 2016) , OpenAI Roboschool, VizDoom (Kempka et al., 2016) , and Unity ML-Agents (Juliani et al., 2018) environment libraries.

Table 3 summarizes the algorithm-focused reinforcement learning software libraries.

Libraries such as Catalyst (Kolesnikov, 2018 ), ChainerRL (chainer, 2017; Tokui & Oono, 2015) , coach (Caspi et al., 2017) , DeepRL (Zhang, 2017) , OpenAI baselines (Dhariwal et al., 2017) , RLgraph (Schaarschmidt et al., 2019 ), RLkit (Pong, 2018) , rlpyt (Stooke & Abbeel, 2019) , RLLib (Liang et al., 2017) , Stable Baselines (Hill et al., 2018) , TensorForce (Kuhnle et al., 2017) , TF-Agents (Guadarrama et al., 2018) , and vel (Tworek, 2018) implement a wide variety of algorithms and are intended to be applied to a variety of RL problems. Most of these libraries also provide some benchmark results for the implemented algorithms to validate their performance. These can be thought of as generalist RL libraries and are the most closely related to SLM Lab.

Other libraries focus on specific algorithms (Dopamine (Castro et al., 2018) , Softlearning (Haarnoja et al., 2018c ), a2c-ppo-acktr-gail (Kostrikov, 2018) , Keras-RL (Plappert, 2016) ), problems (Open-Spiel (Lanctot et al., 2019) , ELF (Tian et al., 2019) , MAgent (Zheng et al., 2017 ), reaver (Ring, 2018) ), components such as loss functions (TRFL (DeepMind, 2018) or scaling training (Horizon (Gauci et al., 2018) ).

The use of configuration files to specify hyperparameters varies. Catalyst, coach, DeepRL, Dopamine, Horizon, reaver, Softlearning, RLgraph, RLLib, and vel use configuration files and provide a number of configured examples. In most cases the network architecture is excluded from the Table 3 : Comparison of RL software libraries. Algorithm acronyms are explained in Supplementary Section A.1. REINFORCE is excluded as are less well-known algorithms. "Benchmark" indicates whether the library reports the performance of their implementations. "Config" indicates whether hyperparameters are specified separately from the implementation and run scripts; "split" indicates that the configuration is divided across multiple files, "partial" indicates that some but not all hyperparameters are included. "Parallel" denotes whether training for any algorithms can be parallelized. "HPO" denotes support for hyperparameter optimization. "Plot" denotes whether the library provides any methods for visualizing results. Almost all libraries include some methods for parallelizing agent training, especially for on-policy methods. However most do not include hyperparameter optimization as a feature. The two exceptions are Stable Baselines (Hill et al., 2018) and Tensorforce (Kuhnle et al., 2017) .

Library:

Finally, many libraries include some tools for visualizing and plotting results. Notably coach (Caspi et al., 2017) "name": "DoubleDQN", 5

"algorithm": { 6 "name": "DoubleDQN", 7

"action_pdtype": "Argmax", 8

"action_policy": "epsilon_greedy", 9

"explore_var_spec": { 10 "name": "linear_decay", 11 "start_val": 1.0, 12 "end_val": 0. • Log policy entropy, Q or V function outputs and check that they change. Looking at Q function values for a random agent helped us to track down an error due to state mutation. • As with all Python code, watch out for mutation. For example, we noticed that the state returned from OpenAI baselines vector environment was mutated by preprocessing for the next time steps. This was because the state was tracked as internal variables and returned directly. To remedy this, the state must be returned as a copy. • Check that the main components of the computation graph are connected as expected. This can be implemented as a decorator around the training step and run when debugging. Conversely, be careful to detach target values. • Visualize trained agents. We forgot to evaluate agents with reward clipping turned off, which led to very significant underestimation of performance for all environments except Pong. We finally realized training was working only after deciding to look at some policies playing the game.

:

URL redacted to preserve anonymity.

URL redacted to preserve anonymity.

Estimated using the training curves provided in Figure7ofSchaul et al. (2015)

A few libraries such as OpenSpiel(Lanctot et al., 2019) and ELF(Tian et al., 2019) implement both.

Figures and Tables:

Figure 2: Reinforce, ActorCritic, and PPO class methods in SLM Lab. + indicates that a method is added or overridden in the class.

Figure 3: Experiment organization in SLM Lab. A Session is a single run of an algorithm on an environment. A Trial is a collection of Sessions. An Experiment is a collection of Trials with different algorithms and/or environments.

Figure 4: Average frames per second as number of vector environments and Hogwild! workers are varied. Each setting was trained using PPO with the same hyperparameters on the RoboschoolAnt environment.

Algorithms Benchmark Config Parallel HPO Plot a2c-ppo-acktr-gail A2C, ACKTR, GAIL, PPO Baselines A2C, ACER, ACKTR, DDPG, DQN, GAIL, HER, PPO, TRPO Catalyst CRA, TD3, SAC, DDPG, QR ChainerRL A3C, ACER, C51, DDPG, DQN+, IQN, NSQ, PPO, Rainbow, TRPO, TD3, SAC coach A3C, ACER, C51, DDPG, DQN+, QR-DQN, NAF, NEC, NSQ, PPO, Rainbow, SAC, configuration file and specified elsewhere. Catalyst, Horizon and RLgraph include all hyperparameters in a single configuration file and are the most similar to SLM Lab. However, RLgraph does not include environment information, and none use the same file for configuring hyperparameter search.

Session The lowest level of the SLM Lab experiment framework: a single training run of one agent on one environment with one set of hyperparameters, all with a fixed random seed.

Table content: <table><tr><td>2. Trial A trial consists of multiple Sessions, with the Sessions varying only in the random</td></tr><tr><td>seed.</td></tr><tr><td>3. Experiment Generates different sets of hyperparameters (according to a spec file, see be-</td></tr><tr><td>low) and runs a Trial for each one. It can be thought of as a study, e.g. "What values of</td></tr><tr><td>n of A2C n-step returns provide the fastest, most stable solution, if the other variables are</td></tr><tr><td>held constant?"</td></tr><tr><td>SLM Lab automatically produces plots for Sessions, Trials, and Experiments for any combination</td></tr><tr><td>of environments and algorithms. It also logs and tracks metrics during training such as rewards,</td></tr><tr><td>loss, exploration and entropy variables, model weights and biases, action distributions, frames-per-</td></tr><tr><td>second and wall-clock time.</td></tr></table>

Episode score at the end of training attained by SLM Lab implementations on discreteaction control problems. Reported episode scores are the average over the last 100 checkpoints, and then averaged over 4 Sessions. A Random baseline with score averaged over 100 episodes is included. Results marked with '*' were trained using the hybrid synchronous/asynchronous version of SAC to parallelize and speed up training time. For SAC, Breakout, Pong and Seaquest were trained for 2M frames instead of 10M frames.

Table content: <table><tr><td>Environment</td><td/><td/><td/><td>Algorithm</td><td/><td/><td/></tr><tr><td/><td colspan="5">Random DQN DDQN+PER A2C (GAE) A2C (n-step)</td><td>PPO</td><td>SAC</td></tr><tr><td>Breakout</td><td>1.26</td><td>80.88</td><td>182</td><td>377</td><td>398</td><td>443</td><td>3.51*</td></tr><tr><td>Pong</td><td>-20.4</td><td>18.48</td><td>20.5</td><td>19.31</td><td>19.56</td><td colspan="2">20.58 19.87*</td></tr><tr><td>Seaquest</td><td>106</td><td>1185</td><td>4405</td><td>1070</td><td>1684</td><td>1715</td><td>171*</td></tr><tr><td>Qbert</td><td>157</td><td>5494</td><td>11426</td><td>12405</td><td>13590</td><td>13460</td><td>923*</td></tr><tr><td>LunarLander</td><td>-162</td><td>192</td><td>233</td><td>25.21</td><td>68.23</td><td>214</td><td>276</td></tr><tr><td>UnityHallway</td><td>-0.99</td><td>-0.32</td><td>0.27</td><td>0.08</td><td>-0.96</td><td>0.73</td><td>0.01</td></tr><tr><td>UnityPushBlock</td><td>-1.00</td><td>4.88</td><td>4.93</td><td>4.68</td><td>4.93</td><td>4.97</td><td>-0.70</td></tr><tr><td colspan="5">3.2 SOFT ACTOR-CRITIC FOR DISCRETE ENVIRONMENTS</td><td/><td/><td/></tr></table>

Episode score at the end of training attained by SLM Lab implementations on continuous control problems. Reported episode scores are the average over the last 100 checkpoints, and then averaged over 4 Sessions. A Random baseline with score averaged over 100 episodes is included. Results marked with '*' require 50M-100M frames, so we use the hybrid synchronous/asynchronous version of SAC to parallelize and speed up training time.

Table content: <table><tr><td>Environment</td><td/><td/><td>Algorithm</td><td/><td/></tr><tr><td/><td colspan="4">Random A2C (GAE) A2C (n-step) PPO</td><td>SAC</td></tr><tr><td>RoboschoolAnt</td><td>52.30</td><td>787</td><td>1396</td><td>1843</td><td>2915</td></tr><tr><td>RoboschoolAtlasForwardWalk</td><td>40.23</td><td>59.87</td><td>88.04</td><td>172</td><td>800</td></tr><tr><td>RoboschoolHalfCheetah</td><td>0.83</td><td>712</td><td>439</td><td>1960</td><td>2497</td></tr><tr><td>RoboschoolHopper</td><td>21.22</td><td>710</td><td>285</td><td>2042</td><td>2045</td></tr><tr><td>RoboschoolInvertedDoublePendulum</td><td>285</td><td>996</td><td>4410</td><td>8076</td><td>8085</td></tr><tr><td>RoboschoolInvertedPendulum</td><td>23.5</td><td>995</td><td>978</td><td>986</td><td>941</td></tr><tr><td>RoboschoolReacher</td><td>-9.24</td><td>12.9</td><td>10.16</td><td colspan="2">19.51 19.99</td></tr><tr><td>RoboschoolWalker2d</td><td>16.06</td><td>280</td><td>220</td><td>1660</td><td>1894</td></tr><tr><td>RoboschoolHumanoid</td><td>-36.16</td><td>99.31</td><td>54.58</td><td colspan="2">2388 2621*</td></tr><tr><td>RoboschoolHumanoidFlagrun</td><td>-6.15</td><td>73.57</td><td>178</td><td colspan="2">2014 2056*</td></tr><tr><td>RoboschoolHumanoidFlagrunHarder</td><td>-8.64</td><td>-429</td><td>253</td><td>680</td><td>280*</td></tr><tr><td>Unity3DBall</td><td>1.24</td><td>33.48</td><td>53.46</td><td colspan="2">78.24 98.44</td></tr><tr><td>Unity3DBallHard</td><td>1.17</td><td>62.92</td><td>71.92</td><td colspan="2">91.41 97.06</td></tr></table>

provides an interactive dashboard for exploring a variety of metrics which are automatically tracked during training.

Table content: <table><tr><td/><td>Adventure</td><td>-0.89</td><td>-0.94</td><td>-0.92</td><td>-0.77</td><td>-0.85</td><td>-0.3</td></tr><tr><td/><td>AirRaid</td><td>486</td><td>1876</td><td>3974</td><td>4202</td><td>3557</td><td>4028</td></tr><tr><td/><td>Alien</td><td>97.0</td><td>822</td><td>1574</td><td>1519</td><td>1627</td><td>1413</td></tr><tr><td/><td>Amidar</td><td>1.8</td><td>90.95</td><td>431</td><td>577</td><td>418</td><td>795</td></tr><tr><td/><td>Assault</td><td>308</td><td>1392</td><td>2567</td><td>3366</td><td>3312</td><td>3619</td></tr><tr><td/><td>Asterix</td><td>307</td><td>1253</td><td>6866</td><td>5559</td><td>5223</td><td>6132</td></tr><tr><td/><td>Asteroids</td><td>1331</td><td>439</td><td>426</td><td>2951</td><td>2147</td><td>2186</td></tr><tr><td/><td>Atlantis</td><td>29473</td><td>68679</td><td>644810</td><td>2747371</td><td>2259733</td><td>2148077</td></tr><tr><td/><td>BankHeist</td><td>13.4</td><td>131</td><td>623</td><td>855</td><td>1170</td><td>1183</td></tr><tr><td/><td>BattleZone</td><td>3080</td><td>6564</td><td>6395</td><td>4336</td><td>4533</td><td>13649</td></tr><tr><td/><td>BeamRider</td><td>355</td><td>2799</td><td>5870</td><td>2659</td><td>4139</td><td>4299</td></tr><tr><td/><td>Berzerk</td><td>212</td><td>319</td><td>401</td><td>1073</td><td>763</td><td>860</td></tr><tr><td/><td>Bowling</td><td>24.14</td><td>30.29</td><td>39.5</td><td>24.51</td><td>23.75</td><td>31.64</td></tr><tr><td/><td>Boxing</td><td>-0.91</td><td>72.11</td><td>90.98</td><td>1.57</td><td>1.26</td><td>96.53</td></tr><tr><td/><td>Breakout</td><td>1.26</td><td>80.88</td><td>182</td><td>377</td><td>398</td><td>443</td></tr><tr><td/><td>Carnival</td><td>905</td><td>4280</td><td>4773</td><td>2473</td><td>1827</td><td>4566</td></tr><tr><td/><td>Centipede</td><td>2888</td><td>1899</td><td>2153</td><td>3909</td><td>4202</td><td>5003</td></tr><tr><td/><td>ChopperCommand</td><td>735</td><td>1083</td><td>4020</td><td>3043</td><td>1280</td><td>3357</td></tr><tr><td/><td>CrazyClimber</td><td>2452</td><td>46984</td><td>88814</td><td>106256</td><td>109998</td><td>116820</td></tr><tr><td/><td colspan="3">Defender 546810 281999</td><td>313018</td><td>665609</td><td>657823</td><td>534639</td></tr><tr><td/><td>DemonAttack</td><td>346</td><td>1705</td><td>19856</td><td>23779</td><td>19615</td><td>121172</td></tr><tr><td/><td>DoubleDunk</td><td>-16.48</td><td>-21.44</td><td>-22.38</td><td>-5.15</td><td>-13.3</td><td>-6.01</td></tr><tr><td/><td>ElevatorAction</td><td>9851</td><td>32.62</td><td>17.91</td><td>9966</td><td>8818</td><td>6471</td></tr><tr><td/><td>Enduro</td><td>0.0</td><td>437</td><td>959</td><td>787</td><td>0.0</td><td>1926</td></tr><tr><td/><td>FishingDerby</td><td>-93.96</td><td>-88.14</td><td>-1.7</td><td>16.54</td><td>1.65</td><td>36.03</td></tr><tr><td/><td>Freeway</td><td>0.0</td><td>24.46</td><td>30.49</td><td>30.97</td><td>0.0</td><td>32.11</td></tr><tr><td/><td>Frostbite</td><td>68.0</td><td>98.8</td><td>2497</td><td>277</td><td>261</td><td>1062</td></tr><tr><td/><td>Gopher</td><td>276</td><td>1095</td><td>7562</td><td>929</td><td>1545</td><td>2933</td></tr><tr><td/><td>Gravitar</td><td>219</td><td>87.34</td><td>258</td><td>313</td><td>433</td><td>223</td></tr><tr><td/><td>Hero</td><td>706</td><td>1051</td><td>12579</td><td>16502</td><td>19322</td><td>17412</td></tr><tr><td/><td>IceHockey</td><td>-9.87</td><td>-14.96</td><td>-14.24</td><td>-5.79</td><td>-6.06</td><td>-6.43</td></tr><tr><td/><td>Jamesbond</td><td>13.0</td><td>44.87</td><td>702</td><td>521</td><td>453</td><td>561</td></tr><tr><td/><td>JourneyEscape</td><td>-18095</td><td>-4818</td><td>-2003</td><td>-921</td><td>-2032</td><td>-1094</td></tr><tr><td/><td>Kangaroo</td><td>54.0</td><td>1965</td><td>8897</td><td>67.62</td><td>554</td><td>4989</td></tr><tr><td/><td>Krull</td><td>1747</td><td>5522</td><td>6650</td><td>7785</td><td>6642</td><td>8477</td></tr><tr><td colspan="3">KungFuMaster MontezumaRevenge A.4 EXAMPLE SPEC FILES 865 0.0</td><td>2288 0.0</td><td>16547 0.02</td><td>31199 0.08</td><td>25554 0.19</td><td>34523 1.08</td></tr><tr><td>1 { 2 3</td><td colspan="2">MsPacman NameThisGame Phoenix "ddqn_per_atari": { 170 2088 1324 "agent": [{</td><td>1175 3915 2909</td><td>2215 4474 8179</td><td>1965 5178 16345</td><td>2158 5795 13586</td><td>2350 6386 30504</td></tr><tr><td>4</td><td>Pitfall</td><td>-301</td><td>-68.83</td><td>-73.65</td><td>-101</td><td>-31.13</td><td>-35.93</td></tr><tr><td/><td>Pong</td><td>-20.4</td><td>18.48</td><td>20.5</td><td>19.31</td><td>19.56</td><td>20.58</td></tr><tr><td/><td>Pooyan</td><td>515</td><td>1958</td><td>2741</td><td>2862</td><td>2531</td><td>6799</td></tr><tr><td/><td>PrivateEye</td><td>-731</td><td>784</td><td>303</td><td>93.22</td><td>78.07</td><td>50.12</td></tr><tr><td/><td>Qbert</td><td>157</td><td>5494</td><td>11426</td><td>12405</td><td>13590</td><td>13460</td></tr><tr><td/><td>Riverraid</td><td>1554</td><td>953</td><td>10492</td><td>8308</td><td>7565</td><td>9636</td></tr><tr><td/><td>RoadRunner</td><td>35.0</td><td>15237</td><td>29047</td><td>30152</td><td>31030</td><td>32956</td></tr><tr><td/><td>Robotank</td><td>1.78</td><td>3.43</td><td>9.05</td><td>2.98</td><td>2.27</td><td>2.27</td></tr><tr><td/><td>Seaquest</td><td>106</td><td>1185</td><td>4405</td><td>1070</td><td>1684</td><td>1715</td></tr><tr><td/><td>Skiing</td><td>-17361</td><td>-14094</td><td>-12883</td><td>-19481</td><td>-14234</td><td>-24713</td></tr><tr><td/><td>Solaris</td><td>2097</td><td>612</td><td>1396</td><td>2115</td><td>2236</td><td>1892</td></tr><tr><td/><td>SpaceInvaders</td><td>164</td><td>451</td><td>670</td><td>733</td><td>750</td><td>797</td></tr><tr><td/><td>StarGunner</td><td>645</td><td>3565</td><td>38238</td><td>44816</td><td>48410</td><td>60579</td></tr><tr><td/><td>Tennis</td><td>-24.0</td><td>-23.78</td><td>-10.33</td><td>-22.42</td><td>-19.06</td><td>-11.52</td></tr><tr><td/><td>TimePilot</td><td>3151</td><td>2819</td><td>1884</td><td>3331</td><td>3440</td><td>4398</td></tr><tr><td/><td>Tutankham</td><td>15.45</td><td>35.03</td><td>159</td><td>161</td><td>175</td><td>211</td></tr><tr><td/><td>UpNDown</td><td>125</td><td>2043</td><td>11632</td><td>89769</td><td>18878</td><td>262208</td></tr><tr><td/><td>Venture</td><td>0.0</td><td>4.56</td><td>9.61</td><td>0.0</td><td>0.0</td><td>11.84</td></tr><tr><td/><td>VideoPinball</td><td>25365</td><td>8056</td><td>79730</td><td>35371</td><td>40423</td><td>58096</td></tr></table>

• Start simple -implement parent algorithms first. e.g. REINFORCE, then A2C, then PPO or SAC, and ensure that each of these work in turn. PPO and SAC were straightforward to debug once A2C was working. • Write tests: Tests have been invaluable in debugging and growing SLM Lab. Especially useful are tests for tricky functions. For example, a test for the GAE calculation was essential for debugging A2C. Additionally, use Continuous Integration to automatically build and test every new code changes in a Pull Request, as is done in SLM Lab. • Check input and output tensor shapes for important calculations. e.g. predicted and target values for Q or V. Broadcasting errors may lead to incorrect tensor shapes which fail silently. • Check the size of the loss at the beginning of training. Is this comparable to other implementations? Tracking this value was essential when debugging DQN. It led us to a bug in the image permutation and we also stopped normalizing the image values as a result.

Table content: <table/>

Peer review (iclr2020):
This paper presents a new RL library called « SLM Lab ». Its most relevant features for RL research are: (1) modularity to help re-use existing components (thus reducing the risk of subtle implementation differences when comparing algorithms), (2) implementations of most popular algorithms like DQN & variants, A3C, PPO, SAC, (3) ability to parallelize both actors (through vectorized environments) and the learner (through distributed gradient descent), and (4) utilities for hyper-parameter optimization, reproducible experiments and reporting. The paper also reports performance over Atari games, Roboschool environments as well as some Unity ML-Agents tasks. Finally, it provides a high-level overview of SLM Lab’s capabilities compared to 23 other RL open source libraries, showing that it is the only one that combines: reporting of the performance of the implemented algorithms, ability to specify hyper-parameters in the config file, parallelization, hyper-parameter optimization, and visualization of the results. Overall this looks like a solid RL library, but I am not convinced that it brings enough novelty to the RL software landscape for a published ICLR paper — it would better fit in a workhop dedicated to ML libraries for instance, thus the weak reject. Table 3 shows that SLM Lab is the only RL library with such a broad offering of features, and this is definitely impressive, but I would argue that many of these features can typically be added to other libraries by plugging in other open source software. For instance there are several tools for experiment management and hyper-parameter optimization (and for RLLib in particular, hyper-parameter optimization is not checked but is straightforward with Ray Tune). TensorBoard can also often be easily used for visualization. The parallelization capabilities of SLM Lab also seem limited: if I understand correctly, actor parallelization can only occur on a single machine, and thus an algorithm like Ape-X or R2D2 could not be implemented. If this is correct then it is a major limitation of the framework, since such parallelization across multiple computers can be extremely useful when environments are slow and costly to run (in CPU / RAM). It is not clear to me to which extent multi-agent is supported. It seems like it is possible to have multiple agents in one environment, but is that enough for general multi-agent RL? (ex: how to specify individual / team rewards? share information between agents? deal with agents not acting all at the same timestep? centralize part of the training / execution?…) I appreciate that the paper presents a discrete version of SAC, mentioning how easy it was to implement thanks to the modular design of SLM Lab, but results from Table 1 do not look very good (especially since it did not work on some of the environments). Relying on the Gumbel-softmax might not be the most robust & stable way to train a discrete SAC — see e.g. the recent « Soft Actor-Critic for Discrete Action Settings » for a different approach. Finally, it is also great to have some benchmarks of the algorithms being implemented, but at least for Atari, I am not aware of previous work using the exact same evaluation setting, so it is hard to tell how they compare to other implementations. In spite of the above, I do not mean to criticize SLM Lab too heavily as from what I can tell it seems to be a a solid library with many useful features, and I am sure many researchers will find it useful in their day-to-day work. Minor points: - Anonymity was clearly violated with the two github links - Having some synthetic result on Atari (like the typically reported median human-normalized score) would be good - I am personally not a fan of large config JSON files due to the lack of comments in JSON - A.5 (« Key Implementation Lessons ») is great! Review update after author feedback: I am on the fence for this paper, but still leaning towards rejection due to the fact I am still not convinced that this library brings that much novelty compared to existing other libraries (although it seems like a nice RL library, I am not sure ICLR is the right venue for talking about it). The authors argue that their benchmark results are a key contribution of the paper, but I do not find these results particularly insightful, especially the Atari ones that are not comparable to previous results (due to using a different evaluation method) and the lack of state-of-the-art algorithms like Rainbow or IQN.