SLM Lab: A Comprehensive Benchmark and Modular Software Framework for Reproducible Deep Reinforcement Learning

1 INTRODUCTION . Progress in reinforcement learning ( RL ) research proceeds only as quickly as researchers can implement new algorithms and publish reproducible empirical results . But it is no secret that modern RL algorithms are hard to implement correctly ( Tucker et al. , 2018 ) , and many empirical results are challenging to reproduce ( Henderson et al. , 2017 ; Islam et al. , 2017 ; Machado et al. , 2017 ) . Addressing these problems is aided by providing better software tools to the RL research community . In this work we introduce SLM Lab,1 a software framework for reinforcement learning research designed for reproducibility and extensibility . SLM Lab is the first open source library that includes algorithm implementations , parallelization , hyperparameter search , and experiment analysis in one framework . After presenting the design and organization of SLM Lab , we demonstrate the correctness of its implementations by producing a comprehensive performance benchmark of RL algorithms across 77 environments . To our knowledge this is the largest single-codebase RL algorithm comparison in the literature . In addition , we leverage the modular design of SLM Lab to introduce a variant of the SAC algorithm for use in discrete-action-space environments and a hybrid synchronous/asynchronous training scheme for RL algorithms . 2 SLM LAB . 2.1 LIBRARY ORGANIZATION . Modularity is the central design choice in SLM Lab , as depicted in Figure 1 . Reinforcement learning algorithms in SLM Lab are built around three base classes : • Algorithm : Handles interaction with the environment , implements an action policy , computes the algorithm-specific loss functions , and runs the training step . • Net : Implements the deep networks that serve as the function approximators for an Algorithm . 1URL redacted to preserve anonymity . • Memory : Provides the data storage and retrieval necessary for training . The deep learning components in SLM Lab are implemented using PyTorch Paszke et al . ( 2017 ) . The Net and Memory classes abstract network training details , data storage , and data retrieval , simplifying algorithm implementations . Furthermore , many Algorithm classes are natural extensions of each other . Modular code is critical for deep RL research because many RL algorithms are extensions of other RL algorithms . If two RL algorithms differ in only a small way , but a researcher compares their performance by running a standalone implementation of each algorithm , they can not know whether differences in algorithm performance are due to meaningful differences between the algorithms or merely due to quirks in the two implementations . Henderson et al . ( 2017 ) showcase this , demonstrating significant performance differences between different implementations of the same algorithm . Modular code is also important for research progress . It makes it as simple as possible for a researcher to implement — and reliably evaluate — new RL algorithms . And for the student of RL , modular code is easier to read and learn from due to its brevity and organization into digestible components . Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) is a good example . When considered as a stand alone algorithm , PPO has a number of different components . However , it differs from the Actor-Critic algorithm only in how it computes the policy loss , runs the training loop , and by needing to maintain an additional actor network during training . Figure 2 shows how this similarity is reflected in the SLM Lab implementation of PPO . The result is that the PPO class in SLM Lab has five overridden methods and contains only about 140 lines of code . Implementing it was straightforward once the ActorCritic class was implemented and thoroughly tested . More importantly , we can be sure that the performance difference between Actor-Critic and PPO observed in experiments using SLM Lab , shown below in Section 3 , are due to something in the 140 lines of code that differentiate ActorCritic and PPO , and not to other implementation differences . Another example of modularity in SLM Lab is that , thanks to the consistent API shared between Algorithm , Net , and Memory subclasses , synchronous parallelization using vector environments ( Dhariwal et al. , 2017 ) can be combined with asynchronous parallelization of an individual RL agent ’ s learning algorithm . This multi-level parallelization is further discussed in Section 3.3 , where we demonstrate its performance benefits . 2.2 EXPERIMENT ORGANIZATION . Reinforcement learning algorithms vary greatly in their performance across different environments , hyperparameter settings , and even within a single environment due to inherent randomness . SLM Lab is designed to easily allow users to study all these types of variability . SLM Lab organizes experiments in the following hierarchy , shown in Figure 3 : 1 . Session The lowest level of the SLM Lab experiment framework : a single training run of one agent on one environment with one set of hyperparameters , all with a fixed random seed . 2 . Trial A trial consists of multiple Sessions , with the Sessions varying only in the random seed . 3 . Experiment Generates different sets of hyperparameters ( according to a spec file , see below ) and runs a Trial for each one . It can be thought of as a study , e.g . “ What values of n of A2C n-step returns provide the fastest , most stable solution , if the other variables are held constant ? ” SLM Lab automatically produces plots for Sessions , Trials , and Experiments for any combination of environments and algorithms . It also logs and tracks metrics during training such as rewards , loss , exploration and entropy variables , model weights and biases , action distributions , frames-persecond and wall-clock time . The metrics are also visualized using TensorBoard ( Abadi et al. , 2015 ) . Hyperparameter search is implemented using Ray Tune ( Liaw et al. , 2018 ) , and the results are automatically analyzed and presented hierarchically in increasingly granular detail . 2.3 REPRODUCIBILITY . The complexity of RL algorithms makes reproducing RL results challenging ( Henderson et al. , 2017 ; Machado et al. , 2017 ) . Every RL researcher knows the difficulty of trying to reproduce an algorithm from its description in a paper alone . Even if code is published along with RL research results , the key algorithm design choices ( or mistakes ( Tucker et al. , 2018 ) ) are often buried in the algorithm implementation and not exposed naturally to the user . In SLM Lab , every configurable hyperparameter for an algorithm is specified in a spec file . The spec file is a JSON file containing a git SHA and all the information required to reproduce a Session , Trial , or Experiment as per Section 2.2 . Reproducing the entirety of an RL experiment merely requires checking out the code at the git SHA and running the saved spec file . The main entries in a spec file are given below . Examples of spec files are given in full in Supplementary Section A.4 . 1. agent - A list ( to allow for multi-agent settings in the future ) , each element of which contains the spec for a single agent . Each agent spec contains the details for its components as described in Section 2.1 : ( a ) algorithm . The main parameters specific to the algorithm , such as the policy type , exploration strategy ( e.g . -greedy , Boltzmann ) , algorithm coefficients , rate decays , and training schedules . ( b ) memory . Specifies which memory to use as appropriate to the algorithm along with any specific memory hyperparameters such as the batch size and the memory size . ( c ) net . The type of neural network , its hidden layer architecture , activations , gradient clipping , loss function , optimizer , rate decays , update method , and CUDA usage . 2. env - A list ( to allow for multi-environment settings in the future ) , each element of which specifies an environment . Each environment spec includes an optional maximum time step per episode , the total time steps ( frames ) in a Session , the state and reward preprocessing methods , and the number of environments in a vector environment . 3. body - Specifies how ( multi- ) agents connect to ( multi- ) environments . 4. meta - The high-level configuration of how the experiment is to be run . It gives the number of Trials and Sessions to run , the evaluation and logging frequency , and a toggle to activate asynchronous training . 5. search - The hyperparameters to search over and the methods used to sample them . Any variables in the spec file can be searched over , including environment variables . 3 RESULTS . Reporting benchmark results is essential for validating algorithm implementations . SLM Lab maintains a benchmark page and a public directory containing all of the experiment data , models , plots , and spec files associated with the reported results.2 We welcome contributions to this benchmark page via a Pull Request . We tested the algorithms implemented in SLM Lab on 77 environments : 62 Atari games and 11 Roboschool environments available through OpenAI gym ( Brockman et al. , 2016 ) and 4 Unity environments ( Juliani et al. , 2018 ) . These environments span discrete ( Table 1 ) and continuous ( Table 2 ) control problems with high- and low-dimensional state spaces . The results we report in each of the tables are the score per episode at the end of training averaged over the previous 100 training checkpoints . Agents were checkpointed every 10k ( Atari ) or 1k ( Roboschool , Unity ) training frames . This measure is less sensitive to rapid increases or decreases in performance , instead reflecting average performance over a substantial number of training frames . To our knowledge , the results we present below are a more comprehensive performance comparison than has been previously published for a single codebase . A full set of learning curves as well as a full table of results for the Atari environments are provided in the supplementary materials . 2URL redacted to preserve anonymity . 3.1 EXPERIMENT DETAILS . A complete set of spec files listing all hyperparameters for all algorithms and experiments are included in the slm_lab/spec/benchmark directory of SLM Lab as well in the experiment data released along with the results . Example spec files listing all of the hyperparameters for PPO and DDQN + PER on the Atari and Roboschool environments are included in Supplementary Section A.4 . For the Atari environments , agents were trained for 10M frames ( 40M accounting for skipped frames ) . For the Roboschool environments , agents were trained for 2M frames except for RoboschoolHumanoid ( 50M frames ) , RoboschoolHumanoidFlagrun ( 100M ) and RoboschoolHumanoidFlagrunHarder ( 100M ) . For the Unity environments , agents were trained for 2M frames . Training was parallelized either synchronously or with a hybrid of synchronous and asynchronous methods . Our results for PPO and A2C on Atari games are comparable the results published by Schulman et al . ( 2017 ) . The results on DQN and DDQN + PER on Atari games are mixed : at the same number of training frames3 we sometimes match or exceed the reported results and sometimes perform worse . This is likely due to two hyperparameter differences . We used a replay memory of size 200,000 compared to 1M in Mnih et al . ( 2015 ) , van Hasselt et al . ( 2015 ) , and Schaul et al . ( 2015 ) . The final output layer of the network is smaller fully-connected layer with 256 instead of 512 units . Finally , our results for SAC confirm the strength of this algorithm compared to PPO for continuous control problems . However the absolute performance is typically worse than the published results from Haarnoja et al . ( 2018b ) . Due to computational constraints , SAC was trained with a replay buffer of 0.2M elements and combined experience replay ( Zhang & Sutton , 2017 ) compared with 1M elements in Haarnoja et al . ( 2018b ) and this is potentially a significant difference .

Summary: This paper presents a new RL library called « SLM Lab ». Its most relevant features for RL research are: (1) modularity to help re-use existing components (thus reducing the risk of subtle implementation differences when comparing algorithms), (2) implementations of most popular algorithms like DQN & variants, A3C, PPO, SAC, (3) ability to parallelize both actors (through vectorized environments) and the learner (through distributed gradient descent), and (4) utilities for hyper-parameter optimization, reproducible experiments and reporting. The paper also reports performance over Atari games, Roboschool environments as well as some Unity ML-Agents tasks. Finally, it provides a high-level overview of SLM Lab’s capabilities compared to 23 other RL open source libraries, showing that it is the only one that combines: reporting of the performance of the implemented algorithms, ability to specify hyper-parameters in the config file, parallelization, hyper-parameter optimization, and visualization of the results.