SLM Lab: A Comprehensive Benchmark and Modular Software Framework for Reproducible Deep Reinforcement Learning
Authors: Wah Loon Keng
Laura Graesser
Milan Cvitkovic
Venue:  (2020)
Keywords: reinforcement learning
machine learning
benchmark
reproducibility
software
framework
implementation issues
parallelization
software platforms

Abstract: We introduce SLM Lab, a software framework for reproducible reinforcement learning (RL) research.  SLM Lab implements a number of popular RL algorithms, provides synchronous and asynchronous parallel experiment execution, hyperparameter search, and result analysis.  RL algorithms in SLM Lab are implemented in a modular way such that differences in algorithm performance can be confidently ascribed to differences between algorithms, not between implementations.  In this work we present the design choices behind SLM Lab and use it to produce a comprehensive single-codebase RL algorithm benchmark.  In addition, as a consequence of SLM Lab's modular design, we introduce and evaluate a discrete-action variant of the Soft Actor-Critic algorithm (Haarnoja et al., 2018) and a hybrid synchronous/asynchronous training method for RL agents.

Under review as a conference paper at ICLR 2020
SLM LAB: A COMPREHENSIVE BENCHMARK AND MODULAR SOFTWARE FRAMEWORK FOR REPRO- DUCIBLE DEEP REINFORCEMENT LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
We introduce SLM Lab, software framework for reproducible reinforcement learning (RL) research. SLM Lab implements number of popular RL algo- rithms, provides synchronous and asynchronous parallel experiment execution, hyperparameter search, and result analysis. RL algorithms SLM Lab are imple- mented in modular way such that differences in algorithm performance confidently ascribed to differences between algorithms, not between implemen- tations. In this work present the design choices behind SLM Lab and use it to produce comprehensive single-codebase RL algorithm benchmark. In addi- tion, as a consequence SLM Lab's modular design, we introduce and evaluate a discrete-action variant of the Soft Actor-Critic algorithm (Haarnoja et al., 2018) and hybrid synchronous/asynchroonus training method for RL agents.
be
INTRODUCTION
Progress in reinforcement learning (RL) research proceeds only as quickly as researchers can im- plement new algorithms and publish reproducible empirical results. But is secret that modern RL algorithms are hard to implement correctly (Tucker et al. 2018), and many empirical results are challenging reproduce (Henderson al., 2017 [slam al., 2017; Machado et al., 2017). Ad- dressing these problems is aided by providing better software tools to the RL. research community.
software framework for reinforcement learning research designed for reproducibility and extensibility, SLM Lab is the first open source library that includes algorithm implementations, parallelization, hyperparameter search, and experiment analysis one framework.
In this work introduce SLM Lab
After presenting the design and organization of SLM Lab, we demonstrate the correctness of its implementations by producing a comprehensive performance benchmark of RL algorithms across 77 environments. our knowledge this is the largest single-codebase RI. algorithm comparison the literature. In addition, we leverage the modular design of SLM Lab to introduce variant of the SAC algorithm for use in discrete-action--nn space environments and a hybrid synchronous/asynchrooou training scheme for RL algorithms.
2
SLM LAB
2.1 LIBRARY ORGANIZATION
Modularity is the central design choice in SLM Lab, as depicted in Figure]] Reinforcement learning algorithms in SLM Lab are built around three base classes:
A1 Algoritae soorithm Handles interaction with the environment, implements an action policy, computes the algorith loss functions, and runs the training step.
Net: Implements the deep networks that serve as the function approximators for an Algorithm.
URL redacted preserve anonymity.

---

Under review as a conference paper at ICLR 2020
Memory: Provides the data storage and retrieval necessary for training.
The deep learning components in SLM Lab are implemented using Py PyTocc Paszke et al. (2017) The Net and Memory classes abstract network training details, data storage, and data retrieval, sim- plifying algorithm implementations. Furthermore, many A gort thm classes are natural extensions of each other.
Algorithm
Memory
Net
Algorithm
Memory
Reinforce
OnPolley Replay
Replay
Recurrem Nat
Vanilla.
Actor- Crilic
OnPolsy Batch Replay
Prioritized Replay
Dueling MLPNet
Q- MLPNet
Dueling ConwNett
Q- ConwNet
DON-
PPO
SAC
Double
Figure SLM Lab classes and their inheritance structure.
Modular code is critical for deep RL research because many RL algorithms are extensions of other RL algorithms. If two RL algorithms differ only a small way, but a researcher compares their performance by running standalone implementation of each algorithm, they cannot know whether differences in algorithm performance are due to meaningful differences between the algorithms or merely due quirks in the two implementations. |endderson al. (2017) showcase this, demonstrat- ing significant performance differences between different implementations of the same algorithm.
Modular code is also important for research progress. It makes simple possible for searcher to implement - and reliably evaluate new RL algorithms. And for the student of RL, modular code easier read and learn from due its brevity and organization into digestible components.
Proximal Policy Optimization (PPO) (Schulman et al. 2017) is a good example. When considered as a stand alone algorithm, PPO has a number of different components. However, it differs from the Actor-Critic algorithm only how it computes the policy loss. runs the training loop, and by needing to maintain an additional actor network during training. Figure 2 shows how this similarity reflected the SLM Lab implementation of PPO.
The result is that the PPO class in SLM Lab has five overridden methods and contains only about 140 lines of code. Implementing it was straightforward once the ActorCrite class was implemented and thoroughly tested. More importantly, we can be sure that the performance difference between Actor-Critic and PPO observed in experiments using SI.M Lab, shown below in Section 3 are due to something in the 140 lines of code that differentiate ActorCritic and PPO, and not to other implementation differences.
Another example of modularity in SLM Lab is that, thanks to the consistent API shared between Algori hm, hm, Net, and Memory subclasses, synchronous parallelization using vector environments be combined with asynchronous parallelization of individual RL agent's 's learning algorithm. This multi-level parallelization is further discussed Section 3.3 where demonstrate its performance benefits.
(Dhariwal et al.] 2017)
2

---

Under review as a conference paper at ICLR 2020
Reinforce
ActorCritic
PPO
+init_ mait_algoritm_peea _aprrss
+init
+ini__nets
+init_nets
+init_nets
+calc__dde _acal__pppramm
caalc_pdparam
+calc_
+calc_policy_lor policy_loss
laaal__ddprrm__batee
-calo_pdparrm__v
+train
#calc_policyy_lee
+calo_y
+update
+calc_ret__advs
+calo_gae_ _w____ee___v____eeee V_targets
+act
aocal__nstep advs_ targets
+sample
+calo_ret advs targets
+train
+calc_ @calc_vallloss
+update
+train
+update
Figure 2: Reinforce, ActorCritic, and PPO class methods in SLM Lab. + indicates that method is added or overridden in the class.
2.2 EXPERIMENT ORGANIZATION
Reinforcement learning algorithms vary greatly in their performance across different environments, hyperparameter settings, and even within single environment due to inherent randomness. SLM Lab designed to easily allow users to study all these types of variability.
SLM Lab organizes experiments in the following hierarchy, shown in Figure
graphi
130
rut ratre
38
Stranc
Session
(b) Trial
(c) Experiment
Figure 3: Experiment organization in SLM Lab. A Session is a single run of an algorithm on an environment. A Trial collection of Sessions. An Experiment collection of Trials with different algorithms and/or environments.
1. Session The lowest level of the SLM Lab experiment framework: single training run of one agent on one environment with one of hyperparameters, all with fixed random seed.
2. Trial A trial consists of multiple Sessions, with the Sessions varying only in the random seed.
3. Experiment Generates different sets of hyperparameters (according to spec file, be- low) and runs a Trial for each one. It can be thought of as study, e.g. "What values of of A2C n-step returns provide the fastest, most stable solution, if the other variables held constant?"
SLM Lab automatically produces plots for Sessions, Trials, and Experiments for any combination of environments and algorithms. It also logs and tracks metrics during training such as rewards, loss, exploration and entropy variables, model weights and biases, action distributions, frames- second and wall-clock time. The metrics also visualized using TensorBoard (Abadi al. 2015). Hyperparameter search is implemented using Ray Tune (Liaw et al.) (018),, and the results are automatically analyzed and presented hierarchically in increasingly granular detail.

---

Under review as a conference paper at ICLR 2020
2.3 REPRODUCIBILITY
The complexity of RL algorithms makes reproducing RL results challenging (Henderson et al. 2017 Machado al. 2017). Every RL researcher knows the difficulty of trying reproduce algorithm from its description in a paper alone. Even if code is published along with RL research results, the key algorithm design choices (or mistakes (Tucker et al. 2018)) are often buried in the algorithm implementation and not exposed naturally to the user.
In SLM Lab, every configurable hyperparameter for an algorithm specified in file. The spec file is a JSON file containing a git SHA and all the information required to reproduce a Session, Trial, or Experiment as per Section 2.2 Reproducing the entirety of an RL experiment merely requires checking out the code at the git SHA and running the saved spec file.
The main entries spee file are given below. Examples of spec files are given in full in Supple- mentary Section A.4
1. agent- list (to allow for multi-agent settings in the future), each element of which con- tains the spec for single agent. Each agent spec contains the details for its components as described in Section 2.
(a) algorithm. The main parameters specific to the algorithm, such as the policy type, exploration strategy (e.g. c-greedy, Boltzmann), algorithm coefficients, rate decays, and training schedules.
(b) memory. Specifies which memory to use as appropriate to the algorithm along with any specific memory hyperparameters such as the batch size and the memory size. (c) net. The type neural network, its hidden layer architecture, activations, gradient clipping, loss function, optimizer, rate decays, update method, and CUDA usage.
2. env A list (to allow for multi-enviromment settings in the future), each element of which specifies an environment. Each environment spec includes an optional maximum time step per episode, the total time steps (frames) in Session, the state and reward preprocessing methods, and the number of environments in a vector environment.
3. body- Specifies how (multi-)agents connect to (multi-)environmmet)
4. meta The high-level configuration of how the experiment is to be run. It gives the number of Trials and Sessions to run, the evaluation and logging frequency, and a toggle to activate asynchronous training.
5. search The hyperparameters search over and the methods used to sample them. Any variables in the spec file can be searched over, including environment variables.
3 RESULTS
Reporting benchmark results is essential for validating algorithm implementations. SLM Lab main- tains a benchmark page and a public directory containing all of the experiment data, models, plots, and spec files associated with the reported results 2 We welcome contributions to this benchmark page via Pull Request.
We tested the algorithms implemented in SLM Lab on 77 environments: 62 Atari games and 11 Roboschool environments available through OpenAl gym (Brockman et al. 2016) and 4 Unity en- vironments (Juliani et al. 2018). These environments span discrete (Table and continuous (Table
control problems with high- and low-dimensional state spaces.
The results we report in each of the tables are the score per episode at the end of training averaged over the previous 100 training checkpoints. Agents were checkpointed every 10k (Atari) 1k (Roboschool, Unity) training frames. This measure is less sensitive to rapid increases or decreases in performance, instead reflecting average performance over substantial number of training frames.
To our knowledge, the results we present below are more comprehensive performance comparison than has been previously published for a single codebase. A full set of learning curves as well as full table of results for the Atari environments are provided in the supplementary materials.
URL redacted to preserve anonymity.
4

---

Under review as a conference paper at ICLR 2020
3.1 EXPERIMENT DETAILS
A complete set of spec files listing all hyperparameters for all algorithms and experiments are in- cluded in the slm 1abl http/ppe//benchaaeee released along with the results. Example spec files listing all of the hyperparameters for PPO and DDQN PER on the Atari and Roboschool environments are included in Supplementary Section A.4
directory of SLM Lab well in the experiment data
For the Atari environments, agents were trained for 10M frames (40M accounting for skipped frames). For the Roboschool environments, agents were trained for 2M frames except for RoboschoolHHmanoid (50M frames), RoboschoolFumannidPhgguu (100M) and RoboschoolHu-- manoidFlagrunHHarkee (100M). For the Unity environments, agents were trained for 2M frames. Training was parallelized either synchronously or with a hybrid of synchronous and asynchronous methods.
Our results for PPO and A2C on Atari games are comparable the results published by |Schulman et al. (2017). The results DQN and DDQN PER Atari games mixed: at the number of trainin frames we sometimes match or exceed the reported results and sometimes perform worse. This is likely due 200,000 compared to IM in Mnih et al. (2015), van Hasselt er al. (2015). and Schaul et al. (2015). The final output layer of the network smaller fully-connected layer with 256 instead of 512 units. Finally, our results for SAC confirm the strength of this algorithm compared to PPO for continuous control problems. However the absolute performance is typically worse than the published results from Haarnoja et al. (2018b). Due to computational constraints, SAC was trained with replay buffer of 0.2M elements and combined experience replay (Zhang & Sutton (2077) compared with 1M elements in Haarnoja et al. (2018b) and this is potentially significant difference.
hyperparameter differences. We used replay memory of size
at the end of training attained by SLM Lab implementations discrete- action control problems. Reported episode scores are the average over the last 100 checkpoints, and then averaged over Sessions. A Random baseline with score averaged over 100 episodes is included. Results marked with ** were trained using the hybrid of SAC to parallelize and speed up training time. For SAC, Breakout, Pong and Seaquest were trained for 2M frames instead of 10M frames.
Table 1: Episode
version
Environment
Algorithm A2C (GAE)
Random
DQN DDQN+PER
A2C (n-step)
PPO
SAC
Breakout
1.26
80.88
182
377
398
443
3.51
Pong
-20.4
18.48
20.5
19.31
19.56
20.58
19.87*
Seaquest
106
1185
4405
1070
1684
1715
171*
Qbert
157
5494
11426
12405
13590
13460
923*
LunarLander
-12
192
233
25.21
68.23
214
276
UnityHallway
-0.99
-0.32
0.27
0.08
-0.96
0.73
0.01
UnityPushBlock
-1.00
4.88
4.93
4.68
4.93
4.97
-0.70
3.2 SOFT ACTOR-CRES CRITIC FOR DISCRETE ENVIRONMENTS
All published results for the Soft Actor-Critic (SAC) algorithm (Haarnoja et al. 2018a b) are for continuous control environments. However, nothing in its algorithmic description makes unsuit- able in principle for use in discrete action-space environments. As a consequence the modular structure of SLM Lab, it straightforward to design and implement discrete variant of SAC. We did so by using policy Nets that produced parameters of a Gumbel-Softmax distribution (Jang al.) 2016 Maddison et al. 2016) from which discrete actions were sampled. The results of this discrete variant of SAC are in Table
On environments for which its training converged, found SAC be of comparable better sample efficiency than all of the other algorithms that we tested. For example, SAC achieves an
Estimated using the training curves provided in Figure of Schaul
2015
5

---

Under review as a conference paper at ICLR 2020
Table 2: Episode score the end of training attained by SLM Lab implementations on continuous control problems. Reported episode scores are the average over the last 100 checkpoints, and then averaged over Sessions. Random baseline with score averaged over 100 episodes included. Results marked with "*' require 50M-100M frames, version of SAC to parallelize and speed up training time.
use the hybrid synchronouu//syycchooouu
Environment
Algorithm
Random
A2C (GAE)
A2C (n-step)
PPO
SAC
Ant
52.30
787
1396
1843
2915
RoboschoolAAtassoraaadd Walk
40.23
59.87
88.04
172
800
RoboschoolHallCheetal
0.83
712
439
1960
2497
RoboschoolHopper
21.22
710
285
2042
2045
RobochholiivverrclDDuuuaaadalld
285
996
4410
8076
8085
RoboschoolInverredPPPddullm
23.5
995
978
986
941
RobaschoolReacher
-9.24
12.9
10.16
19.51
19.99
RoboschoolWWlker22 Walker2d
16.06
280
220
1660
1894
RoboschoolHumanode
-36.16
99.31
54.58
2388
2621* *
Roboschoo/HumanodFPlaee
-6.15
73.57
178
2014
2056
Robashhoo/lummaodddlaaaaaaaaaa
-8.64
-429
253
680
280*
Unil
1.24
33.48
53.46
78.24
98.44
UaallyD
1.17
62.92
71.92
91.41
97.06
average score over the previous 100 checkpoints of around 20 after IM frames on Atari Pong. whereas all other algorithms that we tested require least 2M frames to achieve the same result (plot shown in Supplementary Section A.3) However, even though SAC trained successfully on Pong and Lunar Lander, We also note that while SAC sample efficient is more computationally expensive than the other algorithms, which presents obstacle for extensive performance tuning.
not able to successfully train it all other Atari environments.
3.3 HYBRID SYNCHRONOUS AND ASYNCHRONOUS TRAINING
Synchronous and asynchronous parallelization be combined in SLM Lab to accelerate training, as shown in Figure 4 SLM Lab implements synchronous parallelization within Sessions (Section
2.2 using vector environments (Dhariwal et al.) 2017) and asynchronous parallelization within Tri- als (Section 2.2) using multiple workers, one per Session. There are two available methods for server-worker model in which workers periodically push gradients to a central network and pull copies of the updated parameters.
Trial level parallelization; Hogwild! (Recht et al.) 2011)
training is constrained by data sampling from the environment, then increasing the number of vector environments (synchronous parallelization) speeds up training. But this speed-up saturates as the training step becomes bottleneck and the environment waits for the agent to train. In example shown in Figure 4 the frames per seconds (fps) increases from around 200 for single worker and environment to around 360 for worker and 16 environments, and saturates thereafter. Once fps becomes constrained by the training step, it is beneficial to add workers (asynchronous parallelization) to effectively parallelize the parameter updates. A hybrid of 16 workers each with environments resulted in the maximum fps of around 3800.
4 RELATED WORK
4.1
REPRODUCIBILITY IN REINFORCEMENT LEARNING
The instability of RL algorithms Haarnoja et al. 2018a environment (Henderson al. 2017 Islam al. 2017). as well differences in hyperparameter
randomness in agent policies and the
tuning (Islam et al.] 2017) and implementations Henderson et al.) 2017 Tucker et al, 2018) all contribute the challenge of reproducing RI. results.

---

Under review as a conference paper at ICLR 2020
worker
2 workers
workers
16 workers
4000
3794
3445
3099
3000
Frames per second
2095
1948
1799
2000
1672
1000
572
627
673
202
218
420
245
439
316
368
301
2
4
8
16
32
64
Total parallelism (num vector envs num workers)
Figure 4: Average frames per second as number of vector environments and Hogwild! workers are varied. Each setting was trained using PPO with the same hyperparameters on the RoboschoolAnt environment.
Consequently, the importance of comprchensively documenting all hy hyperaarameter' along with pub- lished results and software is well recognized within the research community (Machado al. 2017 Castro et al. 2018).
4.2
SOFTWARE FOR REINFORCEMENT LEARNING
To date more than twenty reinforcement wewprnentt--erning-gheeee released. They can be organized into two categories: those implementing RL algorithms and those implement RI. environments SLM Lab is algorithn-focused library with built-in integration with the OpenAI gym (Brockman et al., 2016). OpenAI Roboschool, VizDoom (Kempka et al. 2016), and Unity ML-Agents (Juliani al. 2018 environment librarics.
open source software libraries have been
Table summarizes the algorithm-focusee reinforcement learning software libraries.
Libraries such Catalyst (Kolesnikov 2018),
ChainerRL (chainer 2017 Tokui & Oono 2015).
coach (Caspi et al.] (017)) DeepRL (Zhang) 2017). OpenAl baselines (Dhariwal et al. [201]] 7).7. RL-
graph (Schaarschmidt
al. 2019). RLkit (Pong
2018 ripyt (Stooke Abbeel [2019). RLLib
(Liang et 2017), Stable Baselines (Hill et al.
2018). TensorForce (Kuhnle et al. 2017). TF-
Agents (Guadarrama al. 2018). and vel (Tworek] 2018) implement wide variety algorithms and are intended to be applied to a variety of RL problems. Most of these libraries also provide some benchmark results for the implemented algorithms to validate their performance. These can be thought of as generalist RL libraries and are the most closely related to SLM Lab.
Other libraries focus on specific algorithms (Dopamine (Castro et al.) 2018), Softlearnin (Haarnoja al. 2018c), a2c-ppo-acke satooo-cctrrgggg (Kostrikov) 2018). Keras-RI RL [Plapper||2011 (016 problems (Open-
Spiel Lanctot et al.) 2019). ELF (Tian et al. 2019), MAgent (Zheng et al. al.) 2017). reaver (Ring
2018 components such loss functions (TRFL (DeepMind 2018) scaling training (Horizon
(Gauci et al. 2018)).
The use of configuration files specify hyperparameters varies. Catalyst, coach, DeepRL, Dopamine, Horizon, reaver, Softlearning, RLgraph, RLLib, and vel use configuration files and pro- vide number of configured examples. In most cases the network architecture is excluded from the
A few libraries such as OpenSpiel Lanctot
2019 and EI.F Tian al. 2019 implement both.

---

Under review as a conference paper at ICLR 2020
Table Comparison of RL software libraries. Algorithm acronyms are explained in Supplementary Section A.I. REINFORCE is excluded are less well-known algorithms. "Benchmark" indicates whether the library reports the performance of their implementations. "Config" indicates whether hyperparameters are specified separately from the implementation and run scripts; "split" indicates that the configuration is divided across multiple files, "partial" indicates that some but not hyper- parameters are included. "Parallel" denotes whether training for any algorithms can be parallelized. "HPO" denotes support for hyperparameter optimization. "Plot" denotes whether the library pro- vides any methods for visualizing results.
Library
Algorithms
Benchmark
Config
Parallel
HPO
Plot
apo-aaccttt-aail
A2C, ACKTR, GAIL, PPO
X
V
Baselines
A2C, ACER, ACKTR, DDPG. DQN, GAIL, HER, PPO, TRPO
X
X
V
Catalyst
CRA, TD3, SAC, DDPG, QR
V
V
X
V
ChainerRL
A3C. ACER, C51, DDPG,
X
V
DQN+, IQN. NSQ. PPO,
Rainbow, TRPO, TD3, SAC
coach
A3C, ACER, C5L, DDPG.
split
X
DQN+, QR-DQN, NAR, NEC,
NSQ, PPO, Rainbow, SAC,
TD3
DeepRL
A2C. C51, DDPG, DQN+,
split
1
NSQ. OC, PPO, QR-DQN, TD3
T
Dopamine ELF
DQN, C51, Rainbow
split X
A3C, DQN, MCTS, TRPO
Keras-RL
DDPG, DQN+, NAF, CEM,
X
SARSA
Horizon
C51, DQN+, SAC, TD3
X
V
MAgent
A2C, DQN
X
X
OpenSpiel
A2C. DQN, MCTS
X
X
Y
pytorch-rl
A2C. DDPG, DON+, HER
X
reaver
A2C, PPO
split
RLgraph
A2C, Ape-X, DQN+, DQFD,
IMPALA, PPO, SAC
RLkit
DQN+, HER, SAC, TDM, TD3
XI
IX
RLLib
A3C. Ape-X, ARS, DDPG.
partial
DQN, ES, IMPALA. PPO. SAC
ripyt
A2C, DDPG, DON+, CAT-
X
DQN, PPO, TD3, SAC
Softlearning
SAC
V
Stable Baselines
A2C, ACER, ACKTR, DDPG,
DQN, GAIL, HER, PPO, SAC,
TD3, TRPO
TensorForce
A3C, DQN+, DQFD, NAF,
X
X
PPO. TRPO
TF-Agents TRFL
DDPG, DQN+, PPO, SAC, TD3
X
X
XX
X
vel
A2C, ACER, DDPG, DQN+, Rainbow, PPO, TRPO
split
SLM Lab
A2C, A3C, CER, DQN+, PPO, SAC
/
/
main configuration file and specified elsewhere. Catalyst, Horizon and RLgraph include all hyperpa- rameters in single configuration file and are the most similar SLM Lab. However, RI-graph does
8

---

Under review as a conference paper at ICLR 2020
not include environment information, and none use the same file for configuring hyperparameter search.
Almost all libraries include some methods for parallelizing agent training, especially for on on-policy methods. However most do not include hyperparameter optimization as feature. The two excep- tions Stable Baselines (Hill al. 2018 and Tensorforce (Kuhnle al. 2017).
Finally, many libraries include some tools for visualizing and plotting results. Notably coach (Caspi et al. 2017) provides an interactive dashboard for exploring variety of metrics which are automat- ically tracked during training.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving. Michael Isard, Yangging Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg. Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learn- ing heterogeneous systems, 2015. URL https: /twp://wwwwww.o wttp:w/wtw......o.o.ooooo/oo org/ Software available from tensorflow.org.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, 10.01400.,1540, 2016. http:tt:://// /ttp:i////avv../a org/
abs/1606.0155
Itai Caspi, Gal Leibovich, Gal Novik, and Shadi Endrawis. Reinforcement learning coach, Decem- ber 2017. URL
2e022////2noob.113388..99 1134899
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018. URL http: Tla......or//ss//112222.
chainer. Chainerrl. http://////ttttooooooooaaaaaaaaa
2017.
DeepMind, trfl.
2018.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthis Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https: 2017.
Jason Gauci, Edoardo Conti, Yitao Liang, Kittipat Virochsiri, Zhengxing Chen, Yuchen He, Zachary Kaden, Vivek Narayanan, and Xiaohui Ye. Horizon: Facebook's 's open source applied reinforce- ment learning platform. arXiv preprint arXi::1811.00200 2018.
Sergio Guadarrama, Oscar Ramirez Anoop Korattikara, Pablo Castro, Ethan Holly, Sam Fishman, Ke Wang, Ekaterina Gonina, Neal Wu, Chris Harris, Vincent Vanhoucke, and Eugene Brevdo. TF-Agents: A library for reinforcement learning in tensorflow.
/ttp:///gtttttattt com/ ferssofflow/aagett.. 2018. URL httpp::/ggiittub..cmmtteeeett/t/ [Online; accessed 25-June-2019),
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. CoRR, 0bbs/1801.1225
2018a. URL htl.http:/// /ttp::///xxxvv/ /ttpp//////..oo.o/////////////// 1801.1000 01290.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. CoRR, abs/1812.059055, 2018b. URL
1812. 05905/
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. Technical report, 2018c.

---

Under review as a conference paper at ICLR 2020
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. ArXiv, 0ass/100900656// 2017.
Ashley Hill, Antonin Rafin,, Maximilian Ernestus, Adam Gleave, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schul- man, Szymon Sidor, and Yuhuai Wu. Stable baselines. https: /tta::///ghhuae com/hill a/
stable-base ines 2018.
Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of bench-
marked deep reinforcement learning tasks for continuous control. ICML Workshop on Repro- ducibility in Machine Learning. 2017. URL http: wttp://wwwwwxxxxx//////x////////
/ttpp////aa.....rr////////..... 04133
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzatoo with gumbel-softmax, 2016.
URL http::://arxw..oo////aaaaaa http://b///1111// 01144 cite 1611v:1611.01144
Arthur Juliani, Vincent-Pierre Berges, Esh Vekay, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. Unity: A general platform for intelligent agents. CoRR, abs/1809.02627, 2018.
URL
Michal Kempka, Marek Wydmuch, Grzegorz Rune, Jakub Toczek, and Wojeiech Jaskowski. ViZ- Doom: A Doom-based AI research platform for visual reinforcement learning. In IEEE Confer- ence on Computational Intelligence and Games, pp. 341-348, Santorini, Greece, Sep 2016. IEEE. The best paper award.
URL http::/atxiy...ttt///// http://bb/116155 02097
Sergey Kolesnikov. Reproducible and fast dl rl.
cwt.
catalyst 2018.
llya Kostrikov. Pytorch implementations of reinforcement learning algorithms.
https:
github. wwm httm///koott.kko///ttttrrttt
ppwpppppppaaoooppppppppp aeoooaccktraaalaa 2018.
Alexander Kuhnle, Michael Schaarschmidt, and Kai Fricke. Tensorforce: a tensorflow library for applied reinforcement learning. Web page, 2017. URL https://github.r com/
[eersorforree/eesorfoooc
Mare Lanctot, Edward Lockhart, Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Prolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafici, Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, Jnos Kramr, Bart De Vylder, Brennan Saeta, James Bradbury, David Ding, Sebastian Borgeaud, Matthew Lai, Julian Schrit- twieser, Thomas Anthony, Edward Hughes, Ivo Danihelka, and Jonah Ryan-Davis. Openspiel: A framework for reinforcement learning in games, 2019.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Joseph Gonzalez, Ken Gold- berg, and lon Stoica. Ray rllib: A composable and scalable reinforcement learning library. CoRR, abs/1712.0033, 2017. URL http::ttt:///ii org/ abs 11722. 09381
Richard Liaw, Eric Liang Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Sto- ica. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv: 2018.
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. CoRR, abs/ ttss//00900009 2017. URL
1709.06009
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. CoRR, abs/1611.000111 2016. URL httt:://paxxiiz 3xg/abs/1611..0 00712
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529 533, February 2015. ISSN 00280836. URL http://dx. doi www.doo...ooogg. 0ttg/10.1033 1038//aturreeeeee 4236
10

---

Under review as a conference paper at ICLR 2020
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vetito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Matthias Plappert. keras-rl. http:///gtttubb.ccm/eeeett/////s
2016.
Vitchyr Pong. rikit.
2018.
Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinherger (eds.), Advances in Neural Information Processing Systems 24, pp. 693-701. Curran Associates, Inc., 2011. WtRLhttp://pappeee nips ccppperr/ppprr
4390 hogwi 1d a lock free approach to t pdf
ieceaasttc-grddiinne desienttdescent...
Roman Ring.
Reaver: Modular deep reinforcement learning framework.
https://githhi:
com/ inoryy/reaver 2018.
Michael Schaarschmidt, Sven Mika, Kai Fricke, and Eiko Yoneki. RLgraph: Modular Computation Graphs for Deep Reinforcement Learning. In Proceedings of the 2nd Conference Systems and Machine Learning (SysML), April 2019.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. CoRR, abs/1511.000 1.05952, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, has/1707.06341 2017. URL http::/aaxiv.orgg/ absa
06347
Adam Stooke and Pieter Abbeel. ripyt: A research code base for deep reinforcement learning in pytorch. 2019.
Yuandong Tian, Jerry Ma, Qucheng Gong, Shubho Sengupta, Zhuoyuan Chen, James Pinkerton, and Larry Zitnick. ELF opengo: analysis and open reimplementation of alphazero. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 6244-6253, 2019. Ut/htt:://grroeeenngg .ml. presss v97 /97/tianl 9a. html
Sciya Tokui and Kenta Oono. Chainer: next-generation open source framework for deep learning. 2015.
George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard E. Turner, Zoubin Ghahramani, and Sergey Levine. The mirage of action-dependent baselines in reinforcement learning, 2018. URL
http::////ppaeeeeeeeeeeeeettttoo ntt
Jerry Tworek. vel. wttttsss/gstttoooo////oooooonnn
2018.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-
learning. CoRR, abs/1509.00461, 2015. URL
Shangtong
2017.
Shangtong Zhang and Richard S. Sutton. A deeper look experience replay, CoRR,
abs/1712.01275. 2017. URL
Lianmin Zheng, Jiacheng Yang. Han Cai, Weinan Zhang, Jun Wang, and Yong Yu. Magent: A many-agen reinforcement learning platform for artificial collective intelligence. CoRR, 10l//772.00000. 2017. URL arxiv
00600

---

Under review as a conference paper at ICLR 2020
A SUPPLEMENTARY MATERIALS
A.1 ALGORITHM ACRONYMS
The expanded acronymns for all of the algorithms listed in Table are given below:
A2C: Advantage Actor-Critic
ACER: Actor-Critic with Experience Replay
ACKTR: Actor-Critic using Kronecker- Trust Region
Ape-X: Distributed Prioritized Experience Replay
ARS: Augmented Random Search
C51: Categorical DQN
CAT-DQN: Categorical DQN
CRA: Categorical Return Approximation
CEM: Cross Entropy Method
CER: Combined Experience Replay
DDPG: Deep Deterministic Policy Gradients
DQN: Deep Networks
DQN+: Deep Q Network modifications, including some all of the following: Double DQN, Dueling DQN, Prioritized Experience Replay
DQFD: Deep Q-Learning from Demonstrations
ES: Exolutionary Strategies
GAIL: Generative Adversarial Imitation learning
HER: Hindsight Experience Replay
IMPALA: Importance Weighted Actor-Learner Architecture
IQN: Implicit Quantile Networks
MCTS: Monte Carlo Tree Search
NAF: Normalized Advantage Functions
NEC: Neural Episodic Control
NSQ: n-step Q-learning
OC: Option-Critic
PPO: Proximal Policy Optimization
QR: Quantile Regression
QR-DQN: Quantile Regression DQN
TD3: Twin Delayed Deep Deterministic Policy Gradient
TRPO: Trust Region Policy Optimization
SAC: Soft Actor-Critic
A.2
ATARI RESULTS
The table below presents results for 62 Atari games. All agents were trained for 10M frames (40M including skipped frames). Reported results are the episode score at the end of training, averaged over the previous 100 evaluation checkpoints with each checkpoint averaged over 4 Sessions. Agents were checkpointed every 10k training frames. A Random baseline with score averaged over 100 episodes included.
Environment
Algorithm
(Atari, Discrete)
Random
DQN
DDQN+PER
A2C (GAB) A2C (n-step)
PPO
12

---

Under review as a conference paper at ICLR 2020
Adventure
-0.89
-0.94
-0.92
-0.77
-0.85
-0.3
AirRaid
486
1876
3974
4202
3557
4028
Alien
97.0
822
1574
1519
1627
1413
Amidar
1.8
90.95
431
577
418
795
Assault
308
1392
2567
3366
3312
3619
Asterix
307
1253
6866
5559
5223
6132
Asteroids
1331
439
426
2951
2147
2186
Atlantis
29473
68679
644810
2747371
2259733
2148077
BankHeist
13.4
131
623
855
1170
1183
BattleZone
3080
6564
6395
4336
4533
13649
BeamRider
355
2799
5870
2659
4139
4299
Berzerk
212
319
401
1073
763
860
Bowling
24.14
30.29
39.5
24.51
23.75
31.64
Boxing
-0.91
72.11
90.98
1.57
1.26
96.53
Breakout
1.26
80.88
182
377
398
443
Carnival
905
4280
4773
2473
1827
4566
Centipede
2888
1899
2153
3909
4202
5003
ChopperCommant
735
1083
4020
3043
1280
3357
CrazyClimber
2452
46984
88814
106256
109998
116820
Defender
546810
281999
313018
665609
657823
534639
DemonAttack
346
1705
19856
23779
19615
121172
DoubleDunk
-16.48
-2.44
-22.38
-5.15
-13.3
-6.01
Elevator
9851
32.62
17.91
9966
8818
6471
Enduro
0.0
437
959
787
0.0
1926
FishingDerby
-93.96
-88.14
-1.7
16.54
1.65
36.03
Freeway
0.0
24.46
30.49
30.97
0.0
32.11
Frostbite
68.0
98.8
2497
277
261
1062
Gopher
276
1095
7562
929
1545
2933
Gravitar
219
87.34
258
313
433
223
Hero
706
1051
12579
16502
19322
17412
IceHockey
-9.87
14.96
-14.24
-5.79
-6.06
-6.43
Jamesbond
13.0
44.87
702
521
453
561
Jocape
18095
-4888
-2033
.921
-202
-094
Kangaroo
54.0
1965
8897
67.62
554
4989
Krull
1747
5522
6650
7785
6642
8477
KungFuMaster
865
2288
16547
31199
25554
34523
Moneezumaaeveenee
0.0
0.0
0.02
0.08
0.19
1.08
MsPacman
170
1175
2215
1965
2158
2350
NameThisGame
2088
3915
4474
5178
5795
6386
Phoenix
1324
2909
8179
16345
13586
30504
Pitfall
-301
-68.33
-73.65
-101
31.13
-35.93
Pong
-20.4
18.48
20.5
19.31
19.56
20.58
Pooyan
515
1958
2741
2862
2531
6799
PrivateEye
-731
784
303
93.22
78.07
50.12
Qbert
157
5494
11426
12405
13590
13460
Riverraid
1554
953
10492
8308
7565
9636
RoadRunner
35.0
15237
29047
30152
31030
32956
Robotank
1.78
3.43
9.05
2.98
2.27
2.27
Seaquest
106
1185
4405
1070
1684
1715
Sking
17361
14094
12883
19481
-1434
24713
Solaris
2097
612
1396
2115
2236
1892
SpaceInvaders
164
451
670
733
750
797
StarGunner
645
3565
38238
44816
48410
60579
Tennis
-24.0
-23.78
-10.33
-22.42
-19.06
-11.52
TimePilot
3151
2819
1884
3331
3440
4398
Tutankham
15.45
35.03
159
161
175
211
UpNDown
125
2043
11632
89769
18878
262208
Venture
0.0
4.56
9.61
0.0
0.0
11.84
VideoPinball
25365
8056
79730
35371
40423
58096
13

---

Under review as a conference paper at ICLR 2020
WizardOfWor 784
869
328
1516
1247
4283
YarsRevenge
3369
5816
15698
27097
11742
10114
Zaxxon
0.0
442
54.28
64.72
24.7
641
A.3 LEARNING CURVES FOR ALL ALGORITHMS AND ENVIRONMENTS
Each learning curve depicts the episode score. averaged over the previous 100 evaluation check- points, with each checkpoint averaged over 4 Sessions. The shaded area depicts +/- one standard deviation calculated over Sessions.
Pong
5000
190
DDEN
A2C (GMC)
AZC (n-step)
PP1
SAC
124
Qbert
Lumert Lamberr
UnityPasidtlck
-200
-00
Noboschou
Roboschas www.ssssstttisteeaataeeeee
HubcoschooHalli HallChestah
Bubboschoctooper
14:00
1000
1000
500
Rotochhellierttdeedddddd
RobeschoolWaaccoo
1000
1550
10:00
0.54
Robhechhoolumanndd
Unby2CBall
14

---

Under review as a conference paper at ICLR 2020
Uniry Harl
Adventur
Airfard
Allnn
Amiidar
0.4
1000
550
30
124
Acso
Asterix
Asterolds
Attentis
4000
2200
2.54
2003
1500
3.5%
3.5m
Berlcheest
Belthe Bant
Rider
Barzerk
23c
10.0
1050
630
15c
3:53
100
Bowling
Boxing
Breakout
480
250
3000
2050
Centipade
Choppe Command
C CrazyCilber
Delender
1000
700
05
300
100
15

---

Under review as a conference paper at ICLR 2020
Demonkitack
DouolDDunk
EleratorActon
Enduro
-1
FohingDerby
Freeway
Gophor
2000
8000
1550
00.0
-40
4.00
-50
25:00
-50
1D4
2.5
IEN
Cires har
Heeo
trantoadeyy
Jameechend
234
190
-20
180
Kangaroo
3000
200
-100
X00
1000
6M
124
124
MontesumoRovvege
MrPocman
HameThisGome
Phoentx
2003
3.5
5000
5000
400
1000
122
200
-0.5
100
Pitfol
Poayan
Privatellye
15:0
-20
300
20
1EN
Obert
Riverrald
ReadRunner
Robotank
13c
16

---

Under review as a conference paper at ICLR 2020
Seaquesst
Sking
Solars
Spoctiraaees
300
200
100
0153
300
200
180
-20
100
124
124
Upp4ttwn
Ventuue
VideoPanbel
WowaddC/Wer
4200
200
1100
68 124
A.4 EXAMPLE SPEC FILES
1
2
"dden catnn_pee_arrrri
3
"agent": [[
4
"namc": "DoubleDQN", "algorithn";
5
"name"; "DoubleDoN", "action_pdtyype:: "Argmax",
8
"act "action_policy" "epsilcn_greedy"" "explore_var_spect" " "name": "Linear_decay"" "start_vai" 1.0,
9
10
"end 0.01, "start_step": 10000, "end_step"; 1000000 },
13
14
"gamma": 0.99, beptininngbbtchhh.. 1, "traaning_iter": 4, "trraining_frrequeeee"":: 4, "training_start__tte""" 10000
18
21
},
22
"memory": "name": "prioritiiedkeplay"" "alpha": 0.6,
23
24
17

---

Under review as a conference paper at ICLR 2020
25
"epsilon": 0.0001,
"batch_size"; 32, "max_size"; 200000, "usc_cor"; false,
27
28
29
}
30
"net":
"type": "ConvNet",
32
"conv widehri___yyss""
33
132, 8, 4, 0, 11,
34
164, 4, 2, 1l, [64, 3, 1, 1]
35
1, "fa_hid_layers": [256],
37
38
"hid _a "init_fn": null, "batch_norm"" false, "clip_grad_val"; 10.0,
activation": "relu",
39
41
42
"loss_spec"; { "name"; "SmoothLlloss" } Coptim_speatt { "name": "Adam", "ir": 2.5e-5,
43
44
46
47
48
1,
"ir scheduler spec": null, "update_type"" "replace", "update_frequency" 1000, "gpu": true
51
52
53
}1,
"env": [[ 55
56
"namc"; "$[env)",
57
" "frame_op": "concat",
"frame_op_len", 4, reward_scall"" "sign", "num_envs": 16, "max_t": nul1, maax_frame: 1e7
60
61
62
11,
64
"body": I "product": "outer", "num"; 1
65
66
67
68
"mota":
69
"distributed": false, "eval_frequency"" 10000, "log_frequency": 10000, "rigorous_eval"" 0,
70
71
73
"max_session"; 4, "max trax_trial";
74
75
"spec_params"
"env": [
78
"BreakontteFFrrmesttiieee
"PongNoFFameskiigrvt "GbertNoFrameskip
v4", "SeaquesttoFFaaeskkipp
81 },
82
Listing The Double DQN PER spec file for Atari games
1
2
"ppo_atare aaaaaii""
3 "agent": [[
18

---

Under review as a conference paper at ICLR 2020
4
"name" "PPO"
5
algorithm";
6
"name": "PPO", *act "action__ttype": "default", "action_policy": "default", tttploor__va_sppeee mull, "gamma": 0.999 "lam": 0.70,
7
8
9
12
"clip_eps_sped" "name": "no_ dec_dd "start_wal"; 0.10,
"end_val"; 0.10, "start_step", 0, "end step":
16
17
18
20
"name"; "no_decay", "start_val"; 0.01,
21
22
"end_val"; 0.01, "start_step"; "end_step";
23
25
b
26
0.5,
27
"time horizon"i 128,
28
"minibatch size": 256, "training_epoch":
30
},
31
"memory":
32
"name": "OnPollcyBattchReplay""
}
34
"net": {
35
"type": "ConvNet",
36
"shared"; true, hedw_hid__lyers"" [32, 8, 4, 0, 1],
39
[64, 4, 2, 0, 1],
40
[32, 3, 1, 0, 1]
41
1.
"fc hid layers": [512], "hid layers activation"; "init_fn" fn": "orthogonal_", "normalize"; true, "batcch_norm; false, "clip_grad__all" 0.5, sass__sae_opptn"" false, "lcss_spec": "name": "MSELoSs"
43
_ayers_activationn "relu",
44
45
46
48
49
50
1, "actor_optine optoo___ptims_ee": { "name": "Adam", "Ir": 2.5e-4,
52
53
54
55
} peritic_optin_spee. { "name": "Adam",
S7
58
"Ir": 2.5e-4,
59
1,
"Ir si__scheduler_spoe"" "name"; "LincarToZero", "frame"; 1c7
61
62
63
64
"gpu": true
66
11.
67
"env": 11
68
"name": "$ "(len"",
19

---

Under review as a conference paper at ICLR 2020
69
"frame_ op: "concat",
" "frame_op_len" 4,
71
"roward_scale: "sign",
72
"num_cnvs"; 16, "max_t": null, "max_frame": 1e7
73
74
}1,
76
"boay":
77
"product"; "outer",
78
"num": 1
79
meta":
81
"distributed": false, "eval "requueeee:e;"" 10000,
82
83
"log fre__erequen;"; 10000,
trigorous_evalle eval"; 0,
85
"max_scssion"; 4,
86
"max_trial"; 1
87
88
"spec_params";
"env": [
90
ProngNoFrameskip-ee "abertNoframeski
-v4"
91
93
},
94
Listing 2: The PPO spec file for Atari games
1
2 "ppo_roboschool";
3
"agent": [[ "name": "PPO", "algerithm":
4
5
6
"name": "PPO", "action_pdtype"" "detault", "action_policy": "default", "explore_var_spec" null, "gamma": 0.999, 0.99,
7
8
9
10
11
"lam": 0.95,
"name": "no_decay" estart_val"; 0.20,
15
"cnd_val"; 0.20,
16
"start_step"; 0, "end_step";
},
19
ront {
20
"name": "no_decay",
21
"start_val" 0.0,
"end_vai" 0.0,
23
"start "start_ssep"' 0, "end_step"; },
24
25
26
"val_loss_coef"" 1.0,
"time_horizon 2048, mminibatch_sizee 128,
28
29
"training_epoch": 10
30
"memory": "name": "OmPollcyBatchReepayy!
32
33
34
"net": "type": "MIPNet",
35
20

---

Under review as a conference paper at ICLR 2020
36
"shared": false,
"hie_layers"; 1256, 2561, "hit_layars_actiattion "relu", "init_fn"; "orthogonal_", aclip_grad_val"" 0.5, "use_same_optim"" false, "lcss_spar cicss_spec": "name": "MSELoss" 1,
39
40
41
43
44
45
"actor "actorreptim spea": { "name": "Adam", "Ir": 3e-4,
46
48
b "critic "critic_optin_spe "name": "Adam", "Ir": 3e-4,
49
50
52
b "Ir "Ir_scheduler_sppet" null, "gpu": false
53
54
55
}1,
57
"env": [[
58
"name": "$Ienv)",
59
"num_envs" 8,
"max t": null, "max_frame": 266
61
62
}l.
63
"body":
64
"product";
"outer",
"num":
66
67
"mota":
68
false, "distributed": "Log_frequency": 1000, "eval_frequency": 1000,
71
*rigorous_eval"" 0,
72
"max_session"; 4, "max_trial":
73
75
"spec :
76
"env":
77
"Robeschoolant-v1",
koboschoollalllChhettt..."" "Robschholloopee-vv"", "Rbbschool.IvorrroPeooouuuu
"adoocchooIIneeeteeeeeeddddaiadd "toboschoolleeschhrrr"""" Roboschool1Walker222-1
78
80 },
81
Listing 3: The PPO spec file for Roboschool environments (excluding Humanoid)
A.5 KEY IMPLEMENTATION LESSONS
Implementing a number of well-known deep reinforcement learning algorithms taught many im- portant lessons. Such lessons are typically relegated to blog posts and personal conversations, but we feel worth sharing such knowledge the research literature.
Put everything that varies into the spec file. e.g. when implementing A2C we hardcoded state normalization some places but not others. This extended the debugging process by days.
21

---

Under review as a conference paper at ICLR 2020
Log policy entropy, Q or V function outputs and check that they change. Looking Q function values for a random agent helped us to track down an error due to state mutation.
As with all Python code, watch out mutation. returned from OpenAl baselines vector environment was mutated by preprocessing for the next time steps. This was because the state was tracked as internal variables and returned directly. To remedy this, the state must be returned
example, we noticed that the state
copy.
Start simple- implement parent algorithms first. e.g. REINFORCE, then A2C, then PPO or SAC, and ensure that each of these work in turn. PPO and SAC were straightforward to debug once A2C was working.
Write tests: Tests have been invaluable in debugging and growing SLM Lab. Especially useful are tests for tricky functions. For example, a test for the GAE calculation was es- sential for debugging A2C. Additionally, use Continuous Integration to automatically build and test every new code changes in Pull Request, is done in SLM Lab.
Check input and output tensor shapes for important calculations. e.g. predicted and tar- get values for Q or V. Broadcasting errors may lead to incorrect tensor shapes which fail silently.
Check the size of the loss the beginning of training. Is this comparable to other imple- mentations? Tracking this value was essential when debugging DQN. It led us to a bug in the image permutation and we also stopped normalizing the image values result.
Check that the main components of the computation graph connected expected. This can be implemented as a decorator around the training step and run when debugging. Con- versely, be careful to detach target values.
Visualize trained agents. We forgot to evaluate agents with reward clipping turned off, which led to very significant underestimation of performance for all environments except Pong. We finally realized training working only after deciding to look at some policies playing the game.
22


Reviews:
[{"review_id": "SyxZDeUO5r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper provides a description of a new framework for reproducible and efficient RL experiments, as well as benchmarks of many algorithms on popular environments, such as `Atari and Roboschool. \n\nPros:\n- I agree that reproducibility is an extremely important question for the RL research, and thus such a code library is very beneficial for the community.\n- The library is well designed, and allows for creating extensions rather easily in the future.\n- Benchmarks are quite extensive and instructive.\n\nCons:\n-  Comparison with the library [1] is missing (see also [2] for description and benchmarks). As both libraries are focused on reproducibility and flexible implementations of algorithms, such a comparison would support authors claims.\n- I am not sure that ICLR is the right venue for such paper. Perhaps a more specialized conference of a workshop would be better.\n- Anonymity violation\n\nQuestions:\n- How difficult it is to implement distributional algorithms in your framework?\n- What about different exploration strategies? (Boltzmann, epsilon-greedy, parameter noise etc.). I guess it should be quite easy to make it configurable as well\n\n\n[1] https://github.com/catalyst-team/catalyst\n[2] https://arxiv.org/pdf/1903.00027.pdf\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "SyeSd7ZAKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SLM Lab is a software framework for reinforcement learning, which includes many different algorithms, networks, and memory types. The framework is well structured and modular. Thus, it is easily extendable for anyone and can be a pinnacle for future RL research. \n\nThe really like the paper. It is well written, easy to read, and provide a valuable platform / framework to the community, both the scientific community as well as practitioners. Although the scientific contribution may be low in the paper, I think the significance and potential impact of the paper outweigh that. \n\nThe paper also include many results from running the framework in various configurations, showing the flexibility and usefulness of it.\n\nThe code for SLM Lab is released open source, which is very valuable and enables future research in RL. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1xO_s5CKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new RL library called « SLM Lab ». Its most relevant features for RL research are: (1) modularity to help re-use existing components (thus reducing the risk of subtle implementation differences when comparing algorithms), (2) implementations of most popular algorithms like DQN & variants, A3C, PPO, SAC, (3) ability to parallelize both actors (through vectorized environments) and the learner (through distributed gradient descent), and (4) utilities for hyper-parameter optimization, reproducible experiments and reporting. The paper also reports performance over Atari games, Roboschool environments as well as some Unity ML-Agents tasks. Finally, it provides a high-level overview of SLM Lab’s capabilities compared to 23 other RL open source libraries, showing that it is the only one that combines: reporting of the performance of the implemented algorithms, ability to specify hyper-parameters in the config file, parallelization, hyper-parameter optimization, and visualization of the results.\n\nOverall this looks like a solid RL library, but I am not convinced that it brings enough novelty to the RL software landscape for a published ICLR paper — it would better fit in a workhop dedicated to ML libraries for instance, thus the weak reject.\n\nTable 3 shows that SLM Lab is the only RL library with such a broad offering of features, and this is definitely impressive, but I would argue that many of these features can typically be added to other libraries by plugging in other open source software. For instance there are several tools for experiment management and hyper-parameter optimization (and for RLLib in particular, hyper-parameter optimization is not checked but is straightforward with Ray Tune). TensorBoard can also often be easily used for visualization.\n\nThe parallelization capabilities of SLM Lab also seem limited: if I understand correctly, actor parallelization can only occur on a single machine, and thus an algorithm like Ape-X or R2D2 could not be implemented. If this is correct then it is a major limitation of the framework, since such parallelization across multiple computers can be extremely useful when environments are slow and costly to run (in CPU / RAM).\n\nIt is not clear to me to which extent multi-agent is supported. It seems like it is possible to have multiple agents in one environment, but is that enough for general multi-agent RL? (ex: how to specify individual / team rewards? share information between agents? deal with agents not acting all at the same timestep? centralize part of the training / execution?…)\n\nI appreciate that the paper presents a discrete version of SAC, mentioning how easy it was to implement thanks to the modular design of SLM Lab, but results from Table 1 do not look very good (especially since it did not work on some of the environments). Relying on the Gumbel-softmax might not be the most robust & stable way to train a discrete SAC — see e.g. the recent « Soft Actor-Critic for Discrete Action Settings » for a different approach.\n\nFinally, it is also great to have some benchmarks of the algorithms being implemented, but at least for Atari, I am not aware of previous work using the exact same evaluation setting, so it is hard to tell how they compare to other implementations.\n\nIn spite of the above, I do not mean to criticize SLM Lab too heavily as from what I can tell it seems to be a a solid library with many useful features, and I am sure many researchers will find it useful in their day-to-day work.\n\nMinor points:\n- Anonymity was clearly violated with the two github links\n- Having some synthetic result on Atari (like the typically reported median human-normalized score) would be good\n- I am personally not a fan of large config JSON files due to the lack of comments in JSON\n- A.5 (« Key Implementation Lessons ») is great!\n\nReview update after author feedback: I am on the fence for this paper, but still leaning towards rejection due to the fact I am still not convinced that this library brings that much novelty compared to existing other libraries (although it seems like a nice RL library, I am not sure ICLR is the right venue for talking about it). The authors argue that their benchmark results are a key contribution of the paper, but I do not find these results particularly insightful, especially the Atari ones that are not comparable to previous results (due to using a different evaluation method) and the lack of state-of-the-art algorithms like Rainbow or IQN.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: A new software framework fo Deep RL is introduced. This is a useful work for the community, but it is not a research work. I agree with Reviewer4 that somehow it is not a right venue: other papers need to have technical contributions, SOTA, and here - it is difficult but it is another type of work - accurate technical implementation and commenting. I do not feel right to have as it a paper on ICLR. 
Decision: Reject