Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platform
Authors: Shengyi Huang
Jiayi Weng
Rujikorn Charakorn
Min Lin
Zhongwen Xu
Santiago Ontanon
Venue: ICLR 2024 poster (2024)
Keywords: Distributed
Deep Reinforcement Learning
Distributed Deep Reinforcement Learning
Reproducibility
Primary area: reinforcement learning
TL;DR: IMPALA has reproducibility issues; we propose a more reproducible architecture called Cleanba; our Atari-57 experiments shows Cleanba PPO and IMPALA variants outperform torchbeast and moolib's IMPALA.

Abstract: Distributed Deep Reinforcement Learning (DRL) aims to leverage more computational resources to train autonomous agents with less training time. Despite recent progress in the field, reproducibility issues have not been sufficiently explored. This paper first shows that the typical actor-learner framework can have reproducibility issues even if hyperparameters are controlled. We then introduce Cleanba, a new open-source platform for distributed DRL that proposes a highly reproducible architecture. Cleanba implements highly optimized distributed variants of PPO and IMPALA. Our Atari experiments show that these variants can obtain equivalent or higher scores than strong IMPALA baselines in moolib and torchbeast and PPO baseline in CleanRL. However, Cleanba variants present 1) shorter training time and 2) more reproducible learning curves in different hardware settings.

Published as a conference paper at ICLR 2024
CLEANBA: A REPRODUCIBLE AND EFFICIENT DIS- TRIBUTED REINFORCEMENT LEARNING PLATFORM
Shengyi Huang
Jiayi Weng"
Rujikrrn Charakorn'
Min Lin
Zhongwen Xu
Santiago Ontañón
Drexel University
Hugging Face
Gooole
FVISTEC
Sea Al Lab
Tencent Al Lab
costa. hwa.ggouttookk com
ABSTRACT
Distributed Deep Reinforcement Learning (DRL) aims to train autonomous agents in less wall-clock time by leveraging more computational resources. Despite re- cent progress in the field, reproducibility issues have not been sufficiently ex- plored. This paper first shows that the typical actor-learner framework can have reproducibility issues even if hyperparameters are controlled. We then introduce Cleanba, a new open-source platform for distributed DRL that proposes a highly reproducible architecture. Cleanba implements highly optimized distributed vari- ants of PPO (Schulman et al., 2017) and IMPALA (Espeholt et al. 2018). Our Atari experiments show that these variants can obtain equivalent higher scores than strong IMPALA baselines in moolib ib and torchbeast and PPO base- line in CL However, Cleanba variants present shorter training time and 2) more reproducible learning curves in different hardware settings. Cleanba's s source code is available htttts://ggiii httpg//ttu....o//w..ocoee www/wwxyz htwp://////caaaaaae
INTRODUCTION
Deep Reinforcement Learning (DRL) is technique to train autonomous agents perform tasks. In recent years, it has demonstrated remarkable success across various domains, including video games (Mnih et al. 2015), robotics control (Schulman et al. 2017). chip design (Mirhoseınt al. 2021). and large language model tuning (Ouyang et al. 2022). Distributed DRL. (Espeholt et al. 2018 2020) has also become fast-growing paradigm that trains agents in less wall-clock time by leveraging more computing resources. Despite recent progress, reproducibility issues in distributed DRL have not been sufficiently explored. This paper introduces Cleanba, new platform for dis- tributed DRL. that addresses reproducibility issues under different hardware settings.
Reproducibility DRL is challenging issue. Not only are DRL algorithms brittle to hyperparam- eters and neural network architectures (Henderson et al. 2018), implementation details are often crucial for successfully applying DRL but frequently omitted from publications (Engstrom et al.) 2020 Andrychowicz et al. 202T Huang et al. 2022a). Reproducibility issues in distributed DRT. are under-studied and arguably even more challenging. In particular, most high-profile distributed DRL works, such as Apex-DQN (Horgan et al, 2018). IMPALA (Espeholt et al. 2018), R2D2 (Kap- turowski et al. 2019), and Podracer Sebulba Hessel et al. 2021) are not (fully) open-source. Fur- thermore, earlier work pointed out that more actor threads not only improve training speed but cause reproducibility issues different hardware settings could impact the data efficiency in non-linear fashion (Mnih al. 2016).
In this paper, we present a more principled approach to distributed DRL, in which different hardware settings could make training speed slower faster but do not impact data efficiency, thus making scaling results more reproducible and predictable. We first analyze the typical actor-learner architec- ture in IMPALA (Espeholt et al. 2018) and show that its parallelism paradigm could introduce repro- ducibility issues due to the concurrent scheduling of different actor threads. We then propose more reproducible distributed architecture by better aligning the parallelized actor and learner's 's compu-
CCurrently OpenAT.

---

Published as a conference paper at ICLR 2024
tations. Based on this architecture, we introduce Cleanba (meaning CleanRL-style (Huang et al. 2022b) Podracer Sebulba) distributed DRL platform, which aims to be an casy-to-understand distributed DRL infrastructure like CleanRL. but also be scalable as Podracer Sebulba. Cleanba im-
plements a distributed variant of PPO (Schulman et al. 2017) and IMPALA (Espeholt et al. 2018)
with JAX (Bradbury al, 2018) and EnvPool (Weng al. 2022). Next, evaluate Cleanba's
variants against strong IMPALA baselines in moolib (Mella et al. 2022) and torchbeast (Küttler
al. 2019) and PPO baseline in CleanRL (Huang 2022b on 57 Atari games (Bellemare al.
2013
Here are the key results of Cleanba:
1. Strong performance: Cleanba's IMPALA and PPO achieve about 165% median human normalized score (HNS) in Atari with sticky actions, matching monobeast IMPALA's 165% median HNS and outperforming moolib IMPALA's 140% median HNS.
2. Short training time: Under the GPU 10 CPU setting, Cleanba's IMPALA is 6.8x faster than monobeast's IMPALA and 1.2x faster than mooliß li'ss IMPALA. Under a max specification setting, Cleanba's IMPALA (8 GPU and 40 CPU) is 5x faster than monobeast ' IMPALA (1 GPU and 80 CPU) and 2x faster than mool ib's IMPALA (8 GPU and 80 CPU).
3. Highly reproducible: Cleanba shows predictable and reproducible learning curves across 1 and 8 GPU settings given the same of hyperparameters, whereas moo 1 lib's learning curves can be considerably different, even if hyperparameters are controlled to be the same.
4. Highly scalable: Cleanba can linearly scale to multi-node settings, allowing the researchers to leverage hundreds of GPUs (Appendix
To facilitate more transparency and reproducibility, we have made available source code at
2 BACKGROUND
Distributed DRL Systems Utilizing more computational power has been an attractive topic for re- searchers. Earlier DRL methods like DQN (Mnih et al., 2015) were synchronous and typically used single simulation environment, which made them slow and inefficient in using hardware resources. A3C (Mnih et al. 2016) spawns multiple actor threads; each interacts with its own copy of the en- vironment and asynchronously accumulates gradient. To make distributed DRL more scalable, IM- PALA decouples the actors and the learners (Espeholt et al. 2018 2020). The actors produce train- ing data asynchronously, while the learners produce new agent parameters, which are transferred asynchronously to the actor. Actor-learner systems can achieve higher throughput and shorter train- ing wall time than A3C. Additional distributed actor-learner systems include GA3C (Babaeizadeh al.
2017). IMPALA (Espeholt et al. 2018), Apex-DQM (Horgan et al. 2018). R2D2 Kapturowski
et al.
2019). and Podracer Sebulba (Hessel et al. 2021
Reproducibility Issues with Different Hardware Settings Empirical evidence suggests that in- creasing the number of actor threads can enhance the training speed distributed DRL (Mnih et al. (006 Fig. 4)). However, this augmentation is not without its complications. It also impacts data efficiency and final Atari scores (Mnih et al. (2016) Fig. 3)), and these effects could manifest in non-linear manner. While the authors found the side effects of value-based asynchronous methods to be positive and improve data efficiency, the side effects of contemporary distributed DRL systems, such as IMPALA, Apex-DON, and R2D2, across various hardware configurations, have not been sufficiently explored.
Open-source Distributed DRL Infrastructure While many distributed DRL algorithms are not open-source, there have been many notable distributed DRL replications in the open-source soft- ware (OSS) community. These efforts include SEED RL (Espeholt et al. 2020), rlplyt (Stooke & Abbeel 2018), Decentralized Distributed PPO Wijmans et al.) 2020), Sample Factory (Petrenko et al. 2020), HTS-RL (Liu et al. 2020). torchbeast (Küttler et al. 2019) and moolib (Mella al. 20220. Many of them have shown high throughput and good empirical performance in select domains. Nevertheless, most of them either do not have evaluations on 57 Atari games or have various hardware restrictions, leading to reproducibility concerns. mool i b is the only OSS infras-
2

---

Published as a conference paper at ICLR 2024
IMPALA Actor
batch_sias size 32
batch_ size - 32
agent Agent ()
agent Agent ()
rollout_ ()
rollout_ queue (max_ (max_size-l) peram
(max size=1)
def actor
def actor ()
while rollout (agent pargnn,
for range (1, ITER):
if !- 2:
parama param Q get ()
data rollout (params, batch_ aize)
10
rollout_Q.:s put (data)
rol vollout_Q.pa (data)
def Learner ():
def learner
12
for in (1, ITER):
for in range (1, ITER):
data rollout_ g.get many (batch_ size)
dasa =
learn (data)
agent. leatn (cata)
broadcast_ ter______________________eee_ to__ctorrsssscoooooootesseeeeeet (agent param)
param_ O 0-put. (agent. param)
for in range (nun_actors):
param_0 pup (agent.param)
threadlactoz) start
thread (actor) .starrt
chread (learner) .start
thread (learner) attart
Figure The pseudocode for IMPALA architecture (left) and Cleanba's architecture (right). Colors are used to highlight the code differences between the two architectures. The rollout (params, num_envs) function collects rollout data on num_envs independent envi- ronments for num steps steps.
tructure that has both evaluations on 57 Atari games in the standard 200M frames setting and can scale beyond single GPU setting
3 REPRODUCIBILITY ISSUES IN IMPALA
This section shows that IMPALA Espeholt et al. 2018) has non-determinism by nature, which arises from the concurrent scheduling of different actor threads. This don-determinimm could further cause subtle reproducibility issues.
what happens when the learner produces new policy while the actor is in the middle of producing a trajectory? It turns out multiple policy versions could contribute to the actor's rollout data in line of the IMPALA architecture Figure Typically, the faster the policy updates, the more frequently the policies are transferred. However, this impacts the rollout data construction non-trivial way. From reproducibility point of view, it important realize the frequency at which the policies are updated is source of non-determinism.
A natural question arises:
However, non-determinism can be desirable parallel programming because they make programs faster without making outputs significantly different. For example, some of NVIDIA's CuDNN operations are inherently non-determinisitiq What more important is to investigate if this non- determinism could cause reproducibility issues in terms of learning curves. To this end, we manu- facture a specific experiment that magnifies this non-determinism monobeast's IMPALA. For the control group, we
1. decreased the number of trajectories in the batch from 32 reduce training time, thus making the actor's s policy updates more frequent;
2. used 80 actor threads and increased monobeast's 's default unroll length from 20 to 240 to increase the chance of observing the actor's policy updates the middle trajectory.
While SEED RT. also has evaluations 57 Atari games and scale beyond GPU, SEED RL trained the agents for billion frames hours per game.
https: ptttps://doca /ttp///ccsscco//ooo/oooo nwi diai com http:///ww./.....ooo///o/////./.
de/ ndex. haml#
reprocucibility

---

Published as a conference paper at ICLR 2024
Monobeast IMPALA
Monobeast IMPALA (Learner Delayed 1s)
Pong-v5
Pong-v5
Episodic Retur
Value Function Loss
-20
0M
2M
4M
6M
8M
10M 0M 2M
4M
6M
8M
10M
Step
Step
Figure 2: IMPALA's reproducibility issue under different "speedd" settings The y-axes show the episodic return and value function loss of two sets of monobeast experiments that use the exact same hyperparameters, but the orange of experiments has its learner update manually delayed for second to simulate slower learner updates. Note the learning curves across 10 random seeds are non-trivially different, implicating hyperparameters in IMPALA alone cannot always ensure good reproducibility.
For the experimental group, we used the above setting but manually slowed down the policy broad- casting by sleeping the learner for second after the policy updates in order simulate case where the learner is significantly slower (such as when running the learner on CPU).
We found that in the control group, the actors, average, changed their policy versions 12-13 times in the middle of the 240-length trajectory. In the experimental group, because of the manual slowdown broadcasting the learner's policy, the actors, average. changed the policy one time. We note that the results vary on different hardware settings as well. For example, the control group changed their policy versions, on average, eight times when using 40 actor threads. We noted that in moolib, ib, the actor's policy could also change mid-rollout. See Appendix G
Figure demonstrates the empirical effect the experiments. Note that the learning and loss curves looked notably different across ten random seeds, even though the control and experimental group have the exact same hyperparameters. This experiment shows that IMPALA algorithmically could be susceptible reproducibility issues across different hardware settings. While Figure 2 only shows the experimental results on one environment,
primary purpose of to show that this issue exists and is barely predictable. Furthermore, this type of issue be much more subtle and difficult to diagnose at a much larger scale, so it is important that we investigate them.
4 TOWARDS REPRODUCIBLE DISTRIBUTED DRL
Despite these reproducibility issues, the actor-learner architecture is useful because it allows to parallelize the computations of the actors and learners. this work, we address the reproducibility issues mentioned above by decoupling hyperparameters and hardware settings and 2) proposing a synchronization mechanism that makes distributed DRL reproducible.
4.1 DECOUPLING HYPERPARAMETEE AND HARDWARE SETTINGS
As mentioned in the previous section, different numbers of actor threads could make policy updates more or less frequent in the middle of trajectory generation. This is unpredictable and need be the case. A different number of actors also creates different number of simulation environments and thus should be recognized as hyperparameter setting.
To make more clarified setting, advocate decoupling the number of actor threads into two sep- arate typerparameters: 1) the number of environments, and 2) the number CPUs. In this case, use different number of CPUs to simulate given number of environments. This decou- pled interface is readily provided by EnyPool (Weng et al. 2022), which we use in our proposed architecture.

---

TH+1- Figure i is the pseudocode of the architecture and Table Iillustrates how policies get updated. Under the Synchronous Architecture, the actor and learner's computations are sequential: first perform rollout 1 + Dr,, during which the learner stays idle. Given the rollout data, the learner during which the actor stays idle. More generally, the Published as a conference paper at ICLR 2024
Table I: The Synchronous and Cleanba's architecture. Under the Synchronous architecture, the actor and learner's 's computations are sequential and not parallclizable the learner always learns from the rollout data of the latest policy R,
T++1 (c.g..
7T2
53 ) Under Cleanba's architecture,
Dea
D.2
can parallelize the actor and learner's computation at the cost of introducing stale data starting from iteration the learner always learns from the rollout data obtained from the second latest policy T
1++1
(e.g.,
722
R3
Pr;
D
Iteration
3
Synchronous Arch.
R1 + Dey
W1
22
72 + Day
W2
W3
T3 +
Day
Ta
744
De:
Dr2
Dry
Cleanba's 's Arch., Actor Cleanba's Arch., Learner
T Dan
Dx1
HT +
D
D+1
1>11 1
4.2 DETERMINISTIC ROLLOUT DATA COMPOSITION
To address the non-determinism in rollout data composition, we propose our Cleanba' 's architecture, which retains the benefit of parallelizing actor-learner computations but can produce deterministic rollout data composition. At its core, Cleanba's 's architecture is simple mechanism for synchroniz- ing the actor and learner, ensuring the learner performs gradient updates with rollout data of second latest policy.
Let us use the notation n. + Dx; to denote that policy of version is used to obtain rollout data Ti-1 denotes policy of version i is trained with rollout data Dn, obtain new policy DR, Ti D.,
then performs gradient updates D-,
learner always learns from the rollout data of the latest policy क.
D..
To parallelize actor and learner's computation, Cleanba's architecture needs to necessarily intro- duce stale data like IMPALA (Espeholt et al. 2018). In the second iteration of Cleanba's archi- tecture in Figure we skip the param_Q. get call, so T + Dx, happens concurrently with Because Queue. get is blocking when the queue is empty and Queue put is
2TI
R2
Dr.
blocking when the queue full (we set the maximum size be 1), we make sure the actor pro- cess does not perform more rollouts and learner process does not perform more gradient updates. Starting iteration
the learner then learns from the rollout data of the second latest policy Ti 1s+1. As result, Cleanba's 's architecture can parallelize the actor and learner's 's computa- D
tion at the cost of stale data.
Cleanba's architecture above has several benefits. First, easy to reason and reproduce. As highlighted in Table [ we can ascertain the specific policy used for collecting the rollout data, so if we had delayed learner updates like in Section 3 for iteration i, iteration i + 1 would not start until the previous iteration is finished, therefore circumventing IMPALA's reproducibility is- sue. This knowledge about which policy generates the rollout data enhances the transparency and reproducibility of distributed RL and can help us scale up while maintaining good reproducibility principles. Second, Cleanba's 's architecture easy to debug for throughput. For diagnosing through- put, we can evaluate the time taken for roll rollout_O. get () and param_O. get (). If, average, rollout O. get () consumes less time than param Q get () it becomes evident that learning the bottleneck, and vice versa.
5

---

Published as a conference paper at ICLR 2024
Monobeast IMPALA, 1 A100, CPU
Cleanba IMPALA, 1 A100, CPU
Moolib IMPALA, A100, 10 CPU
Cleanba PPO (Sync), 1 A100, 10 CPU
Median
0M
10M
20M
30M
40M
50M
0
200
400
600
800
Steps
(m)
Median
JQM
Mean
Optimallty Gap
Monobeast IMPALA, A100, CPU- IMPALA,
Cleanba IMPALA, A100, 10 GPU
Cleanba PPO (Sync). A100, 10 CPU
1.2
144
1.6
1.8
1,50 1.75
2.00 Normalized Score
10
12
14
0.25 0.27 0.30 0.32
Figure 3: Base experiments. Top figure: the median human-normalized scores of Cleanba variants compared with mool and sonobeast. Bottom figure: the aggregate human normalized score metrics with 95% stratified bootstrap Cls. Higher is better for Median, IQM, and Mean; lower is better for Optimality Gap.
Monobeast IMPALA, 1 A100, 80 CPU
Cleanba IMPALA, 8 A100, 46 CPU Cleanba PPO (Sync), 8 A100, 46 CPU
Moolib IMPALA, 8 A100, 80 CPU
Median
0M
10M
20M
30M
40M
50M
0
20
40
60 Time (m)
80
100
Steps
Median
10M
Mean
Optimality Gap
Monobeast
Moalib MPALA, 8 A100, 80 CPU
Cleanba MPALA, 8 A100, 46 CPU
(Sync),
Normalized Score
Figure 4: Workstation experiments. Top figure: the median Human--ormaille scores of Cleanba variants compared with mool ib. Bottom figure: the aggregate human normalized score metrics with 95% stratified bootstrap Cls.
Based on Cleanba's architecture, this work introduces Cleanba as a reproducible distributed DRL platform. Cleanba is inspired by CleanRI (Huang et al. 2022b) and DeepMind's Sebulba Po- dracer architecture (Hessel et al. 202Ip. Its implementation uses JAX (Bradbury et al. 2018) and EnvPool (Weng al. 2022). both of which are designed to be efficient. To improve the learner's throughput, we allow the use of multiple learner devices via pmap. To improve the system's scalability, we enable running multiple processes on single node or multiple nodes via jax crissributee
EXPERIMENTS
We perform experiments on Atari games (Bellemare et al. 2013). All experiments used 84 x 84 images with greyscale, action repeat of 4, 4 stacked frames, and maximum of 108,000 frames per episode. We followed the recommended Atari evaluation protocol by Machado et al. (2018), which used sticky action with probability of 25%, loss of life signal, and the full action space.

---

Published as a conference paper at ICLR 2024
To make more direct and fair comparison, used the same AWS p4d.24x instances and the same Atari environment simulation setups via EnvPool and compared only the following codebase settings:
1. Monobeast IMPALA: the reference IMPALA implementations in monobeast
2. Moolib IMPALA: the reference IMPALA implementations in Moolib;
3. CleanRL PPO (Sync): the reference PPO implementations
BeanRL(Huang et al.
2022bb;;
4. Cleanba PPO and Cleanba IMPALA: our PPO and IMPALA implementation under the Cleanba Architecture;
5. Cleanba PPO (Syne) and Cleanba IMPALA (Syne) our PPO and IMPALA implementa- tion under the Synchronous Architecture (Table which can be configured by comment- ing out line 7 of the Clcanba's architecture in Figure
Within the 24xlaree instance, also compared two hardware settings:
1. Base experiments uses 10 CPU and A100 setting as base comparison;
2. Workstation experiments uses 46 CPU and A100s for Cleanba experiments, 80 CPU and 8 A100s for moo 1 ib experiments ments.
5 and 80 CPU and 1 A100 for monobeast experi-
Throughout all experiments, the agents used IMPALA's Resnet architecture (Espeholt al. 2018), ran for 200M frames with three random seeds. The hyperparameters and the learning curves can be found in Appendix B We evaluate the experiment results based on median HNS learning curves, interquartile mean (IQM) learning curves, and 95% stratified bootstrap confidence intervals for the mean, median, IQM, and optimality gap (the amount by which the algorithm fails to meet minimum normalized score of I) (Agarwal et al. 2021 To examine scalability in multi-node settings, conduct experiments examining scalability on 16, 32, 64, and 128 A100s (Appendix
5.1 COMPARISON WITH MCOLIB AND MONOBEAST'S IMPALA
Under the base experiments (Figure B Cleanba's 's IMPALA obtains similar level of median HNS as monobeast'ss IMPALA and a higher level of median HNS as moolib's IMPALA. However, Cleanba's IMPALA is 6,8x faster than monobeast?? IMPALA, mostly because Cleanba actors run on GPUs, whereas menobeast's actors run on CPUs. Also, Cleanba's IMPALA 1.2x faster than moolib's IMPALA, but the speedup difference is challenging to explain due to multi- ple confounding factors - Cleanba's variants benefit from JAX's just-in-time compilation, whereas moolis ib benefits from asynchronous operations (e.g., gradient computation and environment steps). Cleanba's PPO (Sync) also obtains a high median HNS but takes longer training time, likely due to the longer training step time spent reusing rollout data times.
Under the workstation experiments (Figure 4. Cleanba's s PPO (Sync) and IMPALA obtain a similar level of median HNS as monobeast.3 's IMPALA and a higher level of median HNS mool ib's IMPALA. However, Cleanba's PPO (Sync) and IMPALA are both faster than monobeast's and mcolib IMPALA. Most prominently, Cleanba's IMPALA 5x faster than monobeast's IM- PALA and 2x faster than moolib's IMPALA.
Additionally, we examine the individual learning curves in Figure 5 and found that Cleanba's variants also produce more consistent learning curves. In comparison, in two hardware settings, moolib's learning curves can be much more unpredictable.
For some experiments, we used p4de.24 24xlare instances but only GPU memory different, which does not affect training speed.
'We wanted test out IMPALA's official
code released in aag.em/inn//caaaaaeeeeeeent but built with tensorf ow which does support the A100 GPU tested in this paper.
'We used more CPUs for moolis ib experiments because 10 CPU per GPU seems to be the default scaling
parameter for mool ib. Also, for the meol ib experiment, we conducted two sets of random seeds. We reported the results with higher IQM and lower median. See Appendix

---

Published as a conference paper at ICLR 2024
Moolia (Resnet CNN) A100, 10 CPU
Cleanba IMPALA, 8 A100, CPU (Sync),
(Resnet CNN)
Cleanba IMPALA, A100, 10 CPU
Cleanba PPO (Sync), A100, 46 CPU
Assault-v5
Asterix-v5
Breakout-v5
Boxing-v5
100
500000
400
2500
250000
200
50
DM
20M
40M
DM
20M
40M
DM
20M
40M
0M
20M
40M
ChopperCommand-vs
DemonAttack.v5
Gravitar-v5
Kangaroo-v5
Episodic Return
20000
4000
100000
10000
10000
50000
2000
0M
20M
40M
0M
20M
40M
0M
20M
40M
0M
20M
40M
KungFuMaster v5
NameThisGame v5
TimePilot vs
DpwDown v5
40000
50000
10000
20000
200000
DM
20M
40M
0M
20M
40M
0M
20M
40M
0M
20M
40M
Steps
Median
IQM
Muan
Optimality Gap
Maalib (Resnet CNN) A100, 10 CPU-
Moalib (Resnet CNN) 8 A100, 80 CPU-
Cleanba IMPALA, A100, 10 CPU-
Cleanba IMPALA, A100,
CPU-
Cleanba PPO (Sync), 1 A100, 10 CPU
Cleanba PPO (Sync), A100, 46 CPU
1.2
14
1.6
1.8
1,50 1.75 2.00 Normalized Score
12 0.24 0.27 0.30 0.33
Figure 5: Reproducible learning curves the Cleanba variants show predictable learning curves in different hardware settings. In comparison, mcol lib's IMPALA's learning curves under the 1 A100, 10 CPU setting (blue curve) and A100. 80 CPU setting (orange curve) are mean- ingfully different, even if they use the same hyperparameters.
5.2 DISCUSSION ABOUT MONOBEAST'S IMPALA
Note that the monobeast experiments are interesting in several ways. First, it produces higher median HNS than mool ib's IMPALA, which is the opposite of what was shown in Mella et al.
(002)) This is probably because Mella et al. (2022) used "comparable environment settings" instead of the same environment settings used in our experiments. Interestingly, we found different Atari wrapper implementations
have non-trivial impact the agent's performance (Appendix D for this reason, we use the same Atari wrapper implementation in the experiments presented this section. Second, the monobeast experiments appear robust in two different hardware settings in practice, despite the reproducibility issues we showed in Section 3 While mon obtained high scores, it is significantly slower in the A100 and 10 CPU settings due poor GPU utiliza- tion. Its codebase also does not support multi-GPU settings and should scale less efficiently with larger networks because actor threads only run on CPUs when compared to moolin variants.
moolib lib and Cleanba's
5.3 SYNCHRONOUS ARCHITECTURE VS CLEANBA ARCHITECTURE
Figure compares the PPO and IMPALA variants between Synchronous and Cleanba architecture and CleanRL's PPO, which the Synchronous architecture by design. We found using Cleanba architecture actually hurts Cleanba PPO's data efficiency. This is an interesting trade-off because the speed benefit of parallelizing actor and learner processes in Cleanba PPO is offset by the lower
8

---

Published as a conference paper at ICLR 2024
CleanRL PPO (Sync), A100, 10 CPU
Cleanba MPALA (Sync), 8 A100, CPU
Cleanba PPO (Sync), A100, 46 CPU
Cleanba MPALA, A100, 46 CPU
Cleanba PPO, A100, 46 CPU
Median
0M 10M
20M
30M
40M
0
50
100
150
200
Steps
Time (m)
Median
IQM
Mean
Optimality Gap
CleanRL PPO (Sync), A100, CPU PPO (Sync), A100, Cleanba PPD, A100, CPU
Cleanba IMPALA (Sync), A100, 46 CPU Cleanba IMPALA,
CPU
1.35 1.50 1.65 1.80
1.80 1.95
2.10
10.4 11.2 12.0
0.24 0.26 0.28
Figure 6: Comparing Cleanba's variants using Cleanba and Synchronous architecture. For PPO, Cleanba's Architecture (orange curve) runs faster but has lower data efficiency than Synchronous architecture (blue curve) For IMPALA, there is no discernible difference between
Synchronous Architecture (red curve) and Cleanba's Architecture (brown curve) This means Cleanba's IMPALA can henefit from the speed-up of parallelizing actor-learner computation with- out paying price for data efficiency under our hyperparameter settings, unlike Cleanba's PPO.
data efficiency. Among many possible causes, the main factor might be that PPO does 16 gradient updates (4 mini-batches and 4 update epochs) per rollout, whereas IMPALA in setting only does 4 gradient updates. In comparison, we noticed Cleanba's IMPALA did not suffer from lower data efficiency compared to Cleanba IMPALA (Sync) architecture, meaning IMPALA can actually benefit from parallelizing actor and learner computations.
LIMITATION
There are several limitations to this work. First, our experiments could not completely control var- ious other confounding settings the reference codebase, such optimizer settings and machine learning framework (c.g., PyTorch, JAX). For example, Cleanba's 's PPO and IMPALA use different learning rates indicated their respective literature, making it difficult to compare PPO and IM- PALA directly. We attempted to make a direct comparison by running Cleanba PPO with Cleanba IMPALA's setting and found made PPO's s data efficiency significantly worse this could sug- gest the IMPALA'S setting is well-tuned for IMPALA but brittle to PPO (Appendix [... Second, our finding that parallelizing actor and learner computation hurts PPO's data efficiency is specific to the PPO's default Atari hyperparameter setting, and it could perhaps be tuned in ways in which opposite findings can be drawn. That said, the main purpose of this work is not hyperparameter tuning. Rather, it is creating codebase that replicates prior results and makes training reproducible, efficient, and scalable across more powerful hardware.
7 CONCLUSION
This paper presents Cleanba, a new distributed deep reinforcement learning platform. Our analy- sis shows that Cleanba's more principled architecture can circumvent reproducibility issues in IM- PALA's architecture. Our Atari experiments demonstrate that Cleanba's PPO and IMPALA accu- rately replicate prior work but have faster training time and highly reproducible across different hardware settings. We believe that Cleanba will be valuable platform for the research community conduct future distributed RL research.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank the following entities for their support.
1. Stability AI's HPC for generously providing much GPU computational resources to this project.
2. Hugging Face's cluster for providing much GPU computational resources to this project.
3. Google's TPU Research Cloud for providing the TPU computational resources.
REPRODUCIBILITY STATEMENT
Ensuring Cleanba's s results are reproducible central theme in our paper. To this end, we have taken several measures to improve reproducibility:
1. Open-source repository: we made source code available at https://gii tiu.b.tt.ub..... com/ ywwyyjnn/ htn/cleabbl The dependencies of the experiments are pinned, and our reposi- tory contains detailed instructions on replicating all Cleanba experiments presented in this paper.
2. Reproducible architecture: as demonstrated in Section 4
Cleanba introduces more prin- cipled approach to understanding distributed DRL and gives clear expectations on where the rollout data comes from, making easier to reason about the reproducibility of dis- tributed DRL.
3. Experiments on different hardware: as demonstrated in Section 5 periments showing Cleanba's PPO and IMPALA variants obtain near-identical data ef- ficiency on different hardware, further demonstrating that this work is highly reproducible.
we also conducted ex-
sum, we have tried to make our work as transparent and reproducible as possible. By leveraging the source code, details provided in the main paper, and appendix, researchers should be well- equipped to reproduce or extend upon our findings.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Informa- fion Processing Systems, 34, 2021.
Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk. Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? large-scale study. In International Conference on Learning Representations, 2021. URL https: opwrrevee.....e
jsn Dzq
Mohammad Babaeizadeh, Iuri Frosio, Stephen Tyree, Jason Clemons, and Jan Kautz. Reinforce- ment learning through asynchronous advantage actor-critic on a GPU, In International Confer- ence on Learning Representations, 2017. URL https:://penrewieew.. net. forum?idd
x E1VGvBcx
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
James Bradbury, Roy Frostig, Peter Hawkins. Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, al. Jax: composable transformations of python+ numpy programs. 2018.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study ppo and trpo. In International Conference on Learning Representations, 2020. URL hetps: //openrevi ew.
fetammmdd=rle. dddr dtN rtPB
10

---

Published as a conference paper at ICLR 2024
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IM- PALA: scalable distributed deep-rl with importance weighted actor-leamer architectures. In Jen- nifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. ICMI. 2018, Stockholmsmüssan Stockholm, Sweden, July 10-15, 2018, vol- ume 80 of Proceedings of Machine Learning Research, pp. 1406-1415, PMLR, 2018. URL attp::/proceedingee mml.. wwees//v0/eeee /ttp:///eeeeeeotttaa..oo html
Lasse Espcholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, and Marcin Michalski. Seed rl: Scalable and efficient deep-rl with accelcrated central inference. In International Confer- ence on Learning Representations, 2020. URL
net. wtt.:e/..teeeoummeemm..
rkgvx1rKwH
Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106.1322 13281, 2021.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Matteo Hessel, Manuel Kroiss, Aidan Clark, lurii Kemaev, John Quan, Thomas Keck, Fabio Viola, and Hado van Hasselt. Podracer architectures for scalable reinforcement learning. arXiv preprint arXiv:2104. 06272, 06272, 2021.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL httpp::/epenreview..nt
Eorum? Brrumidd=HDD--o- 02
Shengyi Huang. Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. ICLR Blog Track, 2022a. URL httts::/iclr- track. black.githbb.. wtt.g/hhh....//////33333333 ppo 1po-implemme wwwppeeeetttiionnoeeoo detaiiss/
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep rein- forcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022b.
URL / wttpp//..w.ppppp/////////222
1342 .tmll
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent ex- perience replay in distributed reinforcement learning. In International conference on learning representations, 2019.
Heinrich Küttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktäschel, and Edward Grefenstette. Torchbeast: A pytorch platform for distributed rl. arXiv preprint axXiv:1910.03552. 2019.
Iou-Jen Liu, Raymond A. Yeh, and Alexander G. Schwing. High-throughput synchronous deep RL. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Marri-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URI. httphhttsss//ppoeeeeeeeeee www.nnng........eeeee. neur ips. 104499000 Abst html
htt.erepp...pppppee//////..e./e. cccppaper//22//haat/
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of larce-batch training. arXiv preprint arXiv: 1612.0666.. 2018

---

Published as a conference paper at ICLR 2024
Vegard Mella, Eric Hambro, Danielle Rothermel, and Heinrich Küttler. moolib: A Platform for Distributed RL. 2022.
ctm/ fac.//bbbooeeesrrr/oooooo.
Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Bric Johnson, Omkar Pathak, Azade Nazi, et al. A graph placement methodol- ogy for fast chip design. Nature, 594(7862):007-212 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518((540):529-55 533- 2015.
Volodymyr Mnih, Adrià Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.). Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19- 24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1928-1933 JMLR.org. 2016. URL http: http:::/ppocceeddee
mlr press / v48 1//48//mn/
6.mml
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint urXiv:2203.0m 20..2222335555 2022.
Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav S. Sukhatme, and Vladlen Koltun. Sam- ple factory: Egocentric 3d control from pixels at 100000 FPS with asynchronous reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Re- search, pp. 7652-7666 7662. PMLR, 2020. URL http: http::/procedingg.. mll press/vi19/ petrenkozoa. html
Martin L Puterman. Markov decision processes: discrete stochustic dynamic programming. John Wiley & Sons, 2014.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust re- gion policy optimization. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, vol- ume 37 of JMLR Workshop and Conference Proceedings, pp. 1889-1897, JMLR.org. 2015. URL schu I man 15 5
http: http:://proceeiieee ngs mm wwwpsrees//vv. v/ /tc///u///uuuucc
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. 2016. URL http:
7/arxiv. abs /0t.//51.... 02438
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017. URL /ttps:://axxi..or/
optimization algorithms. ArXiv preprint,
abs/1707. 06347
Adam Stooke and Pieter Abbeel. Accelerated methods for deep reinforcement learning. arXiv preprint 1rXiv:/803.0281/ 2018.
Jiyy Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, Zhongwen Xu, and Shuicheng YAN. Envpool: A highly parallel reinforcement learning environment execution engine. In siit Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL //oenrevii ew net. forum? wtwoooowwmmmuuurroooroe Hpuuo
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo Learning near cear-perfect pointgoal navigators from 2.5 billion frames. In International Conference on Learning Representations, 2020. URL https: / opw.revie...... net / wow.oommmwiollc00042
12


Reviews:
[{"review_id": "53hhzQ90ce", "reviewer": "Reviewer_fstX", "summary": "The authors introduce CleanBa, a distributed RL library containing distributed implementations of IMPALA and PPO. They demonstrate that IMPALA's actor-learner architecture causes issues with reproducibility \nbecause the learner can do a different number of updates depending on its speed. \n\nTo solve this, they then propose a new architecture, where the policy updates based on the rollouts of the second latest policy. This ensures reproducibility at the cost of more synchronisation and the introduction of stale data.\n\nThey demonstrate that their framework performs well compared to relevant baselines and runs much more quickly.", "questions": "* How does the extra param queue impact the distributed performance (i.e. speed)? i.e. how much cost is paid in speed for fixing the reproducibility issues?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well written and clearly explained.\n* The reproducibility problems identified in IMPALA are interesting and well addressed by their new framework.\n* The empirical results demonstrate the framework is faster than prior methods on comparable hardware.", "weaknesses": "* The authors claim to have built a distributed learning framework, but have only evaluated their method on a single machine. This is a major flaw in their evaluations, and the reason I recommend rejection for this paper. If this is included I will increase my score. \n\n\nTypos: \nAt the end of section 4.2 you write `jax.distibuted` -- I think this should be `jax.distributed` (extra r).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6aGIlYjizE", "reviewer": "Reviewer_Z35M", "summary": "This paper introduces Cleanba, a new reproducible distributed RL training platform. It verifies the reproducibility on PPO and IMPALA in different hardware settings.", "questions": "Please address the above weaknesses.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "It implements a new reproducible and fast distributed RL training platform.", "weaknesses": "1) The paper does not touch the real problems that cause the reproducibility issues. It not only occurs in distributed RL training but also occurs in the single machine training even using the same hyperparameters. Reverb and [1] have reported that replay ratio (the number of gradient updates per environment transition) greatly affects the performance. [1] Revisiting Fundamentals of Experience Replay\n2) The Cleanba shown in Figure 1 does not match the “learner always learns from the rollout data obtained from the second latest policy” (line 7).\n3) “ensuring the learner performs gradient updates with rollout data of second latest policy” assumption is too strong. It may slow down the IMPALA training.\n4) It would be beneficial to provide more reproducibility results in relation to different RL algorithms such as Ape-X, SeedRL, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yviA01iky9", "reviewer": "Reviewer_Ui8X", "summary": "This paper presents cleanba, a jax based software framework for reproducible distributed deep reinforcement learning (DDRL). The authors present errors that arise from existing non-deterministic DDRL frameworks that could harm reproducibility efforts, then propose a deterministic solution (which comes at the cost of stale data). The authors show the stability and reproducibility of cleanba, and evaluate it on a suite of standard Atari benchmarks.", "questions": "- I can’t evaluate the code, since there is no anonymized access to it, but I feel given that this introduces a package, that code quality is relevant. Is the code documented well and complete with type hints and docstrings and the like?\n- How easy is it to extend the framework? What plans are there for other algorithms? What do the development goals look like from here?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Reproducibility is a notorious issue in all of machine learning, but RL especially, and it is good to see more work addressing it\n- The paper is generally clear and understandable\n- Figure/text colouring is generally pretty useful", "weaknesses": "- The abstract’s first sentences feel a bit choppy\n- There are other Sebulba implementations that might be worth comparing to (e.g. https://github.com/instadeepai/sebulba) \n- Fig 2 could motivate the problem better, given that in the current figure both versions end up at very similar scores. If there was a different environment just highlighting that this 1 second lag has a meaningful detriment to final performance that could be really compelling.\n- Although IMPALA is a common algorithm, devoting a short paragraph to it in the background could be helpful (since although many are familiar with it, not as many are with the intricacies as it relates to distributed computing that get mentioned in the next section)\n- In Figure 5, the y axis could have their tickers removed. The absolute numeric scales are not particularly meaningful (I suspect even the most experienced RL researcher will not know if a score of 200,000 on UpNDown is good) and add a lot of clutter to the figure\n- The majority of Fig 1 is highlighted. Maybe it’s better to just highlight the part that is the same? Or remove the highlighting? It just doesn’t add much when the points you want to draw attention to are basically the whole code block.\n- In figure 3, the optimality gap x tickers are overlapping\n- Moolib is a good point of comparison, but I see that this repo is now archived. Are there other DDRL libraries worth comparing to that are more actively maintained?\n- One concern I have is that this paper tries too hard to be an algorithm paper. It presents cleanba, which is a software package that the authors want people to use, but not a lot of time is dedicated to that. More time is dedicated to the analysis of results from cleanba. While these results are somewhat interesting, I think they could be consolidated more (e.g. move the individual atari games to the appendix, and just present HNS, just an idea). Then use that space to actually describe the package of cleanba more (e.g. how does a user interact with it? Extend it? UML diagram perhaps? Etc.). The paper is called “cleanba” for the package, but I can’t say I really know programmatically much about cleanba after reading the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O5exoBYVwy", "reviewer": "Reviewer_eKtC", "summary": "The authors present, Cleanba, a new platform for distributed DRL that addresses reproducibility issues under different hardware settings. One of the biggest challenges in applying deep reinforcement learning to real world problems is reproducibility. In general, the algorithms are not robust to implementation details which are often crucial for successful real-world applications. The authors introduce a new distributed architecture that is reproducible if random seeds are controlled and leverages interleaving the actor and learner’s computations. The main contribution of the paper is that the authors propose a new mechanism to synchronize the actor and learner which reduces the non-determinism and improves reproducibility. While the synchronization mechanism is simple, the experiments show that Cleanba’s variants can obtain equivalent or higher scores than moolib’s IMPALA, but with 1) less training wall time under the 8 GPU setting and 2) more reproducible learning curves in different hardware settings.", "questions": "At first, I was really confused about the novelty and contribution of the paper. In the abstract and introduction, the paper talks about variants of already existing algorithms but does not talk about the novelty of the proposed variant specifically until past mid-way though the paper. It will be better to talk about the contributions first even when they may be simple yet have far reaching implications to reduce training time and improve reproducibility at scale.\n\nIt is important to consider a real-world example when reproducibility is one of the main challenges. Atari is a great way to benchmark but does not present all the variability such as in robotics applications for example to truly highlight issues related to the reproducibility.\n\nIn Section 5.1, the authors state that Cleanba’s IMPALA is 6.8x faster than monobeast’s IMPALA, mostly because Cleanba actors\nrun on GPUs, whereas monobeast’s actors run on CPUs. I think the performance metrics should be based on the same compute hardware CPU only or GPU only.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality & Significance. The proposed synchronization mechanism is simple yet results in significant improvement in training time and reproducibility. Both reduced training time and reproducibility are among the key challenges in applying deep reinforcement learning to real works problems. Therefore despite the simplicity of the proposed mechanism, I believe that the performance improvement is significant. \n\nQuality & Clarity. The authors describe the current state-of-the-art and its pitfalls clearly using clear code samples. They provide empirical results to further their hypothesis on what contribute to the current reproducibility issues in popular distributed reinforcement learning architectures. The experiments are extensive in comparing the proposed architecture to other benchmarks in the literature.", "weaknesses": "Originality & Significance. The authors propose to mechanisms to overcome the reproducibility issues in distributed reinforcement learning: 1) de-coupling hyper-parameters, 2) a new synchronization mechanism to better align actors and learners. The de-coupling mechanism is not novel and already exists in prior literature. The synchronization mechanism is simple yet novel. The performance of the new approach is tested only in Atari which is not a real-world problem where reproducibility is a key challenge. The paper can greatly benefit from a real-world application.\n\n\nQuality & Clarity. The language is a bit off in several sections, specifically in the first paragraph, which makes it difficult to read at times. There are several grammatical errors as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)