IMPACT: Importance Weighted Asynchronous Architectures with Clipped Target Networks
Authors: Michael Luo
Jiahao Yao
Richard Liaw
Eric Liang
Ion Stoica
Venue:  (2020)
Keywords: Reinforcement Learning
Artificial Intelligence
Distributed Computing
Neural Networks

Abstract: The practical usage of reinforcement learning agents is often bottlenecked by the duration of training time. To accelerate training, practitioners often turn to distributed reinforcement learning architectures to parallelize and accelerate the training process. However, modern methods for scalable reinforcement learning (RL) often tradeoff between the throughput of samples that an RL agent can learn from (sample throughput) and the quality of learning from each sample (sample efficiency). In these scalable RL architectures, as one increases sample throughput (i.e. increasing parallelization in IMPALA (Espeholt et al., 2018)), sample efficiency drops significantly. To address this, we propose a new distributed reinforcement learning algorithm, IMPACT. IMPACT extends PPO with three changes: a target network for stabilizing the surrogate objective, a circular buffer, and truncated importance sampling. In discrete action-space environments, we show that IMPACT attains higher reward and, simultaneously, achieves up to 30% decrease in training wall-time than that of IMPALA. For continuous control environments, IMPACT trains faster than existing scalable agents while preserving the sample efficiency of synchronous PPO.

Published as a conference paper at ICLR 2020
IMPACT: IMPORTANCE WEIGHTED ASYNCHRONOUS ARCHITECTURES WITH CLIPPED TARGET NETWORKS
Michael Luo UC Berkeley
Jiahao Yao UC Berkeley jlahaoyao@berrkeeey.ee
michael. www.berkeley..ee..............e. edu
Richard Liaw UC Berkeley
Eric Liang UC Berkeley
Ion Stoica UC Berkeley
ABSTRACT
The practical usage of reinforcement learning agents often bottlenecked by the duration of training time. To accelerate training. practitioners often turn to distributed reinforcement learning architectures to parallelize and accelerate the training process. However, modern methods for scalable reinforcement learning (RL) often tradeoff between the throughput of samples that an RL agent can learn from (sample throughput) and the quality of learning from each sample (sample efficiency). In these scalable RL architectures, as one increases sample throughput (i.e. increasing parallelization in IMPALA (Espeholt al., 201 2018)), sample efficiency drops significantly. To address this. we propose a new distributed reinforcement learning algorithm, IMPACT. IMPACT extends IMPALA with three changes: a target network for stabilizing the surrogate objective, circular buffer, and truncated importance sampling. discrete action-space environments, we show that IMPACT attains higher reward and, simultaneously, achieves up to 30% decrease in training wall-time than that IMPALA. For continuous control environments, IMPACT trains faster than existing scalable agents while preserving the sample efficiency of synchronous PPO.
INTRODUCTION
Proximal Policy Optimization (Schulman et al., 2017) is one of the most sample-efficient on-policy algorithms. However, it relies on a synchronous architecture for collecting experiences, which is closely tied to its trust region optimization objective. Other architectures such as IMPALA can achieve much higher throughputs due to the asynchronous collection of samples from workers. Yet, IMPALA suffers from reduced sample efficiency since cannot safely take multiple SGD steps per batch as PPO can. The new agent, Importance Weighted Asynchronous Architectures with Clipped Target Networks (IMPACT), mitigates this inherent mismatch. Not only the algorithm highly sample efficient, it can learn quickly, training 30 percent faster than IMPALA. At the same time, we propose a novel method to stabilize agents in distributed asynchronous setups and, through our ablation studies, show how the agent can learn in both time and sample efficient manner.
In our paper, we show that the algorithm IMPACT realizes greater gains by striking the balance between high sample throughput and sample efficiency. In experiments, demonstrate in the experiments that IMPACT exceeds state-of-the-ar agents in training time (with same hardware) while maintaining similar sample efficiency with PPO's. The contributions of this paper
follows:
1. We show that when collecting experiences asynchronously, introducing a target network allows for stabilized surrogate objective and multiple SGD steps per batch (Section 3.
2. We show that using a circular buffer for storing asynchronously collected experiences allows for smooth trade-off between real-time performance and sample efficiency (Section 3.2).
3. We show that IMPACT, when evaluated using identical hardware and neural network models, improves both in real-time time and timestep efficiency over both synchronous PPO and IMPALA (Section 4).

---

Published as a conference paper at ICLR 2020
Weights
Worker
Weights
Weights
Worker
Learner
Learner
Learner
Worker
Worker
Worker
Batch
Butter
Target Network
Worker
Worker
Batch
Batch
(a) PPO
(b) IMPALA
(c) IMPACT
Architecture schemes for distributed PPO, IMPALA, and IMPACT. PPO aggregates worker batches into large training batch and the learner performs minibateh SGD. IMPALA workers asynchronously generate data. IMPACT consists batch buffer that takes in worker experience and target's evaluation the experience. The learner samples from the buffer.
Figure I:
BACKGROUND
Reinforcement Learning assumes a Markov Decision Process (MDP) setup defined by the tuple P. 7),) where and A represent the state and action space, 7 E [o, is the discount factor, and p S x A x S + R and Rt S x A + R are the transition dynamics and reward function that models an environment.
Let x(a st) S x A [0, 1] denote a stochastic policy mapping that returns an action distribution given state 9, E S. Rolling out policy π (az]s;) in the environment is equivalent sampling trajectory 7 ~ IP(T). where т == (so, Ag. ..... aT- ST, 0r). We can compactly define state and state-action marginals of the trajectory distribution pr (s{)) and Pr (8t, at) induced by the policy T (aalss).Thhe goal for reinforcement learning aims to maximize the following objective: J(0)
(((e,a))>pp 1E R(s,,))
When 0 parameterizes xa,lss the policy is updated according to the Policy Gradient Theorem (Sutton et al., 2000):
V [() = ())))))))pp()) Vo log (o(a))s)))) (se,
where Ar, (Se; ot) is estimator of the advantage function. The advantage estimator is usually defined the 1-step TD error, ÁT, az) r(st, aq) + +(((s))) (sx)), where Vs::) is an estimation of the value function. Policy gradients, however, suffer from high variance and large stdate-step sizes, oftentimes leading to sudden drops in performance.
2.1 DISTRIBUTED PPO
Per iteration, Proximal Policy Optimization (PPO) optimizes policy ma from target Tew via the following objective function
L( prean
min (r.(0)A,, (r
where r, (0) 50(as (ss) and the clipping hyperparameter. addition, many PPO implementa-
tions use GAB-À as low bias, low variance advantage estimator for Å, (Schulman et al., 201 PPO's surrogate objective contains the importance sampling ratio r0), which can potentially explode if Tee too far from Ta. (Han & Sung, 2017). PPO's surrogate loss mitigates this with the clipping function, which ensures that the agent makes reasonable steps. Alternatively, PPO can also be seen as an adaptive trust region introduced in TRPO (Schulman et al., 2015a).
In Figure la, distributed PPO agents implement a synchronous data-gathenn gatta-gatering scheme. Before data collection, workers updated to Told and aggregate worker batches to training batch Dusin- The learner performs many mini-batch gradient steps on D Once the learner is done, learner weights broadcast all workers, who start sampling again.
2

---

Published as a conference paper at ICLR 2020
2.2 IMPORTANCE WEIGHTED ACTOR-LEARNER ARCHITECTURES
In Figure lb, IMPALA decouples acting and learning by having the learner threads send actions, observations, and values while the master thread computes and applies the gradients from queue of learners experience (Espeholt et al., 2018). This maximizes GPU utilization and allows for increased sample throughput, leading to high training speeds on easier environments such Pong. As the number of learners grows, worker policies begin to diverge from the learner policy, resulting in stale policy gradients. To correct this, the IMPALA paper utilizes V-trace correct the distributional shift:
i-1 I
Us. Vo
Ps (ri41 + VVa (9141) Vø (5:))
where, V is the value network, wo the policy network of the master thread, pay is the policy network of the learner thread, and cj min (a, ma(ay1s;) and p; min IS ratios.
mola.ls)) per la; 4;)
clipped
par (a)la)))))
Algorithm 1 IMPACT
Input: Batch size M, number of workers W, circular buffer size N, replay coefficient K, target update frequency fuugets weight broadcast frequency lrequency learning rates a and 3
1: Randomly initialize network weights (0,w)
2: Initialize target network (8', (f',')) <- (A. ()))
3: Create W workers and duplicate (0, w)) each worker
4: Initialize circular buffer C(N, K)
5: for 1,.., do
6: Obtain batch B of size M traversed & times from C(N, K)
7: If k 0, evaluate B on target f', append target output to B
8: Compute policy and value network gradients
Vono(sylag)
VeJ(0) = M
Σ (1,))
AV-GAE V/LL (Target To)
max Trargor {3) (ay), B BiTworker (s}la,))
1
Ve M
(Vin())) Vv.GAA(S;)) (o)s))
Update policy and value network weights 0 10: If k K, discard batch B from C(N, ((N,K) K)
9:
ar a. VnJ B.
Ift (mod frarget). update target network (0", w') +- w)
12:
If (mod (froquency), broadcast weights to workers
end for
Worker-i
Input: Worker sample batch size $
1: repeat
2:
B
3: fort=1..... S do
4: Store (St, ae, Tt, 81+1) ran by 0g in batch B,
5: end for
6: Send B, to C(N, K)
7: If broadcasted weights exist, set 0, +-
8: until learner finishes
IMPACT ALGORITHM
Like IMPALA, IMPACT separates sampling workers from learner workers. Algorithm I and Figure Ic describe the main training loop and architecture of IMPACT, In the beginning, each worker copies weights from the master network. Then, each worker uses their own policy to collect trajectories

---

Published as a conference paper at ICLR 2020
PPO
Asynchronous PPO
Invariants
wurker
X
Asyne sampling means & works syne with Xemer
Likelihood ratio
worker
werker
xy s//max orge B cemee
Effectiveness
In synchronous PPO, all rollouts are fully on-policy, henee " "udker" is the same as nt "orner"
Since werker may differ per worker, using this ratio results in trust region conflicts across multiple batches.
Since x "urrer is updated after cach batch from the worker, only single SGD step can taken per batch.
The IMPACT objective allows for multiple SGD sleps per asyne batch and has stable trust region.
Figure 2: In asynchronous PPO, there are multiple candidate policies from which the trust region can be defined: (1) Tworker, the policy of the worker process that produced the batch of experiences, (2) Mearerr the current policy of the learner process, and (3) Tearget. the policy of a target network. Introducing the target network allows for both stable trust region and multiple SGD steps per batch of experience collected asynchronously from workers, improving sample efficiency. Since workers can generate experiences asynchronously from their copy of the master policy, this also allows for good real-time efficiency.
and sends the data (s,, a, ) to the circular buffer. Simultancously, workers also asynchronously pull policy weights from the master learner. In the meantime, the target network occasionally synes with the master learner every target iterations. The master learner then repeatedly draws experience from the circular buffer. Each sample weighted by the importance ratio of Twunter as well as clipped
with target network ratio Kwoter,
The target network is used to provide a stable trust region (Figure 2). allowing multiple steps per batch (i.c., like PPO) even in the asynchronous setting (i.e., with the IMPALA architecture). In the next section, we describe the design of this improved objective.
Tuge
3.1
MAXIMAL TARGET- WORKER CLIPPING
PPO gathers experience from previous iteration's policy Tow- and the current policy trains by importance sampling off-policy experience with respect to πy. In the asynchronous setting, worker i's policy, denoted worker. generates experience for the policy network лT. The probability that batch B comes from worker can be parameterized categorical distribution i ~ D(ai, ... On.). include this by adding an extra expectation to the importance-sampler policy gradient objective (IS-PG) (Jie & Abbeel, 2010):
(i)( (anD(a) a
20
A,
workerr;
Since each worker contains different policy, the agent introduces target network for stability (Figure 2). Off-policy agents such as DDPG and DQN update target networks with moving average. For IMPACT, periodically update the target network with the master network. However, training with importance weighted ratio He. can lead to numerical instability, as shown in Figure 3. To
Surget
prevent this, we clip the importance sampling ratio from worker wonker. to target policy, Target
(AIS(0) = D(a)
E (s Tunker
min
Workker,
T
A
Trarget
Tworker;
BinD(a)
E B(s,,a))
A
max TTargets @Twwokker,
where 3
the experiments, we set as a hyperparameter with 2 1 and / S 1.
To see why clipping is necessary, when master network's action distribution changes significantly over few training iterations, worker i's policy, Twarker samples data outside that of target policy, Target. leading to large likelihood ratios, Tanderr The clipping function min( Twonker ,)) pulls back
Turge
Targe
4

---

Published as a conference paper at ICLR 2020
Likelihood Ratios
Target Network Frequency Ablation
P-Tagel P-4torear
1016
Episade Mean Reward
Epis008 Mean Newaro
0003
112
6000
4000
151
2000
2003
Time (9)
4300
10000
2002
4000
Time (6)
8000
3000
12800
(a) Ratio ablation study.
(b) Target update frequency study.
Figure 3: Training curves of the ablation study control benchmarks. In (a), the IMPACT objective outperforms other possible ratio choices for the surrogate loss: R₂
Rs (Targa. In (b), we show target network update frequency robust to range choices. try target network update frequency ftarget equal to the multiple (ranging from 1/16 16) circular buffer and the replay times for each batch in the buffer.
To
Torge
Taukes
N K, product of the size of
large IS ratios to p. Figure 10 in Appendix E provides additional intuition behind the target clipping objective. We show that the target network clipping is lower bound of the IS-PG objective.
For p > 1, the clipped target ratio is larger and serves to augment advantage estimator A.. This incentivizes the agent toward good actions while avoiding bad actions. Thus, higher values of encourages the agent learn faster at the cost of instability.
We use GAE-À A with V-trace (Han Sung, 2019). The V-trace GAE-À modifies the advantage function by adding clipped importance sampling terms to the summation of TD error::
*+n-1
[-11
Av.GAE
Σ (Ay)
IIea
C 6,V,
where min (7, Targe (s,)))
(we use the convention sampled 1-step TD error introduced in V-trace.
IT Cj 1) and 8,V is the importance
Tookeer (0y18))
3.2 CIRCULAR BUFFER
IMPACT uses circular buffer (Figure 4) to emulate the mini-batch SGD used by standard PPO. The circular buffer stores N batches that can traversed at max K times. Upon being traversed K times, batch is discarded and replaced by
worker batch.
For motivation, the circular buffer and the target network are analogous to mini- hini-batching from Told experience in PPO. When target network's 's update frequency N K, the circular buffer is equivalent to distributed PPO's training batch when the learner samples N minibatches for K SGD iterations.
This is in contrast to standard replay buffers, such as in ACER and APE-X, where transitions (St, de, Te, 12 are either uniformly sampled sampled based on priority, and, when the buffer is full, the oldest transitions are discarded (Wang et al., 2016; Horgan et al., 2018).
Figure 4 illustrates an empirical example where tuning K can increase training sample efficiency and decrease training wall-clock time.
EVALUATION
In our evaluation we seek answer the following questions:
1. How does the target-clipping objective affect the performance of the agents compared to prior work? (Section 4.1)
5

---

Published as a conference paper at ICLR 2020
k=3
k=1
PongNo rameskipss Time(s)
Pameestirr Duffer Timesteps
k=1
k=0
N=8
Eoloade Maeodeeeeenn Mean Reward
k=2
K 4
k=2
3
k=3
New Batch
Old Betch kc4
300
500
1300000 300000
200000
100000
Total
Tirsstaps
(a) Circular Buffer.
(b) Wall Clock-time vs. K
(c) Sample Efficiency vs. K
Figure 4: (a): The Circular Buffer in nutshell: N and K correspond buffer size and max times batch can be traversed. Old batches are replaced by worker-generated batches. (b): The performance of IMPACT with different K in terms of time. (c): The performance of IMPACT with different K in
terms of timesteps. IMPACT can achieve greater timestep as well as time efficiency by manipulating K. K outperforms all other settings time and more sample efficient than K 1, 4, 16, 32.
2. How does the IMPACT circular buffer affect sample efficiency and training wall-clock time? (Section 4.2)
3. How does IMPACT compare to PPO and IMPALA baselines in terms of sample and real-time performance? (Section 4.3)
4. How does IMPACT scale with respect to the number of workers? (Section 4.4)
4.1 TARGET CLIPPING PERFORMANCE
We investigate the performance of the clipped-target objective relative to prior work, which includes PPO and IS-PG based objectives. Specifically, we consider the following ratios below:
R
To
R2
To
Rs
Wo
Taryst
Twarkery
max (Targe Tareeer
For all three experiments, we truncate all three ratios with PPO's clipping function: R( clip( R, 1 1-c, 1 +)) and train in an asynchronous setting. Figure 4(a) reveals two important takeaways: first, Ri suffers from sudden drops in performance midway through training. Next, Rg trains stably but does not achieve good performance.
We theorize that Rí fails due to the target and worker network mismatch. During periods of training where the master learner undergoes drastic changes, worker action outputs vastly differ from the learner outputs, resulting in small action probabilities. This creates large ratios in training and destabilizes training. We hypothesize that R2 fails due to different workers pushing and pulling the learner in multiple directions. The learner moves forward with the most recent worker's suggestions without developing a proper trust region, resulting in many worker's suggestions conflicting with each other.
The loss function, R& shows that clipping is necessary and can help facilitate training. By clipping the target-worker ratio, we make sure that the ratio does not explode and destabilize training. Furthermore, we prevent workers from making mutually destructive suggestions by having target network provide singular guidance.
4.1.1 TARGET NETWORK UPDATE FREQUENCY
In Section 3.2, analogy was drawn between PPO's mini- hini-batching mechanism and the circular buffer. Our primary benchmark for target update frequency is n = N K, where N circular buffer size and K is maximum replay coefficient. This is the case when PPO is equivalent to IMPACT.
In Figure 4(b), we test the frequency of updates with varying orders of magnitudes of n. In general, find that agent performance is robust to vastly differing frequencies. However, when 1 ~

---

Published as a conference paper at ICLR 2020
Hopper-v2
Humensid-22
HallCheetah--22
2000
Episodd Mean Roweard
Eatodd Mean Rerward
Episade Mean Rewaar ard
2000
1000
MK
2300
69:00
$009
LOOEE
4000
2010 Time (4)
4000
1990
1003
1000
600
9009
2008
Time 19]
Time
(a) Time
Hopper-u2
Humanoid x2
HallCheetah--22
UPAJA
Episode Modit Rewward
2000
Eatorde Meam Rewaord
Episode Meain Revaard
2000
2000
1000
Timestops
Timocaps
Tiimetoops
(b) Timesteps
Figure 5: IMPACT outperforms baselines in both sample and time efficiency for Continuous Control Domains: Hopper, Humanoid, HalfCheetah.
the agent does not learn. Based on empirical results, we theorize that the agent is able to train long as a stable trust region can formed. On other hand, update frequency is too low, the agent is stranded for many iterations in the same trust region, which impairs learning speed.
4.2 TIME AND SAMPLE EFFICIENCY WITH CIRCULAR BUFFER
Counter to intuition, the tradcoff between time and sample efficiency when K increases is not necessarily true. In Figure 4b and 4c, we show that IMPACT realizes greater gains by striking the balance between high sample throughput and sample efficiency. When K = 2, IMPACT performs the best in both time and sample efficiency. Our results reveal that wall-clock time and sample efficiency can be optimized based on tuning values of K in the circular buffer.
4.3 COMPARISON WITH BASELINES
We investigate how IMPACT attains greater performance in wall clock-time and sample efficiency compared with PPO and IMPALA across six different continuous control and discrete action tasks.
We tested the agent on three continuous environments (Figure 5): HalfCheetah, Hopper, and Hu- manoid on 16 CPUs and I GPU. The policy networks consist of two fully-connected layers of 256 units with nonlinear activation tanh. The critic network shares the same architecture as the policy network. For consistentency, same network architectures were employed across PPO, IMPALA, and IMPACT.
For the discrete environments (Figure 6), Pong, Spacelnvaders, and Breakout were chosen as common benchmarks used in popular distributed RI. libraries (Caspi et al., 2017; Liang et al., 2018). Additional experiments for discrete environments are in the Appendix. These experiments and I GPU. The policy network consists of three 4x4 and one 1lx11 conv layer, with nonlinear activation ReLU. The critic network shares weights with the policy network. The input of the network a stack of four 42x42 down-sampled images of the Atari environment. The continuous and discrete environments are listed in the Appendix B table and respectively.
on 32 CPUs
for
Figures 5 and show the total average return on evaluation rollouts for IMPACT, IMPALA and PPO. We train each algorithm with three different random seeds on each environment for total time
7

---

Published as a conference paper at ICLR 2020
Spacelnvaders www.dedeeesSoooaiii-i---a--
PongNofFreeenkiie
1090
Episode Mean Reward
Epieade Meen Rawward
Sapec
Episodt Mean
090
100
200
4003
18000
2003
1003
2000
19035
100
Time
[s]
(8)
(a) Time
BreakouthoFFrmessip.v
SpacelivederstoFFaaeedde v4 pePAcE
PongNoFFramskiip.
Rewward
Mihul
Uptoode Meen Rewward
1000
Eppecde Mexan Reward
523
629
6.30
123
199
2.0
68 08
32
94
04
12
10/000
200000
20000
Timestops
Timesteps
(b) Timesteps
Figure 6: IMPACT outperforms PPO and IMPALA in both real-time and sample efficiency for Discrete Control Domains: Breakout, SpaceInvaders, and Pong.
of three hours. According the experiments, IMPACT is able to train much faster than PPO and IMPALA in both discrete and continuous domains, while preserving same or hetter sample efficiency than PPO.
Our results reveal that continuous control tasks for IMPACT are sensitive to the tuple (N, K) for the circular buffer. N 16 and K 20 is robust choice for continuous control. Although higher K inhibits workers' sample throughput, increased sample efficiency from replaying experiences results in an overall reduction training wall-clock time and higher reward. For discrete tasks, N and K 2 works best. Empirically, agents learn faster from new experience than replaying old experience, showing how exploration crucial to achieving high asymptotic performance discrete enviornments.
4.4 IMPACT SCALABILITY
Figure shows how IMPACT's performance scales relative to the number workers. More workers means increased sample throughput, which in turn increases training throughput (the rate that learner consumes batches). With the learner consuming more worker data per second, IMPACT can attain better performance in less time. However, number of workers increases, observed increases in performance begin to decline.
5 RELATED WORK
Distributed RL architectures are often used accelerate training. Gorila (Nair et al., 2015) and A3C (Mnih et al., 2016) use workers to compute gradients to be sent to the learner. A2C (Mnih al., 2016) and IMPALA (Espeholt et al., 2018) send experience tuples to the learner. Distributed replay buffers, introduced in ACER (Wang et al., 2016) and Ape-X (Horgan et al., 2018), collect worker-collected experience and define an overarching heuristic for learner batch selection. IMPACT the first to fully incorporate the sample-efficiency benefits of PPO in asynchronous setting.
Surreal PPO (Fan et al., 2018) also studies training with PPO in the asynchronous setting, but do consider adaptation of the surrogate objective nor IS co-correction. Their use of target network for broadcasting weights to workers is also entirely different from IMPACT's. Consequently, IMPACT able to achieve better results in both real-time and sample efficiency.
8

---

Published as a conference paper at ICLR 2020
HaifCheetah-v2
PongNoFrameskii-vv
10000
Episode Mean Reward
8000
Episods
4200
-15
-00
2000
4020
1000
0000
10000
100
200
200
420
500
600
Time
Time (S)
(a) Continuous environment.
(b) Discrete environment.
Figure 7: Performance of IMPACT with respect the number of workers in both continuous and discrete control tasks
Off-pollcy methods, including DDPG and QProp, utilize target networks stabilize learning the Q function (Lillicrap et al., 2015; Gu et al., 2016). This use of a target network is related but different from IMPACT, which the network to define stable trust region for the PPO surrogate objective.
CONCLUSION
In conclusion, introduce IMPACT, which extends PPO with stabilized surrogate objective for asynchronous optimization, enabling greater real-time performance without sacrificing timestep efficiency. We show the importance the IMPACT objective stable training, and show it can outperform tuned PPO and IMPALA baselines in both real-time and timestep metrics.

---

Published as a conference paper at ICLR 2020
REFERENCES
Joshua Achiam. Openai Spinning
spinningup/bench httteelnchhhhm.,,....m..m.m November 2018.
November
Itai Caspi, Gal Leibovich, Gal Novik, and Shadi Endrawis. Reinforcement Learning Coach, December 2017. URL https: : : /tt.i//iiii/////// wrg/10.............. 5281 www.oo.oooooooooo http:///........erooooo....... 1134899.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. IMPALA: Scalable Distributed Deep-RL with Importance Weighted Acto-Learner Architectures. arXiv preprint arXiv:/802.01 2018.
Linxi Fan, Yuke Zhu, Jiren Zhu, Zihua Liu, Orien Zeng. Anchit Gupta, Joan Creus-Costa, Silvio Savarese, and Li Fei-Fei. SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark. In Conference on Robol Learning, pp. 767-782, 2018.
Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E Turner, and Sergey Levine. Q-Prop: Senple-Efficient Policy Gradient with An Off-Policy Critic.. arXiv preprint arXiv: 1.0022477 2016.
Seungyul Han and Youngchul Sung. AMBER: Adaptive Multi-Batch Experience Replay for Continu- ous Action Control. arXiv preprint w7Xiv11710004423, 2017.
Seungyul Han and Youngchul Sung. Rimension-Wise Importance Sampling Weight Clipping for Sample-Efficiee Reinforcement Learning. arXiv preprint frXiv:1905.02363. 2019.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed Prioritized Experience Replay. arXiv preprint arXiv: 10Xxvv/803.000 /80300093,, 2018.
Tang Jie and Pieter Abbeel. On a Connection between Importance Sampling and the Likelihood Ratio Policy Gradient, pp. 1000-1008, 2010.
Eric Liang, Richard Liaw, Robert Nishihara. Philipp Moritz, Roy Fox, Ken Goldberg, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. RLlib; Abstractions for Distributed Reinforcement Learning. In International Conference on Machine Learning (ICML), 2018
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous Control with Deep Reinforcement Learning, arXiv preprint 15Xiv:1509.0 5099992991....... 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. In International conference on machine learning, pp. 1928 1937, 2016.
Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, al. Massively Parallel Methods for Deep Reinforcement Learning. arXiv preprint
0420000 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust Region Policy Optimization. In International conference on machine learning, pp. 1889-1897, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- High-Dimensiooal Continuous Control using Generalized Advantage Estimation. arXiv preprint arXiv: arXiv:1506.00 1506.02438 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.063477 2017.
Richard S Sutton, David A McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In Advances in neural information processing systems, pp. 1057-1063, 2000.
10

---

Published as a conference paper at ICLR 2020
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind Control Suite. arXiv preprint TrXiv:1801.00690 2018.
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample Efficient Actor-Critic with Experience Replay. arXiv preprint arXiv:661.00122,, 2016.
A ADDITIONAL EXPERIMENTS
GbortteFrrmeskiiv v
der v4
Grav LaovtaroFFaaa NoFrrmesiippp- vi
142
Eptocde Mean Roward
PV10
Epecde Mean Reverrd
Pin2
1200
1900
(a) Time
ResemRiderroFF Foweranooooossse-ooooooo
Graw Www.ebrNooForeee www.essooooo
SPHLA
Estodde Moan Rewarrd
Eseodd Meam Rosward
100
2P12
100
Timestaps
Timecaps
Timestops
(b) Timesteps
Figure 8: IMPACT, PPO and IMPALA wallclock time and sample efficiency for Discrete Control Domains: Qbert, BeamRider, and Gravitar.
B HYPER PARAMETERS FOR ALL ENVIRONMENTS
B.1 DISCRETE ENVIRONMENTS
HYPERPARAMETERS
IMPACT
IMPALA
PPO
Clip Parameter
0.3
0.1
Entropy Coeff
0.01
0.01
0.01
Grad Clipping
10.0
40.0
Discount (x)
0.99
0.99
0.99
Lambda (A)
0.995
0.995
Learning Rate
1.0 10-4
10-4
10-5
Minibatch Buffer Size (N)
4
Num SGD Iterations (K)
2
2
Sample Batch Size
50
50
100
Train Batch Size
500
500
5000
SGD Minibatch Size
500
KL Coeff
0.0
0.5
Target
0.01
0.01
Value Function Coeff
1.0
0.5
1.0
Target- Worker Clipping (p)
2.0
Table 1: Hyperparameters for Discrete Environments.
11

---

Published as a conference paper at ICLR 2020
B.2 CONTINUOUS ENVIRONMENTS
HYPERPARAMETERS
IMPACT
IMPALA
PPO
Clip Parameter
0.4
0.3
Entropy Coeff
0.0
0.0
0.0
Grad Clipping
0.5
0.5
Discount (?)
0.995
0.995
0.99
Lambda (A)
0.995
0.995
Learning Rate
3.0 10
1.5 10-5
3.0 10-4
Minibatch Buffer Size (N)
16
Num SGD Iterations' (K)
20
20
Sample Batch Size
1024
1024
1024
Train Batch Size
32768
32768
163840
SGD Minibatch Size
32768
KL Coeff
1.0
1.0
KI. Target
0.04
0.01
Value Function Coeff
1.0
0.5
1.0
Target- Clipping (p)
2.0
Table 2: Hyperparameters for Continuous Control Environments
B.3 HYPERPARAMETER BUDGET
Listed below was the grid search we used for each algorithm obtain optimal hyperparameters. Optimal values were found via grid searching on each hyperparameter separately. We found that IMPACT's optimal hyperparameter values tend hover close either IMPALA's PPO's, which greatly mitigated IMPACT's s budget.
B.3.1 DISCRETE ENVIRONMENT SEARCII
HYPERPARAMETERS
IMPACT
IMPALA
PPO
Clip Parameter
[0.1, 0.2, 0.3]
[0.1, 0.2, 0.3, 0.4)
Grad Clipping
[10, 20, 40]
[2.5, 5, 10, 20, 40, 80]
Learning Rate (10-4)
[0.5, 1.0]
[0.1, 0.3, 0.5, 0.8, 1.0, 3.0, 5.0]
[0.5, 1.0, 3.0, 8.0]
Minibatch Buffer Size (N)
[2,4,8, 16]
Num SGD Iterations (K)
[1.2.4]
[1,2,4,8] [1000, 2500, 5000, 10000] [0.25, 0.5, 1.0. 2.0]
Train Batch Size
Value Function Coeff
[0.5, 1.0, 2.0]
[0.25, 0.5, 1.0, 2.0]
# of Runs
19
17
21
Table 3: Hyperparameter Search for Discrete Environments
'For HalfCheetah-v2, IMPACT and PPO Num SGD Iterations (K) is 32. FFor HalfChectah-v2,, IMPACT Value Function Coeff is 0.5.
MMAAAA difficult finetune due unstable
12

---

Published as a conference paper at ICLR 2020
B.4 CONTINUOUS ENVIRONMENT SEARCH
HYPERPARAMETERS
IMPACT
IMPALA
PPO
Clip Parameter
[0.2, 0.3, 0.4] [0.5, 1.0, 5.0]
[0.1, 0.2, 0.3, 0.4]
Grad Clipping
0.25, 0.5, 1.0, 5.0, 10.0]
Learning Rate (10-4)
[1.0, 3.0, 5.0]
[0.1. 0.15, 0.3, 0.5, 0.8, 1.0, 5.0]
[1.0, 3.0, 5.00
Minibatch Buffer Size (N)
[4,8,16]
Num SGD Iterations (K)
[20,26,32]
[20,26.32] [65536, 98304, 131072, 163840] [0.01, 0.02, 0.04] [0.5, 1.0, 2.0]
Train Batch Size
KL Target
[0.01, 0.02, 0.04]
Value Function Coeff
[0.5, 1.0, 2.0]
[0.5, 1.0, 2.0]
# of Runs
21
17
20
Table 4: Hyperparameter Search for Continuous Environments
C IMPALA TO IMPACT
From IMPACT
10000
IMPALA
IMPALA PPO Objective
IMPALA PPO Objeclive + Replay
Episode Mean Reward
2000
2000
4000
8000
8000
10000
Time (s)
Figure 9: IMPALA to IMPACT: Incrementally Adding PPO Objective, Replay, and Target- Worker Clipping to IMPALA. The experiments done the HalfCheetah-v2 gym environment.
In Figure 9, we gradually add components to IMPALA until the agent is equivalent to IMPACT's. Starting from IMPALA, gradually add PPO's objective function, circular replay buffer, and target-worker clipping. In particular, IMPALA with PPO's objective function and circular replay buffer is equivalent an asynchronous--arriee of PPO (APPO). APPO fails perform as well as synchronous distributed PPO, since PPO is an on-policy algorithm.
D IMPALA IN CONTINUOUS ENVIRONMENTS
In Figure 6, IMPALA performs substantially worse than other agents in continuous environments. We postulate that IMPALA suffers from low asymptotic performance here since its objective is an anpoorance-sampllee version of the Vanilla Policy Gradient (VPG) objective, which is known to suffer from high variance and large update-step sizes. We found that for VPG, higher learning rates encourage faster learning in the beginning but performance drops negative return later in training.
13

---

Published as a conference paper at ICLR 2020
In Appendix B, for IMPALA, we heavily tuned the learning rate, finding that small learning rates stabilize learning at the cost of low asymptotic performance. Prior work also reveals the agents that VPG fail to attain good performance in non-trivial continuous tasks (Achiam, 2018). Our results with IMPALA LA reaches similar performance compared to other VPG-based algorithms. The closest neighhor to IMPALA, A3C uses workers compute gradients from the VPG objective send the learner thread. A3C performs well in InvertedPendulum yet flounders in continuous environments (Tassa al., 2018).
231
nathr
targe
IMPAC
PRO ep-0.1
3.6
IMPACT (psud)
probi dere
3.4
00.3
1.0
01:00
044
0.5
-5
-5
-2
Action Distributions
Likelihood Ratios w.r. Different Objectives
Figure 10: Likelihood ratio (0) for different objective functions, including PPO's. We assume a diagonal Gaussian policy for our policy. Left: Corresponding one dimensional action distributions for Worker i, Target, and Master Learner; Right: Ratio values graphed function of possible action values. IMPACT with PPO clipping a lower bound of PPO.
E THE INTUITION OF THE OBJECTIVE
The following ratios represent the objective functions for different ablation studies. In the plots (Figure 10), set the advantage function to be one, i.e. A. 1.
IS ratio:
me
Å
TTenkkrr
IMPACT target: min
Tuodkerg Turget
TT Tucrkar
Ar
PPO e-clip: min
Ae, clip(
1- E, 1 + e).i)
1-
Faieker
IMPACT target e-clip: min
min
Tourtins
To
Ae, clip
min
Tooskers
Te
1
1 1+++ e)
Kurpet
Torge
Toortery
According to Figure 10, IS ratio is large when Twokker, assigns low probability. IMPACT target c-clip a lower bound of the PPO c-clip. In an distributed asynchronous setting, the trust region suffers from larger variance stemming from off-policy data. IMPACT target e-clip ratio mitigates this by encouraging conservative and reasonable policy-gradient steps.
14


Reviews:
[{"review_id": "SJlsXw06tr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces IMPACT which is a distributed RL algorithm that shortens training time of RL systems while maintaining/ improving the sample efficiency. It is built on top of the famous PPO algorithm (https://arxiv.org/abs/1707.06347). The authors break down the novel component of their model into three categories: target network, circular buffer, and importance sampling. They evaluate the effectiveness of each component through different experiments. \n\nOverall the paper is well-written and the ideas are communicated clearly. I like how the evaluation is done in different environments (discrete and continuous action-space) and improves the results independent of the task settings.  \n\nOne question: you mentioned in section 4.3: \"For fairness, same network hyperparameters were used across PPO, IMPALA, and IMPACT.\" I suppose it would be a fair comparison if you choose the hyperparameters for each algorithm separately ( according to the highest value they achieve on the measured metric.) How did you end up choosing the hyperparameters for your own experiments? Are they fine-tuned for IMPACT?\n\nIt seems like IMPACT is not always doing better than PPO in the discrete control domain as shown in Figure 6. Specifically, in part (a) it looks like PPO is beating both IMPALA and IMPACT for BreakoutNoFrameskip and PongNoFrameskip. It would be nice if authors could do an analysis of these cases and add a discussion as to why this is happening.\n\nOverall I think this is an interesting paper which can motivate more work in this area.\n\n\n------------------------------------------------------------------------------------------------------------------------------------------------\nUpdates:\nI would like to thank the authors for their response. I have read the revised version and it looks good to me.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJx4ydyAKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new distributed algorithm for reinforcement learning. The paper lists three main contributions: a target network for stabilizing the surrogate objective, a circular buffer, and truncated importance sampling. \n\nI'm not that familiar with RL, however I'm very familiar with distributed training in other contexts. Therefore, the significance of the contributions in the RL domain is a bit unclear to me. However, the contributions in the area of distributed training is relatively fair. The introduction of a circular buffer is not very novel. Further, the trade-offs / adaption of update frequency etc. are standard ways to improve performance in distributed training. \n\nThe evaluation of the proposed algorithm is reasonably well done (considering the page limits), with a suitable set of benchmarks (although relatively few). The results are promising and could have a significance for practitioners.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1ePESipKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies a novel way for distributed RL training which combines the data reuse of PPO with the asynchronous updates of IMPALA. The main contribution is the observation that using a target network is necessary for achieving stable learning. I think this is an important result which seems to be validated by another ICLR submission (https://openreview.net/forum?id=SylOlp4FvH). The experimental section could definitely be improved--I was hoping to see more results on Atari or DMLab.\n\nTwo comments:\nThe V-trace equations (page 3 and 5) don't mention any clipping of the importance weights c and rho--can you clarify if this is a typo or if you don't use clipping?\nIt would be great to see experiments showing how learning curves scale with the number of workers.\n\n-----------------------------------------------------------------------------------\nThanks for clarifying and for the extra experiments. I'm keeping my score as I still think it's appropriate.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Byl-Rcma9B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Reinforcement learning (RL) training speed is broadly evaluated on two dimensions:  sample efficiency (the number of environment interactions required) and wall-clock time.  Improved wall-clock training time has been achieved through distributed actors and learners, but often at the expense of sample efficiency.  IMPACT repurposes successful concepts from deep RL - the target network, importance sampling and a replay buffer to demonstrate improvements on both axes in on three continuous environments and three games from the Atari Learning Environment.\n\nPositives\nThis was a well-written paper proposing to address the sample efficiency of distributed RL algorithms.  The diagrams of the algorithm were also well-done.  Improving the sample efficiency of algorithms is an important objective and the approaches followed here are sensible.\n\nAdditionally, the ablations and examination of the sensitive hyperparameters of the algorithm are useful analyses.  These indicate relative insensitivity to the target network update frequency, but both the importance sampling equation and the circular buffer hyperparameters are described.\n\n\nNegatives\nIMPACT introduces additional hyperparameters which are tuned for each continuous control task and discrete control task.  However, there is no description of the hyperparameter tuning budget allocated to IMPACT, PPO, IMPALA.  \n\nRegarding the discrete environment, the game selection should be elaborated upon and if sufficient compute is available, the algorithm should be tested elsewhere.  Specifically, it is atypical (though not necessarily incorrect) to tune specific hyperparameters for each game in the Atari Learning Environment.  Traditionally, algorithms have been justified as robust and useful by the lack of need to tune per game.  Table 4 demonstrates a high degree of tuning for IMPACT due to game-specific changes for clip param, grad clip, lambda, num sgd iter, train_batch_size, value function loss coeff, kl coeff.  However, Table 5 (IMPALA) and Table 6 (PPO) have fewer noted changes.\n\nSmall nits: \n- Define advantage in the policy gradient equation\n- Figure 4 is ahead of Figure 3 in the compiled LaTeX\n\n\nQuestions\n- How were the discrete control games selected?\n- What was the hyperparameter tuning budget for IMPACT versus PPO or IMPALA?\n- If a fixed hyperparameter budget is allocated in advance and new environments are randomly selected, does IMPACT favorably compare to IMPALA and PPO?\n- IMPALA performs remarkably badly in the three continuous control tasks, even on wall-clock time.  What validations have been done here to ensure the algorithm is operating as intended?\n\nI will increase my rating if the robustness and improvements of this algorithm can be validated in randomly chosen games/continuous control environments for a fixed hyperparameter budget for IMPACT and both baselines.  Also, the IMPALA baseline should be validated for the continuous control tasks - it's surprising that this once SOTA-algorithm flounders in even simple tasks like Hopper-v2 or HalfCheetah-v2.\n\n\n-----------------\nUpdate:\nThe authors have addressed my initial concerns carefully through extra experiments and details in the Appendix and I have updated my rating accordingly.  Thanks!\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a novel distributed reinforcement learning algorithm that includes 3 new components: a target network for the policy for stability, a circular buffer, and truncated importance sampling. The authors demonstrate that this improves performance while decreasing wall clock training time.

Initially, reviewers were concerned about the fairness of hyper parameter tuning, the baseline implementation of algorithms, and the limited set of experiments done on the Atari games. After the author response, reviewers were satisfied with all 3 of those issues.

I may have missed it, but I did not see that code was being released with this paper. I think it would greatly increase the impact of the paper at the authors release source code, so I strongly encourage them to do so.

Generally, all the reviewers were in consensus that this is an interesting paper and I recommend acceptance.
Decision: Accept (Poster)