Striving for Simplicity in Off-Policy Deep Reinforcement Learning
Authors: Rishabh Agarwal
Dale Schuurmans
Mohammad Norouzi
Venue:  (2020)
Keywords: reinforcement learning
off-policy
batch RL
offline RL
benchmark

Abstract: This paper advocates the use of offline (batch) reinforcement learning (RL) to help (1) isolate the contributions of exploitation vs. exploration in off-policy deep RL, (2) improve reproducibility of deep RL research, and (3) facilitate the design of simpler deep RL algorithms. We propose an offline RL benchmark on Atari 2600 games comprising all of the replay data of a DQN agent. Using this benchmark, we demonstrate that recent off-policy deep RL algorithms, even when trained solely on logged DQN data, can outperform online DQN. We present Random Ensemble Mixture (REM), a simple Q-learning algorithm that enforces optimal Bellman consistency on random convex combinations of multiple Q-value estimates. The REM algorithm outperforms more complex RL agents such as C51 and QR-DQN on the offline Atari benchmark and performs comparably in the online setting.

Under review as a conference paper at ICLR 2020
STRIVING FOR SIMPLICITY IN OFF-POLICY DEEP REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
This paper advocates the use of offline (batch) reinforcement learning (RL) to help (I) isolate the contributions of exploitation vs. exploration in off-policy deep RL, (2) improve reproducibility of deep RL research, and (3) facilitate the design of simpler deep RL algorithms. We demonstrate that recent off-policy deep RL algorithms, even when trained solely on all of the replay data of an online DQN agent, can outperform online DQN on Atari 2600 games. We also present Random Ensemble Mixture (REM), simple Q-learning algorithm that enforces optimal Bellman consistency on random convex combinations of multiple Q-value estimates. Using the DQN replay dataset, offline REM surpasses the gains from online C51 and outperforms offline QR-DQN. Furthermore, REM performs comparably to QR-DQN in the online RL setting on Atari 2600 games. The DQN replay dataset can serve as an offline RL benchmark and will be released.
INTRODUCTION
Deep neural networks have become critical component of modern reinforcement learning (RL) (Sut-
ton and Barto 2018). The seminal work of Mnih et al. (2013) 2015) on deep Q-networks (DQN)
has demonstrated that it is possible to train neural networks using Q-learning (Watkins and Dayan 1992) to achieve human-level performance in playing Atari 2600 games (Bellemare et al. 2013) directly from raw pixels. Recent progress in mastering Go (Silver et al.) (016)) and advances in
robotic control (Levine et al. 2016 OpenAl et al. 2018: Kalashnikov et al. 2018) present additional supporting evidence for the enormous potential of deep RL.
Off-policy RL algorithms such as Q-learning are attractive because they disentangle data collection and policy optimization and offer more sample efficient solutions than on-policy algorithms (Sutton eet 22000 [Schulman et al. 2015) Mnih et al. (0166). Importantly, off-policy techniques can leverage the vast amount of existing offline logged data for real-world applications such as digital advertis- ing (Strehl et al. 2010 Bottou et al. 2013), education (Mandel et al. (014)), and healthcare (Shortreed 2011). Since online RL is often unsafe to deploy in the real world, offline RL algorithms are the only feasible solution for many practical decision making problems (Dulac-Armold et al.) (2019). Nevertheless, off-policy RL algorithms when combined with neural networks can be unstable or even
divergent Baird 1995 Boyan and Moore
1995. Tiissiklis and Van Roy 1997).
In the absence of theoretical guarantees for off-policy deep RL, recent advances (see Hessel et al. (2018) for an overview) are largely governed by empirical results on a popular benchmark suite of Atari 2600 games (Bellemare et al., 2013). Unfortunately, these empirical results are difficult to reproduce (Henderson et al. 2018 8) and the contribution of novel RL algorithms is often conflated with many other design choices Cllar et al. 2019 Khetarpal et al., 2018). Given the empirical nature up with simpler and reproducible experimental settings and study the relative importance of different components of deep RL algorithms. It is also important to strive for finding successful RL algorithms that are simple as possible.
of deep off- off-policy RL, it is crucial to
This paper advocates the use of offline (batch) RL to help (1) isolate the contributions of exploitation vs. exploration in off-policy deep RL, (2) improve reproducibility of deep RL research, and (3) facilitate the design of simpler off-policy deep RL algorithms. To this end, we propose a new benchmark for offline optimization of RL agents on Atari 2600 games using all of the replay data of DQN agent (Mnih et al. 2015). Using this benchmark, we investigate the following questions:
1. Is it possible to train successful Atari agents based solely on offline data?
2. Can one design simple and effective alternatives to intricate RL algorithms in the offline setting?
3. Are the insights gained from the offline setting useful for developing effective online algorithms?

---

Under review as a conference paper at ICLR 2020
Median Normalized Scores
Median Normalized Scores
119.0%
135.0%
100.0%
Online DQN
119.09
100.0%
Online DQN
60.0%
Oéfine
50.0%
Oifline QR-NIN
Ouline REM
Online QUi-MI
Offiine DGN (Nuluan)
Onlinn Revtstragped-
0.0%
0.0%
0
200
400
600
800
1000
50
100
150
200
Iteration
Iteration
Figure 1: Median normalized online evaluation scores averaged over 5 rums (shown as traces) across stochastic version of 60 Atari 2600 games of (a) offline RL agents trained using the DQN replay dataset containing 200 million frames. Offline REM outperforms offline QR-DQN and surpasses gains from online C51. (b) online RL agents trained for 200 million frames (standard protocol). Online REM with Q-networks performs comparably to online QR BQN, Each iteration corresponds to million frames.
The contributions of this paper can summarized as:
An offline RL benchmark is proposed for evaluating and designing RL algorithms on Atari 2600 games without exploration, based on the logged replay data of a DQN agent comprising 50 million (observation, action, reward, next observation) tuples per game. This reduces the computation cost of the experiments considerably and helps improve reproducibility deep RL research by standardizing training using fixed offline dataset. The replay dataset used in experiments will be released to enable offline optimization of RL algorithms on a common ground.
Contrary recent work (Zhang and Sutton 2017: Fujimoto et al.. 2019), we find that the logged DQN data is sufficient for optimizing strong Atari agents offline without any environment interaction. For instance, QR-DQN (Dabney et al.) (018b) trained offline on the DQN replay dataset significantly outperforms online DQN (Mnih et al. 2015).
A simple and novel Q-learning algorithm called Random Ensemble Mixture (REM) is presented, which enforces optimal Bellman consistency random convex combinations of multiple Q-value estimates. Offline REM, trained using the DQN replay dataset, outperforms more complex RL algorithms such offline QR-DQN and surpasses the gains from online C51 (Bellemare al.) 2017) over online DQN (Figure hT..
We use the insights gained from the offline experiments to develop an online variant of REM, which performs comparably with online QR-DQN (Figure Ip).
OFF-POLICY REINFORCEMENT LEARNING
An interactive environment in reinforcement learning (RL) is typically described Markov decision process (MDP) (S,A, R, 7. (Puterman 1994), with a state space S, an action space AA stochastic reward function R[s a), transition dynamics P(s'|s, ss a) and discount factor policy +(- |8) maps each state s E S to a distribution (density) over actions.
[0, 1). A stochastic
For agent following the policy æ, the action-value function, denoted Q* is defined the expectation of cumulative discounted future rewards, i.e.,
(= = E Σ
80 8, 0g a, 8, P(: 8:-1, 0t-1, ~ T( 8t), R(se, at) 7'r
The goal of RL is to find an optimal policy x that attains maximum expected return, for which QF' (s,a) 2 (T a) for all n, s, a. The Bellman optimality equations (Bellman 1957) characterize the optimal policy in terms of the optimal Q-values, denoted Q" Q via:
as) E
max Q" (=','aa') r ~~ R(s,a), s P(- P(-18,)).
To learn policy from interaction with the environment, Q-learning (Watkins and Dayan 1992) iteratively improves an approximate estimate of Q", denoted Qa. by repeatedly regressing the LHS of (2) to target values defined by samples from the RHS of ( For large and complex state
2

---

Under review as a conference paper at ICLR 2020
spaces, approximate Q-values are obtained using neural network the function approximator. To further stabilize optimization, a target network Qo with frozen parameters may be used for computing the learning target (Mnih et al. 2013). The target network parameters 8' are updated to the current Q-network parameters 0 after a fixed number of time steps. DQN (Mnih et al. 2013 2015) parameterizes Qo with convolutional neural network (LeCun et al. 1998) and uses Q-learning with a target network while following an e-greedy policy with respect to Qo for data collection. DQN minimizes the TD error using the loss L()) on mini-batches of agent's 's past experience tuples, (s, s'), sampled from an experience replay buffer D (Lin 1992) collected during training:
(())= = Eaans'o la (Qo(s,a) - - max (er(s',aa)]
(3)
for |u| S 6, where Io is the Huber loss (Huber 1964 given by (s(u) = [0(12) 20), otherwise.
Q-learnin is an off-policy algorithm (Sutton and Barto 2018) since the learning target can be computed without any consideration of how the experience was generated. In offline (batch) RL (Ernst et al. 2005; Riedmiller 2005 Lange et al. 2012), we assume access to a fixed offline dataset of experiences D, without any further interaction with the environment.
A family of recent off-po deep RL algorithms, which serve as a strong baseline in this paper, include Distributional RL (Bellemare al. 2017: Jaquette 1973) methods. Such algorithms estimate a density over returns for each state-action pair, denoted 2 (s. instead of directly estimating the mean Q (=(ss (s)a). Accordingly, one can express a form of distributional Bellman optimality as
Z (s, a) 2 7Z argmax Q'(s', a')),
where R(s,a), s' P(. s,)), (4)
and D denotes distributional equivalence and ( a') is estimated by taking an expectation with respect to Z'(s (s''a') C51 (Bellemare et al. 2017) approximates 2 (s, a) by using a categorical distribution over a set of pre-specified anchor points, and distributional QR-DQN (Dabney al., 2018b approximates the return density by using a uniform mixture of K Dirac delta functions, i.e.,
K
1 K
Zo(s,a) a) = K
Σ (((sa)). Σ Soi(s.a): and Qo(s.a) = K 1=1
(5)
QR-DQN trains each 0; for SisK match the -quantile of the target return density (RHS
of 44) using the Huber quantile regression loss (Koenker] 20055 Dabney et al.] 2018b). QR-DQN, albeit complex, outperforms C51 and DQN and obtains state-of-the-ar results Atari 2600 games, among agents that do not exploit n-step updates (Sutton) 1988) and prioritized replay (Schaul et al., 2016). This paper avoids using n-step updates and prioritized replay to keep the empirical study simple and focused on deep Q Q-learning algorithms.
OFFLINE OPTIMIZATION OF REINFORCEMENT LEARNING AGENTS
Modern off-policy deep RL algorithms perform remarkably well common benchmarks such the Atari 2600 games (Bellemare et al.) (2013) and continuous control MuJoCo tasks (Todorov et al.) 2012).
2015 Lillicrap et al. (0155) are considered "online" because they alternate between optimizing a policy and using that policy to collect more data. Typically, these algorithms keep a sliding window of most recent experiences in a finite replay buffer (Lin 1992). throwing away stale data to incorporate most fresh (on-policy) experiences. In principle, off aff-policy algorithms can learn from data collected by any policy, however, Zhang and Sutton (2017) assert that using a large replay buffer can significantly hurt the performance of Q-learning algorithms, since it can delay rare on-policy experiences that are important for policy learning.
Such off-policy algorithms (Mnih
This paper revisits offline off-policy RL and investigates whether off-policy deep RL agents trained solely on offline data can be successful. We advocate the use of offline RL to help isolate an RL algorithm's 's ability to exploit experience and generalize its ability to explore effectively by collecting interesting new data. The offline RL setting removes design choices related to the replay buffer and exploration; therefore, it is much simpler to experiment with and reproduce than the typical online off ool-policy learning. Also, in the offline setting, we optimize an objective (e.g., (3) over a fixed dataset opposed to changing replay buffer of an online agent. In this work, mainly experiment
3

---

Under review as a conference paper at ICLR 2020
LSgN
100
Gane
(a)
(b)
Figure 2: Normalized performance improvement (in %) over online DQN (Nature), per game, of (a) offline DQN (Nature) and (b) offline QR-DQV trained using the DQN replay dataset for same number of gradient updates online DQN. The normalized online score for each game is 0.0 and 1.0 for the worse and better performing agent among fully trained online DQN and random agents respectively.
with large and diverse datasets motivated by real-world RL problems where we already have access to (usually continually growing) diverse datasets of logged experiences, e.g,, historical data from recommender systems (Bottou et al. 2013), database of robotic experiences (Dasari et al. 2019) etc.
To facilitate a study of offline RI. with large and diverse datasets, we train several instances of DQN (Nature) agents (Mnih et al., 2015) on stochastic version of 60 Atari 2600 games for 200 million frames each, with frame skip of 4 (standard protocol) and sticky actions enabled (Machado et al. 2018). On each game, we train 5 different agents with random initialization, and store all of the tuples of (observation, action, reward, next observation) encountered during training into 5 replay datasets of 50 million tuples each. These replay datasets are used for training standard off-policy RL agents, offline, without any interaction with the environment during training. We perform an online evaluation of the agents in the intervals of million training frames, and report the best evaluation score for each agent, averaged over runs.
3.1 EXPERIMENTS AND RESULTS
Given the logged replay data of the DQN agent, it is natural to ask whether training offline variant of DQN on this data will recover the performance of online DQN. Furthermore, whether more recent off-policy algorithms
able exploit the DQN replay dataset more effectively than DQN. To investigate these questions, we train DQN (Nature) and distributional QR-DQN (Dabney et al. 2018b agents, offline, on the DQN replay dataset for the same number of gradient updates as online DQN. We use the hyperparameters provided in the Dopamine baselines (Castro et al., 2018) for standardized comparison (Section A.44 and report scores using a normalized scale (Section A.3) We note that the replay datasets include samples from all of the intermediate policies seen during the optimization of the DQN (Nature) agent. Accordingly, we compare the performance of offline agents against the best performing policy among the mixture of data collecting policies.
Results. We find that offline DQN underperforms online DQN on all except a few games while offline QR-DQN outperforms offline DQN and online DQN most of the games (Figure C51 (Bellemare al. 2017) trained offline using the DQN replay dataset also considerably improves upon offline DQN (Figure A.4). These results suggest that DQN (Nature) is somewhat ineffective at exploiting off- policy data. However, offline QR-DQN's impressive performance demonstrates that is possible to optimize strong Atari agents completely offline (see Figure A.6 for learning curves).
3.2 OFFLINE CONTINUOUS CONTROL EXPERIMENTS
In contrast to our offline results on Atari 2600 games with discrete action spaces, discussed above, Fujimoto et al, (2019) find that standard off-policy deep RI. agents are not effective on continuous control problems when trained offline, even when large and diverse replay datasets used. The results of Fujimoto et al. (2019) are based on the evaluation of a standard continuous control agent, called DDPG (Lillicrap et al. 2015), and other more recent continuous control algorithms such as TD3 (Fujimoto et al. 2018) and SAC (Haarnoja et al.) 2018) are not considered in their study.
Motivated by the so-called final buffer setting in Fujimoto et al. (2019) (Section A.2) we train DDPG agent on continuous control MuJoCo tasks (Todorov et al. 2012) for I million time steps and store all of the experienced transitions. Using this dataset, we train standard off-policy agents including
4

---

Under review as a conference paper at ICLR 2020
Actons
Aclona
Actins
Action
Actiace
Actena
Pa.
Σ
Z
2
a2
9,
a2
Shared Network
Shared Network
Neural Network
Network
QR-DQN
Ensemble-DQN
REM
Figure Neural network architectures for DQN, QR-DQN and the proposed expected RL variants, i.e., Ensemble-DQN REM, with same multi-head architecture as QR-DQN. The individual Q-heads share all of the neural network layers except the final fully connected layer.
TD3 and DDPG completely offline. Consistent with our offline results on Atari games, offline TD3 significantly outperforms the data collecting DDPG agent and offline DDPG (Figure A.I)
SEEKING SIMPLER OFF-POLICY RL ALGORITHMS
Section 3.1 finds that recent distributional RI. algorithms such as QR-DQN are effective at exploiting offiine data. However, these algorithms complicated they require generalization of Bellman equations to distributions (4) and typically minimize a probability divergence measure from distributional target (Rowland al. Dabney al.) (018b) instead of scalar TD errors. Moreover, despite recent efforts (Lyle et al. (219)] Bellemare et al. 2019), the source of the gains when using distributional RL algorithms remains unclear. We investigate whether one can develop simple and effective alternatives to more intricate RL algorithms in the offline setting. To this end, we study two deep Q-learning algorithms, Ensemble DQN and REM, which adopt ensembling to improve stability.
4.1 ENSEMBLE-DDE DQQ
Ensemble-DQN is simple extension of DQN that approximates the Q-values via an ensemble of parameterized Q-functions (Fauleer and Schwenker 2015 Osband et al., 2016 Anschel et al. 2017 Each Q-value estimate, denoted Q7 (x, a), is trained against its own target Q (8. a), similar to Bootstrapped-DDo (Osband et al. 2016). The Q-functions are optimized using identical mini- batches in the same order, starting from different parameter initializations. The loss L(0) takes the form,
K
((0)
E. Esanns (es (Qā(s.a) - max (8
(6)
K
where Is is the Huber loss. While Bootstrapped-DQN uses one of the Q-value estimates in each episode to improve exploration, in the offline setting, we are only concerned with the ability of Ensemble-DOo to exploit better and use the mean of the Q-value estimates for evaluation.
4.2 RANDOM ENSEMBLE MIXTURE (REM)
Increasing the number of models used for ensembling typically improves the performance of su- pervised learning models (Shazeer et al., 2017). This raises the question whether one can use an ensemble over exponential number of Q -estimmaess in computationally efficient manner. Inspired by dropout (Srivastava et al.) 2014), we propose Random Ensemble Mixture for RL.
Random Ensemble Mixture (REM) uses multiple parameterized Q-functions to estimate the Q-values, similar to Ensemble-DQu The key insight behind REM is that one can think of a convex combination of multiple Q-value estimates valid Q Qalue estimate itself. This is especially true at the fixed point, where all of the Q-value estimates have converged to an identical Q Q-function. Using this insight, train family of Q-function approximators defined by mixing probabilities (K -1 1)-simllex.
5

---

Under review as a conference paper at ICLR 2020
Median Normalized Scores
Table 1: Median normalized best average eval- uation scores (averaged over 5 runs) across 60 stochastic Atari 2600 games, measured as percent- ages and number games where an offline agent achieves hetter scores than a fully trained online DQN (Nature) agent.
119.0%
Online C51
100.0%
Online DQN
50.0%
Offline RFM
Offiine Avwrgedd RnsamElo-DQ
Offline Ensemble-D@N
Offlnne DQN (Adam)
0.0%
Offline agent
Median
> DQN
200
400 Iteration
600
800
1000
DQN (Adam)
111.9%
41
Ensemble-DQU
111.0%
39
Figure 4: Offline REM vs. Baselines. Normalized scores averaged over rums (shown traces) across stochastic Atari 2600 games of offline agents trained using the DQN replay dataset trained for 5X gradient steps.
Averaged Ensemble-Doe DQN QR-DQN REM
112.1%
43
118.9%
45
123.8%
49
Specifically, for each mini-batch, we randomly draw categorical distribution a, which defines convex combination of the estimates to approximate the optimal Q-function. This approximator is trained against its corresponding target minimize the TD error. The loss [()) takes the form,
(()) = Es..n B Soi Pa
Es
Σ
aa Q5 - --r---mmm max Σ
Qj a')
(7)
where PA represents probability distribution over the standard (K ))-smpleex A = ta E IK 01 + 02 + + ak = 1, ag 2 =0.,.
constraint satisfaction problem based on Bellman optimality con- straints (2) and [(0) can be viewed as an infinite set of constraints corresponding to different mixture probability distributions. For action selection, use the average of the K value estimates as the Q-function, Le. Q(+ Q) = Σx (s, a)/K. REM is easy to implement and analyze (see Proposition 1), and can be viewed we use very simple distribution PA: we first draw set of K values i. i. d. from Uniform (0, 1) and normalize them to get valid categorical distribution, i.e., o'g U(, 1) followed by ag o/// Σ a4.
REM considers Q-learning
simple regularization technique for value-based RL. In experiments,
Proposition 1. Consider the assumptions: (a) The distribution Pa has full support over the entire (K - ---sipplex. (b) Only finite number of distinct Q-functions globally minimize the loss in (3) (c) Q is defined in terms of the MDP induced by the data distribution D. (d) Q" lies in the family of our function approximation. Then, at the global minimum of C()) 7 for a multi-head Q-network:
(i) Under assumptions (a) and (b), all the Q-heads represent identical Q-functions.
(ii) Under assumptions (d)- the common global solution is Q"
The proof of (ii) follows from (i) and the fact that 7 is lower bounded by the TD error attained by Q* The proof of part (i) can be found in the supplementary material.
4.3 EXPERIMENTS AND RESULTS
Asymptotic performance of offline agents. In supervised learning, asymptotic performance matters much more than performance within a fixed budget of gradient updates. Similarly, for a given sample complexity, we prefer RL algorithms that perform the best as long the number of gradient updates feasible. Since the sample efficiency of the offline agents for given dataset is fixed, we train them on the DQN replay dataset for five times as many gradient updates as online DQN.
Comparison with QR-DQN. QR-DQN modifies the DQN (Nature) architecture output K values for each action using a multi head Q-network and replaces RMSProp (Ticleman and Hinton 2012) with Adam (Kingma and Ba 2015) for optimization. To ensure a fair comparison with QR-DQN, we use the same multi-head Q networrk as QR-DQN with K = 200 heads (Figure 33. where each head represents a Q-value estimate for REM and Ensemble-DQNN We also use Adam for optimization.
Additional Baselines. To isolate the gains due to Adam QR-DQN and our proposed variants, compare against DQN baseline which also uses Adam. We also evaluate Averaged Ensemble- DQN, a variant of Ensemble- DQN proposed by Anschel et al. (2017, which uses the average of the predicted target Q-values as the Bellman target for training each parameterized fuuncioon. This

---

Under review as a conference paper at ICLR 2020
5007%
AFTSRI
500%
ESREAROU7
1375
PONG
1207
Q"BERE
3005
BEAQQURTT
Bore
1005
100%
200%
200%
100%
00%
00%
100%
100%
40%
40%
100%
50%
60%
20%
395
00%
1040 25%
60% 100%
10%
20%
50%
100%
176
10%
20%
50% 100%
256
10%
20%
60%
100%
156
10%
25%
65%
100%
Praction of darts
Fructin of Gatta
Fructios of data
Praction of data
Praction of data.
Figure 5: Effect of Dataset Size. Normalized scores (averaged over 5 runs) of QR-DQN and multi-head REM trained offline on stochastic version of Atari 2600 games for 5X gradient steps using only a fraction of the entire DQN replay dataset (200M frames) obtained via randomly subsampling trajectories.
baseline determines whether the random combinations of REM provide any significant benefit over simply using an ensemble of predictors to stabilize the Bellman target.
REM with separate Q-networks. To investigate whether the effectiveness of REM is linked to ensembling techniques and dropout, we compare offline REM with K Q-value estimates computed using K separate Q-networks vs. a multi-head SaeeedQQ-een Q-network for K E (4, 16).
Results. Figure 4 shows the comparison of the additional baselines with REM and Ensemble-DQN and Table summarizes the asymptotic performance results. DQN with Adam noticeably bridges the gap in asymptotic performance between QR-DQN and DQN (Nature) in the offline setting. Offline Ensemble-DQN does not improve upon this strong DQN baseline showing that its naive ensembling approach is inadequate. Furthermore, Averaged Ensemble-DQNN performs only slightly better than Ensemble-DQN In contrast, REM exploits offline data more effectively than other agents, including QR-DQN, when trained for more gradient updates. Surprisingly, using the DQN replay dataset, offline REM surpasses the gains from online C51, a huge improvement over online DQN (Figure [
Based on above empirical results, hypothesize that gains from REM are due to the noise from randomly ensembling Q-value estimates leading to more robust training analogous to dropout. Furthermore, REM with separate Q Q-networks performs better asymptotically and learns faster than REM with a multi-head Q- Q-network in the offline setting (Figure (A.5). This consolidates our belief about effectiveness of REM being linked to dropout as separate Q-networks result in more variation in individual Q- Qeestimates leading to more robust training.
4.4 FROM OFFLINE TO ONLINE REINFORCEMENT LEARNING
Can we combine the insights gained from the offline setting with appropriate design choices (e.g., ex- ploration, replay buffer) to create effective online methods? In online RL, learning and data generation are tightly coupled, i.e., an agent that learns faster also collects more relevant training data.
We ran online REM with K separate Q-networks (with K 4 for computational efficiency) because of the improved convergence speed over multi-head REM in the offline setting. For data collection, we use e-greedy with a randomly sampled Q-estimate from the simplex for each episode, similar to Bootstrapped DQN. To estimate the gains from the REM objective (7) in the online setting, we also evaluate Bootstrapped -DQ with identical modifications (e.g., separate Q-networks) as online REM. Figure Ib show that REM performs on par with QR-DQN and considerably outperforms Bootstrapped-Dee DQN (see Figure A.8 for learning curves). Based on these results, we conclude that the offline setting can help facilitate the design of effective online off-policy RI. algorithms.
5 OFFLINE RL: DATASET SIZE AND COMPOSITION
Our offline learning results indicate that 50 million logged experience tuples per game from DQN (Na- ture) are sufficient to obtain good online performance on most of the Atari 2600 games. One may argue that the size of the fixed replay and its composition play key role in empirical re- sults (De Bruin et al.) 2015). To study the role of the replay buffer size in the success of the offline agents, we perform ablation experiment with variable replay buffer size.
We conducted offline experiments with reduced data obtained via randomly subsampling entire trajectories from the logged DQN experiences, thereby maintaining the same data distribution. Figure 5 presents the performance of the offline REM and QR-DQN agents with N% of the tuples in the DON replay dataset where N {1, 10, 20, 50, 100). As expected, performance tends to increase
7

---

Under review as a conference paper at ICLR 2020
ARTERIX
ISRRAOOU
PoNG
QUBERT
BELAQURRT
Arrrage Scre
148-
1540
(2-
20:00
540
0405
2.0
Eirrstion
Iteration
Iteenttons
Exersiion
Figure 6: Effect of Dataset Quality. Normalized scores (averaged over 3 runs) of QR-DQN and muulti-head REM trained offline on stochastic version of Atari 2600 games for 5X gradient steps using logged data from online DQN trained only for 20M frames (20 iterations). The horizontal line shows the performance of best policy found during DQN training for 20M frames which is significantly worse than fully-trained DQN.
as the fraction of data increases. With N > 10%, REM and QR-DQN still perform comparably to online DQN on most of these games. However, the performance deteriorates drastically for N = 1%.
To see the effect of quality of offline dataset, we perform another ablation where we train offline REM and QR-DQN on the first 20 million frames in the DQN replay dataset which approximate exploration data with suboptimal returns. Similar to our offline results with the entire replay dataset, on most Atari games, offline REM and QR-DQN perform comparably and outperform the best policy amongst the mixture of policies which collected the first 20 million frames (Figure 6 and Figure A.9
6 DISCUSSION AND FUTURE WORK
Since the (observation, action, next observation, reward) tuples in DQN replay dataset stored in the order they were experienced by online DQN during training, various data collection strategies for benchmarking offline RL can be induced by subsampling the replay dataset containing 200 million frames. For example, the first k million frames from the DQN replay dataset emulate exploration data with suboptimal returns (e.g.. Figure [ while the last ic million frames are analogous to near- expert data with stochasticity. Another option is to randomly subsample the entire dataset to create smaller offline datasets with sufficient diversity (e.g., Figure 5). Based on the popularity and ease of experimentation on Atari games, we hope that the DQN replay dataset can be used for benchmarking offline RL. in addition to continuous control setups Fujimoto et al.] (2019): Kumar et al. (2019).
Offline RL tackles the challenge of squeezing the most performance gains out of a fixed batch of data. This indicates the utility of offline RL for creating more sample-eficient RL agents as an online agent should exploit the data collected prior to any given time during online training as much possible before further collecting new data via exploration. The proposed offline benchmark can be utilized for developing better RL algorithms for various data regimes shown in Figure 5
In our offline RL setup, the agents exhibit overfitting on some games (Figure 6). even with 50 million tuples (Figure (..7). (..). i.e., after sufficiently large number of gradient updates, their performance degrades. We also observe divergence w.r.t. Huber loss in our reduced data experiments with N = 1%. Based on these findings, we posit that our offline benchmark can be used for designing techniques akin to regularization for stable pollcy deep RL algorithms with reasonable performance throughout learning (Gareıa and Fernández 2015). a requirement for many practical RL applications.
Finally, investigating Rainbow components, e.g., prioritized replay (Schaul et al.) 2016). n-step updates (Sutton, 1988). double Q-learning (Van Hasselt et al. (016). etc., the offline RL setting and incorporating them with REM are interesting directions. Experience replay-based algorithms can be more sample efficient than model- based approaches (Van Hasselt al., 2019), and using the DQN replay dataset on Atari 2600 games for designing nonpamamet replay models (Pan et al.) 2018) and parametric world models (Kaiser et al. 2019) is quite promising for improving sample-eficcency, We also leave further investigation of the exploitation ability of distributional RL to future work.
7 RELATED WORK
Our work is motivated by whether the complexity of recent off-policy approaches (e.g., Rainbow (Hes- sel et al., )018)) is necessary and is similar in spirit to work by RRaessaran et al. (2017). which attempts to demystify the complexity of on-policy deep RL for continuous control. Our work is mainly related to two subareas of RL known as batch RL and distributional RL. For the sake of clarity, refer to batch RL as offline RL in other sections of the paper.
8

---

Under review as a conference paper at ICLR 2020
Batch Reinforcement Learning. While there has been increasing interest in batch RL (Lange et al. 2012) over the last few years (Jiang and Li 2 Farajtabar et al. 2018 Irpan et al., 2019). 9) much of this focussed on off- off-policy policy evaluation, where the goal is to estimate the performance of a given fixed policy. Similar to (Ernst et al. 2005 Kalyanakrishnan and Stone 2007; Jaques et al. (019). we investigate batch off- off-policy learning, which requires learning good policy given fixed dataset. In our offline setup, we only assume access to samples from the behavior policy and focus on Q-learning methods without any importance sampling correction as opposed to (Swaminathan and Joachims
2015; Liu et al. 2019). Recent work (Fujimoto et al., 2019 Kumar et al. (2119) documents that standard off- policy methods with fixed datasets fail on continuous control MuJoCo tasks. Opposed to Kumar et al. (2019), we focus the offline setting with data collected from a diverse mixture of policies rather than a single Markovian behavior policy. Furthermore, in contrast to Fujimoto et al. (2019), our batch learning results on MuJoCo tasks and Atari 2600 games (Section 3) demonstrate that standard off-policy deep RL algorithms (e.g., TD3 (Fujimoto et al. 2018). QR-DQN (Dabney et al. 2018b)) are quite effective when learning truly off-policy from large and diverse datasets. Zhang and Sutton 2017) show that large replay buffers can hurt the performance of simple Q-learning methods with weak function approximators; they attribute this the "off- "off-policyness" of large buffer which might delay important transitions for learning. However, our results reveal that even with uniform sampling, effective deep Q-learning algorithms (e.g.. REM) can exploit completely off off-policy logged DQN data on Atari 2600 games, given sufficient number of gradient updates.
Distributional Reinforcement Learning. Recently, distributional RL algorithms (e.g., C51 (Belle- mare et 2017). QR-DQN (Dabney et al. (0118)), IQN (Dabney et al. [00 8ab) gained popularity due to their strong performance on Atari 2600 games. We observe that distributional algorithms (e.g. QR- DQN, C51) are quite effective at exploitation and develop much simpler REM (Section 4.2), which as successful as BR-DON. In contrast to distributional RL, much of the theoretical results of Q-learning naturally extends to REM due to its similarity to Q-learning, e.g,, in the tabular setting, REM converges to Q', while popular distributional methods (e.g., QR-DQN, C51) need not learn true return statistics corresponding to the underlying MDP (Rowland et al. 2019).
8 CONCLUSIONS
This work investigates off-policy deep RL using an offline RL benchmark on Atari 2600 games based on logged experiences of a DQN agent. We demonstrate that is possible learn policies with high returns that significantly outperform the policies used to collect replay data, We develop REM, simple variant of ensemble Q-learning, that can effectively exploit large-scale off-policy data and shares much of the success of the more complicated distributional RI. algorithms on Atari 2600 games. The proposed benchmark can serve as testhed for designing simple, stable and sample-efficient RI. algorithms and enable the research community to evaluate off-policy methods common ground. As mentioned by /Sutton and Barto (2018, Chapter 11.10): "The potential for off-policy learning remains tantalizing, the best way to achieve it still mystery."
REFERENCES
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqn: Variance reduction and stahilization for deep reinforcement learning. ICMI, 2017.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. Machine Learning, 1995.
Mare Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade lcarning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 2013.
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. ICMI., 2017.
Mare G Bellemare, Nicolas Le Roux, Pablo Samuel Castro, and Subhodeep Moitra, Distributional reinforcement learning with linear function approximation. arXiv preprint arXiv:1902.0314 2019.
Richard Bellman. Dynamic Programming. Princeton University Press, 1957.
Léon Bottou, Jonas Peters. Joaquin Quiñooneo- Cuinoneeo-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Fd Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. JMI.R, 2013.

---

Under review as a conference paper at ICLR 2020
Justin A Boyan and Andrew W Moore. Generalization in reinforcement learning: Safely approximating the value function. NeurIPS, 1995.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech 7aremba. OpenAl Gym. arXiv preprint arXiv: (606:06000005500 2016.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Mare G Bellemaree Dopamine: A research framework for deep reinforcement learning. ArXtv//8/2.061 2018.
Kaleigh Clary, Emma Tosch, John Foley, and David Jensen. Let's play again: Variability of deep reinforcement learning agents in atari environments. arXiv preprini arXiv:1904.063/2, 2019.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. ICML, 2018a.
Will Dabney, Mark Rowland, Marc G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. AAAI, 2018b.
Sudeep Dasari, Frederik Ebert. Stephen Tian, Suraj Nair. Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. CoRL, 2019.
Tim De Bruin. Jens Kober, Karl Tuyls, and Robert Babuška. The importance of experience replay database composition in deep reinforcement learning. Deep reinforcement learning workshop, NIPS, 2015.
Gabriel Dulac-Amold Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforeement learning. arXiv preprint arXiv: 1904. 12901, 2019.
Damien Emsst Pierre Geurts, and Louis Wchenkel. Tree-based batch mode reinforcement learning. JMLR, 2005.
Mehrdad Farajtahar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. JCML,, 2018.
Stefan Faußer and Friedhelm Schwenker. Neural network ensembles reinforcement learning. Neural Processing Letters, 2015.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error actor-critic methods. ICML, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. ICML, 2019.
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. JMLR, 2015.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- maximum entropy deep reinforcement learning with a stochastic actor, arXiv preprint 1rXiv:1801.01290, 2018
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. 2018
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. AAAI, 2018
PJ Huber. Robust estimation of location parameter. Ann. Math. Stat., 1964.
Alex Irpan, Kanishka Rao, Konstantinos Bousmalis. Chris Harris, Julian Ibarz, and Sergey Levine. Off-policy evaluation via off-pelicy classification. NeurlPS, 2019.
Natasha Jaques, Asma Ghandeharioun. Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza. Noah Jones. Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00445 2019.
Stratton C Jaquette. Markov decision processes with new optimality criterion: Discrete time. The Annals of Statistics, 1973.
Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. ICML, 2016.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski. Dumitru Frhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Model-based reinforcement learning for atari, arXiv preprint arXiv: 190300344 2019.
10

---

Under review as a conference paper at ICLR 2020
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, QT-Opt: Scalable deep reinforcement learning for rision-based robotic manipulation. CoRL, 2018.
Shivaram Kalyanakrishnan and Peter Stone. Batch reinforcement learning in a complex domain. AAMAS, 2007.
Khimya Khetarpal, Zafarali Ahmed, Andre Cianflone, Riashat Islam, and Joelle Pincau. Re-ovaluate: Repro- ducibility evaluating reinforcement learning algorithms. 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Roger Koenker. Quantile Regression. Cambridge University Press, 2005.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing Off-Policy Q-L.eamnng via Bootstrapping Error Reduction. NeurlPS, 2019.
Sascha Lange. Thomas Gabel, and Martin Riedmiller. Batch reinforcement leamnin. Reinforcement learning. 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applicd to document recognition. Pruceedings of IEEE, 1998.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. and-to-end training of deep visuomotor policies. JMLR, 2016.
Timothy P Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning.
2015.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 1992.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with state distribution correction. arXiv preprint
2019.
Clare Lyle, Pablo Samuel Castro, and Mare G Bellemare. A comparative analysis of expected and distributional reinforcement learning. AAAI, 2019.
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 2018.
Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation across representations with applications educational games. AAMAS, 2014.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learming. arXiv:13/2.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement leamning Nature, 2015.
Volodymyr Maih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. JCML, 2016.
OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojeiech Zaremba. Learning dexterous in-hand manipulation. CoRR, 2018.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. NewrIPS, 2016.
Yangchen Pan, Muhammad Zaheer, Adam White, Andrew Patterson, and Martha White. Organizing experience: deeper look at replay mechanisms for sample-based planning in continuous state domains. IJCAI, 2018.
Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994.
Aravind Rajeswaran, Kendall I. Lowrey, Emanuel V Todorov, and Sham M Kakade. Towards generalization and simplicity in continuous control. NeurlPS, 2017.
11

---

Under review as a conference paper at ICLR 2020
Martin Riedmiller. Neural fitted q iteration-firs, experiences with data efficient neural reinforcement learning method. In European Conference on Machine Learning, 2005.
Mark Rowland, Mare G Bellemare, Will Dabney, Rémi Munos, and Yee Whye Teh. An analysis of categorical distributional reinforcement learning, AISTATS, 2018
Mark Rowland, Robert Dadashi, Saurabh Kumar, Rémi Munos, Mare G Bellemare, and Will Dabney, Statistics and samples distributional reinforcement learning. ICML, 2019.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. ICLR, 2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. ICML., 2015.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-expert. layer. arXiv: 0003100010038,8. 2017.
Susan M Shortreed, Fric Laber, Daniel J Lizotte, T Scott Stroup, Joelle Pineau, and Susan A Murphy. Informing sequential clinical decision- makinn through reinforcement learning: empirical study. Machine learning. 2011.
David Silver, Aja Huang, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. JMLR, 2014.
Alexander Strehl, John Langford, Lihong Li, and Sham Kakade. Learning from logged implicit exploration data. NeurlPS, 2010.
Richard S Sutton. Learning predict by the methods of temporal differences. Machine learning, 1988.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. NewrIPS, 2000.
Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. JMLR, 2015.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5 rms-roppp Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learming, 2012.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. IROS, 2012.
John N Tsitsiklis and Benjamin Van Roy. Analysis of temporal-difffereee difffffrrfffeee learning with function approximation. NewrIPS, 1997.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q-leaming, AAAI, 2016.
Hado Van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? NearlPS, 2019.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine Learning, 1992.
Shangtong Zhang and Richard S. Sutton. deeper look experience replay. arXiv preprint 1711:117 2017.
12

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.I PROOFS
Proposition 1. Consider the assumptions: (a) The distribution Pa has full support over the entire (K - -)-siiplex (b) Only a finite number of distinct Q-functions globally minimize the loss (3). (c) Q" is defined in terms of the MDP induced by the data distribution D. (d) Q" lies in the family of our function approximation. Then at the global minimum of (()) for multi-head Q Q-network:
(i) Under assumptions (a) and (b), all the Q-heads represent identical Q-functions.
(i) Under assumptions (a)-d), the common convergence point is Q"
Proof. Part (i): Under assumptions (a) and (b), we would prove by contradiction that each Q-head should be identical to minimize the REM loss (()) (7). Note that we consider two Q-functions to be distinct only they differ on any state 8 in D.
The REM loss L(0)
Banpa pa [A(a, 0)] where L((x, 0) is given by
C(a, 0) Be ls Σ -)) - - max Σ Eouobie,a'))
(8)
If the heads Q8 and O/ don't converge to identical Q-values at the global minimum of L(0), it can be deduced using Lemma that all the Q-functions given by the convex combination a, + Q such that a; + jjj minimizes the loss in (3). This contradicts the assumption that only finite number of distinct Q Q-functions globally minimize the loss in (3). Hence, all Q-heads represent an identical Q- Q-function the global minimum of L(0).
Lemma 1. Assuming that the distribution P has full support over the entire (K - 11)--mmpel A K-1 then at any global minimum of L(0), the Q-function heads Q4 for 1, l,...,KK K minimize L(a, for any AK-1
Proof. Let Qa Ex=1 of 2 (s, corresponding to the convex combination a" (aj. ... .ak) represents one of the global minima of C(a, 8) (8) i.e. L(a",0) = min C(a, 0) a.8
where a E AK-1, Any global minima of ((0) attains a value of C(a".0") or higher since,
[(0) = Earpa [C(xx, 0)] 2 Eamps [C(a", ,0)) 2 C(a",0"
Let Q3- (s, = wy. fo- a) where fo- E R represent the shared features among the Q-heads and wy- R² represent the weight vector in the final layer corresponding the th head. Note that Qa-.o- can also be represented by each of the individual Q-heads using weight vector given by convex combination a of weight vectors (w).,.. ,w ) iee. Q(s, = (EK (e - as,
Let 8¹ be such that Qh: = Qa
.0 for all Q-heads. By definition of Qa ,0, for all a ~ PA- L(a, 0') L(a", 0²) which implies that L(0') L(a" 0*). Hence, of corresponds to one of the global minima of [(0) and any global minima of L(0) attains value of C(a", A*))
Since L(a, 0) > L(a", ((") for any E AK-1, for any 0M such that L(0M) = L(a", implies that L(a,, (²H) L(a", 0*) for any a PA. Therefore, at any global minimum C(0), the Q-function heads Q3 for k = 1, ..... K minimize C(a, 8) for any A
A.2 OFFLINE CONTINUOUS CONTROL EXPERIMENTS
We replicated the final buffer setup as described by Fujimoto et al. (201)): We train DDPG (Lillicrap) ct al. 2015) agent for million time steps three standard MuJoCo continuous control environments in OpenAl gym (Todorov et al. 2012 Brockman et al. 2016), adding (0, 0.5) Gaussian noise to actions for high exploration, and store all experienced transitions. This collection procedure creates dataset with diverse set of states and actions, with the aim of sufficient coverage. Similar to Fujimoto et (2019), we train DDPG across 15 seeds, and select the 5 top performing seeds for dataset collection.
13

---

Under review as a conference paper at ICLR 2020
Offi
Offine D2nG
Offine 500
DDING
Walkerdd
4000
50003
30003
01003
20003
Anerap
1030
1000
0.50
0.25
0,50
1.00
0.00
0.25
0.73
1.50
0.00
1125
01.00
6.75
1.00
Time Stopa (in 1c5)
Time stogs (In LoG)
Time stops (fr 208)
Figure A.I: We examine the performance of DDPG, TD3 and BCQ trained using identical offline data three standard MuloCo environments. We plot the mean performance (evaluated without exploration noise) across runs. The shaded area represents a standard deviation. The bold black line measures the average return of episodes contained in the offline data collected using the DDPG agent (with exploration noise).
Using this logged dataset, we train standard continuous control off-policy actor-critic methods namely DDPG and TD3 (Fujimoto et al.) 2018) completely offline without any exploration. We also train a Contt--Contt deep Q-learning (BCQ) agent, proposed by FFujimoto et al. (2019). which restricts the action space force the offline agent towards behaving close to on-policy w.r.t. a subset of the given data. We use the open source code generously provided by the authors at htttps:: /tt:::/ggttttae com/ s fu jim/ BCQ and https: /ttts://gittull /ggttub......... com/ suu fim TD3, We
use the hyperparameters mentioned in (Fujimoto et al. 2018 2019) except offline TD3 which uses a learning rate of 0.0005 for both the actor and critie.
Figure 1 show that offline TD3 significantly outperforms the hehavior policy which collected the offline data as well the offline DDPG agent. Noticeably, offline TD3 also performs comparably to BCQ, an algorithm designed specifically to learn from arbitrary, fixed offline data. While Fujimoto al. (2019) attribute the failure learn in the offline setting to extrapolation error (i.e., the mismatch between the offline dataset and true state-action visitation of the current policy), our results suggest that failure to learn from diverse offline data may be linked to extrapolation error for only weak exploitation agents such as DDPG.
A.3 SCORE NORMALIZATION
The improvement in normalized performance of an offline agent, expressed as a percentage, over an online DQN (Nature) (Mnih et al. 2015) agent is calculated as: 100 x (ScoreNormalized - 1) where:
ScoreAgent min min(ScoreDQN; Scorenandom)
ScorCNormalized max max (ScorepQN. ScoreRaudom) min(ScorepqN, ScoreRandom)
(10)
Here, the scores are the mean evaluation scores averaged over 5 runs. We chose not to measure performance in terms of percentage of online DQN scores alone because a tiny difference relative to the random agent on some games can translate into hundreds of percent in DQN score difference. Additionally, the max is needed since DQN performs worse than a random agent on the games Solaris and Skiing.
A.4 HYPERPARAMETERS & EXPERIMENT DETAILS
In our experiments, we used the hyperparameters provided in Dopamine baselines (Castro al. 2018) and report them for completeness and ease of reproducibility in Table As mentioned by Dopamine's GitHub repository, changing these parameters can significantly affect performance, without necessarily being indicative of an algorithmic difference. We will also open source our code to further aid in reproducing our results.
The Atari environments (Bellemare et al. 2013) used in our experiments are stochastic due to sticky actions (Machado et al., 2018), í.e., there is 25% chance at every time step that the environment will execute the agent's 's previous action again, instead of the agent's 'ss new action. All agents (online or offline) are compared using the best evaluation score (averaged over runs) achieved during training where the evaluation is done online every training iteration using e-greedy policy with 0.001. We report offline training results with same hyperparameters over 5 random seeds of the DQN replay data collection, game simulator and network initialization.
14

---

Under review as a conference paper at ICLR 2020
Table 2: The hyperparameters used the offline and online RL agents in experiments.
Hyperparameter
setting (for both variations)
Sticky actions
Yes
Sticky action probability
0.25
Grey-scaling
True
Observation down-sampling
(84, 84)
Frames stacked
Frame skip (Action repetitions)
4
Reward clipping
[-]. I]
Terminal condition
Game Over
Max frames per episode
108K
Discount factor
0.99
Mini-batch size
32
Target network update period
every 2000 updates
Training steps per iteration
250K
Update period every Evaluation
steps
0.001
Evaluation steps per iteration
125K
Q network:: channels
32, 64, 64
Q network: filter size
8, 4 4,
Qentwwrkk: stride
1
Q-network: hidden units
512
Multi-head Q-ntwwrkk: number of Q-heads
200
Hardware
Tesla P100 GPU
Hyperparameter
Online
Offline
Min replay size for sampling
20,000
Training (for e-greedy exploration)
0.01
e-decay schedule
250K steps
Fixed Replay Memory
No
Yes
Replay Memory size
1,000,000 steps
50,000,000 steps
Replay Scheme
Uniform
Uniform 200 1000
Training Iterations
200
DQN replay dataset collection. For collecting the offline data used in our experiments, we use online DQN (Nature) (Mnih al.) 2015) with the RMSprop (Tieleman and Hinton (012) optimizer. The DQN replay dataset, BDQN. consists of approximately 50 million experience tuples for each run per game corresponds 200 million frames due to frame skipping four, i.e., repeating selected
action for four consecutive frames. Note that the total dataset size is approximately 15 billion tuples (
50 million tuples agent
5
agents
60 games).
game
Optimizer related hyperparameters. For existing oof-po agents, step size and optimizer were
taken as published. Offline DQN (Adam) and all the offline agents with multi-head Q-network (Fig- use the Adam optimizer (Kingma and Ba 2015) with same hyperparameters as online QR- DDbney et al. 2018b) (Ir 0.00005, Adam 0.01/32). Note that scaling the loss has the same effect as inversely scaling CAdam when using Adam.
ure 3
DQN
Online Agents. For online REM shown in Figure [b. we performed Syper-parameter tuning over Adam in (0.01/32, 0.005/32, 0.001/32) over 5 training games (Asterix. Breakout, Pong. Bert, Seaquest) and evaluated on the full set of 60 Atari 2600 games using the best setting (Ir = 0.00005, CAdam 0.001 132). Online REM uses 4 Q-value estimates calculated using separate Q-networks where each network has the same architecture as originally used by online DQN (Nature). Similar to REM, our version of Bootstrapped-DON also uses separate Q- -entwwrrks and Adam optimizer with identical hyperaparmeters (Ir = 0.00005, EAdam = 0.001/32).
Wall-clock time for offfine experiments. The offline experiments are approximately 3X faster than the online experiments for the same number of gradient steps on a P100 GPU. In Figure and a), the offline agents are trained for 5X gradient steps, thus, the experiments are 1.67X slower than running online DQN for 200 million frames (standard protocol). Furthermore, since the offline experiments do not require any data generation, using tricks from supervised learning such as using much larger batch sizes than 32 with TPUs / multiple GPUS would lead to a significant speed up.
15

---

Under review as a conference paper at ICLR 2020
A.5
ADDITIONAL PLOTS & TABLES
Inprestont Dlg
onakl
DQy
D00
Figure A.2: Normalized Performance improvement (in %) over online DQN (Nature), per game, of (a) offline Sinemblc-DQN and (b) offline REM trained using the DQN replay datasct for same number of gradient steps online DQN. The normalized online score for cach game is 0.0 and 1.0 for the worse and better performing agent among fully trained online DQN and random agents respectively.
Inpreement dag seal
Diflinn qe 1ggN (50)
Dffiine RFM (X0)
asp
Toggroennn dlag
Fanders
Game
Game
(a)
(b)
Figure A.3: Normalized Performance improvement (in %) over online DQN (Nature), per game, of (a) offline QR-DQN (5X) (b) offline REM (5X) trained using the DQN replay dataset for five times many gradient steps online DQN. The normalized online score for each game 0.0 agent among fully trained online DQN and random agents respectively.
the worse and better performing
Offline (Cature)
Office CS1
Daline
ASTRRIIX
BuXAKOUT
Average Store
3003
800
150
3000
30
2000
50
-10-
1000
20
0-
50
100
200
100
200
100
150
200
Iteratioo
Iteration
Iteration SPACK: TNVADERS
14000
Q
BRAQUEST
12003-
Avurage Soore
10000
100301-
3000
5003
10000-
2000
0003
5000
2000
a000
15.0
1000
300
-0000
30
1001
1741
2001
50
103
130
2003
100
150
200
Iteration
Iteration
Iteration
of C51 and DQN (Nature) agents trained offline stochastic version Atari 2600 games using the DQN replay dataset same number of gradient steps as the online DQN agent. The scores are averaged over runs (shown as traces) and smoothed over a sliding window of 5 iterations and error bands show standard deviation.
Figure 4: Offline C51 Offine DQN (Nature). Average online
16

---

Under review as a conference paper at ICLR 2020
Offinne Multi network REM
Offinne Multi REM
Offinee QR DR-DQN
5000
AStERiX
450
BREAROUT
Powa
4260-
430
150
Average
1260-
1200
250
730
1200-
350
-10-
2000
300
300
301
-00
105
1500
500
103
-00
500
Itoration
Iterationn SEAQUERS
Iteration
QEBa
SPACE INVADERS
Areesgee Scores
34200-
07000
14000
10000
13903
8300
9030
10000
1300-
4030
3000
002
200 Iteration
Herratoo
Iseration
(a) REM with 4 Q-value estimates (K 4)
Offline Multi-netwaark RRM
Offline Multi-head REM
Offline QR-DEN
ASTRRIX
BEEA
PONG
Average Soores
8000
12001
2381
200
10--
.000
1000
0-
50
300
2001
300
300
Iteration
Iteration SEAQUEST
Iteration SPACE INVADERS
"EER
10000
10000
20000
10000
Averago Scorea
15400
(1301
6000
0181
2000
2001
100
200
500
400
200
100
200
300
400
500
100
230
300
400
Iteration
Iteration
Iteration
(b) REM with 16 Q-value estimates (K 16)
Figure A.5: REM with Separate Q-networks. Average online scores of offline REM variants with different architectures and QR-DQN trained on stochastic version of 6 Atari 2600 games for 500 iterations using the DQN replay dataset. The scores averaged over 5 rums (shown traces) and smoothed over sliding window of 5 iterations and error bands show standard deviation. The multi-network REM and the multi-head REM employ K Q-value estimates computed using separate Q-networks and Q-heads of multi-head Q-network respectively and are optimized with identical hyperparameters. Multi-network REM improves upon the multi-head REM indicating that the more diverse Q-estimates provided by the separate Q-networks improve performance of REM over Q e-estimaaess provided by the multi head Q-nttwork with shared features.
Table Median normalized scores (Section A.3 across stochastic version of 60 Atari 2600 games, measured percentages number of games where an agent achieves better scores than fully trained ouline DQN (Nature) agent. All the offline agents below are trained using the DQN replay dataset. The entries of the table without any suffix report training results with the five times as many gradient steps online DQN while the entires with suffix (1zz) indicates the same number of gradient steps the online DQN agent. All the offline agents except DQN the multi-head architecture QR-DQN.
Offline agent
Median
DQN
Offiin agent
Median
DQN
DQN (Nature) (1æ)
74.4%
10
DQN (Nature)
83.4%
17
DQN (Adam) (Iz)
104.6%
39
DQN (Adam)
111.9%
41
Ensemble-DQN (1x)
92.5%
26
Ensemble-Doe
111.0%
39
Averaged Ensemble-DQN (1x)
88.6%
24
Averaged Ensemble-Doe
112.1%
43
QR- BR-DQN (1z)
115.0%
44
QR-DQN
118.9%
45
REM (1z)
103.7%
35
REM
123.8%
49
17

---

Under review as a conference paper at ICLR 2020
AMTIAN
AUTESNID
100
Iteration
Horeiten
Joreite
beralie
Jarsike
COUTFES COMDAND
New
Deratian
Terwiee Plassen Sicuer
Detrmut Drtos
Elansmee Actson
Iocation
Zorrsticn
Homeiton Presaat
Elemstioon
StatE Tata GAME
Pooro
PREsAr Era
Iocatioc Qriiee
Heratie
Etaratioo
Riven Rom
Roos Raressos
Serres
SOLATI
SHArE INMADERS
STAR CUUNNER
TIMe PRLCT
AND
VESTURE
VILEO POTBALL
WIZARD
TAKE'S
Figure A.6: Average evaluation scores across stochastic version of 60 Atari 2600 games for online DQN, offline DQN and offline QR-DQN trained for 200 iterations. The offline agents are trained using the DQN replay dataset. The scores are averaged over runs (shown as traces) and smoothed over a sliding window iterations and error bands show standard deviation. The horizontal line shows the performance of the best policy (averaged over 5 runs) found during training of online DQN.
18

---

Under review as a conference paper at ICLR 2020
DARABOUT
CALIRAL
Commisse
Cance
ELEKATOR ACTION
BEEOO.
Janre Desa
Surrice
Bonttee
Jucizes Socurr
Htmatton
Sersiee
Honstto
Tare
Era
Toratio
Sernics
Bacation
Qriirer
Rrves Ram
Rown
Horsiiee
Iteration
Berative
STORE GUTEOE
Tran PILOT
Daradia PENNALL
Zureice Wir
AND Douky
Figure A.7: Average evaluation scores across stochastic version of 60 Atari 2600 games of DQN (Adam), DQN,, QR-DQN and REM agents trained offline using the DQN replay dataset. The horizontal line for online DQN show the best evaluation performance it obtains during training. All the offline agents except DQN use the same multi-hn head architecture with K = 200 heads. The scores are averaged over 5 runs (shown as traces) and smoothed over a sliding window 5 iterations and error bands show standard deviation.
19

---

Under review as a conference paper at ICLR 2020
DederapeoDDg
Halt
ALIES
ADTESNID
Horskonn
Serviles
Hemation
Jomslee
BATTLE ZoweE
benike
Deretica
Dacatoon
Ioralice
Duratioo
Cheocy Cuansoo
Dervites Docras Donk
Jurslie EXCERD
Derrttin TEEIIOO Dacory
Himstoon
DEstoN ArtAcc
Fiocenar
Deeratin
Dianation Zuarns Bortr
Provmmmee
31 S.B.K.
Ice Sooreee
JCCISSS Sooure
KUNG-U MARTER
MONTEROME' REVENGGE
MS.. PSU-MAN
NAME Tats CAME
P030
Pootual
10
Dereiters
Tomsiton
Derrstic
Jomtlc
Etemationn
Figure A.8: Online results. Average evaluation scores across stochastic version of 60 Atari 2600 games of DQN. C51, QR-DQN, Bootstrapped-DQN and REM agents trained online for 200 million game frames (standard protocol). The scores are averaged over 5 runs (shown as traces) and smoothed over a sliding window of 5 iterations and error bands show standard deviation.
20

---

Under review as a conference paper at ICLR 2020
Ali BAll
ALLEn
AUCERADOO
0000
200
Parsttin
Tomaatinn
Joratce
20100
200
Bendias
Temion
Then
1900-
moo
-10--
GRAFITA
Slocnac
ZASIO BAND
JUsTINN Socare
KANGAROO
KING-E MArTER
MOWTESiMen REVENGE
2000
325
20-
180
X0
300
NAME Tare Guare
Possc
Permin
1000-
190
BoaD
BCCOWANK
2001
Sanso
berstien
3.403
2010
Thwn
2000
10207
200
201-
Anin Bove
2000
Figure A.9: Effect of Dataset Quality. Normalized scores (averaged over 3 runs) of QR-DQN and multi-head REM trained offline on stochastic version of 60 Atari 2600 games for 5X gradient steps using logged data from online DQN trained only for 20M frames (20 iterations). The horizontal line shows the performance of best policy found during DQN training for 20M frames which significantly worse than fully-traines DQN.
21


Reviews:
[{"review_id": "SylKrse19r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#rebuttal response\nThanks for your response and the additional experiments. I do think it is a very interesting paper but I keep my score. First, the improvement of their model is not very significant both in Table 1 and the standard RL setting. Second, I am still not clear about how off-policy learning benefits the study on standard RL. Since exploration and exploitation interact with each other, disentanglement between them cannot reflect the original difficulty of RL problems. Different exploration strategies or balancing methods between exploration and exploitation will produce different data. I guess this is why the data generation process is a common concern raised by both Reviewers 1 and 2. I think the response of the authors is not enough to clarify this. In addition, I cannot find Figures 4a and 4b. I think this paper is not ready for publication right now but I encourage the authors to improve this work and resubmit it in the future.\n\n\n#review\n\nThis paper collects a replay dataset of Atari to propose a benchmark for offline (batch) reinforcement learning. Experiments on this dataset show that offline RL algorithms can outperform online DQN. Then, this paper presents a new off-policy RL algorithm, which uses random convex combinations of multiple networks as the Q-value estimator. The experimental results show that it outperforms distributional RL algorithms (i.e., C51 and QR-DQN) in the offline setting and performs comparably in the online setting. \n\nThe idea of randomly combining Q networks is interesting and paper is well written to demonstrate their key ideas. However, I have some concerns.\n\nWhy are off-line RL algorithms necessary? I am not sure this paper is relevant to the community of standard RL. This paper does not show the usage of their method to a standard RL agent. I am happy to change my score if the authors can clarify the motivation of studying the offline performance of conventional RL algorithms and clearly show the improvement over baseline algorithms in a standard RL setting rather than only show insights for potential usage in online RL.\n\nIn my opinion, the results are not significant to support their claims. Such a small gap (4.9% as shown in Table 1) over baseline models may result from considerable hyper-parameter tunning. In addition, the figures and tables in this paper do not show deviations and confidence intervals.\n\nSince REM has similar architectures with attention networks, it will be better to include an attention-based Q network as a baseline model.\n\nIn the right sub-table of Table 1, I wonder why the performance of online QR-DQN in this paper is much lower than that reported in the original papers (media is about 200%).\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SyeV_0S6tr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary*\n\nPaper tackles an important issue, which is benchmarking off-policy deep RL algorithms. The motivation is important, because 1) assessing the performance of off-policy RL algorithm is hard due to the mixed effect of exploration and exploitation while collecting data; hence the paper suggest to fix the data collection process by a standard algorithm (DQN) and then compare the off-policy methods. 2) A fixed dataset will help the reproducibility of the results. \n\nAdditionally the paper assess the performance of some off-policy RL methods (mainly DQN and QR-DQN) and showed that given offline data, these algorithms can obtain a superhuman performance. \n\nBesides the mentioned contribution, authors strived for a simple algorithm (easy to implement), and suggests a new simple algorithm, Random Ensemble Mixture (REM) by training multiple Q-functions and randomly sampling a subset for ensemble estimate. The mina motivation come from supervised learning literature. \n\n*Decision*\n\nI enjoyed reading the paper and I think authors tackled a very important problem. Although a very important step, I vote for a weak reject for this paper as I believe the contribution of the current paper is limited. I made my decision mainly based on the following points:\n\n[Limited Task Diversity]: I agree with the authors that having a fix offline data for testing different algorithms is important; however the suggestion (DQN logged data on Atari2600) is a very limited dataset, in terms of 1) task diversity 2) Data collection strategy. \nFor a benchmark, I would like the dataset to have different strategies for data collection, so that the effect of data collection can be stripped off. In the current assessment, we compare the performance of algorithms condition on the DQN e-greedy data collection procedure. (For example, the right claim to make is: QR-DQN showed a better performance condition on the current data generation process). And it is not clear to me the same observation will hold given a different data generation process (for example a random policy, or a human playing data, or …)\nAlso, I think only focusing on Atari tasks is limited in nature, as many of these games share a similar underlying mechanism. \n\n\n[Overfitting] I believe having a fixed dataset for testing off-policy RL is great, however the current suggestion is very prone to overfitting. I can foresee that if the community start to use this benchmark as a test case, soon we will overfit to this task, in a sense that I try algorithm X on offline data, test on Atari see the results, tune algorithm X test on Atari again, … This way I finally have an algorithm that only learned from offline data, but is performing great on online Atari. But I am overfitting!\n\n\n[Simplicity Trade-Off] Authors focused on getting a simple algorithm, but to me it is unclear why should we optimize for simple algorithm, and my impression of simple as authors have in mind is a “easy to implement” algorithm. \nSimplicity is great, but usually I believe we seek a simple algorithm to be able to get more insight into the theory and build more intuition. However, in this work there is a lot emphasis on simple algorithm, but no extra significant insight into off-policy learning has been gained. Additionally, I found the suggested algorithm ad-hoc and of limited contribution. \n\n\nMinor Comments/ Question that did not significantly contribute to my decision: \n\n[Valid Q-function]: authors mentioned a convex combination of Q-values are a valid Q-value, but never defined a “valid” Q-value. Do authors mean, if Q1 is generated by \\pi_1 and Q2 by \\pi_2 then convex combination of Q1 and Q2, Q^* can be also generated by some policy \\pi^*? \nIf so, this needs clarification. \nAnd if so, I’m not sure why this is important, since in Q-learning the Q-values that we obtain before convergence might be “invalid” (by the definition above, so that no real policy can generate those values exactly), so what is the point of paying attention to a valid Q values in the first place?\n\n*Improvement*\n\nI think some clear improvement avenues for the future are\n1. More diverse set of tasks: I would like to see some non-Atari tasks included in the dataset.\n2. More diverse data collection strategy: Example can be different exploration strategies than e-greedy. Or even random exploration, or maybe data of human players. Since we don’t want to have an algorithm that is good only for state distribution induced by e-greedy exploration.\n3. More diverse task will help with overfitting by itself. But a good benchmark needs a test case that cannot be exploited. \n4. More insight into why REM is outperforming QR-DQN or other algorithms. Currently I am not convinced that we understand why REM is outperforming, or if it’s only because we designed it to be good for this specific task and dataset? \n\n\nGenerally, I believe authors made a very important step, and I really enjoyed reading the paper; however, I think the current contribution is not sufficient to merit a publication. This makes a good workshop paper at this point.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylZbsQTYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigated the offline settings of deep Q-networks, and provided relevant comparison among different agents. The authors also proposed two DQN variants by using ensembles, and showed the improvement (mainly from the random ensemble mixture) on Atari games, when compared with baselines. \n\nIn general, this paper explored an important problem, i.e., offline vs. online settings in DQNs. The experimental comparison also looked interesting, as it showed the empirical evidence on the influence of such settings. A lot of the experiments were performed to support the claims, which I really appreciate. On the other hand, it's a bit unclear to me whether offline should be preferred over its online counterpart, given the results presented and potential computational cost. For the proposed REM method, further explanation isolating from the offline vs. online setting could be helpful to understand why it works. Furthermore, some settings in the experiments need to be clarified. If the authors can address my concern, I would be willing to raise my score.\n\n1. To my knowledge, using the logged experience to train an offline DQN agent is an interesting exploration. This approach can also achieve competitive performance, when compared with the online agent. My concern is whether the authors had considered the cost due to this offline setting, as additional time needs to be spent on training another agent to obtain such logged data, which is not required in the online methods. Moreover, Table 1 suggests that the online agents usually outperformed their offline counterparts, from which I am somewhat in doubt about whether the offline approaches should be preferred.\n2. I am also wondering how the performance depends on the agent used to generate the logged data. In the current version, it is from the Nature DQN, but what will happen if we use a different (can be either better or worse) agent? For example, how about using C51 to generate the logged data?  I hope the authors can provide more evidence and analysis on why the offline settings should work. \n3. When training the agents with the logged experience, how did you handle the new data generated? I understand the offline setting may not consider this issue, but am just thinking if such data can be better used.  \n4. The ensemble based DQNs proposed in Section 4 are very interesting, which can even be a separate contribution, as they do not depend on the online or offline settings. From this perspective, I am a bit curious if the authors have more explanation on the benefits of such methods, e.g., reducing the overestimation bias and gradient noise, as shown in the averaged-DQN and the following softmax-DQN work, \nSong et al., Revisiting the Softmax Bellman Operator: New Benefits and New Perspective, ICML 2019, \nwhich the authors may refer to in their paper.\n5. For the curves in Figures 1 and 4, how about the corresponding running time? Such information could be helpful for others trying to reproduce the results.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers recommend rejecting the submission. There is no basis for acceptance.
Decision: Reject