Striving for Simplicity in Off-Policy Deep Reinforcement Learning

1 INTRODUCTION . Deep neural networks have become a critical component of modern reinforcement learning ( RL ) ( Sutton and Barto , 2018 ) . The seminal work of Mnih et al . ( 2013 ; 2015 ) on deep Q-networks ( DQN ) has demonstrated that it is possible to train neural networks using Q-learning ( Watkins and Dayan , 1992 ) to achieve human-level performance in playing Atari 2600 games ( Bellemare et al. , 2013 ) directly from raw pixels . Recent progress in mastering Go ( Silver et al. , 2016 ) and advances in robotic control ( Levine et al. , 2016 ; OpenAI et al. , 2018 ; Kalashnikov et al. , 2018 ) present additional supporting evidence for the enormous potential of deep RL . Off-policy RL algorithms such as Q-learning are attractive because they disentangle data collection and policy optimization and offer more sample efficient solutions than on-policy algorithms ( Sutton et al. , 2000 ; Schulman et al. , 2015 ; Mnih et al. , 2016 ) . Importantly , off-policy techniques can leverage the vast amount of existing offline logged data for real-world applications such as digital advertising ( Strehl et al. , 2010 ; Bottou et al. , 2013 ) , education ( Mandel et al. , 2014 ) , and healthcare ( Shortreed et al. , 2011 ) . Since online RL is often unsafe to deploy in the real world , offline RL algorithms are the only feasible solution for many practical decision making problems ( Dulac-Arnold et al. , 2019 ) . Nevertheless , off-policy RL algorithms when combined with neural networks can be unstable or even divergent ( Baird , 1995 ; Boyan and Moore , 1995 ; Tsitsiklis and Van Roy , 1997 ) . In the absence of theoretical guarantees for off-policy deep RL , recent advances ( see Hessel et al . ( 2018 ) for an overview ) are largely governed by empirical results on a popular benchmark suite of Atari 2600 games ( Bellemare et al. , 2013 ) . Unfortunately , these empirical results are difficult to reproduce ( Henderson et al. , 2018 ) and the contribution of novel RL algorithms is often conflated with many other design choices ( Clary et al. , 2019 ; Khetarpal et al. , 2018 ) . Given the empirical nature of deep off-policy RL , it is crucial to come up with simpler and reproducible experimental settings and study the relative importance of different components of deep RL algorithms . It is also important to strive for finding successful RL algorithms that are as simple as possible . This paper advocates the use of offline ( batch ) RL to help ( 1 ) isolate the contributions of exploitation vs. exploration in off-policy deep RL , ( 2 ) improve reproducibility of deep RL research , and ( 3 ) facilitate the design of simpler off-policy deep RL algorithms . To this end , we propose a new benchmark for offline optimization of RL agents on Atari 2600 games using all of the replay data of a DQN agent ( Mnih et al. , 2015 ) . Using this benchmark , we investigate the following questions : 1 . Is it possible to train successful Atari agents based solely on offline data ? 2 . Can one design simple and effective alternatives to intricate RL algorithms in the offline setting ? 3 . Are the insights gained from the offline setting useful for developing effective online algorithms ? The contributions of this paper can be summarized as : • An offline RL benchmark is proposed for evaluating and designing RL algorithms on Atari 2600 games without exploration , based on the logged replay data of a DQN agent comprising 50 million ( observation , action , reward , next observation ) tuples per game . This reduces the computation cost of the experiments considerably and helps improve reproducibility of deep RL research by standardizing training using a fixed offline dataset . The replay dataset used in our experiments will be released to enable offline optimization of RL algorithms on a common ground . • Contrary to recent work ( Zhang and Sutton , 2017 ; Fujimoto et al. , 2019 ) , we find that the logged DQN data is sufficient for optimizing strong Atari agents offline without any environment interaction . For instance , QR-DQN ( Dabney et al. , 2018b ) trained offline on the DQN replay dataset significantly outperforms online DQN ( Mnih et al. , 2015 ) . • A simple and novel Q-learning algorithm called Random Ensemble Mixture ( REM ) is presented , which enforces optimal Bellman consistency on random convex combinations of multiple Q-value estimates . Offline REM , trained using the DQN replay dataset , outperforms more complex RL algorithms such as offline QR-DQN and surpasses the gains from online C51 ( Bellemare et al. , 2017 ) over online DQN ( Figure 1a ) . • We use the insights gained from the offline experiments to develop an online variant of REM , which performs comparably with online QR-DQN ( Figure 1b ) . 2 OFF-POLICY REINFORCEMENT LEARNING . An interactive environment in reinforcement learning ( RL ) is typically described as a Markov decision process ( MDP ) ( S , A , R , P , γ ) ( Puterman , 1994 ) , with a state space S , an action spaceA , a stochastic reward function R ( s , a ) , transition dynamics P ( s′|s , a ) and a discount factor γ ∈ [ 0 , 1 ) . A stochastic policy π ( · | s ) maps each state s ∈ S to a distribution ( density ) over actions . For an agent following the policy π , the action-value function , denoted Qπ ( s , a ) , is defined as the expectation of cumulative discounted future rewards , i.e. , Qπ ( s , a ) = E [ ∑∞ t=0 γtrt ∣∣∣ s0 = s , a0 = a , st ∼ P ( · | st−1 , at−1 ) , at ∼ π ( · | st ) , rt ∼ R ( st , at ) ] . ( 1 ) The goal of RL is to find an optimal policy π∗ that attains maximum expected return , for which Qπ ∗ ( s , a ) ≥ Qπ ( s , a ) for all π , s , a . The Bellman optimality equations ( Bellman , 1957 ) characterize the optimal policy in terms of the optimal Q-values , denoted Q∗ = Qπ ∗ , via : Q∗ ( s , a ) = E [ r + γmax a′ Q∗ ( s′ , a′ ) ∣∣∣ r ∼ R ( s , a ) , s′ ∼ P ( · | s , a ) ] . ( 2 ) To learn a policy from interaction with the environment , Q-learning ( Watkins and Dayan , 1992 ) iteratively improves an approximate estimate of Q∗ , denoted Qθ , by repeatedly regressing the LHS of ( 2 ) to target values defined by samples from the RHS of ( 2 ) . For large and complex state spaces , approximate Q-values are obtained using a neural network as the function approximator . To further stabilize optimization , a target network Qθ′ with frozen parameters may be used for computing the learning target ( Mnih et al. , 2013 ) . The target network parameters θ′ are updated to the current Q-network parameters θ after a fixed number of time steps . DQN ( Mnih et al. , 2013 ; 2015 ) parameterizes Qθ with a convolutional neural network ( LeCun et al. , 1998 ) and uses Q-learning with a target network while following an -greedy policy with respect to Qθ for data collection . DQN minimizes the TD error using the loss L ( θ ) on mini-batches of agent ’ s past experience tuples , ( s , a , r , s′ ) , sampled from an experience replay buffer D ( Lin , 1992 ) collected during training : L ( θ ) = Es , a , r , s′∼D [ ` δ ( Qθ ( s , a ) − r − γmax a′ Qθ′ ( s ′ , a′ ) ) ] , ( 3 ) where lδ is the Huber loss ( Huber , 1964 ) given by ` δ ( u ) = { 1 2u 2 for |u| ≤ δ , δ ( |u| − 12δ ) , otherwise. . Q-learning is an off-policy algorithm ( Sutton and Barto , 2018 ) since the learning target can be computed without any consideration of how the experience was generated . In offline ( batch ) RL ( Ernst et al. , 2005 ; Riedmiller , 2005 ; Lange et al. , 2012 ) , we assume access to a fixed offline dataset of experiences D , without any further interaction with the environment . A family of recent off-policy deep RL algorithms , which serve as a strong baseline in this paper , include Distributional RL ( Bellemare et al. , 2017 ; Jaquette , 1973 ) methods . Such algorithms estimate a density over returns for each state-action pair , denoted Zπ ( s , a ) , instead of directly estimating the mean Qπ ( s , a ) . Accordingly , one can express a form of distributional Bellman optimality as Z∗ ( s , a ) D = r + γZ∗ ( s′ , argmaxa′∈A Q ∗ ( s′ , a′ ) ) , where r ∼ R ( s , a ) , s′ ∼ P ( · | s , a ) , ( 4 ) and D= denotes distributional equivalence and Q∗ ( s′ , a′ ) is estimated by taking an expectation with respect to Z∗ ( s′ , a′ ) . C51 ( Bellemare et al. , 2017 ) approximates Z∗ ( s , a ) by using a categorical distribution over a set of pre-specified anchor points , and distributional QR-DQN ( Dabney et al. , 2018b ) approximates the return density by using a uniform mixture of K Dirac delta functions , i.e. , Zθ ( s , a ) : = 1 K K∑ i=1 δθi ( s , a ) , and Qθ ( s , a ) = 1 K K∑ i=1 θi ( s , a ) . ( 5 ) QR-DQN trains each θi for 1 ≤ i ≤ K to match the 2i−12K -quantile of the target return density ( RHS of ( 4 ) ) using the Huber quantile regression loss ( Koenker , 2005 ; Dabney et al. , 2018b ) . QR-DQN , albeit complex , outperforms C51 and DQN and obtains state-of-the-art results on Atari 2600 games , among agents that do not exploit n-step updates ( Sutton , 1988 ) and prioritized replay ( Schaul et al. , 2016 ) . This paper avoids using n-step updates and prioritized replay to keep the empirical study simple and focused on deep Q-learning algorithms . 3 OFFLINE OPTIMIZATION OF REINFORCEMENT LEARNING AGENTS . Modern off-policy deep RL algorithms perform remarkably well on common benchmarks such as the Atari 2600 games ( Bellemare et al. , 2013 ) and continuous control MuJoCo tasks ( Todorov et al. , 2012 ) . Such off-policy algorithms ( Mnih et al. , 2015 ; Lillicrap et al. , 2015 ) are considered “ online ” because they alternate between optimizing a policy and using that policy to collect more data . Typically , these algorithms keep a sliding window of most recent experiences in a finite replay buffer ( Lin , 1992 ) , throwing away stale data to incorporate most fresh ( on-policy ) experiences . In principle , off-policy algorithms can learn from data collected by any policy , however , Zhang and Sutton ( 2017 ) assert that using a large replay buffer can significantly hurt the performance of Q-learning algorithms , since it can delay rare on-policy experiences that are important for policy learning . This paper revisits offline off-policy RL and investigates whether off-policy deep RL agents trained solely on offline data can be successful . We advocate the use of offline RL to help isolate an RL algorithm ’ s ability to exploit experience and generalize vs. its ability to explore effectively by collecting interesting new data . The offline RL setting removes design choices related to the replay buffer and exploration ; therefore , it is much simpler to experiment with and reproduce than the typical online off-policy learning . Also , in the offline setting , we optimize an objective ( e.g. , ( 3 ) ) over a fixed dataset as opposed to a changing replay buffer of an online agent . In this work , we mainly experiment with large and diverse datasets motivated by real-world RL problems where we already have access to ( usually continually growing ) diverse datasets of logged experiences , e.g. , historical data from recommender systems ( Bottou et al. , 2013 ) , database of robotic experiences ( Dasari et al. , 2019 ) etc . To facilitate a study of offline RL with large and diverse datasets , we train several instances of DQN ( Nature ) agents ( Mnih et al. , 2015 ) on stochastic version of 60 Atari 2600 games for 200 million frames each , with a frame skip of 4 ( standard protocol ) and sticky actions enabled ( Machado et al. , 2018 ) . On each game , we train 5 different agents with random initialization , and store all of the tuples of ( observation , action , reward , next observation ) encountered during training into 5 replay datasets of 50 million tuples each . These replay datasets are used for training standard off-policy RL agents , offline , without any interaction with the environment during training . We perform an online evaluation of the agents in the intervals of 1 million training frames , and report the best evaluation score for each agent , averaged over 5 runs .

Summary: Paper tackles an important issue, which is benchmarking off-policy deep RL algorithms. The motivation is important, because 1) assessing the performance of off-policy RL algorithm is hard due to the mixed effect of exploration and exploitation while collecting data; hence the paper suggest to fix the data collection process by a standard algorithm (DQN) and then compare the off-policy methods. 2) A fixed dataset will help the reproducibility of the results.