Towards Consistent Performance on Atari using Expert Demonstrations

1 INTRODUCTION . In recent years , significant advances in the field of deep Reinforcement Learning ( RL ) have led to artificial agents that are able to reach human-level control on a wide array of tasks such as some Atari 2600 games ( Bellemare et al. , 2015 ) . In many of the Atari games , these agents learn control policies that far exceed the capabilities of an average human player ( Gruslys et al. , 2018 ; Hessel et al. , 2018 ; Horgan et al. , 2018 ) . However , learning human-level policies consistently across the entire set of games remains an open problem . We argue that an algorithm needs to overcome three key challenges in order to perform well on all Atari games . The first challenge is processing diverse reward distributions . An algorithm must learn stably regardless of reward density and scale . Mnih et al . ( 2015 ) showed that clipping rewards to the canonical interval [ −1 , 1 ] is one way to achieve stability . However , this clipping operation may change the set of optimal policies . For example , the agent no longer differentiates between striking a single pin or all ten pins in BOWLING . Hence , optimizing the unaltered reward signal in a stable manner is crucial to achieving consistent performance across games . The second challenge is reasoning over long time horizons , which means the algorithm should be able to choose actions in anticipation of rewards that might be far away . For example , in MONTEZUMA ’ S REVENGE , individual rewards might be separated by several hundred time steps . In the standard γ-discounted RL setting , this means the algorithm should be able to handle discount factors close to 1 . The third and final challenge is efficient exploration of the MDP . An algorithm that explores efficiently is able to discover long trajectories with a high cumulative reward in a reasonable amount of time even if individual rewards are very sparse . While each problem has been partially addressed in the literature , none of the existing deep RL algorithms have been able to address these three challenges at once . In this paper , we propose a new Deep Q-Network ( DQN ) ( Mnih et al. , 2015 ) style algorithm that specifically addresses these three challenges . In order to learn stably independent of the reward distribution , we use a transformed Bellman operator that reduces the variance of the action-value function . Learning with the transformed operator allows us to process the unaltered environment rewards regardless of scale and density . We prove that the optimal policy does not change in deterministic MDPs and show that under certain assumptions the operator is a contraction in stochastic MDPs ( i.e. , the algorithm converges to a fixed point ) ( see Sec . 3.2 ) . Our algorithm learns stably even at high discount factors due to an auxiliary temporal consistency ( TC ) loss . This loss prevents the network from prematurely generalizing to unseen states ( Sec . 3.3 ) allowing us to use a discount factor as high as γ = 0.999 in practice . This extends the effective planning horizon of our algorithm by one order of magnitude when compared to other deep RL approaches on Atari . Finally , we improve the efficiency of DQN ’ s default exploration scheme by combining the distributed experience replay approach of Horgan et al . ( 2018 ) with the Deep Q-learning from Demonstrations ( DQfD ) algorithm of Hester et al . ( 2018 ) . The resulting architecture is a distributed actor-learner system that combines offline expert demonstrations with online agent experiences ( Sec . 3.4 ) . We experimentally evaluate our algorithm on a set of 42 games for which we have demonstrations from an expert human player ( see Table 6 ) . Using the same hyper parameters on all games , our algorithm exceeds the performance of an average human player on 40 games , the expert player on 34 games , and state-of-the-art agents on at least 28 games . Furthermore , we significantly advance the state-of-the-art on sparse reward games . Our algorithm completes the first level of MONTEZUMA ’ S REVENGE and it achieves a score of 3997 points on PITFALL ! without compromising performance on dense reward games and while only using 5 demonstration trajectories . 2 RELATED WORK . Reinforcement Learning with Expert Demonstrations ( RLED ) : RLED seeks to use expert demonstrations to guide the exploration process in difficult RL problems . Some early works in this area ( Atkeson & Schaal , 1997 ; Schaal , 1997 ) used expert demonstrations to find a good initial policy before fine-tuning it with RL . More recent approaches have explicitly combined expert demonstrations with RL data during the learning of the policy or action-value function ( Chemali & Lazaric , 2015 ; Kim et al. , 2013 ; Piot et al. , 2014 ) . In these works , expert demonstrations were used to build an imitation loss function ( classification-based loss ) or max-margin constraints . While these algorithms worked reasonably well in small problems , they relied on handcrafted features to describe states and were not applied to large MDPs . In contrast , approaches using deep neural networks allow RLED to be explored in more challenging RL tasks such as Atari or robotics . In particular , our work builds upon DQfD ( Hester et al. , 2018 ) , which used a separate replay buffer for expert demonstrations , and minimized the sum of a temporal difference loss and a supervised classification loss . Another similar approach is Replay Buffer Spiking ( RBS ) ( Lipton et al. , 2016 ) , wherein the experience replay buffer is initialized with demonstration data , but this data is not kept for the full duration of the training . In robotics tasks , similar techniques have been combined with other improvements to successfully solve difficult exploration problems ( Nair et al. , 2017 ; Večerík et al. , 2017 ) . Deep Q-Networks ( DQN ) : DQN ( Mnih et al. , 2015 ) used deep neural networks as function approximators to apply RL to Atari games . Since that work , many extensions that significantly improve the algorithm ’ s performance have been developed . For example , DQN uses a replay buffer to store off-policy experiences and the algorithm learns by sampling batches uniformly from the replay buffer ; instead of using uniform samples , Schaul et al . ( 2015 ) proposed prioritized sampling where transitions are weighted by their absolute temporal difference error . This concept was further improved by Ape-X DQN ( Horgan et al. , 2018 ) which decoupled the data collection and the learning processes by having many actors feed data to a central prioritized replay buffer that an independent learner can sample from . Durugkar & Stone ( 2017 ) observed that due to over-generalization in DQN , updates to the value of the current state also have an adverse effect on the values of the next state . This can lead to unstable learning when the discount factor is high . To counteract this effect , they constrained the TD update to be orthogonal to the direction of maximum change of the next state . However , their approach only worked on toy domains such as Cart-Pole . Finally , van Hasselt et al . ( 2016a ) successfully extended DQN to process unclipped rewards with an algorithm called PopArt , which adaptively rescales the targets for the value network to have zero mean and unit variance . 3 ALGORITHM . In this section , we describe our algorithm , which consists of three components : ( 1 ) The transformed Bellman operator ; ( 2 ) The temporal consistency ( TC ) loss ; ( 3 ) Combining Ape-X DQN and DQfD . 3.1 DQN BACKGROUND . Let 〈X , A , r , p , γ〉 be a finite , discrete-time MDP where X is the state space , A the action space , r the reward function which represents the one-step reward distribution r ( x , a ) of doing action a in state x , γ ∈ [ 0 , 1 ] the discount factor and p a stochastic kernel modelling the one-step Markovian dynamics ( p ( x′|x , a ) is the probability of transitioning to state x′ by choosing action a in state x ) . The quality of a policy π is determined by the action-value function Qπ : X ×A → R , ( x , a ) 7→ Eπ ∑ t≥0 γtr ( xt , at ) | x0 = x , a0 = a , where Eπ is the expectation over the distribution of the admissible trajectories ( x0 , a0 , x1 , a1 , . . . ) obtained by executing the policy π starting from state x and taking action a . The goal is to find a policy π∗ that maximizes the state-value V π ( x ) : = maxa∈AQπ ( x , a ) for all states x , i.e. , find π∗ such that Vπ ∗ ( x ) = supπ V π ( x ) for all x ∈ X . While there may be several optimal policies , they all share a common optimal action-value function Q∗ ( Puterman , 1994 ) . Furthermore , acting greedily with respect to the optimal action-value function Q∗ yields an optimal policy . In addition , Q∗ is the unique fixed point of the Bellman optimality operator T defined as ( T Q ) ( x , a ) : = Ex′∼p ( ·|x , a ) [ r ( x , a ) + γmax a′∈A Q ( x′ , a′ ) ] , ∀ ( x , a ) ∈ X ×A for any Q : X ×A → R. Because T is a γ-contraction , we can learn Q∗ using a fixed point iteration . Starting with an arbitrary function Q ( 0 ) and then iterating Q ( k ) : = T Q ( k−1 ) for k ∈ N generates a sequence of functions that converges to Q∗ . DQN ( Mnih et al. , 2015 ) is an online-RL algorithm using a deep neural network fθ with parameters θ as a function approximator of the optimal action-value function Q∗ . The algorithm starts with a random initialization of the network weights θ ( 0 ) and then iterates θ ( k ) : = argmin θ Ex , a [ L ( fθ ( x , a ) − ( T fθ ( k−1 ) ) ( x , a ) ) ] , ( 1 ) where the expectation is taken with respect to a random sample of states and actions from an experience replay buffer and L is the Huber loss ( Huber , 1964 ) defined as L : R→ R , x 7→ { 1 2x 2 if |x| ≤ 1 |x| − 12 otherwise In practice , the minimization problem in ( 1 ) is only approximately solved by performing a finite and fixed number of stochastic gradient descent ( SGD ) steps1 and all expectations are approximated by sample averages . 3.2 TRANSFORMED BELLMAN OPERATOR . Mnih et al . ( 2015 ) have empirically observed that the errors induced by the limited network capacity , the approximate finite-time solution to ( 1 ) , and the stochasticity of the optimization problem can cause the algorithm to diverge if the variance of the action-value function is too high . In order to reduce the variance , they clip the reward distribution to the interval [ −1 , 1 ] . While this achieves the desired goal of stabilizing the algorithm , it significantly changes the set of optimal policies . For example , consider a simplified version of BOWLING where an episode only consists of a single throw . If the original reward is the number of hit pins and the rewards were clipped , any policy that hits at least a single pin would be optimal under the clipped reward function . Instead of reducing the magnitude of the rewards , we propose to focus on the action-value function instead . We use a function h : R→ R that reduces the scale of the action-value function . Our new operator Th is defined as ( ThQ ) ( x , a ) : = Ex′∼p ( ·|x , a ) [ h ( r ( x , a ) + γmax a′∈A h−1 ( Q ( x′ , a′ ) ) ) ] , ∀ ( x , a ) ∈ X ×A . Proposition 3.1 . Let Q∗ be the fixed point of T and Q : X ×A → R , then 1Mnih et al . ( 2015 ) refer to the number of SGD iterations as target update period . ( i ) If h ( z ) = αz for α > 0 , then T kh Q k→∞−−−−→ h ◦Q∗ = αQ∗ . ( ii ) If h is strictly monotonically increasing and the MDP is deterministic ( i.e. , p ( ·|x , a ) and r ( x , a ) are point measures for all x , a ∈ X ×A ) , then T kh Q k→∞−−−−→ h ◦Q∗ . Proposition 3.1 shows that in the basic cases when either h is linear or the MDP is deterministic , Th has the unique fixed point h ◦Q∗ . We present a proof in Sec . B in the appendix . Furthermore , the fixed point iteration T kh Q converges to h ◦Q∗ for all Q . We treat the case of stochastic MDPs in the appendix ( see Proposition C.1 ) . The following proposition ( see a proof in Sec . B in the appendix ) shows that contracting h can achieve the desired goal of decreasing the scale and variance of the action-value function . Proposition 3.2 . Let h : R→ R be a contraction mapping with fixed point 0 ( i.e. , h ( 0 ) = 0 ) , then 1 . | ( h ◦Q∗ ) ( x , a ) | ≤ |Q∗ ( x , a ) | for all x , a ∈ X ×A . 2 . Var ( ( h ◦Q∗ ) ( X , A ) ) ≤ Var ( Q∗ ( X , A ) ) for all random variables X , A on X ×A . In our algorithm , we use h : z 7→ sign ( z ) ( √ |z|+ 1 − 1 ) + εz with ε = 10−2 where the additive regularization term εz ensures that h−1 is Lipschitz continuous ( see Proposition C.1 ) . We chose this function because it is an invertible contraction mapping with fixed point 0 and has a closed-form inverse ( see Proposition C.2 ) . In practice , DQN minimizes the problem in ( 1 ) by sampling transitions of the form t = ( x , a , r′ , x′ ) from a replay buffer where x ∈ X , a ∼ π ( ·|x ) , r′ ∼ r ( x , a ) , and x′ ∼ p ( x , a ) . Let t1 , ... , tN be N transitions from the buffer with normalized sampling weights w1 , ... , wN , then for k ∈ N the loss function in ( 1 ) using the operator Th is approximated as Ex , a [ L ( fθ ( x , a ) − ( Thfθ ( k−1 ) ) ( x , a ) ) ] ≈ N∑ i=1 wiL ( fθ ( xi , ai ) − h ( r′i + γh−1 ( fθ ( k−1 ) ( x′i , a′i ) ) ) ) = : LTD ( θ ; ( ti ) N i=1 , ( wi ) N i=1 , θ ( k−1 ) ) where a′i : = argmaxa∈A fθ ( k−1 ) ( x ′ i , a ) for DQN and a ′ i : = argmaxa∈A fθ ( x ′ i , a ) for Double DQN ( van Hasselt et al. , 2016b ) .

Summary: The paper proposed three extensions to DQN to improve the learning performance on Atari games. The extensions are transformed Bellman update, temporal consistency loss and expert demonstration. These three extensions together make the proposed algorithm outperform the state-of-the-art results for Atari games. However, these extensions are relatively straightforward and thus the technical contribution is lean.