On-Policy Trust Region Policy Optimisation with Replay Buffers

1 INTRODUCTION . The past few years have been marked by active development of reinforcement learning methods . Although the mathematical foundations of reinforcement learning have been known long before ( Sutton & Barto , 1998 ) , starting from 2013 , the novel deep learning techniques allowed to solve vision based discrete control tasks such as Atari 2600 games ( Mnih et al. , 2013 ) as well as continuous control problems ( Lillicrap et al. , 2015 ; Mnih et al. , 2016 ) . Many of the leading state-of-the-art reinforcement learning methods share the actor-critic architecture ( Crites & Barto , 1995 ) . Actorcritic methods separate the actor , providing a policy , and the critic , providing an approximation for the expected discounted cumulative reward or some derived quantities such as advantage functions ( Baird III , 1993 ) . However , despite improvements , state-of-the-art reinforcement learning still suffers from poor sample efficiency and extensive parameterisation . For most real-world applications , in contrast to simulations , there is a need to learn in real time and over a limited training period , while minimising any risk that would cause damage to the actor or the environment . Reinforcement learning algorithms can be divided into two groups : on-policy and off-policy learning . On-policy approaches ( e. g. , SARSA ( Rummery & Niranjan , 1994 ) , ACKTR ( Wu et al. , 2017 ) ) evaluate the target policy by assuming that future actions will be chosen according to it , hence the exploration strategy must be incorporated as a part of the policy . Off-policy methods ( e. g. , Qlearning ( Watkins , 1989 ) , DDPG ( Lillicrap et al. , 2015 ) ) separate the exploration strategy , which modifies the policy to explore different states , from the target policy . The off-policy methods commonly use the concept of replay buffers to memorise the outcomes of the previous policies and therefore exploit the information accumulated through the previous iterations ( Lin , 1993 ) . Mnih et al . ( 2013 ) combined this experience replay mechanism with Deep Q-Networks ( DQN ) , demonstrating end-to-end learning on Atari 2600 games . One limitation of DQN is that it can only operate on discrete action spaces . Lillicrap et al . ( 2015 ) proposed an extension of DQN to handle continuous action spaces based on the Deep Deterministic Policy Gradient ( DDPG ) . There , exponential smoothing of the target actor and critic weights has been introduced to ensure stability of the rewards and critic predictions over the subsequent iterations . In order to improve the variance of policy gradients , Schulman et al . ( 2015b ) proposed a Generalised Advantage Function . Mnih et al . ( 2016 ) combined this advantage function learning with a parallelisation of exploration using differently trained actors in their Asynchronous Advantage Actor Critic model ( A3C ) ; however , Wang et al . ( 2016 ) demonstrated that such parallelisation may also have negative impact on sample efficiency . Although some work has been performed on improvement of exploratory strategies for reinforcement learning ( Hester et al. , 2013 ) , but it still does not solve the fundamental restriction of inability to evaluate the actual policy , neither it removes the necessity to provide a separate exploratory strategy as a separate part of the method . In contrast to those , state-of-the-art on-policy methods have many attractive properties : they are able to evaluate exactly the resulting policy with no need to provide a separate exploration strategy . However , they suffer from poor sample efficiency , to a larger extent than off-policy reinforcement learning . TRPO method ( Schulman et al. , 2015a ) has introduced trust region policy optimisation to explicitly control the speed of policy evolution of Gaussian policies over time , expressed in a form of Kullback-Leibler divergence , during the training process . Nevertheless , the original TRPO method suffered from poor sample efficiency in comparison to off-policy methods such as DDPG . One way to solve this issue is by replacing the first order gradient descent methods , standard for deep learning , with second order natural gradient ( Amari , 1998 ) . Wu et al . ( 2017 ) used a Kroneckerfactored Approximate Curvature ( K-FAC ) optimiser ( Martens & Grosse , 2015 ) in their ACKTR method . PPO method ( Schulman et al. , 2017 ) proposes a number of modifications to the TRPO scheme , including changing the objective function formulation and clipping the gradients . Wang et al . ( 2016 ) proposed another approach in their ACER algorithm : in this method , the target network is still maintained in the off-policy way , similar to DDPG ( Lillicrap et al. , 2015 ) , while the trust region constraint is built upon the difference between the current and the target network . Related to our approach , recently a group of methods has appeared in an attempt to get the benefits of both groups of methods . Gu et al . ( 2017 ) propose interpolated policy gradient , which uses the weighted sum of both stochastic ( Sutton et al. , 2000 ) and deterministic policy gradient ( Silver et al. , 2014 ) . Nachum et al . ( 2018 ) propose an off-policy trust region method , Trust-PCL , which exploits off-policy data within the trust regions optimisation framework , while maintaining stability of optimisation by using relative entropy regularisation . While it is a common practice to use replay buffers for the off-policy reinforcement learning , their existing concept is not used in combination with the existing on-policy scenarios , which results in discarding all policies but the last . Furthermore , many on-policy methods , such as TRPO ( Schulman et al. , 2015a ) , rely on stochastic policy gradient ( Sutton et al. , 2000 ) , which is restricted by stationarity assumptions , in a contrast to those based on deterministic policy gradient ( Silver et al. , 2014 ) , like DDPG ( Lillicrap et al. , 2015 ) . In this article , we describe a novel reinforcement learning algorithm , allowing the joint use of replay buffers with trust region optimisation and leading to sample efficiency improvement . The contributions of the paper are given as follows : 1. a reinforcement learning method , enabling replay buffer concept along with on-policy data ; 2. theoretical insights into the replay buffer usage within the on-policy setting are discussed ; 3. we show that , unlike the state-of-the-art methods as ACKTR ( Wu et al. , 2017 ) , PPO ( Schulman et al. , 2017 ) and TRPO ( Schulman et al. , 2015a ) , a single non-adaptive set of hyperparameters such as the trust region radius is sufficient for achieving better performance on a number of reinforcement learning tasks . As we are committed to make sure the experiments in our paper are repeatable and to further ensure their acceptance by the community , we will release our source code shortly after the publication . 2 BACKGROUND . 2.1 ACTOR-CRITIC REINFORCEMENT LEARNING . Consider an agent , interacting with the environment by responding to the states st , t ≥ 0 , from the state space S , which are assumed to be also the observations , with actions at from the action space A chosen by the policy distribution πθ ( ·|st ) , where θ are the parameters of the policy . The initial state distribution is ρ0 : S → R. Every time the agent produces an action , the environment gives back a reward r ( st , at ) ∈ R , which serves as a feedback on how good the action choice was and switches to the next state st+1 according to the transitional probability P ( st+1|st , at ) . Altogether , it can be formalised as an infinite horizon γ-discounted Markov Decision Process ( S , A , P , r , ρ0 , γ ) , γ ∈ [ 0 , 1 ) ( Wu et al. , 2017 ; Schulman et al. , 2015a ) . The expected discounted return ( Bellman , 1957 ) is defined as per Schulman et al . ( 2015a ) : ρ ( π ) = Es0 , a0 , ··· [ ∞∑ t=0 γtr ( st , at ) ] ( 1 ) The advantage function Aπ ( Baird III , 1993 ) , the value function V π and the Q-function Qπ are defined as per Mnih et al . ( 2016 ) ; Schulman et al . ( 2015a ) : Aπ ( s , a ) = Qπ ( s , a ) − V π ( s ) , ( 2 ) Qπ ( st , at ) = Est+1 , at+1 , ... [ ∞∑ l=0 γlr ( st+l , at+l ) ] , t ≥ 0 , ( 3 ) V π ( st ) = Eat , st+1 , ... [ ∞∑ l=0 γlr ( st+l , at+l ) ] , t ≥ 0 ( 4 ) In all above definitions s0 ∼ ρ0 ( s0 ) , at ∼ π ( at|st ) , st+1 ∼ P ( st+1|st , at ) , and the policy π = πθ is defined by its parameters θ . 2.2 TRUST REGION POLICY OPTIMISATION ( TRPO ) . A straightforward approach for learning a policy is to perform unconstrained maximisation ρ ( πθ ) with respect to the policy parameters θ . However , for the state-of-the-art iterative gradient-based optimisation methods , this approach would lead to unpredictable and uncontrolled changes in the policy , which would impede efficient exploration . Furthermore , in practice the exact values of ρ ( πθ ) are unknown , and the quality of its estimates depends on approximators which tend to be correct only in the vicinity of parameters of observed policies . Schulman et al . ( 2015a ) , based on theorems by Kakade ( 2002 ) , prove the minorisation-maximisation ( MM ) algorithm ( Hunter & Lange , 2004 ) for policy parameters optimisation . Schulman et al . ( 2015a ) mention that in practice the algorithm ’ s convergence rate and the complexity of maximum KL divergence computations makes it impractical to apply this method directly . Therefore , they proposed to replace the unconstrained optimisation with a similar constrained optimisation problem , the Trust Region Policy Optimisation ( TRPO ) problem : arg max θ ρ ( πθ ) ( 5 ) DKL ( πθold , πθ ) ≤ δ , ( 6 ) where DKL is the KL divergence between the old and the new policy πθold and πθ respectively , and δ is the trust region radius . Despite this improvement , it needs some further enhancements to solve this problem efficiently , as we will elaborate in the next section . 2.3 SECOND ORDER ACTOR-CRITIC NATURAL GRADIENT OPTIMISATION . Many of the state-of-the-art trust region based methods , including TRPO ( Schulman et al. , 2015a ) and ACKTR ( Wu et al. , 2017 ) , use second order natural gradient based actor-critic optimisation ( Amari , 1998 ; Kakade , 2002 ) . The motivation behind it is to eliminate the issue that gradient descent loss , calculated as the Euclidean norm , is dependent on parametrisation . For this purpose , the Fisher information matrix is used , which is , as it follows from Amari ( 1998 ) and Kakade ( 2002 ) , normalises per-parameter changes in the objective function . In the context of actor-critic optimisation it can be written as ( Wu et al. , 2017 ; Kakade , 2002 ) , where p ( τ ) is the trajectory distribution p ( s0 ) ∏T t=0 π ( at|st ) p ( st+1|st , at ) : F = Ep ( τ ) [ ∇θ log π ( at|st ) ( ∇θ log π ( at|st ) ) T ] . ( 7 ) However , the computation of the Fisher matrix is intractable in practice due to the large number of parameters involved ; therefore , there is a need to resort to approximations , such as the Kroneckerfactored approximate curvature ( K-FAC ) method ( Martens & Grosse , 2015 ) , which has been first proposed for ACKTR in ( Wu et al. , 2017 ) . In the proposed method , as it is detailed in Algorithm 1 , this optimisation method is used for optimising the policy .

Summary: The authors introduce a off-policy method for TRPO by suggesting to use replay buffers to store trajectories and sample from them during training. To do this they extend the definition of the Q function to multiple policies where the Q_pi bar is then the expectation over the several policies. They propose the same for the value function and consequently the advantage function.