Stabilizing Off-Policy Reinforcement Learning with Conservative Policy Gradients

1 INTRODUCTION . Reinforcement Learning ( RL ) is a dynamical learning paradigm , in which the algorithm ( also known as the ‘ agent ’ ) learns through sequential interaction with the environment . On each round , the agent performs an action , which transitions it into a new state , and is provided with a state-dependent reward . The goal of the agent is to maximize the cumulative reward it observes throughout these interactions with the environment . When considering tabular RL , in which there is a finite number of states and actions , there exist efficient algorithms with theoretical convergence and stability guarantees [ 29 ; 13 ; 14 ; 34 ; 5 ; 4 ] . However , this is not the case when considering deep RL approaches – when a neural network is used to cope with large state spaces , such as images [ 19 ; 11 ] , and/or large action spaces , for example in continuous control [ 17 ; 27 ] . The issue with deep RL is twofold : ( i ) the optimization process is finite , and as such does not follow many of the requirements for the process to converge , such as decaying learning rates [ 2 ] , and ( ii ) the non-linearity of the function approximators ( e.g. , neural networks ) results in a highly non-convex optimization landscape . Due to this , as can be seen in all empirical works [ 22 ; 26 ; 17 ; 6 ; 9 ] , the learning process exhibits high variance ; during a short time of training , a near-optimal policy may become arbitrarily bad . As a result , in recent years , the stability of RL algorithms has become a major concern of the research community [ 10 ] . Instead of tackling the stability problems in the optimization process , many previous works focused on different aspects of the loss function [ 6 ; 9 ; 30 ] . Then , they usually apply a standard variant of gradient descent algorithm on the modified loss function . While these approaches are capable of finding relatively good policies , they are strictly inferior when compared to their tabular counterparts – the tabular approaches ensure convergence and stability , whereas the practical variants exhibit instability and high variance . In this work , we suggest a novel approach that improves the stability of actor-critic methods . Specifically , we draw the connection between deep RL approaches and their tabular counterparts and highlight the issues inherent in deep RL algorithms . Through this connection , we propose a solution , in the form of a hierarchical training procedure . As opposed to the tabular case , our approach is not ensured to converge , but rather has stability guarantees , i.e. , the stationary distribution of the policies is shown to be stable , meaning that with high probability the performance of the policy improves and does not suffer high degradation . We show , empirically across a range of continuous control tasks in MuJoCo , that our approach indeed results in improved behavior both in terms of stability and overall performance . 2 BACKGROUND AND NOTATION . We consider an infinite-horizon discounted Markov Decision Process [ 23 , MDP ] . An MDP is defined as the 5-tuple ( S , A , P , r , γ ) , where S is the state space , A the action space , Algorithm 1 Policy Iteration 1 : Initialize π0 2 : k = 0 3 : while not converged do 4 : while not converged do . Policy Evaluation 5 : for s ∈ S do 6 : vπk ( s ) = r ( s , πk ( s ) ) + γ ∑ s′∈S P ( s , πk ( s ) , s ′ ) vπk ( s′ ) 7 : for s ∈ S do . Policy Improvement 8 : πk+1 ( s ) ∈ arg maxa∈A r ( s , a ) + γ ∑ s′∈S P ( s , a , s ′ ) vπk ( s′ ) 9 : k = k + 1 10 : return policy πk Algorithm 2 DDPG , TD3 and our Conservative update 1 : Input : algorithm , learning rate α , averaging parameter τ , steps T and exploration N . 2 : Initialize policy θ , critics φ1 , φ2 and replay buffer R 3 : Set θ− = θ , φi− = φ i , t = 0 4 : while t < T do 5 : Collect data and append to replay buffer R 6 : t = t+ 1 7 : { si , ai , ri , si+1 } = R.sample ( ) . Training phase 8 : if DDPG then 9 : φ = φ− α∇φ||ri + γQφ− ( si+1 , πθ− ( si+1 ) ) −Qφ ( si , ai ) ||22 10 : else . TD3 11 : φ1 = φ1 − α∇φ1 ||ri + γminiQφi− ( si+1 , πθ− ( si+1 ) ) −Qφ1 ( si , ai ) || 2 2 12 : φ2 = φ2 − α∇φ2 ||ri + γminiQφi− ( si+1 , πθ− ( si+1 ) ) −Qφ2 ( si , ai ) || 2 2 13 : θ = θ + α∇θQφ1− ( si , πθ ( si ) ) . 14 : φi− = ( 1− τ ) φi− + τφi . Update target networks 15 : if Conservative Update then 16 : if t % Efreq == 0 then 17 : evaluate πθ and πθ− for Enum episodes each 18 : if P ( Jπθ > J π̂θ− ) ≥ 1− p then 19 : θ− = θ 20 : else 21 : θ− = ( 1− τ ) θ− + τθ Colored lines represent similarities between the classic and modern approach . P : S × S ×A 7→ [ 0 , 1 ] is a transition kernel , r : S ×A→ [ rmin , rmax ] is a reward function and γ ∈ ( 0 , 1 ) is the discount factor . The goal of an RL agent is to learn a policy π ( s , a ) that maps states into a distribution over actions . The quality of a policy is measured by its value , vπ ( s ) = Eπ [ ∑∞ t=0 γ tr ( st , at ) | s0 = s ] , which is the average cumulative reward when starting from state s and acting according to π . Another important quantity is the Q-function , Qπ ( s , a ) = Eπ [ ∑∞ t=0 γ tr ( st , at ) | s0 = s , a0 = a ] , which is strongly connected to the value through the relation vπ ( s ) = ∫ a Qπ ( s , a ) π ( s , a ) da . We denote an optimal policy that maximizes the value simultaneously across all of the states by π∗ ∈ arg maxπ vπ and the optimal value function by v∗ = vπ ∗ . When the Q-function and the policy π are represented by a parametric function , we denote them by Qφ and πθ , where φ and θ are the parameters of the function ( e.g . the parameters of a deep neural network ) . While our goal is to maximize vπ , in practice we measure an empirical estimation of Es∼ρvπ ( s ) , where ρ is the initial state distribution . To accommodate for this fact , we define the cumulative reward of a sampled trajectory by Jπ . This is a random variable , as randomness may occur due to ( i ) the initial state distribution , ( ii ) the transition kernel and reward function and ( iii ) the policy . 2.1 CLASSIC REINFORCEMENT LEARNING ALGORITHMS . Policy Iteration [ 12 , PI ] is an algorithm for solving tabular MDPs with convergence guarantees . PI ( Algorithm 1 ) iterates between two steps : ( i ) policy evaluation in which the current policy πk is evaluated , producing the value vπk and ( ii ) policy improvement in which the policy πk+1 is updated greedily w.r.t . vπk . This iterative scheme is proven to converge to the optimal policy . PI schemes have also been extended to the approximate case . Scherrer [ 25 ] provides an overview of such approaches , which are called Approximate Policy Iteration ( API ) . In API we consider the scenario in which ||v̂π̂k − vπk ||∞ ≤ . This approximation error can be caused due to two factors , the first being the inability to estimate the value and the second being the inability to find the greedy policy , i.e. , the arg max at each state . For instance , such errors may occur due to the functional class being used , e.g. , neural networks . Most importantly , the sub-optimality of these approaches is bounded by ||vπ̂∞ − v∗||∞ ≤ ( 1−γ ) 2 [ 25 ] , i.e. , the sub-optimality of the final policy is proportional to the approximation error . 2.2 OFF-POLICY DEEP REINFORCEMENT LEARNING FOR CONTINUOUS CONTROL . DDPG [ 17 ] and its latest improvement TD3 [ 6 ] use neural networks as function approximators in order to find the optimal policy . Both approaches utilize a target network , which is crucial for the success of the methods . While in their work , this was merely considered as a design parameter , we argue that the importance of these target networks lies in the connection to Policy Iteration . Deep Deterministic Policy Gradients [ 17 , DDPG ] ( Algorithm 2 ) aims to directly learn a deterministic policy in a continuous action space . DDPG has both policy evaluation and policy improvement steps . The Q-function , also known as the critic , is trained to estimate the utility of the target policy πθ− , whereas the online policy ( the actor ) is trained to find the 1-step greedy update w.r.t . this estimation . We argue that DDPG and PI are connected through the target network update procedure . In PI , the previous policy πk is evaluated by vπk , followed by an improvement step in which πk+1 becomes the one-step greedy policy w.r.t . this estimate . DDPG follows a similar scheme , in which the critic Qφ estimates the utility of the policy πθ− , followed by the online policy πθ which is updated w.r.t . this estimate . In this work , we compare to TD3 [ 6 ] , an improved version of DDPG . Twin Delayed DDPG [ 6 , TD3 ] improves the DDPG algorithm by tackling the problem of overestimation . Similarly to how Double-DQN [ 32 ] overcame the overestimation problem by introducing an additional Q function , in TD3 they introduce an additional critic . They show , both theoretically and empirically , that by bootstrapping the minimal value between both estimators miniQi ( s , π ( s ) ) they are able to overcome the problem of overestimation . 3 INSTABILITY IN POLICY-BASED DEEP REINFORCEMENT LEARNING . In the previous section , we provided an overview of PI , which has convergence guarantees , and two off-policy DRL variants . However , since in practical applications the policy is represented using a non-linear neural network , a small change in the parameters θ may result in a very large change in the output . We illustrate this issue in Figure 1 , where a state of the art off-policy approach learns to control the Hopper and InvertedPendulum robots . In these figures , we show the individual learning curves over several seeds , without performing temporal smoothing1 , whereas the bold line represents the mean performance across runs . At each evaluation period , we evaluate the performance of the agent across 100 episodes . These results paint a clear picture – while the agent is capable of learning a policy which attains a score of approximately 3500 in Hopper and the optimal score of 1000 in InvertedPendulum , it is highly unstable , i.e. , a policy which was near-optimal at time t might become arbitrarily bad at time t+ 1 . 1Temporal smoothing is a common practice in empirical studies , in which each data-point in the graph represents a moving average over the past N evaluations . While this makes the graphs easier to read , it hides a lot of important information such as the variance of each seed during training .

Summary: This paper proposes an approach called conservative policy gradients to stabilize the training of deep policy gradient methods. At fixed intervals, the current policy and a separate target policy are evaluated with a number of rollouts. The target policy is then updated to match the current policy only if the current policy is better than the target policy. Experiments show that the proposed method, when applied to TD3, reduces the variance in performance through the training.