PPO-CMA: Proximal Policy Optimization with Covariance Matrix Adaptation

1 INTRODUCTION . This paper proposes a new solution to the problem of policy optimization with high-dimensional continuous state and action spaces . This is a problem that has a long history in computer animation , robotics , and machine learning research . More specifically , our work falls in the domain of simulation-based Monte Carlo approaches ; instead of operating with closed-form expressions of the control dynamics , we use a black-box dynamics simulator , sample actions from some distribution , simulate the results , and then adapt the sampling distribution . In recent years , such approaches have achieved remarkable success in previously intractable tasks such as real-time locomotion control of ( simplified ) biomechanical models of the human body ( Wang et al . ( 2010 ) ; Geijtenbeek et al . ( 2013 ) ; Hämäläinen et al . ( 2014 ; 2015 ) ; Liu et al . ( 2016 ) ; Duan et al . ( 2016 ) ; Rajamäki & Hämäläinen ( 2017 ) ) . In 2017 , Proximal Policy Optimization ( PPO ) provided the first demonstration of a neural network policy that enables a simulated humanoid not only to run but also to rapidly switch direction and get up after falling ( Schulman et al . ( 2017 ) ) . Previously , such feats had only been achieved with more computationally heavy approaches that used simulation and sampling not only during training but also in run-time ( Hämäläinen et al . ( 2014 ; 2015 ) ; Rajamäki & Hämäläinen ( 2017 ) ) . PPO has been quickly adopted as the default RL algorithm in popular frameworks like Unity Machine Learning Agents Juliani et al . ( 2018 ) and TensorFlow Agents ( Hafner et al . ( 2017 ) ) . It has also been extended and applied to even more complex humanoid movement skills such as kung-fu kicks and backflips ( Peng et al . ( 2018 ) ) . Outside the continuous control domain , it has demonstrated outstanding performance in complex multi-agent video games 1 . However , like many other reinforcement learning methods , PPO can be sensitive to hyperparameter choices and difficult to tune ( Henderson et al . ( 2017 ) ) . In experiments by ourselves and our colleagues , we have also noticed a tendency to get stuck in local optima . In this paper , we make the following contributions : • We provide visualizations and evidence of how PPO ’ s exploration variance can shrink prematurely , which leads to slow progress or getting stuck in local optima . Figure 1 illustrates this in a simple didactic example . Subsequently , we discuss how a similar exploration problem is solved in the black-box optimization domain by Covariance Matrix Adaptation Evolution Strategy ( CMA-ES ) . CMA-ES dynamically expands the exploration variance on objective function slopes and only shrinks the variance when close to the optimum . • We show how exploration behavior similar to CMA-ES can be achieved in RL with simple changes , resulting in our PPO-CMA algorithm visualized in Figure 1 . As elaborated in Section 4 , a key idea of PPO-CMA is to use separate neural networks for policy mean and variance , and train the mean and variance in separate passes . Our experiments show a significant improvement over PPO in many OpenAI Gym MuJoCo tasks ( Brockman et al . ( 2016 ) ) . In Appendix A , we also investigate solving the variance adaptation problem with simple variance clipping and entropy regularization . The entropy regularization was suggested by Schulman et al . ( 2017 ) but not analyzed in detail . Our results suggest that both approaches can help but they are sensitive to tuning parameters . 2 PRELIMINARIES . 2.1 REINFORCEMENT LEARNING . Algorithm 1 Episodic On-policy Reinforcement Learning ( high-level summary ) 1 : for iteration=1,2 , ... do 2 : while iteration simulation budget N not exceeded do 3 : Reset the simulation to a ( random ) initial state 4 : Run agent on policy πθ for T timesteps or until a terminal state 5 : end while 6 : Update policy parameters θ based on the observed experience [ si , ai , ri , s′i ] 7 : end for We consider the discounted formulation of the policy optimization problem , following the notation of Schulman et al . ( 2015b ) . At time t , the agent observes a state vector st and takes an action at ∼ πθ ( at|st ) , where πθ denotes a stochastic policy parameterized by θ , e.g. , neural network weights . This results in observing a new state s′t and receiving a scalar reward rt . The goal is to find θ that maximizes the expected future-discounted sum of rewards E [ ∑∞ t=0 γ trt ] , where γ is a discount factor in the range ( 0 , 1 ) . A lower γ makes the learning prefer instant gratification instead of long-term gains . The original PPO and the PPO-CMA proposed in this paper collect experience tuples [ si , ai , ri , s′i ] by simulating a number of episodes in each optimization iteration . For each episode , an initial 1https : //blog.openai.com/openai-five/ state s0 is sampled from some application-dependent stationary distribution , and the simulation is continued until a terminal ( absorbing ) state or a predefined maximum episode length T is reached . After an iteration simulation budget N is exhausted , θ is updated . This is summarized in Algorithm 1 . 2.2 POLICY GRADIENT WITH ADVANTAGE ESTIMATION . Policy gradient methods update policy parameters by estimating the gradient g = ∇θE [ ∑∞ t γ trt ] . There are different formulations of the gradient , of which PPO uses the following : g = E [ ∞∑ t=0 Aπ ( st , at ) ∇θ log πθ ( at|st ) ] , ( 1 ) where Aπ is the so-called advantage function . Intuitively , the advantage function is positive if an explored action yields better rewards than expected . Updating θ in the direction of g makes negative advantage actions less likely and positive advantage actions more likely ( Schulman et al . ( 2015b ) ) . In practice , using modern compute graph frameworks like TensorFlow ( Abadi et al . ( 2016 ) ) , one often does not directly operate on the gradients , but instead uses an optimizer like Adam ( Kingma & Ba ( 2014 ) ) to minimize the corresponding loss L = − 1 M M∑ i=1 Aπ ( si , ai ) log πθ ( ai|si ) , ( 2 ) where i denotes minibatch sample index and M is minibatch size . In summary , this type of policy gradient RL simply requires a differentiable expression of πθ and a way to measure Aπ for each explored state-action pair . More specifically , the advantage function is defined as : Aπ ( st , at ) = Q π ( st , at ) − V π ( st ) . ( 3 ) Here , V π is the state value function , i.e. , the expected future-discounted sum of rewards for running the agent on-policy starting from state st. Qπ ( st , at ) is the state-action value function , i.e. , expected sum of rewards for taking action at in state st and then following the policy , Qπ ( st , at ) = r ( st , at ) + γV π ( st+1 ) . Thus , the advantage can also be expressed as Aπ ( st , at ) = r ( st , at ) + γV π ( st+1 ) − V π ( st ) . ( 4 ) In practice , V π is usually approximated by a critic network trained with the observed rewards summed over simulation trajectories . However , plugging such an approximation directly to Equation 4 tends to be unstable due to approximation bias . Instead , same as PPO , we use Generalized Advantage Estimation ( GAE ) ( Schulman et al . ( 2015b ) ) , which is a simple but effective way to estimate Aπ such that one can trade variance for bias . 3 UNDERSTANDING VARIANCE ADAPTATION IN GAUSSIAN POLICY OPTIMIZATION . This paper focuses on the case of continuous control using a Gaussian policy . In other words , the policy network outputs state-dependent mean µθ ( s ) and covariance Cθ ( s ) for sampling the actions . The covariance defines the exploration-exploitation balance . In practice , one often uses a diagonal covariance matrix parameterized by a vector cθ ( s ) = diag ( Cθ ( s ) ) . In the most simple case of isotropic unit Gaussian exploration , C = I , the loss function in Equation 2 becomes : L = − 1 M M∑ i=1 Aπ ( si , ai ) ||ai − µθ ( s ) ||2 , ( 5 ) Following the original PPO paper , this paper uses diagonal covariance . This results in a slightly more complex loss function : L = − 1 M M∑ i=1 Aπ ( si , ai ) ∑ j [ ( ai , j − µj ; θ ( si ) ) 2/cj ; θ ( si ) + 0.5 log cj ; θ ( si ) ] , ( 6 ) where i indexes over a minibatch and j indexes over action variables . 3.1 THE INSTABILITY CAUSED BY NEGATIVE ADVANTAGES . To gain an intuitive understanding of the training dynamics , it is important to note the following : • Equation 5 indicates that the policy is trained to approximate the sampled actions , with each action weighted by Aπ . Equation 5 uses an L2 loss , while Equation 6 uses a Bayesian loss that allows the network to model aleatoric uncertainty , expressed as the free covariance parameters ( Kendall & Gal ( 2017 ) ) . • As elaborated below , actions with a negative Aπ may cause instability , especially when one considers training for several epochs at each iteration using the same data ; as pointed out by Schulman et al . ( 2017 ) , this would be desirable to get the most out of the collected experience . For negative advantages , each minibatch gradient step drives the policy Gaussian further away from the sampled actions . This can easily result in divergence , as shown at the top of Figure 2 . In contrast , the bottom of Figure 2 shows how using only the positive advantage actions is stable ; the policy Gaussian simply approximates the positive advantage action distribution . However , in this case the exploration variance shrinks prematurely , leading to poor final convergence . Note that Figure 2 shows the convergence/divergence over policy gradient iterations ; Appendix D provides an illustration of gradient steps within an iteration . Similar to Figure 1 , Figure 2 depicts a ” stateless ” didactic example , a special case of Algorithm 1 that allows clear visualization of how the action distribution adapts . We set γ = 0 , which simplifies the policy optimization objective E [ ∑∞ t=0 γ trt ] = E [ r0 ] = E [ r ( s , a ) ] , where s , a denote the first state and action of an episode . Further , we use a state-agnostic r ( a ) = −aTa . Thus , Qπ ( s , a ) = −aTa , V π ( s ) = V π = E [ r ( a ) ] ≈ 1N ∑N−1 i=0 r ( ai ) , where i is episode index , and we can compute A π directly using Equation 3 . As everything is agnostic of agent state , a simulator is not even needed and one can feed an arbitrary constant input to a policy network . Equivalently , one can simply replace the policy network with optimized variables for the mean and variance of the action distribution .

Summary: This is in my view a strong contribution to the field of policy gradient methods for RL in the context of continuous control. The method the authors proposed is dedicated to solving the premature convergence issue in PPO through the learning of variance control policy. The authors employ CMA-ES which is usually employed for adaptive Gaussian exploration. The method is simple and yet provides good results on a several benchmarks when compared to PPO.