PPO-CMA: Proximal Policy Optimization with Covariance Matrix Adaptation

1 INTRODUCTION . This paper proposes a new solution to the problem of policy optimization with high-dimensional continuous state and action spaces . This is a problem that has a long history in computer animation , robotics , and machine learning research . More specifically , our work falls in the domain of simulation-based Monte Carlo approaches ; instead of operating with closed-form expressions of the control dynamics , we use a black-box dynamics simulator , sample actions from some distribution , simulate the results , and then adapt the sampling distribution . In recent years , such approaches have achieved remarkable success in previously intractable tasks such as real-time locomotion control of ( simplified ) biomechanical models of the human body ( Wang et al . ( 2010 ) ; Geijtenbeek et al . ( 2013 ) ; Hämäläinen et al . ( 2014 ; 2015 ) ; Liu et al . ( 2016 ) ; Duan et al . ( 2016 ) ; Rajamäki & Hämäläinen ( 2017 ) ) . In 2017 , Proximal Policy Optimization ( PPO ) provided the first demonstration of a neural network policy that enables a simulated humanoid not only to run but also to rapidly switch direction and get up after falling ( Schulman et al . ( 2017 ) ) . Previously , such feats had only been achieved with more computationally heavy approaches that used simulation and sampling not only during training but also in run-time ( Hämäläinen et al . ( 2014 ; 2015 ) ; Rajamäki & Hämäläinen ( 2017 ) ) . PPO has been quickly adopted as the default RL algorithm in popular frameworks like Unity Machine Learning Agents Juliani et al . ( 2018 ) and TensorFlow Agents ( Hafner et al . ( 2017 ) ) . It has also been extended and applied to even more complex humanoid movement skills such as kung-fu kicks and backflips ( Peng et al . ( 2018 ) ) . Outside the continuous control domain , it has demonstrated outstanding performance in complex multi-agent video games 1 . However , like many other reinforcement learning methods , PPO can be sensitive to hyperparameter choices and difficult to tune ( Henderson et al . ( 2017 ) ) . In experiments by ourselves and our colleagues , we have also noticed a tendency to get stuck in local optima . In this paper , we make the following contributions : • We provide visualizations and evidence of how PPO ’ s exploration variance can shrink prematurely , which leads to slow progress or getting stuck in local optima . Figure 1 illustrates this in a simple didactic example . Subsequently , we discuss how a similar exploration problem is solved in the black-box optimization domain by Covariance Matrix Adaptation Evolution Strategy ( CMA-ES ) . CMA-ES dynamically expands the exploration variance on objective function slopes and only shrinks the variance when close to the optimum . • We show how exploration behavior similar to CMA-ES can be achieved in RL with simple changes , resulting in our PPO-CMA algorithm visualized in Figure 1 . As elaborated in Section 4 , a key idea of PPO-CMA is to use separate neural networks for policy mean and variance , and train the mean and variance in separate passes . Our experiments show a significant improvement over PPO in many OpenAI Gym MuJoCo tasks ( Brockman et al . ( 2016 ) ) . In Appendix A , we also investigate solving the variance adaptation problem with simple variance clipping and entropy regularization . The entropy regularization was suggested by Schulman et al . ( 2017 ) but not analyzed in detail . Our results suggest that both approaches can help but they are sensitive to tuning parameters . 2 PRELIMINARIES . 2.1 REINFORCEMENT LEARNING . Algorithm 1 Episodic On-policy Reinforcement Learning ( high-level summary ) 1 : for iteration=1,2 , ... do 2 : while iteration simulation budget N not exceeded do 3 : Reset the simulation to a ( random ) initial state 4 : Run agent on policy πθ for T timesteps or until a terminal state 5 : end while 6 : Update policy parameters θ based on the observed experience [ si , ai , ri , s′i ] 7 : end for We consider the discounted formulation of the policy optimization problem , following the notation of Schulman et al . ( 2015b ) . At time t , the agent observes a state vector st and takes an action at ∼ πθ ( at|st ) , where πθ denotes a stochastic policy parameterized by θ , e.g. , neural network weights . This results in observing a new state s′t and receiving a scalar reward rt . The goal is to find θ that maximizes the expected future-discounted sum of rewards E [ ∑∞ t=0 γ trt ] , where γ is a discount factor in the range ( 0 , 1 ) . A lower γ makes the learning prefer instant gratification instead of long-term gains . The original PPO and the PPO-CMA proposed in this paper collect experience tuples [ si , ai , ri , s′i ] by simulating a number of episodes in each optimization iteration . For each episode , an initial 1https : //blog.openai.com/openai-five/ state s0 is sampled from some application-dependent stationary distribution , and the simulation is continued until a terminal ( absorbing ) state or a predefined maximum episode length T is reached . After an iteration simulation budget N is exhausted , θ is updated . This is summarized in Algorithm 1 . 2.2 POLICY GRADIENT WITH ADVANTAGE ESTIMATION . Policy gradient methods update policy parameters by estimating the gradient g = ∇θE [ ∑∞ t γ trt ] . There are different formulations of the gradient , of which PPO uses the following : g = E [ ∞∑ t=0 Aπ ( st , at ) ∇θ log πθ ( at|st ) ] , ( 1 ) where Aπ is the so-called advantage function . Intuitively , the advantage function is positive if an explored action yields better rewards than expected . Updating θ in the direction of g makes negative advantage actions less likely and positive advantage actions more likely ( Schulman et al . ( 2015b ) ) . In practice , using modern compute graph frameworks like TensorFlow ( Abadi et al . ( 2016 ) ) , one often does not directly operate on the gradients , but instead uses an optimizer like Adam ( Kingma & Ba ( 2014 ) ) to minimize the corresponding loss L = − 1 M M∑ i=1 Aπ ( si , ai ) log πθ ( ai|si ) , ( 2 ) where i denotes minibatch sample index and M is minibatch size . In summary , this type of policy gradient RL simply requires a differentiable expression of πθ and a way to measure Aπ for each explored state-action pair . More specifically , the advantage function is defined as : Aπ ( st , at ) = Q π ( st , at ) − V π ( st ) . ( 3 ) Here , V π is the state value function , i.e. , the expected future-discounted sum of rewards for running the agent on-policy starting from state st. Qπ ( st , at ) is the state-action value function , i.e. , expected sum of rewards for taking action at in state st and then following the policy , Qπ ( st , at ) = r ( st , at ) + γV π ( st+1 ) . Thus , the advantage can also be expressed as Aπ ( st , at ) = r ( st , at ) + γV π ( st+1 ) − V π ( st ) . ( 4 ) In practice , V π is usually approximated by a critic network trained with the observed rewards summed over simulation trajectories . However , plugging such an approximation directly to Equation 4 tends to be unstable due to approximation bias . Instead , same as PPO , we use Generalized Advantage Estimation ( GAE ) ( Schulman et al . ( 2015b ) ) , which is a simple but effective way to estimate Aπ such that one can trade variance for bias . 3 UNDERSTANDING VARIANCE ADAPTATION IN GAUSSIAN POLICY OPTIMIZATION . This paper focuses on the case of continuous control using a Gaussian policy . In other words , the policy network outputs state-dependent mean µθ ( s ) and covariance Cθ ( s ) for sampling the actions . The covariance defines the exploration-exploitation balance . In practice , one often uses a diagonal covariance matrix parameterized by a vector cθ ( s ) = diag ( Cθ ( s ) ) . In the most simple case of isotropic unit Gaussian exploration , C = I , the loss function in Equation 2 becomes : L = − 1 M M∑ i=1 Aπ ( si , ai ) ||ai − µθ ( s ) ||2 , ( 5 ) Following the original PPO paper , this paper uses diagonal covariance . This results in a slightly more complex loss function : L = − 1 M M∑ i=1 Aπ ( si , ai ) ∑ j [ ( ai , j − µj ; θ ( si ) ) 2/cj ; θ ( si ) + 0.5 log cj ; θ ( si ) ] , ( 6 ) where i indexes over a minibatch and j indexes over action variables . 3.1 THE INSTABILITY CAUSED BY NEGATIVE ADVANTAGES . To gain an intuitive understanding of the training dynamics , it is important to note the following : • Equation 5 indicates that the policy is trained to approximate the sampled actions , with each action weighted by Aπ . Equation 5 uses an L2 loss , while Equation 6 uses a Bayesian loss that allows the network to model aleatoric uncertainty , expressed as the free covariance parameters ( Kendall & Gal ( 2017 ) ) . • As elaborated below , actions with a negative Aπ may cause instability , especially when one considers training for several epochs at each iteration using the same data ; as pointed out by Schulman et al . ( 2017 ) , this would be desirable to get the most out of the collected experience . For negative advantages , each minibatch gradient step drives the policy Gaussian further away from the sampled actions . This can easily result in divergence , as shown at the top of Figure 2 . In contrast , the bottom of Figure 2 shows how using only the positive advantage actions is stable ; the policy Gaussian simply approximates the positive advantage action distribution . However , in this case the exploration variance shrinks prematurely , leading to poor final convergence . Note that Figure 2 shows the convergence/divergence over policy gradient iterations ; Appendix D provides an illustration of gradient steps within an iteration . Similar to Figure 1 , Figure 2 depicts a ” stateless ” didactic example , a special case of Algorithm 1 that allows clear visualization of how the action distribution adapts . We set γ = 0 , which simplifies the policy optimization objective E [ ∑∞ t=0 γ trt ] = E [ r0 ] = E [ r ( s , a ) ] , where s , a denote the first state and action of an episode . Further , we use a state-agnostic r ( a ) = −aTa . Thus , Qπ ( s , a ) = −aTa , V π ( s ) = V π = E [ r ( a ) ] ≈ 1N ∑N−1 i=0 r ( ai ) , where i is episode index , and we can compute A π directly using Equation 3 . As everything is agnostic of agent state , a simulator is not even needed and one can feed an arbitrary constant input to a policy network . Equivalently , one can simply replace the policy network with optimized variables for the mean and variance of the action distribution .

Summary: This paper proposes an improvement of the PPO algorithm inspired by some components of the CMA-ES black-box optimization method. The authors evaluate the proposed method on a few Mujoco domains and compare it with PPO method using simpler exploration strategies. The results show that PPO-CMA less likely to getting stuck in local optima especially in Humanoid and Swimmer environments.