On Proximal Policy Optimization's Heavy-Tailed Gradients 

1 INTRODUCTION . As Deep Reinforcement Learning ( DRL ) methods have made strides on such diverse tasks as game playing and continuous control ( Berner et al. , 2019 ; Silver et al. , 2017 ; Mnih et al. , 2015 ) , policy gradient methods ( Williams , 1992 ; Sutton et al. , 2000 ; Mnih et al. , 2016 ) have emerged as a popular alternative to dynamic programming approaches . Since the breakthrough results of Mnih et al . ( 2016 ) demonstrated the applicability of policy gradients in DRL , a number of popular variants have emerged ( Schulman et al. , 2017 ; Espeholt et al. , 2018 ) . Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) —one of the most popular policy gradient methods—introduced the clipped importance sampling update , an effective heuristic for off-policy learning . However , while their stated motivation for clipping draws upon trust-region enforcement , the behavior of these methods tends to deviate from its key algorithmic principle ( Ilyas et al. , 2018 ) , and exhibit sensitivity to implementation details ( Engstrom et al. , 2019 ) . More generally , policy gradient methods are brittle , sensitive to both the random seed and hyperparameter choices , and poorly understood ( Ilyas et al. , 2018 ; Engstrom et al. , 2019 ; Henderson et al. , 2017 ; 2018 ; Islam et al. , 2017 ) . The ubiquity of these issues raises a broader concern about our understanding of policy gradient methods . In this work , we take a step forward towards understanding the workings of PPO , the most prominent and widely used deep policy gradient method . Noting that the heuristics implemented in PPO are evocative of estimation techniques from robust statistics in outlier-rich settings , we conjecture that the heavy-tailed distribution of gradients is the main obstacle addressed by these heuristics . We perform a rigorous empirical study to understand the causes of heavy-tailedness in PPO gradients . Furthermore , we provide a novel perspective on the clipping heuristics implemented in PPO by showing that these heuristics primarily serve to alleviate heavy-tailedness in gradients . Our first contribution is to analyze the role played by each component of the PPO objective in the heavy-tailedness of the gradients . We observe that as the training proceeds , gradients of both the actor and the critic loss get more heavy-tailed . Our findings show that during on-policy gradient steps the advantage estimates are the primary contributors to the heavy-tailed nature of the gradients . Moreover , as off-policyness increases ( i.e . as the behavioral policy and actor policy diverge ) dur- ing training , the likelihood-ratios that appear in the surrogate objective exacerbates the heavy-tailed behavior . Subsequently , we demonstrate that the clipping heuristics present in standard PPO implementations ( i.e. , gradient clipping , actor objective clipping , and value loss clipping ) significantly counteract the heavy-tailedness induced by off-policy training . Finally , motivated by this analysis , we present an algorithm that uses Geometric Median-of-Means ( GMOM ) , a high-dimensional robust aggregation method adapted from the statistics literature . Without using any of the objective clipping and gradient clipping heuristics implemented in PPO , the GMOM algorithm nearly matches PPO ’ s performance on MuJoCo ( Todorov et al. , 2012 ) continuous control tasks . 2 PRELIMINARIES . We define a Markov Decision Process ( MDP ) as a tuple ( S , A , R , γ , P ) , where S represent the set of environments states , A represent the set of agent actions , R : S ×A → R is the reward function , γ is the discount factor , and P : S × A × S → R is the state transition probability distribution . The goal in reinforcement learning is to learn a policy πθ : S × A → R+ , parameterized by θ , such that the expected cumulative discounted reward ( known as returns ) is maximized . Formally , π∗ : = argmaxπ Eat∼π ( ·|st ) , st+1∼P ( ·|st , at ) [ ∑∞ t=0 γ tR ( st , at ) ] . Policy gradient methods directly parameterize the policy ( also known as actor network ) . Since directly optimizing the cumulative rewards can be challenging , modern policy gradient algorithms typically optimize a surrogate reward function . Often the surrogate objective includes a likelihood ratio to allow importance sampling from a behavior policy π0 while optimizing policy πθ . For example , Schulman et al . ( 2015a ) optimize : max θ E ( st , at ) ∼π0 [ πθ ( at , st ) π0 ( at , st ) Aπ0 ( st , at ) ] , ( 1 ) where Aπθ = Qπθ ( st , at ) − Vπθ ( st ) . Here , Q-function , i.e . Qπθ ( s , a ) , is the expected discounted reward after taking an action a at state s and following πθ afterwards and Vπθ ( s ) is the value estimate ( implemented with a critic network ) . However , the surrogate is indicative of the true reward function only when πθ and π0 are close in distribution . Different policy gradient methods ( Schulman et al. , 2015a ; 2017 ; Kakade , 2002 ) attempt to enforce the closeness in different ways . In Natural Policy Gradients ( Kakade , 2002 ) and Trust Region Policy Optimization ( TRPO ) ( Schulman et al. , 2015a ) , authors utilize a conservation policy iteration with an explicit divergence constraint which provides provable lower bounds guarantee on the improvements of the parameterized policy . On the other hand , PPO ( Schulman et al. , 2017 ) implements a clipping heuristic on the likelihood ratio of the surrogate reward function to avoid excessively large policy updates . Specifically , PPO optimizes the following objective : max θ E ( st , at ) ∼π0 [ min ( clip ( ρt , 1− , 1 + ) Âπ0 ( st , at ) , ρtÂπ0 ( st , at ) ) ] , ( 2 ) where ρt : = πθ ( at , st ) π0 ( at , st ) . We refer to ρt as likelihood-ratios . Due to a minimum with the unclipped surrogate reward , the PPO objective acts as a pessimistic bound on the true surrogate reward . As in standard PPO implementation , we use Generalized Advantage Estimation ( GAE ) ( Schulman et al. , 2015b ) . Moreover , instead of fitting the value network via regression to target values : LV = ( Vθt − Vtarg ) 2 , ( 3 ) standard implementations fit the value network with a PPO-like objective : LV = max { ( Vθt − Vtarg ) 2 , ( clip ( Vθt , Vθt−1 − ε , Vθt−1 + ε ) − Vtarg ) 2 } , , ( 4 ) where is the same value used to clip probability raitos in PPO ’ s loss function ( Eq . 9 ) . PPO uses the following training procedure : At any iteration t , the agent creates a clone of the current policy πθt which interacts with the environment to collect rollouts B ( i.e. , state-action pairs { ( si , ai ) } Ni=1 ) . Then the algorithm optimizes the policy πθ and value function Vθ for a fixed K gradient steps on the sampled data B . Since at every iteration the first gradient step is taken on the same policy from which the data was sampled , we refer to these gradient updates as on-policy steps . And as for the remaining K − 1 steps , the sampling policy differs from the current agent , we refer to these updates as off-policy steps . Throughout the paper , we consider a stripped-down variant of PPO ( denoted PPO-NOCLIP ) that consists of policy gradient with importance weighting ( Eq . 1 ) , but has been simplified as follows : i ) no likelihood-ratio clipping , i.e. , no objective function clipping ; ii ) value network optimized via regression to target values ( Eq . 3 ) without value function clipping ; and iii ) no gradient clipping . Overall PPO-NOCLIP uses the objective summarized in App . A . One may argue that since PPONOCLIP removes the clipping heuristic from PPO , the unconstrained maximization of Eq . 1 may lead to excessively large policy updates . In App . I , we empirically justify the use of Eq . 1 by showing that with the small learning rate used in our experiments ( tuned hyperparameters in Table 1 ) , PPONOCLIP maintains a KL based trust-region like PPO throughout the training . We elaborate this in App . I . 2.1 FRAMEWORK FOR ESTIMATING HEAVY-TAILEDNESS . We now formalize our setup for studying the distribution of gradients . Throughout the paper , we use the following definition of the heavy-tailed property : Definition 1 ( Resnick ( 2007 ) ) . A non-negative random variable w is called heavy-tailed if its tail probability Fw ( t ) : =P ( w ≥ t ) is asymptotically equivalent to t−α ∗ as t → ∞ for some positive number α∗ . Here α∗ determines the heavy-tailedness and α∗ is called tail index of w. For a heavy-tailed distribution with index α∗ , its α-th moment exists only if α < α∗ , i.e. , E [ wα ] < ∞ iff α < α∗ . A value of α∗ = 1.0 corresponds to a Cauchy distribution and α∗ = ∞ ( i.e. , all moments exist ) corresponds to a Gaussian distribution . Intuitively , as α∗ decreases , the central peak of the distribution gets higher , the valley before the central peak gets deeper , and the tails get heavier . In other words , the lower the tail-index , the more heavy-tailed the distribution . However , in the finite sample setting , estimating the tail index is notoriously challenging ( Simsekli et al. , 2019 ; Danielsson et al. , 2016 ; Hill , 1975 ) . In this study , we explore three estimators as heuristic measures to understand heavy tails and nonGaussianity of gradients ( refer to App . B for details ) . ( i ) Alpha-index estimator which measures alpha-index for symmeteric α-stable distributions . This estimator is derived under the ( strong ) assumption that the stochastic Gradient Noise ( GN ) vectors are coordinate-wise independent and follow a symmetric alpha-stable distribution . ( ii ) Anderson-Darling test ( Anderson & Darling , 1954 ) on random projections of GN to perform Gaussianity testing ( Panigrahi et al. , 2019 ) . To our knowledge , the deep learning literature has only explored these two estimators for analyzing the heavytailed nature of gradients . ( iii ) Finally , in our work , we propose using Kurtosis . To quantify the heavy-tailedness relative to a normal distribution , we measure kurtosis ( fourth standardized moment ) of the gradient norms . Given samples { Xi } Ni=1 , the kurtosis κ is given by : κ = ∑N i=1 ( Xi − X̄ ) 4/N ( ∑N i=1 ( Xi − X̄ ) 2/N ) 2 , where X̄ is the empirical mean of the samples . With a slight breach of notation , we use kurtosis to denote κ1/4 . It is well known that for a Pareto distribution with shape α ≥ 4 , the lower the tail-index ( shape parameter α ) the higher the kurtosis . For α < 4 , since the fourth moment is non-existent , kurtosis is infinity . While for Gaussian distribution , the kurtosis value is approximately 1.31 . In App . B , we show behavior of kurtosis on Gaussian and Pareto data with varying sample sizes and tail-indices for Pareto data .

Summary: This paper performs an empirical analysis of the heavy-tailedness of the PPO gradients across MuJoCo environments. They find that PPO gradients are heavy-tailed, which means that they are sensitive to outliers, which means computing the expected gradient is hard. The paper studies two causes of this issue -- advantage estimation errors and the harm caused by optimizing density ratios in a sampled setting, and shows that removing either of them with the other issue controlled for helps PPO. They then propose to use a standard robust mean estimation technique to obtain a robust gradient mean estimator that is plugged into PPO, and performs sort of somewhat worse than vanilla PPO.