Samples Are Useful? Not Always: denoising policy gradient updates using variance explained

Policy gradient algorithms in reinforcement learning optimize the policy directly and rely on efficiently sampling an environment . However , while most sampling procedures are based on sampling the agent ’ s policy directly , self-performance measures could be used to improve sampling before each policy update . Following this line of thoughts , we propose SAUNA , a method where transitions are rejected from the gradient updates if they do not meet a particular criterion , and kept otherwise . The criterion , Vex , is the fraction of variance explained by the value function V : a measure of the discrepancy between V and the returns . In this work , Vex is used to estimate the impact transitions will have on learning : it refines sampling by simplifying the underlying state space and improves policy gradient methods . In this paper : ( a ) We introduce Vex , the criterion used for denoising policy gradient updates . We refer to this procedure as transition dropout and explore its implications on learning . ( b ) We conduct experiments across a variety of benchmark environments , including continuous control problems . SAUNA clearly improves performance . ( c ) We investigate how Vex reliably selects samples with most positive impact on learning . ( d ) We study how this criterion can work as a dynamic tool to adjust the ratio between exploration and exploitation . 1 INTRODUCTION . Learning to control agents in simulated environments has been a challenge for decades in reinforcement learning ( RL ) ( Nguyen & Widrow , 1990 ; Werbos , 1989 ; Schmidhuber & Huber , 1991 ; Robinson & Fallside , 1989 ) and has recently led to a lot of research efforts in this direction ( Mnih et al. , 2013 ; Burda et al. , 2019 ; Ha & Schmidhuber , 2018 ; Silver et al. , 2016 ; Espeholt et al. , 2018 ) , notably in policy gradient methods ( Schulman et al. , 2016 ; Silver et al. , 2014 ; Lillicrap et al. , 2016 ; Mnih et al. , 2016 ) . Despite the definite progress made , policy gradient algorithms still heavily suffer from sample inefficiency ( Kakade , 2003 ; Wu et al. , 2017 ; Schulman et al. , 2017 ; Wang et al. , 2017 ) . In particular , many policy gradient methods are subject to use as much experience as possible in the most efficient way . However , the quality of the sampling procedure also determines the final performance of the agent . We make the hypothesis that not all experiences are worth to use in the gradient update . Indeed , while perhaps trajectory simulations should be as rich as possible , some transitions may instead add noise to the gradient update , diluting relevant signals and hindering learning . The central idea of SAUNA is to reject transitions that are not informative for the particular task at hand . For that purpose , we use a measure of the discrepancy between the estimated state value and the observed returns . This discrepancy is formalized with the notion of the fraction of variance explained Vex ( Kvålseth , 1985 ) . Transitions for which Vex is close to zero are those for which the correlation between the value function V and the observed returns is also close to zero . SAUNA keeps transitions where there is either strong correlation or lack of fit between V and the returns , while avoiding the learning signals to be diluted by the dropped out samples . We will examine the impact of transition dropout ( inspired by Srivastava et al . ( 2014 ) and Freeman et al . ( 2019 ) ) and its theoretical implications . We consider on-policy methods for their unbiasedness and stability compared to off-policy methods ( Nachum et al. , 2017 ) . However , our method can be applied to off-policy methods as well , and we leave this investigation open for future work . The contributions of this paper are summarized as follows : 1 . We propose to move from a simple policy-based sampling procedure based to one taking into account the agent ’ s ability in an environment measured by Vex . We explore how the use of Vex can drive the alignment between the samples used to update the policy and the agent ’ s progress , while simplifying the underlying state space with transition dropout . 2 . We provide a method that transforms policy gradient algorithms by assuming that not all samples are useful for learning and that these disturbing samples should , therefore , be rejected . While our method is a simple extension of policy gradient algorithms , it adds a variance criterion to the optimization problem and introduces a novel rejection sampling procedure . 3 . By combining ( 1 ) and ( 2 ) , we obtain a learning algorithm that is empirically effective in learning neural network policies for challenging control tasks . Our results extend the stateof-the-art in using reinforcement learning for high-dimensional continuous control . We also evaluate the theoretical implications of our method . 2 PRELIMINARIES . We consider a Markov Decision Process ( MDP ) with states s ∈ S , actions a ∈ A , transition distribution st+1 ∼ P ( st , at ) and reward function r ( s , a ) . Let π = { π ( a|s ) , s ∈ S , a ∈ A } denote a stochastic policy and let the objective function be the traditional expected discounted reward : J ( π ) , E τ∼π [ ∞∑ t=0 γtr ( st , at ) ] , ( 1 ) where γ ∈ [ 0 , 1 ) is a discount factor ( Puterman , 1994 ) and τ = ( s0 , a0 , s1 , . . . ) is a trajectory sampled from the environment . Policy gradient methods aim at modelling and optimizing the policy directly ( Williams , 1992 ) . The policy π is generally implemented with a function parameterized by θ . In the sequel , we will use θ to denote the parameters as well as the policy . In deep reinforcement learning , the policy is represented in a neural network called the policy network and is assumed to be continuously differentiable with respect to its parameters θ . 2.1 FRACTION OF VARIANCE EXPLAINED : Vex The fraction of variance that the value function explains about the returns corresponds to the proportion of the variance in the dependent variable V that is predictable from the independent variable st. We define Vexτ as the fraction of variance explained for a trajectory τ : Vexτ , 1− ∑ t∈τ ( R̂t − V ( st ) ) 2 ∑ t∈τ ( R̂t −R ) 2 , ( 2 ) where R̂t and V ( st ) are respectively the return and the expected return from state st ∈ τ , and R is the mean of all returns in trajectory τ . In statistics , this quantity is also known as the coefficient of determination R2 and it should be noted that this criterion may be negative for non-linear models ( Kvålseth , 1985 ) , indicating a severe lack of fit of the corresponding function : • Vexτ = 1 : V perfectly explains the returns - V and the returns are correlated ; • Vexτ = 0 corresponds to a simple average prediction - V and the returns are not correlated ; • Vexτ < 0 : V provides a worse fit to the outcomes than the mean of the returns . One can have the intuition that Vexτ close to 1 is interesting because it gives samples from an exercised behavior while Vexτ < 0 corresponds to a high mean-squared error for the value function , meaning the agent will learn from the corresponding samples . On the other hand , Vexτ close to 0 does not provide any valuable information for a correct fitting of the state value function . We will demonstrate that Vex is a relevant indicator for assessing self-performance in RL . 2.2 POLICY GRADIENT METHODS : PPO AND A2C . We use PPO ( Schulman et al. , 2017 ) throughout this work . We also use A2C , a synchronous variant of Mnih et al . ( 2016 ) , to demonstrate SAUNA ’ s performance . Below we provide some details regarding PPO , as we use it generously in this work . In previous work , PPO has been tested on a set of benchmark tasks and has produced impressive results in many cases despite a relatively simple implementation . At each iteration , the new policy θnew is obtained from the old policy θold : θnew ← argmax θ E st , at∼πθold [ LPPO ( st , at , θold , θ ) ] . ( 3 ) We use the clipped version of PPO whose objective function is : LPPO ( st , at , θold , θ ) = min ( πθ ( at|st ) πθold ( at|st ) Aπθold ( st , at ) , g ( , A πθold ( st , at ) ) ) , ( 4 ) where g ( , A ) = { ( 1 + ) A , A ≥ 0 ( 1− ) A , A < 0 . ( 5 ) A is the advantage function , A ( s , a ) , Q ( s , a ) − V ( s ) . The expected advantage function Aπθold is estimated by an old policy and then re-calibrated using the probability ratio between the new and the old policy . In Eq . 4 , this ratio is constrained to stay within a small interval around 1 , making the training updates more stable . 2.3 RELATED WORK . Our method incorporates three key ideas : ( a ) function approximation with a neural network combining or separating the actor and the critic with an on-policy setting , ( b ) transition dropout reducing signal dilution in gradient updates while simplifying the underling MDP and ( c ) using Vexτ as a measure of correlation between the value function and the returns to allow for better sampling and more efficient learning . Below , we consider previous work building on some of these approaches . Actor-critic algorithms essentially use the value function to alternate between policy evaluation and policy improvement ( Sutton & Barto , 1998 ; Barto et al. , 1983 ) . In order to update the actor , many methods adopt the on-policy formulation ( Peters & Schaal , 2008 ; Mnih et al. , 2016 ; Schulman et al. , 2017 ) . However , despite their important successes , these methods suffer from sample complexity . In the literature , research has also been conducted in prioritization sampling . While Schaul et al . ( 2016 ) makes the learning from experience replay more efficient by using the TD error as a measure of these priorities in an off-policy setting , our method directly selects the samples on-policy . Schmidhuber ( 1991 ) is related to our method in that it calculates the expected improvement in prediction error , but with the objective to maximize the intrinsic reward through artificial curiosity . Instead , our method estimates the expected fraction of variance explained and filters out some of the samples to improve the learning efficiency . Finally , motion control in physics-based environments is a long-standing and active research field . In particular , there are many prior work on continuous action spaces ( Schulman et al. , 2016 ; Levine & Abbeel , 2014 ; Lillicrap et al. , 2016 ; Heess et al. , 2015 ) that demonstrate how locomotion behavior and other skilled movements can emerge as the outcome of optimization problems . 3 SAUNA : A DYNAMIC TRANSITION DROPOUT METHOD 3.1 ESTIMATING Vex While sampling the environment , SAUNA rejects samples where V ( st ) is not correlated with the return from st . Therefore , Vexτ must be estimated at each timestep so we define Vexθ ( st ) as the prediction of Vexτ under parameters θ at state st ∈ τ . In addition , for shared parameters configurations , an error term on the value estimation is added to the PPO objective . The final objective becomes : L ( st , at , θold , θ ) = E [ LPPO ( st , at , θold , θ ) − c1 ( Vθ ( st ) − R̂t ) 2 − c2 ( Vexθ ( st ) − Vexτ ) 2 ] , ( 6 ) where c1 and c2 are respectively the coefficient for the squared-error loss of the value function and of the fraction of variance explained function . For cases where the network is not shared between the policy and the value function , Vexτ is added to the value function network . Appendix A illustrates how Vexθ ( st ) is embedded in the original architecture . The rest of the network is unchanged , making it very easy to use SAUNA without altering the complexity of existing policy gradient methods . 3.2 USING Vex FOR DYNAMIC TEMPORAL ABSTRACTION Mechanistically , SAUNA results in dropping out several transitions from a trajectory τ , before each gradient update , and as a function of the starting state st of those transitions . The mechanism is analogous to the method of dropout in deep learning but articulated here by a dropout in the state space of the MDP . To evaluate the theoretical implications of the transition dropout , our method can be formulated using the Options framework ( Sutton et al. , 1999 ; Precup , 2000 ) thoroughly applied to policy gradient methods in Smith et al . ( 2018 ) . Previous work shows that by reasoning at several levels of abstraction , reinforcement learning agents are able to infer , learn , and plan more effectively . We detail below how SAUNA can be theoretically understood using the Options framework . In this work , we take semi-Markov options that may depend on the entire preceding sequence . The agent is given access to a set of options , indexed by σ. Interestingly , the framework can be reduced as follows : all options share the same policy , πσ ( at|st ) = πθ ( at|st ) , introduced in section 2 , the same initiation set and the same termination condition which we parameterize as the function βθ ( st ) . βθ ( st ) represents the probability of terminating an option and is defined as follows : βθ ( st ) = { 0 if bθ ( s0 , . . . , st ) ≥ r 1 if bθ ( s0 , . . . , st ) < r with bθ ( s0 , . . . , st ) = |Vext | |Ṽex0 : t−1|+ 0 , ( 7 ) where for simplicity we rewrite Vexθ ( st ) as Vext , Ṽex0 : t−1 is the median of Vexθ between timesteps 0 and t− 1 , 0 = 10−8 is to avoid division by zero and r is the dropout parameter . One may legitimately ask why not use directly bθ ( st ) = |Vext | . The rationale is practical : the ratio becomes a standardized measure as the agent learns , stabilized by the median , more robust to outliers than the mean . Fig . 1a illustrates an example of transition dropout dynamics in SAUNA while Fig . 1b depicts the behavior of Eq . 7 . The termination function βθ ( st ) dynamically selects the transitions for which Vext is either high or low , but not in between . Those transitions should have a strong impact on learning : a high absolute score means that the samples correspond to a state st for which the value function is highly correlated with the returns or does not fit them properly . With this modification , the resulting gradients should be affected accordingly , which we investigate in section 4.2.2 .

Summary: This paper proposes a novel way to denoise the policy gradient by filtering the samples to add by a criterion "variance explained". The variance explained basically measures how well the learn value function could predict the average return, and the filter will keep samples with a high or low variance explained and drop the middle samples. This new mechanism is then added on top of PPO to get their algorithm SAUNA. Empirical results show that it is better than PPO, on a set of MuJoCo tasks and Roboschool.