Title: SAMPLES ARE USEFUL? NOT ALWAYS: DENOISING POLICY GRADIENT UPDATES USING VARIANCE EXPLAINED

Abstract: Policy gradient algorithms in reinforcement learning optimize the policy directly and rely on efficiently sampling an environment. However, while most sampling procedures are based on sampling the agent's policy directly, self-performance measures could be used to improve sampling before each policy update. Following this line of thoughts, we propose SAUNA, a method where transitions are rejected from the gradient updates if they do not meet a particular criterion, and kept otherwise. The criterion, V ex , is the fraction of variance explained by the value function V : a measure of the discrepancy between V and the returns. In this work, V ex is used to estimate the impact transitions will have on learning: it refines sampling by simplifying the underlying state space and improves policy gradient methods. In this paper: (a) We introduce V ex , the criterion used for denoising policy gradient updates. We refer to this procedure as transition dropout and explore its implications on learning. (b) We conduct experiments across a variety of benchmark environments, including continuous control problems. SAUNA clearly improves performance. (c) We investigate how V ex reliably selects samples with most positive impact on learning. (d) We study how this criterion can work as a dynamic tool to adjust the ratio between exploration and exploitation.

1. INTRODUCTION:

Learning to control agents in simulated environments has been a challenge for decades in reinforcement learning (RL) (Nguyen & Widrow, 1990; Werbos, 1989; Schmidhuber & Huber, 1991; Robinson & Fallside, 1989) and has recently led to a lot of research efforts in this direction (Mnih et al., 2013; Burda et al., 2019; Ha & Schmidhuber, 2018; Silver et al., 2016; Espeholt et al., 2018) , notably in policy gradient methods (Schulman et al., 2016; Silver et al., 2014; Lillicrap et al., 2016; Mnih et al., 2016) . Despite the definite progress made, policy gradient algorithms still heavily suffer from sample inefficiency (Kakade, 2003; Wu et al., 2017; Schulman et al., 2017; Wang et al., 2017) .

In particular, many policy gradient methods are subject to use as much experience as possible in the most efficient way. However, the quality of the sampling procedure also determines the final performance of the agent. We make the hypothesis that not all experiences are worth to use in the gradient update. Indeed, while perhaps trajectory simulations should be as rich as possible, some transitions may instead add noise to the gradient update, diluting relevant signals and hindering learning.

The central idea of SAUNA is to reject transitions that are not informative for the particular task at hand. For that purpose, we use a measure of the discrepancy between the estimated state value and the observed returns. This discrepancy is formalized with the notion of the fraction of variance explained V ex (Kvålseth, 1985) . Transitions for which V ex is close to zero are those for which the correlation between the value function V and the observed returns is also close to zero. SAUNA keeps transitions where there is either strong correlation or lack of fit between V and the returns, while avoiding the learning signals to be diluted by the dropped out samples. We will examine the impact of transition dropout (inspired by Srivastava et al. (2014) and Freeman et al. (2019) ) and its theoretical implications. We consider on-policy methods for their unbiasedness and stability compared to off-policy methods (Nachum et al., 2017) . However, our method can be applied to off-policy methods as well, and we leave this investigation open for future work.

The contributions of this paper are summarized as follows:

1. We propose to move from a simple policy-based sampling procedure based to one taking into account the agent's ability in an environment measured by V ex . We explore how the use of V ex can drive the alignment between the samples used to update the policy and the agent's progress, while simplifying the underlying state space with transition dropout. 2. We provide a method that transforms policy gradient algorithms by assuming that not all samples are useful for learning and that these disturbing samples should, therefore, be rejected. While our method is a simple extension of policy gradient algorithms, it adds a variance criterion to the optimization problem and introduces a novel rejection sampling procedure. 3. By combining (1) and ( 2), we obtain a learning algorithm that is empirically effective in learning neural network policies for challenging control tasks. Our results extend the stateof-the-art in using reinforcement learning for high-dimensional continuous control. We also evaluate the theoretical implications of our method.

2. PRELIMINARIES:

We consider a Markov Decision Process (MDP) with states s ∈ S, actions a ∈ A, transition distribution s t+1 ∼ P (s t , a t ) and reward function r(s, a). Let π = {π(a|s), s ∈ S, a ∈ A} denote a stochastic policy and let the objective function be the traditional expected discounted reward:

EQUATION

where γ ∈ [0, 1) is a discount factor (Puterman, 1994) and τ = (s 0 , a 0 , s 1 , . . . ) is a trajectory sampled from the environment. Policy gradient methods aim at modelling and optimizing the policy directly (Williams, 1992) . The policy π is generally implemented with a function parameterized by θ. In the sequel, we will use θ to denote the parameters as well as the policy. In deep reinforcement learning, the policy is represented in a neural network called the policy network and is assumed to be continuously differentiable with respect to its parameters θ.

2.1. FRACTION OF VARIANCE EXPLAINED: V ex:

The fraction of variance that the value function explains about the returns corresponds to the proportion of the variance in the dependent variable V that is predictable from the independent variable s t . We define V ex τ as the fraction of variance explained for a trajectory τ :

EQUATION

where Rt and V (s t ) are respectively the return and the expected return from state s t ∈ τ , and R is the mean of all returns in trajectory τ . In statistics, this quantity is also known as the coefficient of determination R 2 and it should be noted that this criterion may be negative for non-linear models (Kvålseth, 1985) , indicating a severe lack of fit of the corresponding function:

• V ex τ = 1: V perfectly explains the returns -V and the returns are correlated; • V ex τ = 0 corresponds to a simple average prediction -V and the returns are not correlated; • V ex τ < 0: V provides a worse fit to the outcomes than the mean of the returns. One can have the intuition that V ex τ close to 1 is interesting because it gives samples from an exercised behavior while V ex τ < 0 corresponds to a high mean-squared error for the value function, meaning the agent will learn from the corresponding samples. On the other hand, V ex τ close to 0 does not provide any valuable information for a correct fitting of the state value function. We will demonstrate that V ex is a relevant indicator for assessing self-performance in RL.

2.2. POLICY GRADIENT METHODS: PPO AND A2C:

We use PPO (Schulman et al., 2017) throughout this work. We also use A2C, a synchronous variant of Mnih et al. (2016) , to demonstrate SAUNA's performance. Below we provide some details regarding PPO, as we use it generously in this work. In previous work, PPO has been tested on a set of benchmark tasks and has produced impressive results in many cases despite a relatively simple implementation. At each iteration, the new policy θ new is obtained from the old policy θ old :

θ new ← argmax θ E st,at∼π θ old L PPO (s t , a t , θ old , θ) .

(3)

We use the clipped version of PPO whose objective function is:

EQUATION

where

EQUATION

A is the advantage function, A(s, a) Q(s, a) -V (s). The expected advantage function A π θ old is estimated by an old policy and then re-calibrated using the probability ratio between the new and the old policy. In Eq. 4, this ratio is constrained to stay within a small interval around 1, making the training updates more stable.

2.3. RELATED WORK:

Our method incorporates three key ideas: (a) function approximation with a neural network combining or separating the actor and the critic with an on-policy setting, (b) transition dropout reducing signal dilution in gradient updates while simplifying the underling MDP and (c) using V ex τ as a measure of correlation between the value function and the returns to allow for better sampling and more efficient learning. Below, we consider previous work building on some of these approaches.

Actor-critic algorithms essentially use the value function to alternate between policy evaluation and policy improvement (Sutton & Barto, 1998; Barto et al., 1983) . In order to update the actor, many methods adopt the on-policy formulation (Peters & Schaal, 2008; Mnih et al., 2016; Schulman et al., 2017) . However, despite their important successes, these methods suffer from sample complexity.

In the literature, research has also been conducted in prioritization sampling. While Schaul et al. (2016) makes the learning from experience replay more efficient by using the TD error as a measure of these priorities in an off-policy setting, our method directly selects the samples on-policy. Schmidhuber (1991) is related to our method in that it calculates the expected improvement in prediction error, but with the objective to maximize the intrinsic reward through artificial curiosity. Instead, our method estimates the expected fraction of variance explained and filters out some of the samples to improve the learning efficiency.

Finally, motion control in physics-based environments is a long-standing and active research field. In particular, there are many prior work on continuous action spaces (Schulman et al., 2016; Levine & Abbeel, 2014; Lillicrap et al., 2016; Heess et al., 2015) that demonstrate how locomotion behavior and other skilled movements can emerge as the outcome of optimization problems.

3. SAUNA: A DYNAMIC TRANSITION DROPOUT METHOD:



3.1. ESTIMATING V ex:

While sampling the environment, SAUNA rejects samples where V (s t ) is not correlated with the return from s t . Therefore, V ex τ must be estimated at each timestep so we define V ex θ (s t ) as the prediction of V ex τ under parameters θ at state s t ∈ τ . In addition, for shared parameters configurations, an error term on the value estimation is added to the PPO objective. The final objective becomes:

EQUATION

)

where c 1 and c 2 are respectively the coefficient for the squared-error loss of the value function and of the fraction of variance explained function. For cases where the network is not shared between the policy and the value function, V ex τ is added to the value function network. Appendix A illustrates how V ex θ (s t ) is embedded in the original architecture. The rest of the network is unchanged, making it very easy to use SAUNA without altering the complexity of existing policy gradient methods.

3.2 USING V ex FOR DYNAMIC TEMPORAL ABSTRACTION Mechanistically, SAUNA results in dropping out several transitions from a trajectory τ , before each gradient update, and as a function of the starting state s t of those transitions. The mechanism is analogous to the method of dropout in deep learning but articulated here by a dropout in the state space of the MDP. To evaluate the theoretical implications of the transition dropout, our method can be formulated using the Options framework (Sutton et al., 1999; Precup, 2000) thoroughly applied to policy gradient methods in Smith et al. (2018) . Previous work shows that by reasoning at several levels of abstraction, reinforcement learning agents are able to infer, learn, and plan more effectively. We detail below how SAUNA can be theoretically understood using the Options framework.

In this work, we take semi-Markov options that may depend on the entire preceding sequence. The agent is given access to a set of options, indexed by σ. Interestingly, the framework can be reduced as follows: all options share the same policy, π σ (a t |s t ) = π θ (a t |s t ), introduced in section 2, the same initiation set and the same termination condition which we parameterize as the function β θ (s t ). β θ (s t ) represents the probability of terminating an option and is defined as follows:

EQUATION

where for simplicity we rewrite V ex θ (s t ) as V ex t , Ṽex 0:t-1 is the median of V ex θ between timesteps 0 and t -1, 0 = 10 -8 is to avoid division by zero and r is the dropout parameter. One may legitimately ask why not use directly b θ (s t ) = |V ex t |. The rationale is practical: the ratio becomes a standardized measure as the agent learns, stabilized by the median, more robust to outliers than the mean.

a a σ a a a a V, π, 𝒱 ex t = 0 1 2 3 4 5 6 β θ (s t ) = 1 0 0 0 0 1 1 V, π, 𝒱 ex V, π, 𝒱 ex V, π, 𝒱 ex 𝒱 ex t = .1 .8 .7 .6 .9 -.1 .1 drop drop drop (a) 1 1 0 Ṽex 0:t-1 𝒱 ex t -1 r (b)

Figure 1 : (a) Red solid arrows illustrate the transitions affecting the gradient update. We illustrate the termination condition mechanism with an example. For t = (1, 2, 3, 4), V ex t is large enough so β θ (s t ) = 0. The transitions sampled by policy π σ affect the gradient update. The others are dropped. Note that an option σ can be one-step long, i.e. β θ (s t ) = 1, β θ (s t+1 ) = 0, β θ (s t+2 ) = 1. (b) Grey area samples: kept for the gradient update. White area samples: dropped (r = 0.3).

Fig. 1a illustrates an example of transition dropout dynamics in SAUNA while Fig. 1b depicts the behavior of Eq. 7. The termination function β θ (s t ) dynamically selects the transitions for which V ex t is either high or low, but not in between. Those transitions should have a strong impact on learning: a high absolute score means that the samples correspond to a state s t for which the value function is highly correlated with the returns or does not fit them properly. With this modification, the resulting gradients should be affected accordingly, which we investigate in section 4.2.2.

3.3. PUTTING IT ALL TOGETHER: SAUNA SAMPLING PROCEDURE:

In this work, we hypothesize that low correlated transitions dilute the valuable signal information of the transitions with high absolute correlation. The temporal abstraction provided by the Options framework is leveled dynamically thanks to a termination function exploiting the self-performance measure V ex t . Instead of keeping these noisy transitions, the sampling procedure drops them until the trajectory is T -steps long. Algorithm 1 illustrates how learning is achieved with SAUNA applied to PPO, in particular, the fitting of the V ex function in Eq. 10 and the transition dropout in the if statement. We choose to depict a configuration where the parameters between the policy network, the value function and the V ex function are not shared, since from this configuration the shared parameter case is direct.

Algorithm 1 SAUNA sampling procedure in PPO.

Initialize policy parameters θ 0 Initialize value function parameters φ 0 and V ex function parameters ψ 0 for k = 0, 1, 2, . . . do

For each update k

Initialize trajectory τ to capacity T while size(τ

) ≤ T do For each timestep t a t ∼ π θ k (s t ), v t = V φ k (s t ), V ex t = V ex ψ k (s t

) execute action a t and observe reward r t+1 and next state s t+1 if

EQUATION

EQUATION

PPO sampling SAUNA dropout 4 HOW DOES SAUNA LEARN?

For ease of reproducibility and sharing, we have forked the original baselines repository from Ope-nAI and modified the code to incorporate our method1 . The complete list of hyperparameters and details of our implementation are given in Appendix E and F respectively. A discussion about additional experiments whose results are non-positive, but which we think contribute positively to this paper, can be found in Appendix B.

4.1. SAUNA IN THE CONTINUOUS DOMAIN: MUJOCO AND ROBOSCHOOL:

To verify the generalizability of our method, we study SAUNA against PPO and A2C. We compare SAUNA (PPO+V ex in red) with its natural baseline PPO (PPO in blue). We use 6 simulated robotic tasks from OpenAI Gym (Brockman et al., 2016) using MuJoCo (Todorov et al., 2012) . The two hyperparameters required by our method (r = 0.3 from Eq. 2 and c 2 = 0.5 from Eq. 6) and all the others (identical to those in Schulman et al. (2017) ) are exactly the same for all tasks. We made this choice within a clear and objective framework of comparison between the two methods. Thus, we have not optimized the rest of the hyperparameters for SAUNA, and its reported performance is not necessarily the best that could be obtained with more intensive tuning. From the graphs reported in Fig. 2 , we see that our method outperforms all continuous control tasks.

We also present in Table 1 the scores obtained for each task on the experiments with PPO and A2C.

The graphs for A2C are reported in Appendix C.1. We then experiment with the more difficult, high-dimensional continuous domain environment of Roboschool (Klimov & Schulman, 2017) with different neural network capacities for the model. When resources are limited in terms of number of parameters, it seems natural that dropping out samples based on their predicted learning impact allows to reduce noise in the gradient update and accelerate learning. And as a result, our method is faster and more efficient than the baseline. When resources are not limited, the gap closes towards the end of the training and our method performs as well as the baseline.

4.2. UNDERSTANDING THE IMPACT OF SAUNA ON LEARNING:



4.2.1. WHAT ADVANTAGE DOES DROPOUT GIVE?:

We further study the impact of dropping out noisy samples by conducting additional experiments in predicting V ex while omitting the filtering step before the gradient update: the if statement in Algorithm 1 is removed and all transitions are collected in τ . Indeed, the SAUNA algorithm could improve the agent's performance by simply training the shared network to optimize the variance explained head. Fig. 3 (full results are provided in Appendix D) demonstrates the positive effects of dropping out the samples. In addition, we studied the number of sample dropouts per task and their evolution throughout the training. On average, SAUNA rejects 5-10% of samples at the beginning of training which reduces to 2-6% at the end. . Red is our method PPO+V ex , Orange is PPO+V ex without the filtering out of noisy samples. Line: average performance. Shaded area: standard deviation.

4.2.2. HOW DOES SAUNA IMPACT THE GRADIENTS?:

Prior to the gradient updates, SAUNA removes the transitions for which the state value is poorly correlated with the returns. By doing so, we hypothesized that information signals from samples with significant V ex would be less diluted by dropped out samples. One can easily see by the graphs that the norm of the gradients is affected by such a change in the learning procedure. Fig. 4 shows that SAUNA dropout generates larger gradients. Hence policy updates have bigger steps, which eventually results in better performance. One may wonder quite rightly why performance is not damaged, since larger gradients could hinder learning. The performance results suggest that the gradients contain more useful information from each of the transitions that passed SAUNA sampling. In other terms, the relevant information for the task at hand is less diluted, the gradients are more qualitative and have been partially denoised.

4.2.3. HALFCHEETAH: A QUALITATIVE STUDY:

Under review as a conference paper at ICLR 2020

4.3. ROBOSCHOOL:

We then experimented with the more difficult, high-dimensional continuous domain environment of Roboschool: RoboschoolHumanoidFlagrunHarder-v1. The purpose of this task is to allow the agent to run towards a flag whose position varies randomly over time. It is left to fall and is continuously bombarded by white cubes that push it out of its path. In Fig. 4a , the same fully-connected network as for the MuJoCo experiments (2 hidden layers each with 64 neurons) is used. In Fig. 4b , the network is composed of 3 hidden layers with 512, 256 and 128 neurons. We trained those agents with 32 parallel actors. In both experiments, our method performs better and faster at the beginning. Then, when the policy and value functions benefit from a larger network, the gap closes, and our contribution does as well as its baseline. When resources are limited parameter-wise, it seems natural that filtering out samples based on their predicted training impact allows to remove noise from the gradient update and accelerate learning.

Finally, we investigated further and conducted the same experiment as with the larger network (3 hidden layers with 512, 256 and 128 neurons), but with 128 actors in parallel instead of 32. The total number of timesteps is unchanged and is still equal to 100M. Results are reported in Fig. 4c : our method is still faster and achieves better performance than the baseline.

4.4 VARIANCE EXPLAINED V ex : CASE STUDY While studying HalfCheetah-v2, we observed that for a number of seeds, PPO was converging to a local minimum forcing the agent to move forward on its back. This is a well-known behavior (Lapan, 2018) . However, we observed that our method made it possible to leave from, or at least to avoid these local minima. Those particular deterministic environments can be generated reproducibly with specific seeds. This is illustrated in Fig. 5 where we can look at still frames of two agents trained for 10 6 timesteps on identically seeded environments. In HalfCheetah, a well-known behavior (Lapan, 2018) is that for multiple seeds PPO is converging to a local minimum forcing the agent to move forward on its back. However, we observed that SAUNA made it possible to leave from, or at least to avoid these local minima. This is illustrated in Fig. 5a where we can see still frames of two agents trained with PPO and SAUNA for 10 6 timesteps on identically seeded environments. Their behavior is entirely different. Looking at V ex in Fig. 5b , we can see that the graphs differ quite interestingly. The orange agent seems to find very quickly a local minimum on its back while the green agent's V ex varies much more. This seems to allow the latter to explore more states than the former and finally find the fastest way forward. Supported by the previous study, we can infer that SAUNA agents are better able to explore interesting states while exploiting with confidence the value given to the states observed so far.

5. DISCUSSION:

Intuitively, for the policy update, our method will only use qualitative samples that provide the agent with (a) reliable and exercised behavior (high V ex ) and (b) challenging states from the point of view of correctly predicting their value (low V ex ). The SAUNA algorithm keeps samples with high learning impact, rejecting other noisy samples from the gradient update.

5.1. DENOISING POLICY GRADIENT UPDATES AND THE POLICY GRADIENT THEOREM:

Policy gradient algorithms are backed by the policy gradient theorem (Sutton et al., 2000) :

EQUATION

As long as the asymptotic stationary regime is not reached, it is not reasonable to assume the sampled states to be independent and identically distributed (i.i.d.). Hence, it seems intuitively better to ignore some of the samples for a certain period, to allow the most efficient use of information. One can understand SAUNA as making gradient updates more robust through dropout, especially when the update is low and the noise can be dominant. Besides, not taking all samples reduces the bias in the state distribution d π . Therefore, it now seems more reasonable to consider the sampled states i.i.d., which we theoretically need for the policy gradient theorem.

5.2. IMPACT OF V ex ON THE SHARED NETWORK PARAMETERS:

The shared network predicts V ex in conjunction with the value function and the policy. Therefore, as its parameters are updated through gradient descent, they converge to one of the objective function minima (hopefully, a global minimum). This parameter configuration integrates V ex , predicting how much the value function has fit the observed samples, or informally speaking how well the value function is doing for state s t . This new head tends to lead the network to adjust predicting a quantity relevant for the task. Instead of using domain knowledge for the task, the method rather introduces problem knowledge by constraining the parameters directly.

6. CONCLUSION:

Policy gradient methods optimize the policy directly through gradient ascent with the objective to maximize the expected return. We have introduced a new, lightweight and agnostic method technically applicable to any policy gradient method using a neural network as function approximation.

The central idea of this paper is that samples that are uncorrelated with the return are dropped out and not considered for the policy update. Those low correlated samples are ignored by SAUNA, the mechanism is reflected in a dynamic temporal abstraction controlled by the estimated variance explained of each state. The relevant signal information being less diluted, this results in a denoising effect on the gradients, ultimately leading to improved performance.

We demonstrated the effectiveness of our method when applied to two policy gradient methods on several standard benchmark environments. We also established that samples can be removed from the gradient update without hindering learning but can, on the opposite, improve it. We further studied the impact that such a modification in the sampling procedure has on learning. Several open topics warrant further study. The influence of SAUNA on the distribution of states could, for instance, be theoretically investigated. We also find the effort (Cobbe et al., 2019; Zhang et al., 2018) to go further towards generalization in RL very promising, and we think SAUNA could be useful in these problems as a way to regularize policy gradient methods. In Fig. 6 we rewrite V ex θ (s t ) as V ex t . On the right side of the figure is illustrated the V ex t head that is added to the shared or the separate network configurations. Note that even if V ex τ is defined for a sampled trajectory τ , the model predicts its value at each state s t ∈ τ .

B ADDITIONAL EXPERIMENTS WITH NON-POSITIVE RESULTS:

Atari domain. We tested our method on the Atari 2600 domain (Bellemare et al., 2013) without observing any improvement in learning. By comparing the two algorithms where the method of filtering the samples is used or not, we could not observe any difference, as some tasks were better performed by one method and others by the other.

Mean of V ex . Although Ṽex , the median of V ex , is more expensive to calculate, we observe that it gives much better results than if we use its mean in Eq. 7. Using the median helps (Kvålseth, 1985) because the distribution of V ex is not normal and includes outliers that will potentially produce misleading results.

Non-empirical V ex . We also experimented with using the real values of V ex in Eq. 7 when calculating Ṽex 0:t-1 , instead of the predicted ones. This has yielded less positive results, and it is likely that this is due to the difference between the predicted and actual values at the beginning of learning, which has the effect of distorting the ratio in Eq. 7.

Adjusting state count. In order to stay in line with the policy gradient theorem (Sutton et al., 2000) , we have worked to adjust the distribution of states d π to what it really is, since some states that the agent has visited are not included in the gradient update. We adjusted it using the ratio between the number of states visited and the actual number of transitions used in the gradient update, but this did not improve the learning, and instead, we observed a decrease in performance.

Random dropout. We experimented with dropping out at random, and before each gradient update, a number of samples corresponding to the same approximate number of samples that SAUNA drops. This resulted in a decrease in performance compared to PPO, as one can expect.

Finally, we investigated further and conducted the same experiment with the larger network (3 hidden layers with 512, 256 and 128 neurons), but with 128 actors in parallel instead of 32. Results are reported in Fig. 8c : our method still learns faster and achieves better performance than the baseline.

D ADDITIONAL EXPERIMENTS ON THE ADVANTAGE OF TRANSITION:



DROPOUT:

In order to identify the effects of the training of the V ex head and the filtering out of sample, we verify the hypothesis that filtering out noisy samples does improves the performance. To do so, in Section 4.2.1, we conduct experiments where the network predicts V ex but where the noisy samples are not filtered out: the if statement in Algorithm 1 is removed and all transitions are collected in τ . In Fig. 9 , we see that when the noisy samples are not filtered out the performance is worst than the baseline, confirming the positive denoising impact of filtering out the variance-selected samples. 

E HYPERPARAMETERS:

We have tuned the hyperparameters of our method by performing a grid search and selecting the best combinations by considering those with the largest consensus. 

:

Code is available here: https://github.com/iclr2020-submission/denoising-gradient-updates

Figures and Tables:

Figure 2: Comparison of SAUNA with PPO on 6 MuJoCo environments (10 6 timesteps, 6 different seeds). Red is our method PPO+V ex . Line: average performance. Shaded area: standard deviation.

Figure3: Comparison of SAUNA with PPO on 3 MuJoCo environments (10 6 timesteps, 6 different seeds). Red is our method PPO+V ex , Orange is PPO+V ex without the filtering out of noisy samples. Line: average performance. Shaded area: standard deviation.

Figure 4: Gradients L1-norm from the (a) first layer and (b) last layer of the shared parameters network for PPO and when SAUNA is applied to PPO. Task: HalfCheetah-v2.

Figure 4: Comparison of our method with PPO-Clip on the more challenging Roboschool environment (10 8 timesteps, 6 different seeds). Red is our method PPO+Vex. Line: average performance. Shaded area: standard deviation.

Figure 5: Example of a deterministic environment where PPO converges towards a local minimum (top row) while our method does not (bottom row).The behavior is entirely different for the agent trained with PPO and the agent trained with PPO combined with our method denoising the policy gradient updates. If we look at the explained vari-7

Figure9: Comparison of our method with PPO on 6 MuJoCo environments (10 6 timesteps, 6 different seeds). Red is our method PPO+V ex , Orange is PPO+V ex without the filtering out of noisy samples. Line: average performance. Shaded area: standard deviation.

Average total reward of the last 100 episodes over 6 runs on the 6 MuJoCo environments on PPO and A2C. Boldface mean ± std indicate better mean performance.

Table content: <table><tr><td>Task HalfCheetah Hopper InvertedDoublePendulum InvertedPendulum Reacher Swimmer</td><td>PPO 2277 ± 432 2106 ± 133 6100 ± 143 532 ± 19 -7.5 ± 0.8 99.5 ± 5.4</td><td>PPO+V ex 2929 ± 169 2250 ± 73 6893 ± 350 609 ± 24 -7.2 ± 0.3 100.8 ± 10.4</td><td>A2C 1389 ± 157 1367 ± 110 4151 ± 67 686 ± 15 -9.2 ± 0.8 44.1 ± 10.3</td><td>A2C+V ex 1731 ± 147 1627 ± 97 5132 ± 409 684 ± 10 -8.5 ± 0.7 59.0 ± 5.5</td></tr></table>

A ILLUSTRATION OF THE SAUNA ARCHITECTURE

Table content: <table><tr><td>v t</td><td>π t a t ∼ π t</td><td>v t</td><td>π t a t ∼ π t</td><td>ex t</td></tr><tr><td colspan="2">Shared network</td><td colspan="2">Shared network</td><td/></tr><tr><td/><td>s t</td><td/><td>s t</td><td/></tr><tr><td colspan="2">PPO (shared network)</td><td colspan="2">SAUNA (shared network)</td><td/></tr><tr><td>π t a t ∼ π t</td><td>v t</td><td>π t a t ∼ π t</td><td>v t</td><td>ex t</td></tr><tr><td>Policy network</td><td>Value network</td><td>Policy network</td><td colspan="2">Value network</td></tr><tr><td>s t</td><td>s t</td><td>s t</td><td>s t</td><td/></tr><tr><td colspan="2">PPO (separate network)</td><td colspan="3">SAUNA (separate network)</td></tr><tr><td colspan="4">Figure 6: Network-agnostic variance explained head.</td><td/></tr></table>

Hyperparameters used both in PPO and SAUNA. The two last hyperparameters are only relevant for our method.

Table content: <table><tr><td>Hyperparameter Horizon (T ) Adam stepsize Nb. epochs Minibatch size Discount (γ) GAE parameter (λ) Clipping parameter ( ) VF coef (c 1 ) V ex coef (c 2 ) Dropout threshold (r)</td><td>Value 2048 (MuJoCo), 512 (Roboschool) 3 • 10 -4 10 (MuJoCo), 15 (Roboschool) 64 (MuJoCo), 4096 (Roboschool) 0.99 0.95 0.2 0.5 0.5 0.3</td></tr></table>

Peer review (iclr2020):
In their post-review revision, the authors have added a much clearer motivation for SAUNA, along with extra experiments that validate and clarify the approach. The revision is a significantly better paper than the original, and I am updating my score accordingly. ------------- This paper introduces *the fraction of variance explained* V^{ex} as a measure of the ability of the value function to fit the true reward. Given the mean per-timestep reward as a baseline, V^{ex} measures the proportion of the variance in reward that is captured by the value function. In this paper, the authors introduce a filtering condition aimed to ensure that no single transition excessively reduces the fraction of variance explained. This filtering condition relies on a prediction of the variance explained, which comes from an extra head added to the standard PPO architectures, and its parameters are updated along with all other model parameters at the end of each trajectory. Intuitively, I can believe that learning will be more stable under the condition that no single transition leads to too great a divergence between the predicted reward and true reward. However, I do not understand the authors* assertion that this filtering procedure removes noisy samples (how can we characterize these samples as noise?). I*m also insufficiently familiar with the related literature to properly gauge the theoretical implications of this modification (see low confidence below). The paper compares learning accuracy over time of PPO with and without the variance explained filtering. It seems that the filtering does improve learning for the MuJoCo environments, as well as low resource models for a harder task from the Roboschool environment but it is hard to state definitively that the new method is better. I commend the authors on their discussion of non-positive Atari results in the appendix and I agree that it contributes significantly to the paper. With the caveat that this paper is quite far from the realm of my expertise. I think that the approach is intriguing but, in the absence of any theoretical justification for the approach, I*m not sure that the empirical results are sufficiently convincing for ICLR. I also believe that the paper would be easier to understand with a more thorough investigation of the effect of the filtering on the learning procedure. Questions for the authors: - what proportion of samples are rejected - how does this proportion change over the course of learning - how was the V^{ex} threshold of 0.3 chosen