SAMPLES ARE USEFUL? NOT ALWAYS: DENOISING POLICY GRADIENT UPDATES USING VARIANCE EXPLAINED
Year: 2019

Abstract:
Policy gradient algorithms in reinforcement learning optimize the policy directly and rely on efficiently sampling an environment. However, while most sampling procedures are based on sampling the agent’s policy directly, self-performance measures could be used to improve sampling before each policy update. Following this line of thoughts, we propose SAUNA, a method where transitions are rejected from the gradient updates if they do not meet a particular criterion, and kept otherwise. The criterion, Vex, is the fraction of variance explained by the value function V : a measure of the discrepancy between V and the returns. In this work, Vex is used to estimate the impact transitions will have on learning: it refines sampling by simplifying the underlying state space and improves policy gradient methods. In this paper: (a) We introduce Vex, the criterion used for denoising policy gradient updates. We refer to this procedure as transition dropout and explore its implications on learning. (b) We conduct experiments across a variety of benchmark environments, including continuous control problems. SAUNA clearly improves performance. (c) We investigate how Vex reliably selects samples with most positive impact on learning. (d) We study how this criterion can work as a dynamic tool to adjust the ratio between exploration and exploitation.

text: Policy gradient algorithms in reinforcement learning optimize the policy directly and rely on efficiently sampling an environment. However, while most sampling procedures are based on sampling the agent’s policy directly, self-performance measures could be used to improve sampling before each policy update. Following this line of thoughts, we propose SAUNA, a method where transitions are rejected from the gradient updates if they do not meet a particular criterion, and kept otherwise. The criterion, Vex, is the fraction of variance explained by the value function V : a measure of the discrepancy between V and the returns. In this work, Vex is used to estimate the impact transitions will have on learning: it refines sampling by simplifying the underlying state space and improves policy gradient methods. In this paper: (a) We introduce Vex, the criterion used for denoising policy gradient updates. We refer to this procedure as transition dropout and explore its implications on learning. (b) We conduct experiments across a variety of benchmark environments, including continuous control problems. SAUNA clearly improves performance. (c) We investigate how Vex reliably selects samples with most positive impact on learning. (d) We study how this criterion can work as a dynamic tool to adjust the ratio between exploration and exploitation.

heading: 1 INTRODUCTION
text: Learning to control agents in simulated environments has been a challenge for decades in reinforcement learning (RL) (Nguyen & Widrow, 1990; Werbos, 1989; Schmidhuber & Huber, 1991; Robinson & Fallside, 1989) and has recently led to a lot of research efforts in this direction (Mnih et al., 2013; Burda et al., 2019; Ha & Schmidhuber, 2018; Silver et al., 2016; Espeholt et al., 2018), notably in policy gradient methods (Schulman et al., 2016; Silver et al., 2014; Lillicrap et al., 2016; Mnih et al., 2016). Despite the definite progress made, policy gradient algorithms still heavily suffer from sample inefficiency (Kakade, 2003; Wu et al., 2017; Schulman et al., 2017; Wang et al., 2017).
In particular, many policy gradient methods are subject to use as much experience as possible in the most efficient way. However, the quality of the sampling procedure also determines the final performance of the agent. We make the hypothesis that not all experiences are worth to use in the gradient update. Indeed, while perhaps trajectory simulations should be as rich as possible, some transitions may instead add noise to the gradient update, diluting relevant signals and hindering learning.
The central idea of SAUNA is to reject transitions that are not informative for the particular task at hand. For that purpose, we use a measure of the discrepancy between the estimated state value and the observed returns. This discrepancy is formalized with the notion of the fraction of variance explained Vex (Kvålseth, 1985). Transitions for which Vex is close to zero are those for which the correlation between the value function V and the observed returns is also close to zero. SAUNA keeps transitions where there is either strong correlation or lack of fit between V and the returns, while avoiding the learning signals to be diluted by the dropped out samples. We will examine the impact of transition dropout (inspired by Srivastava et al. (2014) and Freeman et al. (2019)) and its theoretical implications. We consider on-policy methods for their unbiasedness and stability
compared to off-policy methods (Nachum et al., 2017). However, our method can be applied to off-policy methods as well, and we leave this investigation open for future work.
The contributions of this paper are summarized as follows:
1. We propose to move from a simple policy-based sampling procedure based to one taking into account the agent’s ability in an environment measured by Vex. We explore how the use of Vex can drive the alignment between the samples used to update the policy and the agent’s progress, while simplifying the underlying state space with transition dropout.
2. We provide a method that transforms policy gradient algorithms by assuming that not all samples are useful for learning and that these disturbing samples should, therefore, be rejected. While our method is a simple extension of policy gradient algorithms, it adds a variance criterion to the optimization problem and introduces a novel rejection sampling procedure.
3. By combining (1) and (2), we obtain a learning algorithm that is empirically effective in learning neural network policies for challenging control tasks. Our results extend the stateof-the-art in using reinforcement learning for high-dimensional continuous control. We also evaluate the theoretical implications of our method.

heading: 2 PRELIMINARIES
text: We consider a Markov Decision Process (MDP) with states s ∈ S , actions a ∈ A, transition distribution st+1 ∼ P (st, at) and reward function r(s, a). Let π = {π(a|s), s ∈ S, a ∈ A} denote a stochastic policy and let the objective function be the traditional expected discounted reward:
J(π) , E τ∼π [ ∞∑ t=0 γtr (st, at) ] , (1)
where γ ∈ [0, 1) is a discount factor (Puterman, 1994) and τ = (s0, a0, s1, . . . ) is a trajectory sampled from the environment. Policy gradient methods aim at modelling and optimizing the policy directly (Williams, 1992). The policy π is generally implemented with a function parameterized by θ. In the sequel, we will use θ to denote the parameters as well as the policy. In deep reinforcement learning, the policy is represented in a neural network called the policy network and is assumed to be continuously differentiable with respect to its parameters θ.
2.1 FRACTION OF VARIANCE EXPLAINED: Vex
The fraction of variance that the value function explains about the returns corresponds to the proportion of the variance in the dependent variable V that is predictable from the independent variable st. We define Vexτ as the fraction of variance explained for a trajectory τ :
Vexτ , 1− ∑ t∈τ ( R̂t − V (st) )2 ∑ t∈τ ( R̂t −R
)2 , (2) where R̂t and V (st) are respectively the return and the expected return from state st ∈ τ , and R is the mean of all returns in trajectory τ . In statistics, this quantity is also known as the coefficient of determination R2 and it should be noted that this criterion may be negative for non-linear models (Kvålseth, 1985), indicating a severe lack of fit of the corresponding function:
• Vexτ = 1: V perfectly explains the returns - V and the returns are correlated; • Vexτ = 0 corresponds to a simple average prediction - V and the returns are not correlated; • Vexτ < 0: V provides a worse fit to the outcomes than the mean of the returns.
One can have the intuition that Vexτ close to 1 is interesting because it gives samples from an exercised behavior while Vexτ < 0 corresponds to a high mean-squared error for the value function, meaning the agent will learn from the corresponding samples. On the other hand, Vexτ close to 0 does not provide any valuable information for a correct fitting of the state value function. We will demonstrate that Vex is a relevant indicator for assessing self-performance in RL.

heading: 2.2 POLICY GRADIENT METHODS: PPO AND A2C
text: We use PPO (Schulman et al., 2017) throughout this work. We also use A2C, a synchronous variant of Mnih et al. (2016), to demonstrate SAUNA’s performance. Below we provide some details regarding PPO, as we use it generously in this work. In previous work, PPO has been tested on a set of benchmark tasks and has produced impressive results in many cases despite a relatively simple implementation. At each iteration, the new policy θnew is obtained from the old policy θold:
θnew ← argmax θ E st,at∼πθold
[ LPPO (st, at, θold, θ) ] . (3)
We use the clipped version of PPO whose objective function is:
LPPO(st, at, θold, θ) = min ( πθ(at|st) πθold(at|st) Aπθold (st, at), g( , A πθold (st, at)) ) , (4)
where
g( , A) = { (1 + )A,A ≥ 0 (1− )A,A < 0. (5)
A is the advantage function, A(s, a) , Q(s, a) − V (s). The expected advantage function Aπθold is estimated by an old policy and then re-calibrated using the probability ratio between the new and the old policy. In Eq. 4, this ratio is constrained to stay within a small interval around 1, making the training updates more stable.

heading: 2.3 RELATED WORK
text: Our method incorporates three key ideas: (a) function approximation with a neural network combining or separating the actor and the critic with an on-policy setting, (b) transition dropout reducing signal dilution in gradient updates while simplifying the underling MDP and (c) using Vexτ as a measure of correlation between the value function and the returns to allow for better sampling and more efficient learning. Below, we consider previous work building on some of these approaches.
Actor-critic algorithms essentially use the value function to alternate between policy evaluation and policy improvement (Sutton & Barto, 1998; Barto et al., 1983). In order to update the actor, many methods adopt the on-policy formulation (Peters & Schaal, 2008; Mnih et al., 2016; Schulman et al., 2017). However, despite their important successes, these methods suffer from sample complexity.
In the literature, research has also been conducted in prioritization sampling. While Schaul et al. (2016) makes the learning from experience replay more efficient by using the TD error as a measure of these priorities in an off-policy setting, our method directly selects the samples on-policy. Schmidhuber (1991) is related to our method in that it calculates the expected improvement in prediction error, but with the objective to maximize the intrinsic reward through artificial curiosity. Instead, our method estimates the expected fraction of variance explained and filters out some of the samples to improve the learning efficiency.
Finally, motion control in physics-based environments is a long-standing and active research field. In particular, there are many prior work on continuous action spaces (Schulman et al., 2016; Levine & Abbeel, 2014; Lillicrap et al., 2016; Heess et al., 2015) that demonstrate how locomotion behavior and other skilled movements can emerge as the outcome of optimization problems.
3 SAUNA: A DYNAMIC TRANSITION DROPOUT METHOD
3.1 ESTIMATING Vex
While sampling the environment, SAUNA rejects samples where V (st) is not correlated with the return from st. Therefore, Vexτ must be estimated at each timestep so we define Vexθ (st) as the prediction of Vexτ under parameters θ at state st ∈ τ . In addition, for shared parameters configurations, an error term on the value estimation is added to the PPO objective. The final objective becomes:
L(st, at, θold, θ) = E [ LPPO(st, at, θold, θ)− c1 ( Vθ(st)− R̂t )2 − c2 (Vexθ (st)− Vexτ )2 ] , (6)
where c1 and c2 are respectively the coefficient for the squared-error loss of the value function and of the fraction of variance explained function. For cases where the network is not shared between the policy and the value function, Vexτ is added to the value function network. Appendix A illustrates how Vexθ (st) is embedded in the original architecture. The rest of the network is unchanged, making it very easy to use SAUNA without altering the complexity of existing policy gradient methods.
3.2 USING Vex FOR DYNAMIC TEMPORAL ABSTRACTION
Mechanistically, SAUNA results in dropping out several transitions from a trajectory τ , before each gradient update, and as a function of the starting state st of those transitions. The mechanism is analogous to the method of dropout in deep learning but articulated here by a dropout in the state space of the MDP. To evaluate the theoretical implications of the transition dropout, our method can be formulated using the Options framework (Sutton et al., 1999; Precup, 2000) thoroughly applied to policy gradient methods in Smith et al. (2018). Previous work shows that by reasoning at several levels of abstraction, reinforcement learning agents are able to infer, learn, and plan more effectively. We detail below how SAUNA can be theoretically understood using the Options framework.
In this work, we take semi-Markov options that may depend on the entire preceding sequence. The agent is given access to a set of options, indexed by σ. Interestingly, the framework can be reduced as follows: all options share the same policy, πσ(at|st) = πθ(at|st), introduced in section 2, the same initiation set and the same termination condition which we parameterize as the function βθ(st). βθ(st) represents the probability of terminating an option and is defined as follows:
βθ(st) = { 0 if bθ(s0, . . . , st) ≥ r 1 if bθ(s0, . . . , st) < r
with bθ(s0, . . . , st) = |Vext |
|Ṽex0:t−1|+ 0 , (7)
where for simplicity we rewrite Vexθ (st) as Vext , Ṽex0:t−1 is the median of Vexθ between timesteps 0 and t− 1, 0 = 10−8 is to avoid division by zero and r is the dropout parameter. One may legitimately ask why not use directly bθ(st) = |Vext |. The rationale is practical: the ratio becomes a standardized measure as the agent learns, stabilized by the median, more robust to outliers than the mean.
Fig. 1a illustrates an example of transition dropout dynamics in SAUNA while Fig. 1b depicts the behavior of Eq. 7. The termination function βθ(st) dynamically selects the transitions for which Vext is either high or low, but not in between. Those transitions should have a strong impact on learning: a high absolute score means that the samples correspond to a state st for which the value function is highly correlated with the returns or does not fit them properly. With this modification, the resulting gradients should be affected accordingly, which we investigate in section 4.2.2.

heading: 3.3 PUTTING IT ALL TOGETHER: SAUNA SAMPLING PROCEDURE
text: In this work, we hypothesize that low correlated transitions dilute the valuable signal information of the transitions with high absolute correlation. The temporal abstraction provided by the Options framework is leveled dynamically thanks to a termination function exploiting the self-performance measure Vext . Instead of keeping these noisy transitions, the sampling procedure drops them until the trajectory is T -steps long. Algorithm 1 illustrates how learning is achieved with SAUNA applied to PPO, in particular, the fitting of the Vex function in Eq. 10 and the transition dropout in the if statement. We choose to depict a configuration where the parameters between the policy network, the value function and the Vex function are not shared, since from this configuration the shared parameter case is direct.
Algorithm 1 SAUNA sampling procedure in PPO. Initialize policy parameters θ0 Initialize value function parameters φ0 and Vex function parameters ψ0
for k = 0, 1, 2, . . . do . For each update k
Initialize trajectory τ to capacity T while size(τ ) ≤ T do . For each timestep t
at ∼ πθk(st), vt = Vφk(st), Vext = Vexψk(st) execute action at and observe reward rt+1 and next state st+1 if |V ex t |
|Ṽex0:t−1|+ 0 ≥ r then
collect transition (st, at, rt, vt, st+1,Vext ) in τ else
continue without collecting the transition
Gradient Update
θk+1 ← argmax θ ∑ t∈τ min ( πθ (at|st) πθk (at|st) Aπθk (st, at) , g ( , A πθk (st, at)) ) (8)
φk+1 ← argmin φ ∑ t∈τ ( Vφk (st)− R̂t )2 (9)
ψk+1 ← argmin ψ ∑ t∈τ ( Vexψk (st)− V̂exτ )2 (10)
PPO sampling
SAUNA dropout

heading: 4 HOW DOES SAUNA LEARN?
text: For ease of reproducibility and sharing, we have forked the original baselines repository from OpenAI and modified the code to incorporate our method1. The complete list of hyperparameters and details of our implementation are given in Appendix E and F respectively. A discussion about additional experiments whose results are non-positive, but which we think contribute positively to this paper, can be found in Appendix B.

heading: 4.1 SAUNA IN THE CONTINUOUS DOMAIN: MUJOCO AND ROBOSCHOOL
text: To verify the generalizability of our method, we study SAUNA against PPO and A2C. We compare SAUNA (PPO+Vex in red) with its natural baseline PPO (PPO in blue). We use 6 simulated robotic tasks from OpenAI Gym (Brockman et al., 2016) using MuJoCo (Todorov et al., 2012). The two hyperparameters required by our method (r = 0.3 from Eq. 2 and c2 = 0.5 from Eq. 6) and all the others (identical to those in Schulman et al. (2017)) are exactly the same for all tasks. We made this
1Code is available here: https://github.com/iclr2020-submission/denoising-gradient-updates
choice within a clear and objective framework of comparison between the two methods. Thus, we have not optimized the rest of the hyperparameters for SAUNA, and its reported performance is not necessarily the best that could be obtained with more intensive tuning.
From the graphs reported in Fig. 2, we see that our method outperforms all continuous control tasks. We also present in Table 1 the scores obtained for each task on the experiments with PPO and A2C. The graphs for A2C are reported in Appendix C.1.
We then experiment with the more difficult, high-dimensional continuous domain environment of Roboschool (Klimov & Schulman, 2017) with different neural network capacities for the model. When resources are limited in terms of number of parameters, it seems natural that dropping out samples based on their predicted learning impact allows to reduce noise in the gradient update and accelerate learning. And as a result, our method is faster and more efficient than the baseline. When resources are not limited, the gap closes towards the end of the training and our method performs as well as the baseline.

heading: 4.2 UNDERSTANDING THE IMPACT OF SAUNA ON LEARNING

heading: 4.2.1 WHAT ADVANTAGE DOES DROPOUT GIVE?
text: We further study the impact of dropping out noisy samples by conducting additional experiments in predicting Vex while omitting the filtering step before the gradient update: the if statement in Algorithm 1 is removed and all transitions are collected in τ . Indeed, the SAUNA algorithm could improve the agent’s performance by simply training the shared network to optimize the variance explained head. Fig. 3 (full results are provided in Appendix D) demonstrates the positive effects of dropping out the samples. In addition, we studied the number of sample dropouts per task and their evolution throughout the training. On average, SAUNA rejects 5-10% of samples at the beginning of training which reduces to 2-6% at the end.

heading: 4.2.2 HOW DOES SAUNA IMPACT THE GRADIENTS?
text: Prior to the gradient updates, SAUNA removes the transitions for which the state value is poorly correlated with the returns. By doing so, we hypothesized that information signals from samples with significant Vex would be less diluted by dropped out samples. One can easily see by the graphs that the norm of the gradients is affected by such a change in the learning procedure. Fig. 4 shows that SAUNA dropout generates larger gradients. Hence policy updates have bigger steps, which eventually results in better performance.
(a) (b)
Figure 4: Gradients L1-norm from the (a) first layer and (b) last layer of the shared parameters network for PPO and when SAUNA is applied to PPO. Task: HalfCheetah-v2.
One may wonder quite rightly why performance is not damaged, since larger gradients could hinder learning. The performance results suggest that the gradients contain more useful information from each of the transitions that passed SAUNA sampling. In other terms, the relevant information for the task at hand is less diluted, the gradients are more qualitative and have been partially denoised.
4.2.3 HALFCHEETAH: A QUALITATIVE STUDY
Under review as a conference paper at ICLR 2020
4.3 ROBOSCHOOL
We then experimented with the more difficult, high-dimensional continuous domain environment of Roboschool: RoboschoolHumanoidFlagrunHarder-v1. The purpose of this task is to allow the agent to run towards a flag whose position varies randomly over time. It is left to fall and is continuously bombarded by white cubes that push it out of its path.
(a) (b) (c)
Figure 4: Comparison of our method with PPO-Clip on the more challenging Roboschool environment (108 timesteps, 6 different seeds). Red is our method PPO+Vex. Line: average performance. Shaded area: standard deviation.
In Fig. 4a, the same fully-connected network as for the MuJoCo experiments (2 hidden layers each with 64 neurons) is used. In Fig. 4b, the network is composed of 3 hidden layers with 512, 256 and 128 neurons. We trained those agents with 32 parallel actors. In both experiments, our method performs better and faster at the beginning. Then, when the policy and value functions benefit from a larger network, the gap closes, and our contribution does as well as its baseline. When resources are limited parameter-wise, it seems natural that filtering out samples based on their predicted training
impact allows to remove noise from the gradient update and accelerate learning.
Finally, we investigated further and conducted the ame exp riment as with the larger network (3
hidden layers with 512, 256 and 128 neurons), but with 128 actors in parallel instead of 32. The total number of timesteps is unchanged and is still equal to 100M. Results are reported in Fig. 4c: our method is still faster and achieves better performance than the baseline.
4.4 VARIANCE EXPLAINED Vex: CASE STUDY
While studying HalfCheetah-v2, we observed that for a number of seeds, PPO was converging to a local minimum forcing the agent to move forward on its back. This is a well-known behavior (Lapan, 2018). However, we observed that our method made it possible to leave from, or at least to avoid these local minima. Those particular deterministic environments can be generated reproducibly with specific seeds. This is illustrated in Fig. 5 where we can look at still frames of two agents trained for 106 timesteps on identically seeded environments.
Figure 5: Example of a deterministic environment where PPO converges towards a local minimum (top row) while our method does not (bottom row).
The behavior is entirely different for the agent trained with PPO and the agent trained with PPO
combined with our method denoising the policy gradient updates. If we look at the explained vari-
7
(a) (b)
Figure 5: (a) Example of PPO getting trapped in a local minimum (top row) while PPO+Vex reaches a better optimum (bottom row). (b) Vex score for PPO (orange) and SAUNA (green).
In HalfCheetah, a well-known behavior (Lapan, 2018) is that for multiple seeds PPO is converging to a local minimum forcing the agent to move forward on its back. However, we observed that SAUNA made it possible to leave from, or at least to avoid these local minima. This is illustrated in Fig. 5a where we can see still frames of two agents trained with PPO and SAUNA for 106 timesteps on
identically seeded environments. Their behavior is entirely different. Looking at Vex in Fig. 5b, we can see that the graphs differ quite interestingly. The orange agent seems to find very quickly a local minimum on its back while the green agent’s Vex varies much more. This seems to allow the latter to explore more states than the former and finally find the fastest way forward. Supported by the previous study, we can infer that SAUNA agents are better able to explore interesting states while exploiting with confidence the value given to the states observed so far.

heading: 5 DISCUSSION
text: Intuitively, for the policy update, our method will only use qualitative samples that provide the agent with (a) reliable and exercised behavior (high Vex) and (b) challenging states from the point of view of correctly predicting their value (low Vex). The SAUNA algorithm keeps samples with high learning impact, rejecting other noisy samples from the gradient update.

heading: 5.1 DENOISING POLICY GRADIENT UPDATES AND THE POLICY GRADIENT THEOREM
text: Policy gradient algorithms are backed by the policy gradient theorem (Sutton et al., 2000): ∇θL(θ) =∝ ∑ s∈S dπ(s) ∑ a∈A Qπ(s, a)∇θπθ(a|s). (11)
As long as the asymptotic stationary regime is not reached, it is not reasonable to assume the sampled states to be independent and identically distributed (i.i.d.). Hence, it seems intuitively better to ignore some of the samples for a certain period, to allow the most efficient use of information. One can understand SAUNA as making gradient updates more robust through dropout, especially when the update is low and the noise can be dominant. Besides, not taking all samples reduces the bias in the state distribution dπ . Therefore, it now seems more reasonable to consider the sampled states i.i.d., which we theoretically need for the policy gradient theorem.
5.2 IMPACT OF Vex ON THE SHARED NETWORK PARAMETERS
The shared network predicts Vex in conjunction with the value function and the policy. Therefore, as its parameters are updated through gradient descent, they converge to one of the objective function minima (hopefully, a global minimum). This parameter configuration integrates Vex, predicting how much the value function has fit the observed samples, or informally speaking how well the value function is doing for state st. This new head tends to lead the network to adjust predicting a quantity relevant for the task. Instead of using domain knowledge for the task, the method rather introduces problem knowledge by constraining the parameters directly.

heading: 6 CONCLUSION
text: Policy gradient methods optimize the policy directly through gradient ascent with the objective to maximize the expected return. We have introduced a new, lightweight and agnostic method technically applicable to any policy gradient method using a neural network as function approximation. The central idea of this paper is that samples that are uncorrelated with the return are dropped out and not considered for the policy update. Those low correlated samples are ignored by SAUNA, the mechanism is reflected in a dynamic temporal abstraction controlled by the estimated variance explained of each state. The relevant signal information being less diluted, this results in a denoising effect on the gradients, ultimately leading to improved performance.
We demonstrated the effectiveness of our method when applied to two policy gradient methods on several standard benchmark environments. We also established that samples can be removed from the gradient update without hindering learning but can, on the opposite, improve it. We further studied the impact that such a modification in the sampling procedure has on learning. Several open topics warrant further study. The influence of SAUNA on the distribution of states could, for instance, be theoretically investigated. We also find the effort (Cobbe et al., 2019; Zhang et al., 2018) to go further towards generalization in RL very promising, and we think SAUNA could be useful in these problems as a way to regularize policy gradient methods.

heading: B ADDITIONAL EXPERIMENTS WITH NON-POSITIVE RESULTS
text: Atari domain. We tested our method on the Atari 2600 domain (Bellemare et al., 2013) without observing any improvement in learning. By comparing the two algorithms where the method of filtering the samples is used or not, we could not observe any difference, as some tasks were better performed by one method and others by the other.
Mean of Vex. Although Ṽex, the median of Vex, is more expensive to calculate, we observe that it gives much better results than if we use its mean in Eq. 7. Using the median helps (Kvålseth, 1985) because the distribution of Vex is not normal and includes outliers that will potentially produce misleading results.
Non-empirical Vex. We also experimented with using the real values of Vex in Eq. 7 when calculating Ṽex0:t−1, instead of the predicted ones. This has yielded less positive results, and it is likely that this is due to the difference between the predicted and actual values at the beginning of learning, which has the effect of distorting the ratio in Eq. 7.
Adjusting state count. In order to stay in line with the policy gradient theorem (Sutton et al., 2000), we have worked to adjust the distribution of states dπ to what it really is, since some states that the agent has visited are not included in the gradient update. We adjusted it using the ratio between the number of states visited and the actual number of transitions used in the gradient update, but this did not improve the learning, and instead, we observed a decrease in performance.
Random dropout. We experimented with dropping out at random, and before each gradient update, a number of samples corresponding to the same approximate number of samples that SAUNA drops. This resulted in a decrease in performance compared to PPO, as one can expect.

heading: C ADDITIONAL EXPERIMENTS WITH SAUNA IN THE CONTINUOUS DOMAIN
text: C.1 MUJOCO: COMPARISON OF A2C+SAUNA AND A2C
In Fig. 7 we compare A2C+SAUNA (A2C+Vex in red) with A2C (A2C in green) on again 6 simulated robotic tasks from OpenAI Gym using the MuJoCo physics engine.
C.2 ROBOSCHOOL: COMPARISON OF PPO+Vex AND PPO
We experiment here with the more difficult, high-dimensional continuous domain environment of Roboschool (Klimov & Schulman, 2017): RoboschoolHumanoidFlagrunHarder-v1. The purpose of this task is to allow the agent to run towards a flag whose position varies randomly over time. It is continuously bombarded by white cubes that pushes it out of its path, and if it does not hold itself up it is left to fall.
In Fig. 8a, the same fully-connected network as for the MuJoCo experiments (2 hidden layers each with 64 neurons) is used. In Fig. 8b, the network is composed of a larger 3 hidden layers with 512, 256 and 128 neurons. We trained those agents with 32 parallel actors. In both experiments, SAUNA performs better and learns faster at the beginning. Then, only when the policy and value functions benefit from a larger network, the gap closes, and our method does as well as the baseline. When resources are limited in terms of the number of parameters, it seems natural that filtering out samples based on their predicted training impact helps to reduce noise in the gradient update and accelerate learning.
Finally, we investigated further and conducted the same experiment with the larger network (3 hidden layers with 512, 256 and 128 neurons), but with 128 actors in parallel instead of 32. Results are reported in Fig. 8c: our method still learns faster and achieves better performance than the baseline.

heading: D ADDITIONAL EXPERIMENTS ON THE ADVANTAGE OF TRANSITION DROPOUT
text: In order to identify the effects of the training of the Vex head and the filtering out of sample, we verify the hypothesis that filtering out noisy samples does improves the performance. To do so, in Section 4.2.1, we conduct experiments where the network predicts Vex but where the noisy samples are not filtered out: the if statement in Algorithm 1 is removed and all transitions are collected in τ . In Fig. 9, we see that when the noisy samples are not filtered out the performance is worst than the baseline, confirming the positive denoising impact of filtering out the variance-selected samples.

heading: E HYPERPARAMETERS
text: We have tuned the hyperparameters of our method by performing a grid search and selecting the best combinations by considering those with the largest consensus.
F IMPLEMENTATION DETAILS
Unless otherwise stated, the policy network used for MuJoCo and Roboschool tasks is a fullyconnected multi-layer perceptron with 2 hidden layers of 64 units. For Atari, the network is shared between the policy and the value function and is the same as in Mnih et al. (2016). The architecture for the Vex function head is the same as for the value function head.

heading: G CLIPPED SURROGATE OBJECTIVE DETAILS
text: In Eq. 4, we use the following standard definitions for the advantage function:
Aπ(s, a) = Qπ(s, a)− V π(s), (12) where
Qπ (st, at) = Est+1:∞ at+1:∞ [ ∞∑ l=0 γlrt+l ] and V π (st) = Eat:∞ st+1:∞ [ ∞∑ l=0 γlrt+l ] . (13)

heading: H AN ANALOGY WITH SAUNAS
text: Saunas originated in Northern Europe and are thought to date back to 7000 BC. Their use helps to release impurities [filtered out noisy samples] and improves the regeneration of cells [improved policy gradient updates]. Their temperatures could be fatal if not regulated by humidity [Vex criterion].

Review summaries:
This paper proposes a novel way to denoise the policy gradient by filtering the samples to add by a criterion "variance explained". The variance explained basically measures how well the learn value function could predict the average return, and the filter will keep samples with a high or low variance explained and drop the middle samples. This new mechanism is then added on top of PPO to get their algorithm SAUNA. Empirical results show that it is better than PPO, on a set of MuJoCo tasks and Roboschool.