Samples Are Useful? Not Always: denoising policy gradient updates using variance explained
Authors: Yannis Flet-Berliac
Philippe Preux
Venue:  (2020)
Keywords: reinforcement learning
policy gradient
sampling

Abstract: Policy gradient algorithms in reinforcement learning optimize the policy directly and rely on efficiently sampling an environment. However, while most sampling procedures are based solely on sampling the agent's policy, other measures directly accessible through these algorithms could be used to improve sampling before each policy update. Following this line of thoughts, we propose the use of SAUNA, a method where transitions are rejected from the gradient updates if they do not meet a particular criterion, and kept otherwise. This criterion, the fraction of variance explained Vex, is a measure of the discrepancy between a model and actual samples. In this work, Vex is used to evaluate the impact each transition will have on learning: this criterion refines sampling and improves the policy gradient algorithm. In this paper: (a) We introduce and explore Vex, the criterion used for denoising policy gradient updates. (b) We conduct experiments across a variety of benchmark environments, including standard continuous control problems. Our results show better performance with SAUNA. (c) We investigate why Vex provides a reliable assessment for the selection of samples that will positively impact learning. (d) We show how this criterion can work as a dynamic tool to adjust the ratio between exploration and exploitation.

Under review as a conference paper at ICLR 2020
SAMPLES ARE USEFUL? NOT ALWAYS: DENOISING POLICY GRADIENT UPDATES USING VARIANCE EXPLAINED
Anonymous authors Paper under double-bling brind review
ABSTRACT
Policy gradient algorithms in reinforcement learning optimize the policy directly and rely on efficiently sampling an environment. However, while most sampling procedures are based on sampling the agent's policy directly, self-performance measures could be used to improve sampling before each policy update. Follow- ing this line of thoughts, we propose SAUNA, method where transitions are rejected from the gradient updates if they do not meet particular criterion, and kept otherwise. The criterion, VEz, is the fraction of variance explained by the value function V: measure of the discrepancy between V and the returns. In this work, Vr is used to estimate the impact transitions will have on learning: it refines sampling by simplifying the underlying state space and improves policy gradient methods. In this paper: (a) We introduce yea, the criterion used for de- noising policy gradient updates. We refer this procedure transition dropout and explore its implications on learning. (b) We conduct experiments across vari- ety of benchmark environments, including continuous control problems. SAUNA (c) We investigate how yez reliably selects sam- ples with most positive impact on learning.
clearly improves performance.
(d) We study how this criterion can work as dynamic tool to adjust the ratio between exploration and exploitation.
INTRODUCTION
Learning to control agents in simulated environments has been challenge for decades in rein-
forcement learning (RL) (Nguyen & Widrow 1990; Werbos 1989
Schmidhuber & Huber [991
Robinson & Fallside 1989) and has recently led to lot of research efforts in this direction (Miih
et al. 2013 Burda et al. 2019 Ha & Schmidhuber 2018: Silver et al.
2016
Espeholt et al.)
2018
notably in policy gradient methods (Schulman et al. 2016: Silver al.
2014
Lillicrap al.
2016
Mnih et al. 2016). Despite the definite progress made, policy gradient algorithms still heavily suffer
from sample inefficiency (Kakade) 20033 Wu et al. 2017 Schulman al.
2017 Wang al. 2017).
In particular, many policy gradient methods are subject to use as much experience as possible in the most efficient way. However, the quality of the sampling procedure also determines the final performance of the agent. We make the hypothesis that not all experiences are worth to use in the gradient update. Indeed, while perhaps trajectory simulations should be as rich as possible, some transitions may instead add noise to the gradient update, diluting relevant signals and hindering learning.
The central idea of SAUNA is to reject transitions that are not informative for the particular task hand. For that purpose, we use measure of the discrepancy between the estimated state value and the observed returns. This discrepancy is formalized with the notion of the fraction of variance explained yez (Káálseth 1985). Transitions for which your is close to zero are those for which the correlation between the value function V and the observed returns is also close to zero. SAUNA keeps transitions where there is either strong correlation or lack of fit between V and the returns, while avoiding the learning signals to be diluted by the dropped out samples. We will examine the impact of transition dropout (inspired by SSriaastaa et al.] (2014) and Freeman et al. (2019) and its theoretical implications. We consider on-policy methods for their unbiasedness and stability

---

Under review as a conference paper at ICLR 2020
compared off-policy methods (Nachum et al. 2017). However, method can be applied to off-policy methods as well, and we leave this investigation open for future work.
The contributions of this paper are summarized follows:
We propose to move from a simple policy-based sampling procedure based one taking into account the agent's ability in an environment measured by yea. We explore how the use of yea can drive the alignment between the samples used to update the policy and the agent's progress, while simplifying the underlying state space with transition dropout.
2. We provide a method that transforms policy gradient algorithms by assuming that not all samples useful for learning and that these disturbing samples should, therefore, be rejected. While our method is a simple extension of policy gradient algorithms, it adds a variance criterion to the optimization problem and introduces novel rejection sampling procedure.
3. By combining (1) and (2), we obtain a learning algorithm that is empirically effective in learning neural network policies for challenging control tasks. Our results extend the state- of-the-art using reinforcement learning for high-dimensional continuous control. We also evaluate the theoretical implications of our method.
2 PRELIMINARIES
We consider a Markov Decision Process (MDP) with states s E S, actions a E A. transition distribution 8 ~ P(s,, a,) and reward function r(s, a). Let {alsss, S,a A} denote a stochastic policy and let the objective function be the traditional expected discounted reward:
(TT
E TIII
y'r (St, az)
(1)
where 7 E [0, 1) is a discount factor (Puterman 1994) and T == (5g, 40 81....) is a trajectory sampled from the environment. Policy gradient methods aim at modelling and optimizing the policy directly (Williams) 1992}. The policy x is generally implemented with a function parameterized by 0. In the sequel, we will use 0 to denote the parameters well the policy. In deep reinforcement learning, the policy is represented in neural network called the policy network and is assumed to be continuously differentiable with respect its parameters 0.
2.1 FRACTION OF VARIANCE EXPLAINED: Vea
The fraction of variance that the value function explains about the returns corresponds to the pro- portion of the variance in the dependent variable V that is predictable from the independent variable st. We define De as the fraction of variance explained for trajectory
Σxer (Ree (V(v)
Ver A 1
(2)
EIEr [Re
R
where RS and (s;) are respectively the return and the expected return from state 8t E т, and R the mean of all returns in trajectory In statistics, this quantity is also known the coeffi- cient of determination H²2 and should be noted that this criterion may be negative for non-linear models (vvälseth 1985). indicating severe lack of fit of the corresponding function:
e V perfectly explains the returns V and the returns are correlated; r
yer 0 corresponds to simple average prediction V and the returns not correlated;
VEr < 0: V provides a worse fit to the outcomes than the mean of the returns.
One can have the intuition that V close to is interesting because it gives samples from ercised behavior while yer < 0 corresponds to high mean-squared error for the value function, meaning the agent will learn from the corresponding samples. On the other hand, yea close to 0 does not provide any valuable information for a correct fitting the state value function. We will demonstrate that vez is relevant indicator for assessing self self-performance in RL.
2

---

Under review as a conference paper at ICLR 2020
2.2 POLICY GRADIENT METHODS: PPO AND A2C
We use PPO (Schulman et al. 2017) throughout this work. We also use A2C, synchronous vari- of Mnih al. (2016), to demonstrate SAUNA's performance. Below we provide some details regarding PPO, as we use it generously in this work. In previous work, PPO has been tested on a set of benchmark tasks and has produced impressive results in many cases despite a relatively simple implementation. At each iteration, the new policy &new is obtained from the old policy Ooid:
&new f- argmax
E
PPO (st, as, Ood. 0)
(3)
We use the clipped version of PPO whose objective function is:
1 T.PPO (st, ar, Ootd: 0) min To((el()) A at); g(c, A (r0a)((s)
(4)
Toaia (01
where
g(c, A) = { (1+c)A - c, <.
(5)
A is the advantage function, A(s, a) A Q(s.a) - (s). The expected advantage function a is estimated by an old policy and then re-calibrated using the probability ratio between the new and the old policy. In Eq. 4 this ratio is constrained to stay within a small interval around 1, making the training updates more stable.
2.3 RELATED WORK
Our method incorporates three key ideas: (a) function approximation with neural network com- bining or separating the actor and the critic with an on-policy setting, (b) transition dropout reducing signal dilution in gradient updates while simplifying the underling MDP and (c) using V as mea- sure of correlation between the value function and the returns to allow for better sampling and more efficient learning. Below, we consider previous work building on some of these approaches.
Actor-critic algorithms essentially use the value function to alternate between policy evaluation and policy improvement (Sutton & Barto 1998 Barto al. 1983). In order to update the actor, many methods adopt the on-policy formulation (Peters & Schaal, 2008 Mnih et al. 2016 Schulman et al., 2017). However, despite their important successes, these methods sufter from sample complexity.
In the literature, research has also been conducted in prioritization sampling. While Schaul et al. (206)) makes the learning from experience replay more efficient by using the TD error as mea- sure of these priorities in an off- off-policy setting, our method directly selects the samples on-policy. Schmidhuber (199T) is related to our method in that it calculates the expected improvement in pre- diction error, but with the objective to maximize the intrinsic reward through artificial curiosity. Instead, our method estimates the expected fraction of variance explained and filters out some of the samples to improve the learning efficiency.
Finally, motion control in physics-based environments is long-standing and active research field. In particular, there are many prior work on continuous action spaces (Schulman et al.) 2016 Levine al. 2016 Heess al.) (2015) that demonstrate how locomotion behavior
& Abbeel, and other skilled movements can emerge as the outcome of optimization problems.
3 SAUNA: A DYNAMIC TRANSITION DROPOUT METHOD
3.1 ESTIMATING yeu
While sampling the environment, SAUNA rejects samples where V(st) not correlated with the return from &z Therefore, ye" must be estimated at each timestep so we define ()*())) as the pre- diction y under parameters at state s, E T. In addition, for shared parameters configurations, an error term the value estimation is added to the PPO objective. The final objective becomes:
L(88, oz Oold 0) = PPO (P (Rc, or Oold. Saaa00 0) c1 (Vo(s)) R.) - (VFT(x)) - )3 (6)
3

---

Under review as a conference paper at ICLR 2020
where and 02 are respectively the coefficient for the squared-error loss of the value function and of the fraction of variance explained function. For cases where the network is not shared between the policy and the value function, Ve is added to the value function network. Appendix (reendix/jillustatttee how (*e) is embedded in the original architecture. The rest the network unchanged, making it very easy
SAUNA without altering the complexity of existing policy gradient methods.
3.2 USING Dor FOR DYNAMIC TEMPORAL ABSTRACTION
Mechanistically, SAUNA results in dropping out several transitions from trajectory т, before each gradient update, and as a function of the starting state S, of those transitions. The mechanism is analogous to the method of dropout in deep learning but articulated here by dropout in the state space of the MDP. To evaluate the theoretical implications of the transition dropout, our method can be formulated using the Options framework (Sutton al., 1999. Precup 2000) thoroughly applied to policy gradient methods in Smith et al. (2018). Previous work shows that by reasoning at several levels of abstraction, reinforcement learning agents are able to infer, learn, and plan more effectively. We detail below how SAUNA can be theoretically understood using the Options framework.
In this work, we take semi-Markov options that may depend on the entire preceding sequence. The agent is given access to set of options, indexed by a. Interestingly, the framework can be reduced as follows: all options share the same policy, Ty (ar (t) = To (ag (sr), introduced section 2 the same initiation set and the same termination condition which we parameterize the function 39g (s)). (x())) represents the probability of terminating an option and is defined as follows:
(s() = {
if bo 80
8t) 2 r 8t) <r
with bo ,
8t)
if bo 80
D O:t
+
where for simplicity we rewrite (x²) ) as V. Vor-1 is the median of V between timesteps 0 and t - 1, 6n = 10-8 is to avoid division by zero and is the dropout parameter. One may legitimately
ask why not use directly bo(st) | The rationale is practical: the ratio becomes standardized measure as the agent learns, stabilized by the median, more robust to outliers than the mean.
V.s, goe V.s, gote V.x, goee V. grrs
14
14
drop
drop
drop
==
1
3 (a)
(455)
7/7
(b)
Figure 1: (a) Red solid arrows illustrate the transitions affecting the gradient update. We illustrate the termination condition mechanism with an example. For t (1, 2, 4), VT is large enough so (st) = 0. The transitions sampled by policy To affect the gradient update. The others are dropped. Note that an option can be one-step long, i.e. (ox) = 1, 30 (s++1) 0. (o 1. (b) Grey area samples: kept for the gradient update. White area samples: dropped (r = 0.3).
Fig. Ia illustrates an example of transition dropout dynamics in SAUNA while Fig. lb depicts the behavior of Eq. 7 The termination function (p(st) dynamically selects the transitions for which V either high or low, but not in between. Those transitions should have a strong impact on learning: high absolute score means that the samples correspond to state s, for which the value function is highly correlated with the returns or does not them properly. With this modification, the resulting gradients should be affected accordingly, which investigate in section 4.2.2
4

---

Under review as a conference paper at ICLR 2020
3.3 PUTTING IT ALL TOGETHER: SAUNA SAMPLING PROCEDURE
In this work, we hypothesize that low correlated transitions dilute the valuable signal information of the transitions with high absolute correlation. The temporal abstraction provided by the Options framework is leveled dynamically thanks to a termination function exploiting the self-performance measure DE Instead of keeping these noisy transitions, the sampling procedure drops them until the trajectory is I' steps long. Algorithm Tillustrates how learning is achieved with SAUNA applied PPO, in particular, the fitting of the Ve function in Eq. I0 and the transition dropout in the if statement. We choose to depict a configuration where the parameters between the policy network, the value function and the Vea function are not shared, since from this configuration the shared parameter case is direct.
Algorithm SAUNA sampling procedure in PPO.
Initialize policy parameters Og
Initialize value function parameters do and Ve function parameters voo
for A 2, do
For each update A
Initialize trajectory capacity T while size(T) < T do
b. For each timestep /
0t To, (st); 122 VoL (8x). V/ = Ven (st)
execute action ag and observe reward T141 and next state 81+1
if
2 r then
PPO sampling
1 1
collect transition (st, at, Tt, Ut, 8 Ve") in
else
continue without collecting the transition
SAUNA dropout
Gradient Update
Te (0elst)
Ox+1 f- arginax Σ
min
T (ot, (e, A (8t, ail)
(8)
To. (arl (8))
tET
Vol (st) R2g) 2
Dx+1 argmin
the+1
argmin
yer (8t) - )ecz))
(10)
1ET
HOW DOES SAUNA LEARN?
For ease reproducibility and sharing, we have forked the original baselines repository from Ope- nAI and modified the code to incorporate our method" The complete list of hyperparameters and discussion about ad-
details of our implementation are given in Appendix F ditional experiments whose results are non-positive, but which we think contribute positively to this paper, can be found in Appendix
4.1 SAUNA IN THE CONTINUOUS DOMAIN: MuJoCo AND ROBOSCHOOL.
To verify the generalizability of our method, study SAUNA against PPO and A2C. We compare SAUNA (PPO+)e> in red) with its natural baseline PPO (PPO in blue). We use simulated robotic tasks from OpenAl Gym (Brockman al., 2016) using MuJoCo (Todorov al.) 2012). The two
hyperparameters required by our method (r 0.3 from Eq. 2 and Co = 0.5 from Eq. 6) and all the
others (identical those in Schulman et 201 are exactly the same for all tasks. We made this
Code available here: htw.://wwwtttttooee/o//mmmmmnaae

---

Under review as a conference paper at ICLR 2020
choice within clear and objective framework of comparison between the two methods. Thus, have not optimized the rest of the hyperparameters for SAUNA, and its reported performance is not necessarily the best that could be obtained with more intensive tuning.
HalfCheatah
Hopper
InvertedDouhaaonddum
2009
2000
6000
1000
190
3000
2000
40000
80082
InveriodPendulum
Reacher
Swimmer
1909
-11
208
40
20
201000
100000
A0012
00:000
110003
20000
40000
X1001
A10010
110000
20:000
00/000
#000/
0000
100000
Figure 2: Comparison of SAUNA with PPO on 6 MuloCo environments (10' timesteps, 6 different seeds). Red is our method PPO+ Line: average performance. Shaded area: standard deviation.
From the graphs reported in Fig. we see that our method outperforms all continuous control tasks. We also present in Table the scores obtained for each task on the experiments with PPO and A2C. The graphs for A2C are reported in Appendix C.I
Table 1: Average total reward of the last 100 episodes over runs on the MuJoCo environments on PPO and A2C, Boldface mean + std indicate better mean performance.
Task
PPO
PPO+Yer
A2C
A2C+yer
HalrCheetah
2277 + 432
2929 169
1389 157
1731 + 147
Hopper
2106 + 133
2250 + 73
1367 110
1627 97
DovertedDoublePenduuul
6100 + 143
6893 350
4151 67
5132 + 409
InvertedPendulie
532 - 19
609 = 24
686 + 15
684 + 10
Reacher
-7.5
-7.2 = 0.3
0.8
0.7
Swimmer
99.5 + 5.4
100.8 + 10.4
44.1 + 10.3
59.0 + 5.5
We then experiment with the more difficult, high-dimensional continuous domain environment of Roboschool (Klimov & Schulman 2017) with different neural network capacities for the model. When resources are limited in terms of number of parameters, it seems natural that dropping out samples based on their predicted learning impact allows to reduce noise in the gradient update and accelerate learning. And as result, our method is faster and more efficient than the baseline. When resources are not limited, the gap closes towards the end of the training and our method performs as well as the baseline.
4.2
UNDERSTANDING THE IMPACT OF SAUNA ON LEARNING
4.2.1 WHAT ADVANTAGE DOES DROPOUT GIVE?
We further study the impact of dropping out noisy samples by conducting additional experiments in predicting you while omitting the filtering step before the gradient update: the if statement in Al- gorithm [ is removed and all transitions are collected in т. Indeed, the SAUNA algorithm could improve the agent's 's performance by simply training the shared network to optimize the variance explained head. Fig. (full results are provided in Appendix D) demonstrates the positive effects of dropping out the samples. In addition, evolution throughout the training. On average, SAUNA rejects 5- 10% of samples at the beginning of training which reduces to 2-6% the end.
studied the number of sample dropouts per task and their

---

Under review as a conference paper at ICLR 2020
Imvotto@Doobblaaardduuuu
InvertedPandulum
Reacher
190
470
-11
200
-0
+0000
40000
800000
Comparison of SAUNA with PPO on 3 MuJoCo environments (10% timesteps, 6 different seeds). Red is our method PPO+ Orange is PPO+yee without the filtering out noisy samples. Line: average performance. Shaded area: standard deviation.
Figure 3:
4.2.2 How DOES SAUNA IMPACT THE GRADIENTS?
Prior to the gradient updates. SAUNA removes the transitions for which the state value is poorly correlated with the returns. By doing so, we hypothesized that information signals from samples with significant yes would be less diluted by dropped out samples. One can easily see by the graphs that the norm of the gradients is affected by such change in the learning procedure. Fig, [ shows that SAUNA dropout generates larger gradients. Hence policy updates have bigger steps, which eventually results in better performance.
2.204
2,001
(a)
(b)
Figure 4: Gradients L1-norm from the (a) first layer and (b) last layer of the shared parameters network for PPO and when SAUNA is applied PPO. Task: HalfCheetah-v2.
One may wonder quite rightly why performance is not damaged, since larger gradients could hinder learning. The performance results suggest that the gradients contain more useful information from each of the transitions that passed SAUNA sampling. In other terms, the relevant information for the task at hand is less diluted, the gradients more qualitative and have been partially denoised.
4.2.3 HALFCHEETAI: A QUALITATIVE STUDY
Aul Ma
200000
600000
500000
1000000
(a)
(b)
Figure 5: (a) Example of PPO getting trapped in a local minimum (top row) while PeO+ reaches better optimum (bottom row). (b) Vee score for PPO (orange) and SAUNA (green).
In HalfCheetah, a well-known behavior Lapan 2018) is that for multiple seeds PPO is converging to local minimum forcing the agent move forward its back. However, observed that SAUNA made it possible to leave from, or at least to avoid these local minima. This is illustrated in Fig. Ba where see still frames of two agents trained with PPO and SAUNA for 10 timesteps
7

---

Under review as a conference paper at ICLR 2020
identically seeded environments. Their behavior is entirely different. Looking at V in Fig. 5b can see that the graphs differ quite interestingly. The orange agent seems to find very quickly a local minimum its back while the green agent's ' yea varies much more. This seems to allow the latter to explore more states than the former and finally find the fastest way forward. Supported by the previous study, we can infer that SAUNA agents are better able to explore interesting states while exploiting with confidence the value given to the states observed so far.
5 DISCUSSION
Intuitively, for the policy update, our method will only use qualitative samples that provide the agent with (a) reliable and exercised behavior (high ye*) and (b) challenging states from the point of view of correctly predicting their value (low V=*). The SAUNA algorithm keeps samples with high learning impact, rejecting other noisy samples from the gradient update.
5.1 DENOISING POLICY GRADIENT UPDATES AND THE POLICY GRADIENT THEOREM
Policy gradient algorithms are backed by the policy gradient theorem (uttonn et al. 2000):
VoL(O) =X Σ dh (s) Σ Q" (
(11)
As long the asymptotic stationary regime is not reached, is not reasonable to assume the sampled states to be independent and identically distributed di.i.d.). Hence, it seems intuitively better to ignore some of the samples for certain period, to allow the most efficient use of information. One can understand SAUNA as making gradient updates more robust through dropout, especially when the update is low and the noise can be dominant. Besides, not taking all samples reduces the bias in the state distribution d". Therefore, it now seems more reasonable to consider the sampled states i.i.d., which we theoretically need for the policy gradient theorem.
5.2 IMPACT OF yea ON THE SHARED NETWORK PARAMETERS
The shared network predicts yor in conjunction with the value function and the policy. Therefore, as its parameters are updated through gradient descent, they converge to one of the objective function minima (hopefully, global minimum). This parameter configuration integrates yea, predicting how much the value function has fit the observed samples, or informally speaking how well the value function is doing for state 8t. This new head tends to lead the network adjust predicting quantity relevant for the task. Instead of using domain knowledge for the task, the method rather introduces problem knowledge by constraining the parameters directly.
CONCLUSION
Policy gradient methods optimize the policy directly through gradient ascent with the objective to maximize the expected return. We have introduced a new, lightweight and agnostic method techni- cally applicable to any policy gradient method using a neural network function approximation. The central idea of this paper is that samples that are uncorrelated with the return are dropped out and not considered for the policy update. Those low correlated samples are ignored by SAUNA. the mechanism is reflected in a dynamic temporal abstraction controlled by the estimated variance explained of each state. The relevant signal information being less diluted, this results in denoising effect on the gradients, ultimately leading to improved performance.
We demonstrated the effectiveness of our method when applied to two policy gradient methods on several standard benchmark environments. We also established that samples can be removed from the gradient update without hindering learning but can, on the opposite, improve it. We further studied the impact that such a modification in the sampling procedure has learning. Several open topics warrant further study. The influence SAUNA on the distribution states could, for instance, be theoretically investigated. We also find the effort (Cobbe et al.] 2019 Zhang et al. 2018) to go further towards generalization in RL very promising, and we think SAUNA could be useful in these problems as way regularize policy gradient methods.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, 13(5):33-846, 1983.
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning envi- ronment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606. 01500, 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning. In International Conference on Machine Learning, pp. 1282-1289, 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pp. 1406-1415, 2018.
Daniel Freeman, Luke Metz, and David Ha. Learning to predict without looking ahead: World models without forward prediction. In Advances in Neural Information Processing Systems, 2019.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, pp. 2450-2462, 2018
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learn- ing continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pp. 2944-2952, 2015.
Sham Machandranath Kakade. the sample complexity of reinforcement learning. PhD thesis, University of London, 2003.
Oleg Klimov and J Schulman. Roboschool, 2017.
Tarald O. Kvålseth. Cautionary Note about R². The American Statistician, 39(4):279-2)) 1985.
Max Lapan. Deep Reinforcement Learning Hands-One On: Apply modern RL methods, with deep Q- networks, value iteration, policy gradients, TRPO, AlphaGo Zero and more. Packt Publishing, 2018.
Sergey Levine and Pieter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems, pp. 1071-1079, 2014.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national Conference on Learning Representations, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference Machine Learning, pp. 193871937, 2016.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2775-2785, 2017.

---

Under review as a conference paper at ICLR 2020
Demrick Nguyen and Bernard Widrow. The truck backer-upper: An example of self-learning in neural networks. In Advanced Neural Computers. pp. 11-19, 1990.
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural networks: the official journal of the International Neural Network Society, 21(4):682-697, 2008.
Doina Precup. Temporal abstraction in reinforcement learning. Ph. D. thesis, University of Mas- sachusetts, 2000.
Martin L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Sons, Inc., New York, NY, USA, 1st edition, 1994. ISBN 0471619779.
Anthony Robinson and F Fallside. Dynamic reinforcement driven emror propagation networks with application to game playing. In Conference of the Cognitive Science Society, pp. 836-843, 1989.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference Learning Representations, 2016.
Jürgen Schmidhuber. Curious model-building control systems. IEEE International Joint Confer- on Neural Networks, pp. 1458-145 1463, 1991.
Jürgen Schmidhuber and Rudolf Huber. Learning to generate artificial fovea trajectories for target detection. International Journal of Neural Systems, 2(1/2):135-141, 1991.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In International Con- ference on Learning Representations, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707. 777706647, 2017.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference Muchine Learning, pp- 387-395, 2014.
David Silver, Aja Huang, Christopher Maddison, Arthur Guez, Laurent Sifre, George Driessche, Julian Schrittiieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukeuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529:484, 2016.
Matthew Smith, Herke Hoof, and Joëlle Pineau. An inference-based policy gradient method for learning options. In International Conference on Machine Learning, pp. 4710-4719, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1)119291195 2014.
Richard S. Sutton and Andrew G. Barto. Introduction to reinforcement learning. Cambridge: MIT Press. 1998.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181- 211, 1999.
Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Infor- mation Processing Systems, pp. 1057-1063, 2000.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033 IEEE, 2012.
10

---

Under review as a conference paper at ICLR 2020
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukeuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. In International Conference on Learning Representations, 2017.
Paul J. Werbos. Neural networks for control and system identification. In IEEE Conference on Decision and Control,, pp. 260-265, 1989.
Ronald J. Williams. Simple statistical gradient-following algorithms for connecionissreenfoorrennttt learning. Machine learning, 8(3-4):229-22) 256, 1992.
Yuhuai Wu, Elman Mansimov, Roger Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using cronecker-factorie approximation. In Advances in Neural Information Processing Systems, pp. 5279-5288, 2017.
Amy Zhang, Nicolas Ballas, and Joëlle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv:1806.07937 2018.

---

Under review as a conference paper at ICLR 2020
A
ILLUSTRATION OF THE SAUNA ARCHITECTURE
a, ~ V
O
V
7
Shared network
Shared network
S
S
PPO (shared network)
SAUNA (shared network)
G ~ #
R4
1/r
V
Policy network
Value network
Policy network
Value network
5
S
PPO (separate network)
SAUNA (separate network)
Figure 6: Network-agnostic variance explained head.
In Fig. 6 we rewrite "/ (8x) as Vea On the right side of the figure is illustrated the V head that added to the shared the separate network configurations. Note that even if Ve is defined for sampled trajectory the model predicts its value at each state E T..
B ADDITIONAL EXPERIMENTS WITH NON-POSITIVE RESULTS
Atari domain. We tested our method on the Atari 2600 domain (Bellemare et al. 2013) without observing any improvement in learning. By comparing the algorithms where the method of filtering the samples is used or not, we could not observe any difference, as some tasks were better performed by one method and others by the other.
Mean of yes Although Dez, the median of yer is expensive to calculate, observe that it gives much better results than we use its mean in Eq. 7 Using the median helps (Kvålseth) 1985) because the distribution of vez is not normal and includes outliers that will potentially produce misleading results.
Non-empirical yor. We also experimented with using the real values of ver in Eq. when calcu- lating Vor instead of the predicted ones. This has yielded less positive results, and it is likely that this is due to the difference between the predicted and actual values at the beginning of learning, which has the effect of distorting the ratio in Eq.
Adjusting state count. order to stay in line with the policy gradient theorem (Sutton et al. 2000), have worked adjust the distribution of states d" to what it really is, since states that the agent has visited are not included in the gradient update. We adjusted it using the ratio hetween the number of states visited and the actual number of transitions used in the gradient update, but this did not improve the learning, and instead, we observed decrease in performance.
Random dropout. We experimented with dropping out at random, and before each gradient update, a number of samples corresponding to the same approximate number samples that SAUNA drops. This resulted in a decrease in performance compared PPO,
expect.
12

---

Under review as a conference paper at ICLR 2020
C ADDITIONAL EXPERIMENTS WITH SAUNA IN THE CONTINUOUS DOMAIN
C.I MuJoCo: COMPARISON OF A2C+SAUNA AND A2C
HalfCheetah
Hopper
InvertedDoubleeenddde
2000
1007
120
3000
2000
2000.0
40008
60008
100006
200004
40098
10000
180006
200054
400003
00000
000800
196909
InvertadFanddluumm
Reacher
Swimmer
-11
208008
G0000E
130000
430000
130008 20000
60083
100000
Figure 7: Comparison of SAUNA with A2C on 6 MuJoCo environments (109 timesteps, 6 different seeds). Red is our method Ve Line: average performance. Shaded area: standard deviation.
In Fig. 7 we compare A2C+SAUNA (A2C+Ve* in red) with A2C (A2C in green) on again simu- lated robotic tasks from OpenAl Gym using the MuloCo physics engine.
C.2 ROBOSCHOOL: COMPARISON OF PPO+ AND PPO
We experiment here with the more difficult, high-dimensional continuous domain environment of Roboschool (Klimov & Schulman 2017): RoboschoollHumaniddFF hww.ammnniidddgnttrarrrrrr... The purpose of this task is to allow the agent to run towards a flag whose position varies randomly over time. It continuuusly bombarded by white cubes that pushes it out of its path, and it does hold itself up it is left to fall.
80
130
(a)
(b)
(c)
Figure 8: Comparison of SAUNA with PPO the more challenging Roboschool environment (10* timesteps, different seeds). Red is our method Peo+ Line: average performance. Shaded area: standard deviation.
In Fig. Sal the same fully-connected network as for the MuJoCo experiments (2 hidden layers each with 64 neurons) is used. In Fig. [b] the network is composed of larger 3 hidden layers with 512, 256 and 128 neurons. We trained those agents with 32 parallel actors. In both experiments, SAUNA performs better and learns faster the beginning. Then, only when the policy and value functions benefit from a larger network, the gap closes, and our method does as well as the baseline. When resources are limited in terms of the number of parameters, it seems natural that filtering out samples based on their predicted training impact helps to reduce noise in the gradient update and accelerate learning.
13

---

Under review as a conference paper at ICLR 2020
Finally, we investigated further and conducted the same experiment with the larger network (3 hid- den layers with 512, 256 and 128 neurons), but with 128 actors in parallel instead of 32. Results are reported in Fig. Sc our method still learns faster and achieves better performance than the baseline.
D ADDITIONAL EXPERIMENTS ON THE ADVANTAGE OF TRANSITION DROPOUT
In order to identify the effects of the training of the y head and the filtering out of sample, we verify the hypothesis that filtering out noisy samples does improves the performance. To do so, in Section 1.2.1 we conduct experiments where the network predicts V but where the noisy sam- ples are not filtered out: the if statement Algorithm is removed and all transitions are collected in In Fig. 9 we see that when the noisy samples are not filtered out the performance is worst than the baseline, confirming the positive denoising impact of filtering out the aariance-selected samples.
HalfChhedah
Hopper
Invette@DoublFandduuum
8000
1500
5000
4000
200
500
1000
20:000
100003
20/000
1/0000
20:000
41001
InvertedPonduur Pondulum
Reacher
Swimmer
600
00
600
-12
20000
Figure 9: Comparison of our method with PPO on MuJoCo environments (10 timesteps, 6 dif- ferent seeds). Red is our method PPO+Ye2, Orange is PPO+yer without the filtering out of noisy samples. Line: average performance. Shaded area: standard deviation.
E HYPERPARAMETERS
We have tuned the hyperparameters of method by performing grid search and selecting the best combinations by considering those with the largest consensus.
Hyperparameter
Value
Horizon (7)
2048 (MuJoCo), 512 (Robaschool)
Adam stepsize
3- 10-4
Nb. epochs
10 (MuJoCo), 15 (Roboschool)
Minibatch size
64 (MuJoCo), 4096 (Roboschool)
Discount (7)
0.99
GAE parameter (A)
0.95
Clipping parameter (e)
0.2
VF coef (ci)
0.5
Vea coef (C2)
0.5
Dropout threshold (r)
0.3
Table 2: Hyperparameters used both PPO and SAUNA. The two last hyperparameters are only relevant for our method.
14

---

Under review as a conference paper at ICLR 2020
Hyperparameter
Value
Horizon (T)
2048
Adam stepsize
10
Nb. epochs
10
Minibatch size
64
Nb. envs
Discount (?)
0.99
VF coef (c)
0.5
y coef (c2)
0.5
Dropout threshold (r)
0.35
Table 3: Hyperparameters used both in A2C and SAUNA. The last hyperparameters are only relevant for our method.
F IMPLEMENTATION DETAILS
Unless otherwise stated, the policy network used for MuJoCo and Roboschool tasks is fully- connected multi-layer perceptron with hidden layers of 64 units. For Atari, the network is shared between the policy and the value function and is the same in Mnih al. (2066). The architecture for the yea function head is the same as for the value function head.
G CLIPPED SURROGATE OBJECTIVE DETAILS
In Eq. we use the following standard definitions for the advantage function:
A (s, = (= (*(s,,
(12)
where
DC
Q" (815
Σ
x'ret and V V* (s.)
E
E'rren
(13)
300
H AN ANALOGY WITH SAUNAS
Saunas originated in Northern Europe and are thought to date back to 7000 BC. Their use helps to lease impurities [filtered out noisy samples] and improves the regeneration of cells [improved policy gradient updates]. Their temperatures could be fatal if not regulated by humidity [ver criterion].
15


Reviews:
[{"review_id": "r1giq6Bk9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In their post-review revision, the authors have added a much clearer motivation for SAUNA, along with extra experiments that validate and clarify the approach. \n\nThe revision is a significantly better paper than the original, and I am updating my score accordingly.\n\n-------------\n\nThis paper introduces 'the fraction of variance explained' V^{ex} as a measure of the ability of the value function to fit the true reward. Given the mean per-timestep reward as a baseline, V^{ex} measures the proportion of the variance in reward that is captured by the value function.\n\nIn this paper, the authors introduce a filtering condition aimed to ensure that no single transition excessively reduces the fraction of variance explained. This filtering condition relies on a prediction of the variance explained, which comes from an extra head added to the standard PPO architectures, and its parameters are updated along with all other model parameters at the end of each trajectory.\n\nIntuitively, I can believe that learning will be more stable under the condition that no single transition leads to too great a divergence between the predicted reward and true reward. However, I do not understand the authors' assertion that this filtering procedure removes noisy samples (how can we characterize these samples as noise?). I'm also insufficiently familiar with the related literature to properly gauge the theoretical implications of this modification (see low confidence below).\n\nThe paper compares learning accuracy over time of PPO with and without the variance explained filtering. It seems that the filtering does improve learning for the MuJoCo environments, as well as low resource models for a harder task from the Roboschool environment but it is hard to state definitively that the new method is better. I commend the authors on their discussion of non-positive Atari results in the appendix and I agree that it contributes significantly to the paper.\n\nWith the caveat that this paper is quite far from the realm of my expertise. I think that the approach is intriguing but, in the absence of any theoretical justification for the approach, I'm not sure that the empirical results are sufficiently convincing for ICLR. I also believe that the paper would be easier to understand with a more thorough investigation of the effect of the filtering on the learning procedure.\n\n\nQuestions for the authors:\n\n- what proportion of samples are rejected\n\n- how does this proportion change over the course of learning\n\n- how was the V^{ex} threshold of 0.3 chosen", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkgfXyihYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a simple modification to policy gradient methods that relies on the variance explained to filter samples. The paper contains experiments showing empirical gains to the method, as well as some evidence that filtering rather than simply predicting more quantities is making a difference. \n\nI put a \"weak accept\" score. This method is novel, afaict, and is based on interesting statistical hypotheses. I think showing why this method is relevant could be better executed. There are some gains compared to a PPO baseline, but the gains are somewhat incremental and may only apply to PPO, as other PG methods aren't tested.\n\nDetailed comments:\n- is this method always relevant? Are there environments where it makes more sense to use? (in terms of reward density/variance, exploration difficulty, etc.) Policy gradient algorithms for which it makes more sense to add?\n- The conclusion in particular claims much more than what is in the paper:\n>> \"applicable to any policy gradient method\", technically yes, but it remains untested\n>> \"SAUNA removes noise\", also remains to be seen. A graph showing this would add a lot to this paper\n>> \"We [..] studied the impact [..] on both the exploitation [..] and on the exploration\", Figure 5 is a single data point, where one run for one environment got out of a \"well-known\" local minima. To really convince readers of this you would need to test your method on multiple environments with actual exploration and sparse rewards (Mujoco locomotion tasks do not satisfy those requirements, even though they has local minima, this is far from exploration as commonly understood in RL)\n- This method is very much related to prioritized experience replay and others, as observed in 2.2, yet no comparison is made (PER can be implemented online using two exponential moving averages to estimate the distribution of TD error, used to do rejection sampling). Simpler baselines could also have been tested against, e.g. simply rejecting samples with high TD error.\n- It's not clear how the threshold of 0.3 was chosen, nor what its effect is empirically. \n- Estimating V^ex vs using empirical V^ex seems to make a big difference. It's not obvious why we should be estimating V^ex at all, and I think this deserves more analysis.\n- I've seen Roboschool cited, you can use @misc{klimov2017roboschool, title={Roboschool}, author={Klimov, Oleg and Schulman, J}, year={2017} }", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylQuBY6tB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel way to denoise the policy gradient by filtering the samples to add by a criterion \"variance explained\". The variance explained basically measures how well the learn value function could predict the average return, and the filter will keep samples with a high or low variance explained and drop the middle samples. This new mechanism is then added on top of PPO to get their algorithm SAUNA. Empirical results show that it is better than PPO, on a set of MuJoCo tasks and Roboschool.\n\nFrom my understanding, this paper does not show a significant contribution to the related research area. The main reason I tend to reject this paper is that the motivation of their proposed algorithm is very unclear, lack of theoretical justification and the empirical justification is restricted on PPO -- one policy gradient method.\n\n1) It's unclear to me how it goes to the final algorithm, and what is the intuition behind it. Second 3.1 is easy to follow but the following part seems less motivated. In section 3.2 it's unclear to me why we need to fit a parametric function of Vex. In section 3.2, it's unclear to me why the filter condition is defined as Eq (7). The interpretation is a superficial explanation of what Eq 7 means but does not explain why I should throw out some of my samples, why high and low score means samples are helpful for learning and score in between does not?\n\n2) This paper argues the filter condition improves PG algorithms by denoising the policy gradient. This argument is not justified at all except a gradient noise plot in one experimental domain in figure 5b. That's not enough to support the argument that what this process is really doing. Some theoretical understanding of what the dropped/left samples will do is helpful.\n\n3) The method of denoising the policy gradient is expected to help policy gradient methods in general. It's important to show at least one more PG algorithm (DDPG/REINFORCE/A2C) where the proposed method can help, for verifying the generalizability of algorithm.\n\nIn general, I feel that the content after section 3.1 could be presented in a much more principled way. It should provide not just an algorithm and some numbers in experiments, but also why we need the algorithm, what's the key insight of designing this algorithm, what the algorithm really did by the algorithm mechanism itself instead of just empirical numbers.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors aim to improve policy gradient methods by denoising the gradient estimate. They propose to filter the transitions used to form the gradient update based on a variance explains criterion. They evaluate their method in combination with PPO and A2C, and demonstrate improvements over the baseline methods.

Initially, reviewers were concerned about the motivation and explanation of the method. The authors revised the paper by clarifying the motivation and providing a justification based on the options framework. Furthermore, the authors included additional experiments investigating the impact of their approach on the gradient estimator, showing that with their filtering, the gradient estimator had larger magnitude.

Reviewers found the justification via the options framework to be a stretch, and I agree. The authors should explain how the options framework leads to dropping gradient terms. At the moment, the paper describes an algorithm using the options framework, however, they don't connect the policy gradients of that algorithm to their method. Furthermore, the authors should more clearly verify the claims about reducing noise in the gradient estimate. While the additional experiments on the norm are nice, the authors should go further. For example, if the claim is that the variance of the gradient estimator is reduced, then that should be verified. Finally, there are many approaches for reducing the variance of the policy gradient (Grathwohl et al. 2018, Wu et al 2018, Liu et al. 2018) and no comparisons are made to these approaches.

Given the remaining issues, I recommend rejection for this paper at this time, however, I encourage the authors to address these issues and submit to a future venue.

Decision: Reject