Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In their post-review revision, the authors have added a much clearer motivation for SAUNA, along with extra experiments that validate and clarify the approach. 

The revision is a significantly better paper than the original, and I am updating my score accordingly.

-------------

This paper introduces 'the fraction of variance explained' V^{ex} as a measure of the ability of the value function to fit the true reward. Given the mean per-timestep reward as a baseline, V^{ex} measures the proportion of the variance in reward that is captured by the value function.

In this paper, the authors introduce a filtering condition aimed to ensure that no single transition excessively reduces the fraction of variance explained. This filtering condition relies on a prediction of the variance explained, which comes from an extra head added to the standard PPO architectures, and its parameters are updated along with all other model parameters at the end of each trajectory.

Intuitively, I can believe that learning will be more stable under the condition that no single transition leads to too great a divergence between the predicted reward and true reward. However, I do not understand the authors' assertion that this filtering procedure removes noisy samples (how can we characterize these samples as noise?). I'm also insufficiently familiar with the related literature to properly gauge the theoretical implications of this modification (see low confidence below).

The paper compares learning accuracy over time of PPO with and without the variance explained filtering. It seems that the filtering does improve learning for the MuJoCo environments, as well as low resource models for a harder task from the Roboschool environment but it is hard to state definitively that the new method is better. I commend the authors on their discussion of non-positive Atari results in the appendix and I agree that it contributes significantly to the paper.

With the caveat that this paper is quite far from the realm of my expertise. I think that the approach is intriguing but, in the absence of any theoretical justification for the approach, I'm not sure that the empirical results are sufficiently convincing for ICLR. I also believe that the paper would be easier to understand with a more thorough investigation of the effect of the filtering on the learning procedure.


Questions for the authors:

- what proportion of samples are rejected

- how does this proportion change over the course of learning

- how was the V^{ex} threshold of 0.3 chosen
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a novel way to denoise the policy gradient by filtering the samples to add by a criterion "variance explained". The variance explained basically measures how well the learn value function could predict the average return, and the filter will keep samples with a high or low variance explained and drop the middle samples. This new mechanism is then added on top of PPO to get their algorithm SAUNA. Empirical results show that it is better than PPO, on a set of MuJoCo tasks and Roboschool.

From my understanding, this paper does not show a significant contribution to the related research area. The main reason I tend to reject this paper is that the motivation of their proposed algorithm is very unclear, lack of theoretical justification and the empirical justification is restricted on PPO -- one policy gradient method.

1) It's unclear to me how it goes to the final algorithm, and what is the intuition behind it. Second 3.1 is easy to follow but the following part seems less motivated. In section 3.2 it's unclear to me why we need to fit a parametric function of Vex. In section 3.2, it's unclear to me why the filter condition is defined as Eq (7). The interpretation is a superficial explanation of what Eq 7 means but does not explain why I should throw out some of my samples, why high and low score means samples are helpful for learning and score in between does not?

2) This paper argues the filter condition improves PG algorithms by denoising the policy gradient. This argument is not justified at all except a gradient noise plot in one experimental domain in figure 5b. That's not enough to support the argument that what this process is really doing. Some theoretical understanding of what the dropped/left samples will do is helpful.

3) The method of denoising the policy gradient is expected to help policy gradient methods in general. It's important to show at least one more PG algorithm (DDPG/REINFORCE/A2C) where the proposed method can help, for verifying the generalizability of algorithm.

In general, I feel that the content after section 3.1 could be presented in a much more principled way. It should provide not just an algorithm and some numbers in experiments, but also why we need the algorithm, what's the key insight of designing this algorithm, what the algorithm really did by the algorithm mechanism itself instead of just empirical numbers.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a simple modification to policy gradient methods that relies on the variance explained to filter samples. The paper contains experiments showing empirical gains to the method, as well as some evidence that filtering rather than simply predicting more quantities is making a difference. 

I put a "weak accept" score. This method is novel, afaict, and is based on interesting statistical hypotheses. I think showing why this method is relevant could be better executed. There are some gains compared to a PPO baseline, but the gains are somewhat incremental and may only apply to PPO, as other PG methods aren't tested.

Detailed comments:
- is this method always relevant? Are there environments where it makes more sense to use? (in terms of reward density/variance, exploration difficulty, etc.) Policy gradient algorithms for which it makes more sense to add?
- The conclusion in particular claims much more than what is in the paper:
>> "applicable to any policy gradient method", technically yes, but it remains untested
>> "SAUNA removes noise", also remains to be seen. A graph showing this would add a lot to this paper
>> "We [..] studied the impact [..] on both the exploitation [..] and on the exploration", Figure 5 is a single data point, where one run for one environment got out of a "well-known" local minima. To really convince readers of this you would need to test your method on multiple environments with actual exploration and sparse rewards (Mujoco locomotion tasks do not satisfy those requirements, even though they has local minima, this is far from exploration as commonly understood in RL)
- This method is very much related to prioritized experience replay and others, as observed in 2.2, yet no comparison is made (PER can be implemented online using two exponential moving averages to estimate the distribution of TD error, used to do rejection sampling). Simpler baselines could also have been tested against, e.g. simply rejecting samples with high TD error.
- It's not clear how the threshold of 0.3 was chosen, nor what its effect is empirically. 
- Estimating V^ex vs using empirical V^ex seems to make a big difference. It's not obvious why we should be estimating V^ex at all, and I think this deserves more analysis.
- I've seen Roboschool cited, you can use @misc{klimov2017roboschool, title={Roboschool}, author={Klimov, Oleg and Schulman, J}, year={2017} }
review_title: Official Blind Review #1

Meta-review: The authors aim to improve policy gradient methods by denoising the gradient estimate. They propose to filter the transitions used to form the gradient update based on a variance explains criterion. They evaluate their method in combination with PPO and A2C, and demonstrate improvements over the baseline methods.

Initially, reviewers were concerned about the motivation and explanation of the method. The authors revised the paper by clarifying the motivation and providing a justification based on the options framework. Furthermore, the authors included additional experiments investigating the impact of their approach on the gradient estimator, showing that with their filtering, the gradient estimator had larger magnitude.

Reviewers found the justification via the options framework to be a stretch, and I agree. The authors should explain how the options framework leads to dropping gradient terms. At the moment, the paper describes an algorithm using the options framework, however, they don't connect the policy gradients of that algorithm to their method. Furthermore, the authors should more clearly verify the claims about reducing noise in the gradient estimate. While the additional experiments on the norm are nice, the authors should go further. For example, if the claim is that the variance of the gradient estimator is reduced, then that should be verified. Finally, there are many approaches for reducing the variance of the policy gradient (Grathwohl et al. 2018, Wu et al 2018, Liu et al. 2018) and no comparisons are made to these approaches.

Given the remaining issues, I recommend rejection for this paper at this time, however, I encourage the authors to address these issues and submit to a future venue.

Decision: Reject