Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors propose a new stochastic reduced variance policy gradient estimator, which combines a baseline GPOMDP estimator with a control variate integrating past gradients by importance re-weighting. The authors establish the sample complexity of gradient descent using the proposed estimator, and further demonstrate its effectiveness through some simple empirical results.

I believe this paper is a good contribution for ICLR. The result is relevant and interesting, and extends recent ideas around reduced-variance policy gradient estimators. The paper is overall easy to read, and presents its ideas clearly. Some detailed comments:

- The wording of theorem 4.5 and corollary 4.7 could be somewhat clarified. In particular, I did not see \Phi defined in the main text, and given its definition in the appendix, I believe theorem 4.5 could be stated simply in terms of J, avoiding any additional notation. Similarly, corollary 4.7 could be stated somewhat more clearly, and in particular, the choice of S should be made explicit. In the appendix, I could not find a definition for \phi.

- The empirical results presented are interesting, although I wish they were more comprehensive. In particular, it would be valuable to more exhaustively evaluate the impact of the hyper-parameters N, B and m. The authors should also clarify how the current values were chosen. Given that the theoretical results also apply to projected gradient descent, it would be interesting to see empirical results in that case.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper studies the theory of sample efficiency in reinforcement learning, which is of great importance and has a potentially large audience. 

The strong points of the paper:
1. This paper proposed a new algorithm stochastic variance reduced policy gradient algorithms. This paper establishes better sample complexity compared with existing work. The key part of the proposed algorithm for variance reduction is to have step-wise importance weights to deal with the inconsistency caused by varying trajectory distribution.  
2. This paper provides experimental results verifies the efficiency and effectiveness of the proposed algorithm. 
3. In addition, parameter-based exploration extension is discussed in the appendix, which enjoys the same order of sample complexity under mild assumptions and gives better empirical performance. 
4. This paper is easy to follow. In particular, there are a lot of discussions comparing this work with existing work. 

The weak points of the paper:
1. In section 3, it is not quite clear how the reference policy is defined, and the \theta^s is not clearly defined when s >= 1.
2. In the main part of the paper, the discussion in Remark 4.6 and the following Corollary 4.7 is not quite clear.


Some minor comments of the paper:
1. In introduce page 3, We note that a recent work by .... by a fator of H. --> Here H should be defined as the Horizon. 
2. There is one additional parenthesis in Theorem 4.5.  
3. In  Corollary 4.7, T is not defined. 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary: 

The paper proposed a policy gradient method called SRVR-PG, which based on stochastic recursive gradient estimator. It shows that the complexity is better than that of SVRPG. Some experiments on standard environments are provided to show the efficiency of the algorithm over GPOMDP and SVRPG.   

Comments:

1) I think you may ignore the highly-related work: Yang and Zhang, "Policy Optimization with Stochastic Mirror Descent", June 2019. Since both papers are highly-related, I would suggest the author(s) have some discussions to differentiate two papers. 

2) Could you please provide the reasons why you are only choosing two methods (GPOMDP and SVRPG) to compare? There are many policy gradient algorithms such as A3C, A2C, ... which have not mentioned or discussed in this paper. Are they completely different here? Is there no way to compare the performance among them with SRVR-PG? 

3) I am not sure if you are using the right word "novel" to describe your method. Basically, you adopt the existing estimator based on SARAH/SPIDER in optimization algorithms into RL problems. I do not think the word "novel" is proper here since it is not something total new. Notice that, the complexity result achieved in this paper is also matched the one for SARAH/SPIDER/SpiderBoost in nonconvex optimization. 

4) There are some discussion on choosing a batch-size in the experimental part. However, I do not see the discussion on the learning rate. The choice of parameters for GPOMDP and SVRPG may also need to be discussed. 

5) In Papini et al., 2018, for the experiment part, they use a snapshot policy to sample in the inner loop, and use early stopping inner loop. Moreover, they also check variance to recover the backup policy when it is blowup. Do you apply any trick to your experiments? I wonder if your numerical experiments are totally followed on your theory. 

Minor comments: 
- Redundancy ")" in \eta*S*m in Theorem 4

review_title: Official Blind Review #1

Meta-review: The paper introduces a policy gradient estimator that is based on stochastic recursive gradient estimator. It provides a sample complexity result of O(eps^{-3/2}) trajectories for estimating the gradient with the accuracy of eps.
This paper generated a lot of discussions among reviewers. The discussions were around the novelty of this work in relation to SARAH (Nguyen et al., ICML2017), SPIDER (Fang et al., NeurIPS2018) and the work of Papini et al. (ICML 2018). SARAH/SPIDER are stochastic variance reduced gradient estimators for convex/non-convex problems and have been studied in the optimization literature.
To bring it to the RL literature, some adjustments are needed, for example the use of importance sampling (IS) estimator. The work of Papini et al. uses IS, but does not use SARAH/SPIDEH, and it does not use step-wise IS.

Overall, I believe that even though the key algorithmic components of this work have been around, it is still a valuable contribution to the RL literature.

Decision: Accept (Poster)