Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a novel policy improvement lower bound that provides theoretical evidence for the off-policy sample reuse of PPO algorithms. Specifically, the theorem justifies the clip 
term in PPO as a regularization in the lower bound. 
main_review: ### Methodology

The proposed learning rate schedule is similar to the adaptive Adaptive KL penalty coefficient in PPO. Could the author comment on which schema has better control on the TV? 


Presumably, the older policy will have larger TV divergence comparing to the more updated policies, 
is there a better approach than uniform weight? Does this weighting strategy affect a lot?

### Experiments

The experimental results of this work are relatively weak. 1) The common practice in the literature of sample-efficiency algorithms [continuous action space] benchmark six mujoco environments. 2) The only baseline in this work is PPO. It would be helpful to also compare with off-policy algorithms [TD3, SAC, DDPG] as this work enables the sample reuse. 

When comparing to the baselines, I think the fair comparison would be set the replay ratio same as the baseline. Currently, the replay ratio is B times of PPO. For off-policy approaches, the replay ratio can affect performance a lot. 

Furthermore, PPO is not restricted to continuous action environments. It would be more solid to also benchmark the proposed algorithm on discrete action space environments [e.g., Atari]. 

It would be interesting to show the balance between wall clock time and sample efficiency. How does the proposed regularization affect the wall clock time? 

Figure 2 indicates the performance drop is aligned with the change in total variation distance is inspiring. 

It looks like most of the performance gain is from the adaptation of learning rate based on the results of 
PPO-Adapt in the appendix. It would be helpful to see what is the performance of GePPO without learning rate adaptation. 




limitations_and_societal_impact: There is no potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a generalization of PPO to the off-policy setting.

PPO has been extensively used with multiple policy updates, making it inherently off-policy and it was no clear how the algorithm should be modified in order to preserve its theoretical guarantees. 
main_review: The adaptation or the bound from Achiam follows easily when considering a distribution over the last M policies. The additional theoretical results help understanding how one should choose this distribution or adapt the learning rate of the policy in a more principled way. The authors perform an ablation, showing that the benefits in performance come both from the adapted clipping mechanism and from the reuse of past data.

I think it would be interesting to compare this also to the case when the generalized clipping mechanism is missing

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work generalizes PPO to the off-policy setting. First the policy improvement bound from Achiam et al. (2017) was generalized to accomodate for several prior policies. The authors also made a connection between the total variation distance and the PPO clipping term. A new algorithm GePPO is proposed which is shown to be more sample efficient compared to standard PPO.
main_review: Overall I think this is a very well-written paper. To the best of my knowledge, this work is the first to draw a direct connection between the monotonic policy improvement theorem and the clipped PPO algorithm, including deriving a relationship between the clipping mechanism and the total variational distance. The generalization of the policy improvement bound to past policies is also quite innovative. Most of the proofs in this paper directly extend upon the proofs in Achiam et al. (2017) and do appear to be correct.  Different components of the new algorithm were thoroughly analyzed. Experiments presented in this paper are mostly quite well designed though I think a comparison with some popular off-policy algorithms and also TRPO (which is a more direct application of the policy improvement bound) would be of interest. I also particularly like the fact that the authors are very forthcoming with the limitations of their paper, since the proposed algorithm is a variation of PPO, I do not expect it to achieve SOTA performance but I believe this work does provide additional useful insights into the PPO algorithm and serves as a useful jumping off point for the community for further research.

limitations_and_societal_impact: See main review
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: 
This paper extends PPO to an off-policy case. They develop policy improvement bound for the off-policy setting. 

main_review: 
Generally, I like the idea of the paper. Extending PPO to off-policy case is an interesting and promising aspect. The paper has provided concise theorem derivation and an elegant realization of off-policy PPO. I think it's important and valuable.

I have gone through the detail of the derivation and the theorems seem correct for me. I like the insightful derivation in Theorem 1, Lemma 3, and Definition 1. 

However, I do not give a high rating as the paper does not demonstrate the effectiveness of the methods (see the detail of the comment below).







The main weakness of the paper is the experiment.
- There are abundant environments in Mujoco and Atari. Since the author has provided a very general algorithm, I expect the author to conduct experiments on more environments to verify the effectiveness of the newly proposed methods.
- The comparison methods is missing. 
A) Comparison with other PPO variants. There are several methods (as you have stated in related work) which can improve PPO while require equal computation as PPO. I expect the author to compare with these methods or extend their methods to these settings (since the method in the paper is very general and can also be extended to these PPO variants. I should not blame the author for this, but the author should verify the effectiveness of the proposed method). 
B) Comparison with other off-policy methods, such as DDPG, SAC. One shortcoming of PPO is its on-policy limitation. I expect the proposed off-policy PPO could be competitive with other off-policy methods.
- Alation study on the parameter \epsilon. By the way, I don't understand why \epsilon is not presented in Algorithm 1 (Input) and APPENDIX Table 2.




Minor questions:
In my view, one critical component for success is the estimation of A^{\pi_k}(s,a) using the off-policy data of \pi_{k-i}. They use the V-trace method for estimation. I expect the author to discuss on this (not necessary).


limitations_and_societal_impact: N/A
review_title: null

Meta-review: After reading each other's reviews and the authors' feedback, the reviewers discussed the merits and flaws of the paper. 
The authors' answers solved most of the doubts exposed in the reviews, and the reviewers agree that this paper can be accepted for publication.
I want to congratulate the authors and invite them to modify their paper following the reviewers' suggestions.
Decision: Accept (Poster)