Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new approach to regularize on-policy methods utilizing two soft trust region terms. The first term encourages the new policy to stay close to old policy like PPO and the second term enforces the new policy to stay close to the combination of past policies called virtual policy. The virtual policy is built by a convex combination of past policies in the memory through attention weights. The motivation to use attention is to ensure the virtual policy has good performance on current data. To evaluate their method, this paper uses 3 control tasks (i.e. Pendulum, LunarLander and Bipedal- Walker), MiniGrid library for sparse reward environments evaluation, 6 Mujoco tasks, and 6 Atari games.
main_review: I agree with the authors that enforcing policy updates in constrained on-policy methods considering only previous policy can be suboptimal and it is better to utilize the history of past policies to make the update more efficient and it can be a right direction to improve the performance of on-policy methods. That being said, there are some concerns in terms of novelty, related works, and experimental results: 

- While using attention weights to combine the past policies is interesting and shows some promising results and I consider this is the main novelty of this paper, the idea of improving trust region constraint in on-policy methods has been extensively studied in recent years ( which is not a weakness though). However, one of the main concerns with this paper is it fails to discuss and compares with two important related works [1,2] which are more relevant to this paper. Both [1,2] proposed ways to improve constrained on-policy methods by introducing new ways to impose trust regions constraint: [1] propose a way to adaptively adjusts the clipping range within the trust region and [2] introduced a way to adaptively enforce soft trust region between current policy and all previous ones. Comparing and discussing these works in this paper, can help to highlight the contribution of this work and to better situate this work against them.
 
- Although this paper used various benchmarks to compare with their work, I found the experimental results are not conclusive. Specifically, it is not clear which method is the best performing method on Atari and major experiments ( e.g. this method only shows better results Gopher and Enduro not others). Another concern is that the choice of Atari games seems very arbitrary. What is the logic to select only these games? Does your method only show good performance on these environments, not the rest?  In addition, it is really hard to come to any conclusion just by looking at Table 2 without seeing the learning curves for these experiments that summarize these results.  I have to acknowledge that the authors did a good job in regard to the ablation studies. But unfortunately this has not been the case for comparison with other methods. It is better to show clear improvement on two benchmarks rather than 5 different ones without a clear conclusion. 

- The paper is written fairly well, except for the experimental section as it doesn't have a smooth flow as other sections. I highly suggest improving presentation in that section.

- It is not clear to me how the choice of $\beta$ affects the two trust regions. Author explained on page 4, but it is confusing. Can the authors explain in detail why Eq2 makes sense? In addition, you said "The intuition is that we encourage the enforcement of the constraint when $\pi_\theta$ is too far from $\theta_{old}$ using the distance between $\theta_{old}$ and $\theta_{\phi}$ as a reference" if that is the case, why shouldn't you use beta ($1-\beta$) for previous policy and ($\beta$) for virtual policy then?


[1] Trust Region-Guided Proximal Policy Optimization
https://arxiv.org/abs/1901.10314

[2] P3O: Policy-on Policy-off Policy Optimization
https://arxiv.org/abs/1905.01756 

summary_of_the_review: In general, this paper's proposed method has merits. But unfortunately, it missed some important related works and experimental results are not conclusive enough yet.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new trust-region based policy optimization method with two trust regions constraints for policy updates which enforces the proximity of the current policy to (1) the old policy prior to the current update and (2) a virtual policy constructed using an attention mechanism and memory buffer which stores several past policies. In addition a new algorithm based the the aforementioned ideas + minibatch updates is also proposed, which is shown to perform well on a series of high-dimensional control tasks.
main_review: I think this is quite a well-written paper. While other works have explored the idea of enforcing trust region on all or several past policies, the use of trust region constraints along with an attention mechanism for past policies is to the best of my knowledge novel. My detailed comments are as follows:

- The presentation of the paper is mostly clear and overall I found it quite easy to read.
- The experiments are very well-run, and I think it is one of the main strength of this paper. The authors demonstrated results on a large set of environments. The ablation studies are most appreciated and included discussions on most of the key components of the algorithm.
- Can you comment briefly on the intuition behind conditional writing?
- How did you determine the context vector features? You also mentioned in your ablation studies that using fewer context features causes a loss of information. Did you find any features that were particularly important?
- (Minor) I suggest the authors change the heading of Section 2, Constrained Policy Optimization (CPO) (Achiam et al. 2017) is a well known paper for constrained RL and the heading may cause some confusion.
- (Minor typo) Last paragraph of page 3: ...dynamically 'weigh' the whole KL....
summary_of_the_review: Overall, I think this paper proposed some very novel and insightful ideas with well-run experiments that would be beneficial to the community. I think this paper passes the bar for acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper concerns the suboptimality of the "previous" policy in the trust-region policy optimization algorithm. And the authors propose MemoryConstrained Policy Optimization (MCPO) that uses an additional trust-region relying on the virtual policy, the one that represents the history of policies.
main_review: Pros: The overall paper is easy to follow, and the motivation is also attractive. The suboptimality of the previous policy in the trust region indeed cases undesirable constraints for follow-up optimization. Experimental results also validate the effectiveness algorithm.

Cons: 1. The major concern I have is how MCPO addresses such suboptimality problems. Use the example in the paper, when the previous policy is at the local optimum, the new policy will also be constrained around the local optimum. But how MCPO policy can escape from this optimum? There might be some history policies that explore other regions of the landscape, how are these policies guaranteed to be chosen as the additional constraint? Otherwise, adding the worse history policies which lie in the convex hull of the local optimum will even harm the policy optimization.\
2. It would be helpful to provide some theorem support, maybe under ideal cases (e.g., when the memory is not constrained).\
3. In experiments Fig. 2 and 3, the curves are not converging. Could you run for more iterations?

summary_of_the_review: The motivation of the suboptimality is sound, but the algorithm to solve this problem is not very convincing.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this work, the authors propose the use a trust-region method in the vein of Proximal Policy Optimization (PPO). In contrast to previous work, the author propose to use two trust regions instead of a single one. Instead of constraining the policy to stay near to the previous policy, the author's proposed algorithm includes an extra virtual policy constructed from a memory buffer of past policies.
main_review: I have several concerns with this paper. In its current form, the paper is somewhat confusing to the reader, and could use from a clearer rewrite. More importantly, the novelty of the paper appears to be minor and the the design decisions of the author's algorithm do not seem to be very well explained.

- Why is there a need to have two trust regions? Following the author's argument for their  virtual policy, would it not be sufficient to use simply a properly designed trust region with respect to to it?

- The design of the virtual policy seems to be completely ad-hoc and unmotivated. Why are the policies in the memory buffer mixed by a weighted average of their parameters? A big reason to use the KL divergence is due to the fact that similarity in the parameter space may not lead to similarity in policy output (specially for parametrization such as neural networks). 
summary_of_the_review: In its current form, I cannot recommend this paper for publication.
review_title: null

Meta-review: Two trust region constrained optimization for policy gradient RL, where the second trust region is based on a virtual policy built from a memory buffer and using an attention mechanism to combine prior policies. The reviewers agree that the paper is well written, the idea is novel, and the paper is extensively evaluated. The authors are commended for running the additional baselines during the rebuttal period.

However, the paper still contains some shortcomings, specifically, the results are somewhat inconclusive even after the rebuttal. While it is not expected that the method wins across the board, it is important to provide an analysis of the limitations of the method. When is the algorithm appropriate to use, and when is it not? 

To make the paper stronger, in the next version of the paper should:
- move the theory in the main text (Appendix C).
- provide the analysis of the algorithm and its limitations.
Decision: Reject