Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work addresses the problem of policy optimization in policy gradient methods (PG) in deep RL. Due to the off-policy nature of using mini-batch data to make updates to the policy, the policy updates can become very far from the areas where data has been collected. In this work, the authors propose to construct a so-called virtual policy that is a weighted mixture of several past policies, where the weights are learned via an attention network. The policy updates are then regularized to stay within a KL-distance region around the virtual policy and the previous behaviour or “old policy”. In this way, if the old policy performs poorly or is stuck in a local optimum, the new policy will be encouraged towards the virtual policy. The authors provide empirical evidence of their proposed method having better performance than some previous works that also form trust regions or constraints of similar kinds (i.e. TRPO, PPO, and the like) on a variety of domains.
strengths_and_weaknesses: The proposed method of keeping a virtual policy that keeps track of past policies in order to avoid the pitfalls of simply keeping close to a potentially bad or suboptimal old policy is straightforward and sensible. I also think the use of a learned attention network to weigh these policies is a good idea. The empirical evaluation also shows good results on a wide variety of domains.
I think the work lacks in a theoretical understanding of the impact of the proposed method. Does the virtual policy really help escape local optima? Are the improvements observed in performance really due to the virtual policy or are they due to for example, an improved optimization with the new objective? 

Strengths:
- Good idea, simple to implement and use, asymptotic results, and learning curves (in the appendix) appear to consistently show improved performance over baselines.

Weaknesses:
- In-depth understanding of the impact of virtual policy as stated above.

Writing:
- Overall, the writing quality is good, and the paper is easy to follow.
- Definitions of setting is missing, for example I did not realize we were in a finite-horizon setting until I read Alg 1. 
- L168: typo: optimize $\theta$ and *$\phi$* alternately
- L178: N or M (L113-115) for number of past policies?
- Section 4.2 results are in the Appendix but talked about in the main body. I understand space constraints but I would recommend only talking in-depth about results that can be shown in the main body for improved readability.


questions: - One potential issue I see with the way Eqn (2) chooses which new policy to include in its memory is that if the list of policies becomes very uniform $D(\theta_{old}, \psi)$ would become very small and many policies would get added, potentially making the usage of the memory useless. Have the authors observed something like this in their experimentation and how do they address it?
- L253-254: Regarding Hopper “prefers an average restriction”. Not sure I understand this point. What does it mean that it prefers an average restriction? To my understanding, since MCPO favours better-performing policies, this means that the ~middle policy performed the best, but on average all the policies were not very bad since "Mean $\psi$" also performed similarly. Are the authors saying this is related to the environment? If so, what kind of qualities of the environment might lead to this? Or is this more related to the optimization procedure? 
- How do we know if the regularization is working, i.e. the new policy is closer to $\psi$ when the old policy becomes stuck or bad, and that this is the reason for improved results? I think some hand-designed experiments that demonstrate or visualize the effect of having the virtual policy would be interesting (similar to Figure 1 but more comprehensive, i.e. including situations where $\theta_{old}$ is forced to be a bad or locally optimum policy)

limitations: The authors discuss some reasonable limitations in their work.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper considers the trust-region-based policy optimization methods in deep reinforcement learning for which the policy update (from an old policy to a new one) at each iteration should be constrained by a trust region. This paper argues that for this family of methods the trust region constraint could impede the policy update since the old policy might not necessarily be a good one to define the trust region. The paper then proposes an improvement by incorporating into the TRPO objective a second type of trust region, which is defined as the KL divergence between the target policy and a linear combination of some history policies, i.e., *virtual trust region*. In order to effectively find such linear combination, the paper presents an attention-based mechanism to learn the combination coefficients based on some contextual features of the policy. The paper also empirically evaluates the augmented trust region method, Memory-Constrained Policy Optimization (MCPO) on both discrete and continuous control tasks.  
strengths_and_weaknesses: **Strengths**: 

[clarity] The paper presents an interesting idea of “extrapolating” the trust region by introducing a virtual one. The way of learning an attention-based mechanism to determine the trust region is also inspiring. 


**Weaknesses**: 

[originality] Lack of convincing evidence to motivate the work and method: The paper argues that the trust region defined according to the old policy could lead to “suboptimal policy update” (line 34 to 36, line 103 to 104). This argument is interesting. It would be more convincing if the paper could provide any empirical evidence to support such argument. Also, it is unclear how the “new policy may fall into a local optimum”. The trust region constraint itself is to ensure that updating policy could result in a performance improvement. That is, as long as the policy can be improved, regardless how much the improvement is, the trust region is then effective and the policy update is valid. Thus, it remains unclear to me how such local optimum would really impact the training. 

[clarity & significance] Lack of important technical details: One of major contributions of this paper is introducing “virtual trust region”, which is defined as the KL divergence between the psi policy and the new policy. The psi policy is constructed based on the context vector (Tab 4). However, it remains largely unknown what the distance between $\theta$ and $\psi$ is (feature dim 1, 2 & 3), how the return of a new policy (before it is updated) is computed (feature dim 6, 10, 11 & 12). Furthermore, it is also unclear how the psi weights are optimized. It is not straightforward how Eq. (5) is maximized w.r.t. $\varphi$. The lack of these important details could downplay the significance of this work. 

[experimental results] Lack of insightful results to support how virtual trust region works: The paper presents many empirical results of MCPO. Most of them are from the lens of a whole policy training process, e.g., empirical returns. It is, however, largely unclear what role the virtual trust region plays at each policy iteration. Given that the main idea of the paper is about how a “bad” old policy would affect the trust region constraint, it would make more sense to dig into the trust region analysis (e.g., empirical KL estimates) at each iteration, rather than the empirical returns (one such analysis can be found in PPO paper, Figure 2). Moreover, the ablation studies presented in the current version are very inconclusive. No detailed training curves or analytical statistics are available. 
questions: Q1: After digging into Eq. (3), I found the virtual trust region could be replaced by the cross entropy between $\pi_\psi$ and $\pi_\theta$ without affecting the maximization. In this sense, having the “virtual trust region” could be effectively imposing an entropy regularization. Could it be possible that having an entropy regularization (i.e., maximum entropy) for TRPO and all other trust-region based methods would yield a similar empirical performance as MCPO? We can try decaying entropy coefficient or tuning the coefficient based on the learning progress. 


Q2: (line 151) How to estimate $R_t(\psi)$ from $A_t(\theta_{old})$ via importance sampling given that their state distribution are also different? 

Q3: (algo 1 line 6) How to compute $\psi$ via Eq. 1 using $\psi_{old}$, $\theta$, and $\theta_{old}$? 

Q4: How the parameter $\varphi$ is updated when optimizing Eq. (6)? 

Q5: (line 149) Why the $\alpha_t$ is defined as the ratio of return exponents? This $\alpha_t$ is different from what’s being used in Eq. 3 as it does not condition on $s_t$. 

Q6: (line 255 to 256) the paper states that “conservative updates are less important in some tasks”. Does it mean the virtual trust region constraint could impede training in some environments? This would be contradictory to the underlying motivation of this paper. 
limitations: yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Most on-policy methods used a constraint only on the distance to the most current previous policy, but the constraint can deteriorate the performance of the current policy if the previous policy has worse performance than before. To tackle this issue, this paper proposes a new on-policy method called MCPO that uses two trust region constraints: constraint on 1) distance to the most current previous policy, and constraint on 2) distance to a good policy, called virtual policy, which is computed using previous policies stored in a policy memory. The parameters of the virtual policy are a weighted expectation of those of previous policies in the policy memory. The weights in the expectation are parameterized by a neural network called the attention network, whose input is a 12-dim feature vector, and the neural network is updated to maximize the estimated performance of the virtual policy. Empirical results show that the proposed MCPO outperforms other on-policy methods in most environments and show that all components of MCPO are essential to have the best performance through ablation studies.
strengths_and_weaknesses: Strength
1. The main ideas are 1) constraining both the previous policy and the virtual policy, 2) defining and optimizing the virtual policy using the attention network, and 3) constructing the 12-dim feature vector for input to the attention network, and these are simple but interesting.

2. Numerous empirical results show the proposed MCPO outperforms other on-policy algorithms in most environments and show the performance of variants of MCPO to ensure that all components of the method are essential to achieve the performance. 


Weakness
1. Most of the components of the proposed method are designed heuristically based on some intuitions. For example, designing $\alpha_t$, switching $\beta$-rule, and the rule of storing a policy in the policy memory are all heuristically chosen methods. Although the authors showed the performance of MCPO with $\alpha_t = 0$, with annealing $\beta$, or with frequent writing and uniform writing, it is not enough to give reasons for why these methods improve the performance.
questions: 1. In the ablation study, are there more results of more variants of MCPO for each component?

2. Is there any reason for selecting BipedalWalkerHardcore for the experiments in the ablation study? Is there any result performed on other environments for ablation study? 
limitations: While the authors give numerous experimental results to show the effectiveness of the proposed method, as aforementioned in weakness, it is still seen that it is a combination of simple and heuristic methods. In this case, I recommend two ways to the authors. The first is adding more results with various heuristic methods. In the ablation study, the authors considered only one simplest variant of the proposed one to compare with MCPO. To give reasons for why the chosen methods are reasonable, I think that the authors should measure the performance of at least 3 or more variants of MCPO for each component, and compare them with MCPO. The second way is to find the explanation of what the chosen methods mean through deriving equations. For example, considering the meaning of the term $(1-\alpha_t) KL[\pi_{old}, \pi_\theta] + \alpha_t KL[\pi_\psi, \pi_\theta]$ when $\alpha_t$ is the softmax of the true return of $\pi_\psi$ and $\pi_\theta$ can give a reasonable explanation for choosing the proposed components. If one of these is given, it helps readers understand the effectiveness of the chosen methods.
review_title: null

Meta-review: metareview: The paper addresses a constrained policy optimization through an addition of a virtual trust region trained with an attention. Evaluations show that the proposed method outperforms other on-policy methods most of the time.

The reviewers agree that the method is novel and effective, and the evaluations are extensive. The revised paper added ablation studies that show effects of different components of the method, although some reviewers would like to see deeper analysis. 
confidence: Less certain
Decision: Accept