Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel strategy for safe reinforcement learning in constrained MDPs (cMDPs). At a high level, they provide a bound on performance difference that, compared to (Achiam et al., 2017), accounts for the approximation error when applying a Taylor approximation. Then, they propose a novel algorithm based on their theory, and experimentally evaluate its performance.
strengths_and_weaknesses: Pros
- Important problem

Cons
- Incremental contribution compared to (Achiam et al., 2017)

The theoretical contributions in this paper appear a bit limited. Compared to prior work (Achiam et al., 2017), they are only providing guarantees on the approximation error. Intuitively, such a guarantee should follow pretty straightforwardly from Taylor’s theorem. If there are additional technical challenges that must be overcome, the authors should make this clear.

The algorithmic contribution is also unclear. They claim to propose a new algorithm that performs a projection based on their theory to ensure correctness. However, prior work (Achiam et al., 2017) uses a similar strategy. I’m wondering if the authors can clarify how their proposed approach differs from this existing strategy.

Finally, their experimental results also appear mixed. On MuJoCo, they do perform well, but there are several cases where existing baselines outperform their approach. The same issue holes for SafetyGym; here, they perform best on 3/4 environments, but CPO apparently significantly outperforms their approach on CarGoal1-v0 (though it is hard to tell since they cut off the graph).

questions: Can you clarify the novelty compared to (Achiam et al., 2017), both in terms of the theory and the algorithm?
limitations: None
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new safe RL method based on the constrained updated projection framework(CUP).  The authors use CUP to generalize the surrogate objective and cost(constraints) functions and solve them by the first-order method, which facilities the safety analysis for RL. The authors also compared against some safe RL baselines to show their effectiveness. 
strengths_and_weaknesses: Strengths:
The paper derives the theoretical bounds for the surrogate functions of objective and constraints to achieve monotonic policy improvement with an asymptotic safety guarantee. They claimed this work is the first to show a rigorous theoretical analysis to extend the surrogate functions w.r.t GAE. 

Weaknesses:
1. The paper may miss a significant part of the literature review and experiment comparison.  I did a quick search for the related works. 
a. CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee(last year's ICML paper), claims it can achieve optimality and zero constraints violation in a sub-linear rate. Although the authors mentioned it in the paper briefly, they didn't compare it in the experiment. In CRPO, the baselines include primal-dual optimization (PDO) and interior point optimization (IPO), the author should explain why they didn't compare with these methods either.

My understanding of safe RL does not only include the solutions for the constrained optimization problem(like CPO) but also some methods leveraging the formal methods(such as barrier function, and control barrier function, temporal logic), which this paper didn't cover at all. Some references here:
b. IPO: Interior-point Policy Optimization under Constraints (by adding the barrier penalty)
c. End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks.
d. Temporal Logic Guided Safe Reinforcement Learning Using Control Barrier Functions.

2. Another major concern is how the safety 'guarantee' is expressed in this paper. For example, in CRPO, they show that the safety violation is in a sub-linear rate while this paper only claimed they can have asymptotic constraint satisfaction, without quantitative results. It is even worse when compared to the formal methods-based approaches where they can really achieve rigorous safety guarantees.  Thus, the claim of the safety guarantee in the paper is too strong.  

 
questions: Please refer to the detailed weakness points. 
limitations: Missing a significant literature review and comparison. 
The claim of safety guarantees is too strong for the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work extends the performance difference bounds for Constrained Policy Optimization (CPO, Achiam et al, 2017) to a generalized setting that includes GAE in the bound. This provides a theoretical justification for using GAE in CPO-like methods as well as allows for a new approach, CUP, that explicitly uses these new general bounds with an implementation that does not depend on the convex approximation. The authors then validate their new approach on a variety of safe RL tasks and baselines and show the effectiveness of their approach.  
strengths_and_weaknesses: # Strengths :

- The results in the paper are original and bridge the missing justification for the use of GAE in CPO-based methods. The paper is also mostly well written for the most part. 

- The empirical study in the paper is done rigorously which further demonstrates the effectiveness of their method. Moreover, the code is provided in the supplemental that will also help future research in this area.

# Weaknesses:

- The aspects of the paper can be improved when it comes to clarity. For instance, one of the major contributions of the work is how the optimize the objective function via the first-order method (L:208-212, L:245), however, most of the details regarding this have been pushed to the Appendix instead. I would recommend the authors to expand a bit more on these aspects. Though the results are novel, most of the proof methodology seems to stem directly from the CPO proof techniques. Maybe some of the details that are direct extensions of CPO like L:177-182 can be moved to the Appendix instead and space can be provided to move the Algorithm to the main paper.

- More details should be provided on the comparison with FOCOPS. It seems like FOCOPS is the most competitive baseline, with some similarities with CUP approach such as the update rule for $\nu$ is the same as FOCOPS (L:296). In Sec 6.2, while the plots for Fig 3 (a) make sense, they are not particularly interesting as they follow the same rule as FOCOPS. A more interesting comparison could have been on the choice of initial $\nu$ and the choice of $\eta$.


questions: - Significance of Remark 1 is not clear. Comparison with Kakade has already been made in CPO. The generalized bounds with GAE don't really provide any new insights w.r.t. that comparison or do they? Will it be better to instead have a remark on comparison with CPO instead? 
 
limitations: The authors don't really discuss the limitations of their approach. Are there any scenarios where it makes sense (say computationally) to use any of the other baseline rather than the CUP approach?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 1. A new GAE style surrogate function for solving CMDP problem
2. The theory covers the previous well-known results
3. Do not require to compute the second oder information, unlike CPO and PCPO.

strengths_and_weaknesses: Pros
1. The theory looks technically-sound
2. The related work section is good. I like the comparison table of different prior algorithm shown in the appendix.
3. In the experiment section, the claim that PCPO and CPO uses the convex relaxation results of larger errors is back up by the experiment in figure 2.
4. Nice ablation studies on the projection parameter $v$, the step size, and the cost penalty. It clearly shows that the algorithm is stable, easy to optimize, and can follow the cost constraint threshold well no matter how hard the threshold is.

Cons
1. In algorithm 1 in the appendix, the protection step requires to compute the KL divergence between the old policy and the target policy, does this require to compute second order information to do an update? If this is the case, then the contribution of using less computation would not hold.
2. The idea of using projection has been extensively used in PCPO paper. The difference between PCPO and this work is the theoretical analysis. In addition, formulating the GAE problem has been discussed in Zhong et al paper. The difference between Zhong et al and this work is again the theoretical analysis.
questions: Please answer my questions in the weaknesses. 
limitations: 1. The similarity to PCPO
review_title: null

Meta-review: metareview: This paper studies safe reinforcement learning and proposes a new policy optimization method with a rigorous safety guarantee. During the author-reviewer discussion period, the reviewers' concerns were mostly resolved. The reviewers have reached the consensus that the contribution of the proposed method is sufficient to be borderline accepted. I recommend it for acceptance and suggest the authors incorporate the reviewers' comments into the final version.
confidence: Certain
Decision: Accept