Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a framework QCRL to constrain the quantile of the distribution of cumulative sum cost, which is equal to guarantee a target probability of outage event that the cumulative sum cost exceeds a given threshold. This work is a good complement work for the constrained reinforcement learning area by considering a "new" quantile constraint. By constraining the quantile (instead of the expectation) of the cumulative sum cost,  we can have "precise" control over the target outage probability, which is very critical in practice , especially when high-cost event is unsafe.

This paper can be seen as a great work following WCSAC, the main contributions of this paper are: (1) better estimation for the quantile of the sum cost (2) focusing directly on the constraint on quantile.
strengths_and_weaknesses: I find it is easy to understand the ideas, motivation, and methods of this paper, results are also well-understandable. Overall, I think this paper gives a very critical and interesting setting for constrained RL theoretically and practically. 



questions: 
In this paper,  three different constrained RL problems and related algorithms are discussed:  (1) ExpCp (2) ProbCP ( equivalent to QuantCP) (3) CVaR-CP.  The manuscript would benefit from adding a formal formulation for CVaR constraints problems with the similar type as the other two problems.  Besides, the WCSAC is not for ProbCP,  but for CVaR-CP.

The second question is about the experiments. The authors compare three different methods, QCPO, PPO_Lag, and WCSAC. However, these three methods are for three different problems (ExpCP, ProCP, CVaR-CP) respectively. It is not surprising that QCPO is the best in terms of controlling outage probability. Is that fair to compare the performance for different methods when given different constraints? What are these experiments designed for? 

The third issue is about the results on DynamicEnv-v0.  WCSAC does get a bad estimation of the tail probability such that we can not compare the return of WCSAC with QCPO later.  How about adding another experiment of WCSAC by replacing the quantile and tail estimation part with the proposed method in section 4.1.  

Another question is about the range of outage probability $\epsilon_0$? If the $\epsilon$ is smaller (say 0.05, 0.02), could the proposed method still work well? What if the $\epsilon$ is very large (say 0.5)?
 
The last question is about the conclusion "the implementation algorithm learns an optimal policy while it keeps satisfying the outage probability constraint during its learning process".   I am confused about whether the constraint is always satisfied when you use a Lagrange-based optimization and also keep updating the Lagrange multiplier $\lambda$ during the learning procedure. It is also obvious from the experimental results that the constraint is not always satisfied, especially for the early stage. 

#################
update:  Thanks for authors' response, which has addressed most of my concerns.  I would like to see the results of WCSAC with a Weibull approximation included in the final version.  Therefore, I would like to keep my score as 7.
limitations: yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper considers the problem of solving a constrained MDP under quantile constrained, i.e., that the tail probability of cost is below a certain desired value.  The problem falls under the general framework of risk constrained reinforcement learning and is relevant to systems where a stranger metric than expected cost constraints are desired.  The approach is to derive a proxy for the policy gradient of the quantile probability, and utilize it in a policy gradient approach to ensure the constraint is met.  The derivation of the appropriate policy gradient term is the major contribution.
strengths_and_weaknesses: Strengths

+ The paper makes a novel observation in obtaining the policy gradient of the Lagrange dual of the risk constrained problem. The policy gradient in this case depends not he tall probability itself, as well as a const that is upper bounded by what appears to be a policy independent term.  

+ The authors use a standard PPO codebase to implement their approach and verify its performance in relevant scenarios.

Weaknesses


- The imprecations of Theorem 3 are unclear.  It appears that the result depends on deterministic transitions, and the availability of a base policy $\pi'$  whose KL divergence from $\pi$ should be less than $\delta.$  How doe we obtain $\pi'$?  Furthermore, the actual problem does not have deterministic transitions.  How does this result apply to the problem?

- Related to the above question, in section 4, step 2 is on estimating $X^\pi(s)$ to compute the cost $\tilde{c}^{\pi'}_u(s,a).$  How is this done?  The last paragraph of the section in step 3 says that we must compute the additional cost, again without showing how to do it.  Appendix E.2 only presents loss functions, not how this step is accomplished.  

- The above seem to be fundamental to the analytical claims of the approach as well as its implementation.  Otherwise, the implementation would become some kind of constraint on the rate function of the occupancy measure generated by the policy.  I am not sure if the implantation is consistent with the theory developed.


questions: Please see weaknesses.  Addressing the gaps in the logic, namely deterministic transition, nearness of base policy to current policy and hence finding delta and using the bound in in the implementation should be addressed.  I would increase my score if this is satisfactorily addressed.
limitations: Yes, it appears to be sufficient.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper "Quantile Constrained Reinforcement Learning" studies the problem of reinforcement learning for constraining outage probability. The paper converts this constraint into an equivalent quantile constraint. The policy gradient theorem can not be directly applied, due to the lack of a closed form gradient under this constraint. The paper uses large deviations principle and derives an "approximate" gradient for the quantile. Using this, a policy gradient based reinforcement learning algorithm "Quantile Constrained Policy Optimization" (QCPO) is proposed. For theoretical result, an improvement theorem is provided. The experimental results show the improvement over existing methods.


strengths_and_weaknesses: So far the literature is mainly focused on expected cumulative sum cost constraint, very few policy gradient results exist for VAR and CVAR.  It is not a straightforward derivation of policy gradient for risk constrained problems as in expected cost constraint. This paper tries to overcome this issue by computing approximate gradients. The technical contribution is sound, and the paper is readable. These results are significant in the risk constrained RL literature. That said, in the main paper, the theoretical results have many assumptions. The repeated usage of phase "under the mild assumptions" is misleading (Theorem 1, Corollary 1, Lemma 1, Theorem 2, etc). Authors should clearly state the assumptions. In line 204, transition dynamics of the CMDP are assumed deterministic for the theorems to work. This has to be stated as an assumption. The experimental evaluation part is too simple and weak.
questions: 1. Please state the assumptions used in the theorems clearly.

2. The improvements shown in the approximations degrade as the complexity of environments increase. For example, the algorithm seems to perform better in simple button environment compared to other algorithms, but in Dynamic Env, its performance benefits are not much.
limitations: The limitations are not discussed in the paper. The authors are suggested to include a discussion in the paper about the possible limitations. The problem has a probability constraint, hence, with a small probability, the catastrophic events happen.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a framework, named Quantile Constrained RL (QCRL), to constrain the quantile of the distribution of cumulative
sum cost that is a necessary and sufficient condition to satisfy the outage constraint. The paper converts the outage probability constraint in (ProbCP) into a quantile constraint which is equivalent to the original probabilistic constraint and solve the equivalent optimization using  Lagrange multiplier. 
strengths_and_weaknesses: Strength:

1. This paper considers the more general outage probability constraint in (ProbCP) than the classic expectation constraint (ExpCP) and modifies the optimization problem using quantile constraint.
2. The paper applies policy gradient theorem to the quantile and provides theoretical results for approximating the gradient of the quantile. 

Weakness:

1. This paper does not fully clarify why it is necessary to employ the outage probability constraint in practice and what may be the potential impact of this setting to the real-world examples.
2. The empirical part is not strong enough to show the advantage of the proposed method, and some more realistic and interesting examples such as the autonomous driving or the telecommunication system that are mentioned by the authors may be helpful. 
questions: 1. How to know that the underlying distribution of the sum of costs is right-tailed in practice if we do not have much information about the true environment. How does this method perform in practice if the model assumption is mis-specified. For example, what if the real distribution is left-tailed or multi-modal?
2. Similar to Q1, how to choose proper values of $\epsilon_0$ and $d_{th}$ if there is no much information about the environment.
3. The experiments are done with 10 different seeds while the variance seems to be large. Is the proposed method sampling efficient and how the estimations especially the quantiles are affected by the data sample?
4. There are a lot of studies about Distributional RL which uses the Huber loss to learn the quantiles. Can this approach be applied here?
limitations: This paper considers a more general more general outage probability constraint for Constrained Reinforcement Learning. However, this paper fails to provide enough insight about this setup and the experimental results do not provide a strong support.
review_title: null

Meta-review: metareview: This paper made novel contributions in developing policy gradient methods in constrained RL. Unlike previous approaches that focused on expected cost constraint this work studied a more chance-constrained setting and proposed a Quantile Constrained RL (QCRL) framework that is based on distributional RL and Large Deviation Principle (LDP). Experiments show that this newly developed approach outperforms SOTA both in learning performance and safety guarantees. Therefore, without major controversies the review committee has the consensus of accepting this paper to be published at NeurIPS22.

confidence: Certain
Decision: Accept