Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary
In this paper, the authors proposed a new constrained policy optimization algorithm and a worst-case version of the constrained MDP framework. Tho proposed constrained policy optimization algorithm is based on CPO, and a novel advantage function (CSAE) based on the concept of a "safe" state. Experiments in control simulation tasks are provided.

In general, the main novel contribution of this work is the cost-sensitive advantage estimation (CSAE), which is later justified by the empirical study. However, I think the problem setting needs further clarification, and the new advantage estimation and reward shaping are not justified enough.  

Pros:
1. The connection between the CSAE with a new form of reward is interesting. I think it provides more intuition or a different understanding of the advantage function.

2. The CVaR version of CMDP is also a novel and interesting setting.

Cons:
1. It is not clear enough what is the problem formulation that this work (or CSAE algorithm) aims to solve. Before section 4 it seems that the goal is to maximize the reward while keeping the *trajectory-level* cost under a budget. However, later the CSAE algorithm leverages the concept of "safety" of states. While the trajectory-level cost may or may not be decomposed into some state-level binary constrained, it seems either the algorithm design needs more justification or the problem setting is misleading. 
Furthermore, I think the choice of $\alpha_t$ in the algorithm needs to be discussed extensively: instead of saying "e.g. C_t > 0", it needs to discuss how to choose the threshold, how the state-level constrained relate to the constraints on the discounted sum, etc. 

2. Since the new advantage function is linked to the new reward form in Eq (5), it needs more discussion about why this form of reward is a natural one. It isn't obvious why using the average reward for the "unsafe" state-action pairs can penalize them.

3. The paper proposed two contributions but I did not follow how they are connected to each other. WCMDP can be applied to any constrained policy optimization algorithm. 
review_title: New constrained RL algorithm. Setting is unclear, and the two constributions seem disconnected.

final_score: rating: 4: Ok but not good enough - rejection
review_content: 

#### Summary:

In this work, the authors extend the CPO (Achiam et al) with a Cost-Sensitive Advantage estimation technique that eliminates the negative effect of unsafe high-reward state-action transitions. The authors then modify the CMDP objective with a CVAR based objective function and provide empirical results on the Mujoco based safety environments.   


#### Strengths:

- The idea of incorporating the notion of including safe/unsafe states to build a new advantage estimation is an interesting one, and I believe the link to Cost-Sensitive Advantage Estimation and reward shaping is novel.

- The visualizations of the trajectory of the final policy returned by the algorithm were insightful to see. 

#### Weakness:

- **Unclear setting:** The authors start with the motivation with the setting of the Constrained MDPs, but through-out the draft the discussion is carried in terms of safe and unsafe states. Note that, while the safe-state formulation can be defined using the CMDP framework, the opposite is not always true.  


- **Technical flaws:** The method relies on an important assumption that is not discussed in enough detail. The paper assumes that the agent has knowledge about what states are safe and which states are unsafe ($\alpha_t$ variable in Eq 4). Note that, this is an important distinction, as for CMDPs defining what states are unsafe (or what states to avoid) is a big challenge in itself (Altman, 1999). 
 For the purpose of this work, the authors use a hard-coded safety distinction ($\apha_t = \mathbb{1}[C(s_t,a_t,s_{t+1})] > 0 ) and though this works for the examples chosen in this work (the examples based on CPO) there is no reason this will be true in general. In most cases, an optimal policy of CMDP is the stochastic policy, and directly avoiding some states is not always possible, and usually, the safe or unsafe states for CMDP are a function of the policy. 
 
 This is also related to the author's claim that their proposed method is "free of hyper-parameter tuning and easy to deploy". In this case, the authors have already abstracted the difficult aspect of finding safe and unsafe states by including that information in the $\alpha$ variable, and the authors assume that they always have access to this variable. As I mentioned above this is not always true. Furthermore, if the agent already knows safe and unsafe state distinction, there are much simpler and computationally efficient methods (compared to CPO) are available that can be deployed, for instance, (Dalal et al, 2018) that the authors mention in the manuscript.   
  
- **Incremental nature of work**: The proposed method relies heavily on the CPO (Achiam et al), both for the theoretical and empirical results and baselines. Combined with the above flaw of hard-coding the unsafe states, it is not clear what is the novelty of this work. The Worst-Case formulation uses a CVAR based objective instead of expectation based. While that is an interesting formulation, there are no comments made on how that changes the quality of the solution compared to the regular objective. Are the optimal policy in regular CMDP and CVAR-based objective different? How tough the problem becomes? 
  
- **Reproducibility**: There is no mention of code release for empirically intense work. 

- **Baselines:** The authors mention that CPO is the SOTA, but that is not true. From (Ray et al, 2019), we know that Lagrange based approaches can actually perform better than CPO when trained properly. 
 
#### References:

- Ray, Alex, Joshua Achiam, and Dario Amodei. "Benchmarking safe exploration in deep reinforcement learning." arXiv preprint arXiv:1910.01708 (2019).
review_title: Recommendation to Reject

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors' rebuttal has addressed some of my confusion regarding the paper, which is greatly appreciated. The additional baseline of early termination would still be interesting to have, though I agree it's not critical for the presented line of work. In general, I think the work is interesting and will keep my current score (6).

=================================

The paper introduces a learning algorithm for training a control policy to complete certain tasks while satisfying some safety constraints. The main ideas in this work are to first use a cost-sensitive advantage function, where the advantage values for the unsafe states are set to zero. Second, a more conservative estimation of the safety cost is proposed to further improve the safety of the robot during the learning process. The authors demonstrated that with the proposed modified reward function, the algorithm would obtain a controller that completes the task while being safe. The proposed algorithm is evaluated on a set of simulated control problems and with both proposed components used, the algorithm achieves better performance in terms of both task completion and safety than prior methods.

The paper solves an important problem of training a performant control policy while taking the safety of the robot into consideration. The paper is well written and the experiments show good results compared to prior methods.

However, I do have a few questions about the paper:
1. The paper mentioned first about zeroing the advantage for the unsafe states and showed the equivalence of doing that to a shaped reward which replaces the original reward with the shaped reward. I'm wondering if this equivalence also holds for the safe states? For example, within a trajectory that ends in an unsafe state, a prior safe state's advantage will take the original reward of the unsafe state into consideration if only the unsafe state's advantage is modified, which might be different from the shaped reward version. It would be great if the authors could help me understand this part better.
2. In the implementation of the work, is it using the modified advantage function in Eq. 4, or the shaped reward in Eq. 5? It's a bit confusing in that the title of the paper suggests the use of Eq. 4, while section 4.2 seems to be saying it's using the shaped reward? If it's using the shaped reward version, is there a reason not to use Eq. 4 directly?
3. It seems a simple baseline to compare to is to terminate the rollout when the robot enters an unsafe state, as is done in various OpenAI Gym tasks. There seems to be some analogy in this strategy and the proposed one in that when the rollout is terminated, the policy will not have any updates regarding the unsafe states, i.e. zero gradient, and thus corresponds to zero advantage in policy gradient formulation. I do note that there are still many differences between the two methods, but I feel it's worth trying given the simplicity of the approach and how it's been effective in teaching the robots to run upright in existing problems.

Overall, I think it's an interesting approach with good results.



review_title: review for "Learning Safe Policies with Cost-sensitive Advantage Estimation"

final_score: rating: 7: Good paper, accept
review_content: Summary: The authors propose to improve a safe RL algorithm, constrained policy optimizaiton, that can learn the optimal safe policy while exploring unsafe states less often during the training process. In particular, they dampen the estimated advantage associated with unsafe states, which encourages the RL algorithm to explore safe states more often during the learning process. In addition, the authors aim to find a policy that satisfies the constraints with high probability, rather than only in expectation, by considering the worst-case constraints. The empirical results show that a safe RL algo that dampens the advantage and respects worst-case constraints are able to learn policies with large returns and avoid unsafe states.

Pros:
+ The authors present a simple modification of a safe RL method to make it learn faster and more safely. The empirical results validate this claim.

Major concerns:

1. The authors describe how their method replaces the reward for an unsafe state as the average reward, and claim that this is generally better than previous reward reshaping methods, e.g. setting the reward for unsafe states to zero. The authors only present a single example that compares the two approaches (Fig 4); However, it's not clear if the average reward is a wider range of scenarios. I'd like to see a more comprehensive comparison in Fig 1 to justify this claim. Also, can the authors give an explanation for why the mean is a more suitable penalization of unsafe states or is more universally applicable, compared to other reward reshaping methods?

2. The authors compare the cumulative safe reward of different RL algorithms. However, the theoretical results in Theorem 1 compare the cumulative reshaped rewards. It is somewhat difficult to interpret the result in Theorem 1 since the reshaped reward uses the average reward for unsafe states, rather than setting the reward to zero. Given the similarity between there two rewards, can the authors establish a bound on the cumulative safe reward instead, as this seems to be the quantity we are truly interested in?

3. In the empirical analysis, the improvement from CSAE to CSAE-WC was quite large, whereas the improvement from CPO to CSAE was smaller. To understand how much dampening the advantage improves on the worst-case constraints, can the authors do an ablation study where the safe RL algo does not dampen the advantage function but respects the worst case constraints?

Minor concerns:

1. Please plot the constraint threshold in Fig 1 row 2. It's hard to tell if the methods have satisfied the constraints, since there is no reference line.
2. There is a typo when referring to the rows in Fig 1 -- see last paragraph on page 6. It should say "third and fourth rows", rather than "second and third rows".
review_title: Review of "Learning Safe Policies with Cost-sensitive Advantage Estimation"

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose two contributions:
1 - A Cost-Sensitive Advantage Estimation procedure called CSAE which is intended to differentiate more clearly safe and unsafe state-actions pairs with high rewards;
2 - A beta-quantile worst-case formulation of the Constrained MDP called WCMDP, and a variant of CPO  to solve this problem instead of its usual expectation formulation.
I liked the second part which prevents constraint violation with high confidence but I found the part on CSAE hard to follow.
The notion of safety being defined here in term of trajectory budget, not in term of state, I had difficulties to understand the notion of "safe-states" or "safe state action pairs"  when opposed to the notion of "safe-policies" as defined for instance in (Achiam et al. 2017). This notion of safe-sates seems however to be a key element for the definition of the CSAE estimate which is a variant of the "Generalized Advantage Estimation" (Schulman et al. 2016) where the TD error is nullified on "unsafe states".
Is it because the considered policies are stationary ?
Probably for the same reason I did not understand the claim of the authors on page 2 that "CPO sacrifices too much on the expected return".  
The proposed algorithm is detailed in the Appendix, its main difference with the CPO algorithm is the formulation of the primal-dual problem of equation (14) which rely on a beta-worst case discounted cost (and CSAE) instead of the expected discounted cost.
Theorem 1 gives a TRPO-like lower bound similar to the ones in  (Schulman et al. 2016).
The experiments in section 5 are performed on the same environments as (Achiam et al. 2017), allowing for a comparison of the two methods and underlining clearly the improvement of CSAE-WC against CPO and other baselines. The improvement from CPO to CSAE seems less clear to me.
What would be the performance of a simple beta-worst case version of CPO without the CSAE machinery ?

Minor remarks:
A few typos did add to my confusion when delving into the paper, for instance on equation (2) and on page 4 when introducing the gradient CSAE the authors forgot the log in the policy gradient formula, a strange mistake that is hopefully not committed in the appendix.
The TD error delta_t should be defined before  equation (3) on page 4.


review_title: A worst-case (i.e. CVaR) variant of "Constrained Policy Optimization" (Achiam et al. 2017)

Meta-review: The reviewers appreciate the importance of enforcing safety in RL, and the technical directions considered in the paper related to incorporating cost in advantage estimation.  However, they express several concerns about the formulation of the problem considered and the consistency of the approach, as well as the somewhat incremental contribution w.r.t. CPO.  Three reviewers recommend rejection.
Decision: Reject