Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes an approach to learning safe reinforcement learning policies using a simulator. The main idea is to consider essentially two distinct reward functions on the same MDP, one quantifying the risk/safety level of the states, and the other quantifying the usual reward. A policy is first trained to optimize safety, and then a policy is trained to optimize reward, subject to remaining in the safe region, i.e., subject to a constraint given by the Q function for the safety MDP. If the policy enters an unsafe state, it is required to follow the policy for the safety MDP; the paper includes some theoretical results demonstrating that this is an adequate way to ensure the expected safety remains above a given threshold.

Actually, the paper also proposes to decompose the safety reward into scores w.r.t. various events such as colliding with the various objects in the environment, and uses a linear combination of these component scores to obtain an aggregate safety score. This approach provides estimates of the safety scores in novel environments, for example where the configuration of the environment is changed, or with a different number of obstacles. Experiments suggest that the resulting approach is much computationally lighter than model-predictive control, and obtains the safest execution across the various methods considered. The reward obtained remains competitive overall.

The one downside here is the assumption of a simulator, which is pretty strong. It is true that pure sampling alone is never going to be able to provide a strong safety guarantee, and some kind of assumption is going to be necessary to obtain a strong guarantee, so I don't fault the use of some assumption. At the same time, the existence of the simulator does not make the problem trivial, as the optimization remains challenging, and the relative efficiency of the proposed approach is an argument in its favor. The approach is pretty effective overall and has some nice generalization properties. I recommend acceptance.

One question I have is whether the decomposition of the environment's threats into different factors could be used with other methods: if the dimension of the state space is sufficiently small, then maybe for example the synthesis (model checking/reachability) methods could be feasibly applied.
review_title: A computationally inexpensive method to obtain strong safety using a simulator.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper attempts to propose a practical method for solving constrained MDP via decomposing the original problem into two MDPs. In the first phase, it uses a generative model to solve the MDP with the original constraint being the cost. The policy is then used to define secure actions to help solving the second MDP that maximizes the reward. While the method does not guarantee to find the optimal safe policy, the authors claim (and show in few experiments) that it can perform better than classical methods.

While the approach seems simply and intuitive, it is not clear whether the contributions are significant for the community. The decomposition seems straightforward and the algorithmic innovation does not seem to be significant. All the difficulties of solving the thread function is being resolved by assuming the access to a generative model. In particular, the authors further consider the case where solving the R-MDP is simpler: the danger is described by known risky events and hence sampling rare events with the generative model is easier. As a result, the experiments primarily focus on navigation tasks. More thorough experiments would be appreciated. In addition, beyond MPC, is it possible to add other baselines that use a learning based approach with access to generative model? If possible, I think this will help to further clarify the benefit of the overall idea of decomposition.   
review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: The paper studies problems that can be modelled as constrained markov decision processes (CMDPs). It proposes to solve the CMDP by decomposing it into a pair of MDPs; i) a reconnaissance MDP (R-MDP) that is trained with the help of a generative model, and ii) a planning MDP (P-MDP) that is trained given a threat function (i.e., previously trained for the R-MDP). The decomposition approach is tested over some classical benchmarks.

Strengths:

i) The motivation, organization and the overall writing of the paper are clear.

Weakness:

i) The paper seems to be missing out related works [1,2,3,4,…] that can solve C-(PO)MDPs, which could have been used as baselines (i.e., instead of DQN). Therefore it is not clear if the following claim is true or not: “Although our method does not guarantee to find the optimal solution of the CMDP problem, there has not been any study to date that has succeeded in solving a CMDP in dynamical environments as high-dimensional as the ones discussed in this study.”. Overall, I would say this is the weakest part of the paper.

ii) It is not clear if the selected experimental benchmarks are challenging. Looking at Table 1 and 2, it seems either i) the selected domains do not benefit from long-term planning, and/or ii) the proposed method learns short-sighted policies. That is in Table 2 for N=15, the proposed method performs similar to MPC over 3 steps.

iii) The experimental results require better presentation. For example, since one of the main arguments is that the proposed methodology can solve high-dimensional problems, it would be great to note the dimensionality of the benchmarks in the beginning of section 4. Moreover, none of the figures are readable and are left mostly unexplained.

References:

[1] Reinforcement Learning for MDPs with Constraints, Geibel, ECML 2016.

[2] Monte-Carlo Tree Search for Constrained MDPs, Chen et al., IJCAI 2018.

[3] Column Generation Algorithms for Constrained POMDPs, Walraven and Spaan, JAIR 2018.

[4] Hindsight Optimization for Hybrid State and Action MDPs, Raghavan et al. AAAI 2017.
review_title: Review of RL with Safety Constraints

Meta-review: This paper investigates safe reinforcement learning with distinct reward function and safety function. The authors present theoretical analysis and simulation results. The representation of safety is a critical step. The authors define the safety function values based on various events and use linear combination of them to construct safety score. Theoretical guarantees on safety and efficiency are presented. Simulation results also show safety and efficiency of the method. 

This was a tricky case as the paper is borderline. Based on reviewers comments, we decided that the paper is not ready for publication in its current form and would benefit from another round revisions. 

Decision: Reject