Reconnaissance for reinforcement learning with safety constraints

Abstract: Practical reinforcement learning problems are often formulated as constrained Markov decision process (CMDP) problems, in which the agent has to maximize the expected return while satisfying a set of prescribed safety constraints. In this study, we consider a situation in which the agent has access to the generative model which provides us with a next state sample for any given state-action pair, and propose a model to solve a CMDP problem by decomposing the CMDP into a pair of MDPs; \textit{reconnaissance} MDP (R-MDP) and \textit{planning} MDP (P-MDP). In R-MDP, we train threat function, the Q-function analogue of danger that can determine whether a given state-action pair is safe or not. In P-MDP, we train a reward-seeking policy while using a fixed threat function to determine the safeness of each action. With the help of generative model, we can efficiently train the threat function by preferentially sampling rare dangerous events. Once the threat function for a baseline policy is computed, we can solve other CMDP problems with different reward and different danger-constraint without the need to re-train the model. We also present an efficient approximation method for the threat function that can greatly reduce the difficulty of solving R-MDP. We will demonstrate the efficacy of our method over classical approaches in benchmark dataset and complex collision-free navigation tasks.

Reviews and author responses:

We thank the reviewers for taking their time to read our paper and making comments and suggestions. 

Our modification is summarized as follows:
- We emphasize the crash rate in Table 1 and Table 2 to better convey our intention. Also, we added the tables with more penalties in Appendix H.
- We stated the dimensionality of the environments in the first paragraph of Sec. 4.2. We also described the details of it in Appendix F.
- We mentioned clearly that the baseline DQN method with Lagrange multiplier in the high-dimensional experiments is essentially the same as the one proposed in Geibel et al., 2005.
- We added the explanation in the captions and also made the figures larger to increase the visibility.

We are sorry about the size of the figures. We will add more captions and enlarge the figures in the revision. Also, we will describe the dimensionality in the proper part of section 4. Thank you for your suggestion.

> Looking at Table 1 and 2, it seems either i) the selected domains do not benefit from long-term planning, and/or ii) the proposed method learns short-sighted policies.

We disagree with the above statement. This is an experiment in which the long-sighted policies are especially important. 
As for our comparison against MPC, please look at the value in parenthesis that represents the crashing rate. Although MPC and our method both achieve reward on the same scale, our crash rate (2%)  is significantly lower than that of the MPC (45%). This means that MPC cannot avoid the collision when the prediction horizon is short. 

In this experiment, the agents are rewarded for every step that reduces the distance from the goal, and is penalized 50pts for each crash.  Should we penalize the agent more severely for each crash, the difference between our method and the rival will be clearer. In the revision, we plan to either make such a change or emphasize more about the difference in crash rate.


To our knowledge, there has not been a notable research that uses the generative model in a way much different from MPC. Moreover, most existing algorithms use generative models to evaluate the outcome of proposed actions in real-time, and the real-time computation aspect makes it challenging for such algorithms to scale to high-dimensional problems. Our most important algorithmic innovation is the isolation of the reward seeking phase from the threat assessing phase that extensively requires the simulation. We present a novel way to use the generative model. In addition, our method only uses the generative model during the R-MDP training phase. The P-MDP phase as well as the real-time execution do not need to access the generative model, and this feature further provides transferability as discussed in the introduction and demonstrated in Section 4.3. 

Thank you for the comments.
In our understanding,  one of the greatest merits of the generative model in constrained MDP is its ability to allow the user to access the rare dangerous event.  It is therefore indeed our intention to sell our algorithm as a strategy that endeavors to make utmost use of the generator, and we hope our contribution to be evaluated on the extent to which we are succeeding in this goal. Please note that assuming the accessibility of generative models does not make the problem trivial. Even for the methods specific to autonomous driving, for example, where we know what is a risky event in advance and can utilize the power of generative models, collision avoidance is still a difficult problem. The video we provided in the supplementary would help you to see how difficult the task is, especially for the Jam environment.

Thank you for pointing out related studies.
First, we would like to note that we do make a comparison against Geibel and Wysotzki, 2005, which is almost the same as the method in [1] published in 2006. Note that [1] was actually published in 2006, just a year after Geibel and Wysotzki, 2005. Both of them belong to a family of classical methods in solving a CMDP by using Lagrangian multipliers. We will mention it clearly in the revision. The method in [3] is also based on Lagrange multipliers, and it attempts to solve the CMDP as LP problem using the incremental column-generation method to optimize the lagrange multiplier coefficient. We believe that we can make sufficient comparative study with all these Lagrange multiplier methods by comparing our method against the baseline of DQN-based Lagrange multiplier method [5]. 

Second, the methods in [2] and [4] are not suitable for the high-dimensional stochastic environments we considered. The method proposed in [2] is a version of Monte Carlo tree search, whose variation of the future state can quickly explode especially when there are many randomly moving obstacles in the system. The approach of [4] requires constraints to be “linear” which is not the case for the complex environments we consider in this paper.

Lastly, we would like to point out that none of the cited papers has verified their performance in as high dimensional environment as the Jam environment used in our paper, the state of which is represented by 392 real values. More particularly, [2] used deterministic MuJoCo-based environment like Ants; [3] used low-dimensional environments (|S|: up to 60) for evaluation; [4] also used lower-dimensional (at most 100 real values) environments.

The video we provided in the supplementary would help you to see how difficult the task is, especially for the Jam environment.

[1] Reinforcement Learning for MDPs with Constraints, Geibel, ECML 2016. 

[2] Monte-Carlo Tree Search for Constrained MDPs, Chen et al., IJCAI 2018. 

[3] Column Generation Algorithms for Constrained POMDPs, Walraven and Spaan, JAIR 2018. 

[4] Hindsight Optimization for Hybrid State and Action MDPs, Raghavan et al. AAAI 2017. 

[5] Consideration of risk in reinforcement learning, Heger, Machine learning proceedings 1994. 

Thank you very much for understanding the significance of our approach. With regard to your question, indeed, when the decomposed state space is small and when the danger is defined in terms of events (indicator function-based constraints), one could potentially apply existing control algorithms like reachability analysis to verify possible collisions to compute the threat function. We would like to investigate efficient ways to use our decomposition together with other control algorithms in future work to accelerate threat function learning. 

Summary: The paper studies problems that can be modelled as constrained markov decision processes (CMDPs). It proposes to solve the CMDP by decomposing it into a pair of MDPs; i) a reconnaissance MDP (R-MDP) that is trained with the help of a generative model, and ii) a planning MDP (P-MDP) that is trained given a threat function (i.e., previously trained for the R-MDP). The decomposition approach is tested over some classical benchmarks.

Strengths:

i) The motivation, organization and the overall writing of the paper are clear.

Weakness:

i) The paper seems to be missing out related works [1,2,3,4,…] that can solve C-(PO)MDPs, which could have been used as baselines (i.e., instead of DQN). Therefore it is not clear if the following claim is true or not: “Although our method does not guarantee to find the optimal solution of the CMDP problem, there has not been any study to date that has succeeded in solving a CMDP in dynamical environments as high-dimensional as the ones discussed in this study.”. Overall, I would say this is the weakest part of the paper.

ii) It is not clear if the selected experimental benchmarks are challenging. Looking at Table 1 and 2, it seems either i) the selected domains do not benefit from long-term planning, and/or ii) the proposed method learns short-sighted policies. That is in Table 2 for N=15, the proposed method performs similar to MPC over 3 steps.

iii) The experimental results require better presentation. For example, since one of the main arguments is that the proposed methodology can solve high-dimensional problems, it would be great to note the dimensionality of the benchmarks in the beginning of section 4. Moreover, none of the figures are readable and are left mostly unexplained.

References:

[1] Reinforcement Learning for MDPs with Constraints, Geibel, ECML 2016.

[2] Monte-Carlo Tree Search for Constrained MDPs, Chen et al., IJCAI 2018.

[3] Column Generation Algorithms for Constrained POMDPs, Walraven and Spaan, JAIR 2018.

[4] Hindsight Optimization for Hybrid State and Action MDPs, Raghavan et al. AAAI 2017.

This paper attempts to propose a practical method for solving constrained MDP via decomposing the original problem into two MDPs. In the first phase, it uses a generative model to solve the MDP with the original constraint being the cost. The policy is then used to define secure actions to help solving the second MDP that maximizes the reward. While the method does not guarantee to find the optimal safe policy, the authors claim (and show in few experiments) that it can perform better than classical methods.

While the approach seems simply and intuitive, it is not clear whether the contributions are significant for the community. The decomposition seems straightforward and the algorithmic innovation does not seem to be significant. All the difficulties of solving the thread function is being resolved by assuming the access to a generative model. In particular, the authors further consider the case where solving the R-MDP is simpler: the danger is described by known risky events and hence sampling rare events with the generative model is easier. As a result, the experiments primarily focus on navigation tasks. More thorough experiments would be appreciated. In addition, beyond MPC, is it possible to add other baselines that use a learning based approach with access to generative model? If possible, I think this will help to further clarify the benefit of the overall idea of decomposition.   

This paper proposes an approach to learning safe reinforcement learning policies using a simulator. The main idea is to consider essentially two distinct reward functions on the same MDP, one quantifying the risk/safety level of the states, and the other quantifying the usual reward. A policy is first trained to optimize safety, and then a policy is trained to optimize reward, subject to remaining in the safe region, i.e., subject to a constraint given by the Q function for the safety MDP. If the policy enters an unsafe state, it is required to follow the policy for the safety MDP; the paper includes some theoretical results demonstrating that this is an adequate way to ensure the expected safety remains above a given threshold.

Actually, the paper also proposes to decompose the safety reward into scores w.r.t. various events such as colliding with the various objects in the environment, and uses a linear combination of these component scores to obtain an aggregate safety score. This approach provides estimates of the safety scores in novel environments, for example where the configuration of the environment is changed, or with a different number of obstacles. Experiments suggest that the resulting approach is much computationally lighter than model-predictive control, and obtains the safest execution across the various methods considered. The reward obtained remains competitive overall.

The one downside here is the assumption of a simulator, which is pretty strong. It is true that pure sampling alone is never going to be able to provide a strong safety guarantee, and some kind of assumption is going to be necessary to obtain a strong guarantee, so I don't fault the use of some assumption. At the same time, the existence of the simulator does not make the problem trivial, as the optimization remains challenging, and the relative efficiency of the proposed approach is an argument in its favor. The approach is pretty effective overall and has some nice generalization properties. I recommend acceptance.

One question I have is whether the decomposition of the environment's threats into different factors could be used with other methods: if the dimension of the state space is sufficiently small, then maybe for example the synthesis (model checking/reachability) methods could be feasibly applied.

Meta-review: This paper investigates safe reinforcement learning with distinct reward function and safety function. The authors present theoretical analysis and simulation results. The representation of safety is a critical step. The authors define the safety function values based on various events and use linear combination of them to construct safety score. Theoretical guarantees on safety and efficiency are presented. Simulation results also show safety and efficiency of the method. 

This was a tricky case as the paper is borderline. Based on reviewers comments, we decided that the paper is not ready for publication in its current form and would benefit from another round revisions. 


Decision: withdrawn-rejected-submissions