Title: Solving Parameter-Robust Avoid Problems with Unknown Feasibility using Reinforcement Learning
Primary area: reinforcement learning
Keywords: Reinforcement Learning, Optimal Control, Deep Reachability

Abstract: Recent advances in deep reinforcement learning (RL) have seen impressive success in finding solutions to high-dimensional reachability problems. However, the process by which the original reachability problem is translated into the RL problem is largely overlooked. The goal of the reachability problem is to maximize the set of states that a system can remain safe in for all time, but this can be misaligned with the goal in RL. Specifically, RL aims to minimizes the expected cost over the user-specified initial state distribution, which can lead to poor policy performance for low probability initial states. One approach is to treat the original problem as a robust optimization problem over a set of initial states, or more generally, a set of static task parameters. However, specifying feasible static parameters is difficult since the feasible set is often unknown. We propose Feasibility-Guided Exploration (FGE), a method that simultaneously identifies a subset of feasible static parameters and learns a policy to solve the reachability problem over this set of static task parameters. Empirical results demonstrate that our approach learns policies that give larger safe sets compared to existing methods on a suite of avoid problems in the Mujoco simulator.