Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies

1 INTRODUCTION . Deep reinforcement learning ( RL ) has achieved superior performance in several domains such as games ( Mnih et al. , 2013 ; Silver et al. , 2016 ) and robotic control ( Levine et al. , 2016 ; Rajeswaran et al. , 2017 ) . However , in these complex applications , learning policies from scratch often requires tremendous amounts of time and computation power . To alleviate this issue , one would like to leverage a baseline policy available from a teacher or a previous task . However , the baseline policy may be sub-optimal for the new application and is not guaranteed to produce actions that satisfy given constraints on safety , fairness , or other costs . For instance , when you drive an unfamiliar vehicle , you do so cautiously to ensure safety , while at the same time you adapt your driving technique to the vehicle characteristics to improve your ‘ driving reward ’ . In effect , you ( as the agent ) gradually adapt a baseline policy ( i.e. , prior driving skill ) to avoid violating the constraints ( e.g. , safety ) while improving your driving reward ( e.g. , travel time , fuel efficiency ) . This problem is challenging because directly leveraging the baseline policy , as in DAGGER ( Ross et al. , 2011 ) or GAIL ( Ho & Ermon , 2016 ) , may result in policies that violate the constraints since the baseline is not guaranteed to satisfy them . To ensure constraint satisfaction , prior work either adds a hyper-parameter weighted copy of the imitation learning ( IL ) objective ( i.e. , imitating the baseline policy ) to the RL objective ( Rajeswaran et al. , 2017 ; Gao et al. , 2018 ; Hester et al. , 2018 ) , or pre-trains a policy with the baseline policy and then fine-tunes it through RL ( Mülling et al. , 2013 ; Chernova & Thomaz , 2014 ) . Both these approaches incur the cost of weight tuning to satisfy the cost constraint and do not ensure constraint satisfaction during training . In this work , to learn from the baseline policy while satisfying constraints , we propose an iterative algorithm that performs policy updates in three stages . The first step updates the policy to maximize expected reward using trust region policy optimization ( e.g. , TRPO ( Schulman et al. , 2015 ) ) . This can , however , result in a new intermediate policy that is too far from the baseline policy and one that may not satisfy the constraints . The second step performs a projection in policy space to control 1Code is available at : https : //sites.google.com/view/spacealgo . the distance between the current policy and the baseline policy . This distance is updated each episode depending on the reward improvement and constraint satisfaction , allowing the learning algorithm to explore without being overly restricted by the ( potentially constraint-violating ) baseline policy ( Rajeswaran et al. , 2017 ) . This also enables the baseline policy to influence the learning without the computational burden of learning a cost function for the baseline policy ( Kwon et al. , 2020 ) . The third step ensures constraint satisfaction at every iteration by performing a projection onto the set of policies that satisfy the given constraints . This ensures recovery from infeasible ( i.e. , constraint-violating ) states ( e.g. , due to approximation errors ) , and eliminates the need for tuning weights for auxiliary cost objective functions ( Tessler et al. , 2018 ) . We call our algorithm Safe Policy Adaptation with Constrained Exploration ( SPACE ) . This paper ’ s contributions are two-fold . We first analyze our proposed SPACE algorithm and provide a finite-time guarantee for its convergence . We also provide an analysis of controlling the distance between the learned policy at iteration k and the baseline policy to ensure both feasibility of the optimization problem and safe exploration by the learning agent . Second , we empirically compare SPACE with state-of-the-art algorithms on five different control tasks , including two Mujoco environments with safety constraints from Achiam et al . ( 2017 ) , two challenging traffic management tasks with fairness constraints from Vinitsky et al . ( 2018 ) , and one human demonstration driving task with safety constraints from Brockman et al . ( 2016 ) . In all tasks , SPACE outperforms the state-of-the art safe RL algorithm , projection-based constrained policy optimization ( PCPO ) in Yang et al . ( 2020 ) , averaging 40 % more reward with 10 times fewer cost constraint violations . This shows that SPACE leverages the baseline policy to achieve better learning efficiency while satisfying the cost constraint . 2 RELATED WORK . Policy optimization with constraints . Learning constraint-satisfying policies has been explored in the context of safe RL ( Garcia & Fernandez , 2015 ) , ( Hasanbeig et al. , 2020 ; Junges et al. , 2016 ; Jansen et al. , 2020 ) . Prior work either uses a conditional-gradient type of approach ( Achiam et al. , 2017 ) , adds a weighted copy of the cost objective in the reward function ( Tessler et al. , 2018 ; Chow et al. , 2019 ; Fujimoto et al. , 2019 ; Stooke et al. , 2020 ) , adds a safety layer to the policy ( Dalal et al. , 2018 ) ( Avni et al. , 2019 ) , or concerns about the chanced constraints ( Fu & Prashanth L , 2018 ; Zheng & Ratliff , 2020 ) . Perhaps the closest work to ours is Projection-based Constrained Policy Optimization ( PCPO ) ( Yang et al. , 2020 ) , which also uses projections in policy space to ensure constraint satisfaction . However , PCPO does not have the capability to safely exploit prior information ( through a baseline policy ) . The lack of using prior policies in PCPO makes it sampleinefficient . In addition , our SPACE algorithm ’ s update dynamically sets distances between policies while PCPO does not . This update is important to effectively and safely learn from the baseline policy . Furthermore , we provide a safety guarantee to ensure the feasibility of the optimization problem while PCPO does not . Merely adding an IL objective in the reward objective of PCPO can not learn efficiently , as shown in experiments . This analysis allows us to advance towards the practical use of RL in real applications , which PCPO and other algorithms have never done before . Policy optimization with the initial safe set . Wachi & Sui ( 2020 ) ; Sui et al . ( 2015 ) ; Turchetta et al . ( 2016 ) assume that the initial safe set is given , and the agent explores the environment and verifies the safety function from this initial safe set . There is no baseline policy here . In contrast , our assumption is to give a baseline policy to the agent . Both assumptions are reasonable as they provide an initial understanding of the environment . Leveraging baseline policies for RL . Prior work has used baseline policies to provide initial information to RL algorithms to reduce or avoid undesirable situations . This is done by either : initializing the policy with the baseline policy ( Driessens & Džeroski , 2004 ; Smart & Kaelbling , 2000 ; Koppejan & Whiteson , 2011 ; Abbeel et al. , 2010 ; Gao et al. , 2018 ; Le et al. , 2019 ; Vecerik et al. , 2017 ; Jaques et al. , 2019 ) ; or providing a teacher ’ s advice to the agent ( Garcia & Fernández , 2012 ; Quintı́a Vidal et al. , 2013 ; Abel et al. , 2017 ; Zhang et al. , 2019 ) . However , such works often assume that the baseline policy is constraint-satisfying ( Sun et al. , 2018 ; Balakrishna et al. , 2019 ) . In contrast , our SPACE algorithm safely leverages the baseline policy without requiring it to satisfy the specified constraints . Pathak et al . ( 2015 ) ; Bartocci et al . ( 2011 ) also modify the existing known models ( policies ) based on new conditions in the context of the formal methods . In contrast , we solve this problem using projections in the policy space . Learning from logged demonstration data . To effectively learn from demonstration data given by the baseline policy , Wu et al . ( 2019 ) ; Brown et al . ( 2019 ) ; Kwon et al . ( 2020 ) assess the demonstration data by either : predicting their cost in the new task using generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) ; or directly learning the cost function of the demonstration data . However , these approaches require a large number of training samples from the new task . In addition , the learned cost function is not guaranteed to recover the true one . This may result in driving the agent to undesirable situations . In contrast , SPACE controls the distance between the learned and baseline policies to ensure reward improvement and constraint satisfaction . Table 1 catalogs related work on safe RL and IL . We can differentiate these methods on several key axes : ( 1 ) whether they optimize reward and/or ensure cost satisfaction , ( 2 ) whether they use on-policy methods ( i.e. , query the baseline policy ) or off-policy methods ( i.e. , learn from a batch of demonstration data ) , ( 3 ) whether the baseline policy or demonstration data is optimal for the agent ’ s own objective , and ( 4 ) the technical approach of the work . 3 PROBLEM FORMULATION . We frame our problem as a constrained Markov Decision Process ( CMDP ) ( Altman , 1999 ) , defined as a tuple < S , A , T , R , C > . Here S is the set of states , A is the set of actions , and T specifies the conditional probability T ( s′|s , a ) that the next state is s′ given the current state s and action a . In addition , R : S × A → R is a reward function , and C : S × A → R is a constraint cost function . The reward function encodes the benefit of using action a in state s , while the cost function encodes the corresponding constraint violation penalty . A policy is a map from states to probability distributions on A . It specifies that in state s the selected action is drawn from the distribution π ( s ) . The state then transits from s to s′ according to the state transition distribution T ( s′|s , a ) . In doing so , a reward R ( s , a ) is received and a constraint cost C ( s , a ) is incurred , as outlined above . Let γ ∈ ( 0 , 1 ) denote a discount factor , and τ denote the trajectory τ = ( s0 , a0 , s1 , · · · ) induced by a policy π . Normally , we seek a policy π that maximizes a cumulative discounted reward JR ( π ) . = Eτ∼π [ ∑∞ t=0 γ tR ( st , at ) ] , ( 1 ) while keeping the cumulative discounted cost below hC JC ( π ) . = Eτ∼π [ ∑∞ t=0 γ tC ( st , at ) ] ≤ hC . ( 2 ) Here we consider an additional objective . We are provided with a baseline policy πB and at each state s we measure the divergence between π ( s ) and πB ( s ) . For example , this could be the KL-divergence D ( s ) . = DKL ( π ( s ) ‖πB ( s ) ) . We then seek a policy that maximizes Eq . ( 1 ) , satisfies Eq . ( 2 ) , and ensures the discounted divergence between the learned and baseline policies is below hD : JD ( π ) . = Eτ∼π [ ∑∞ t=0 γ tD ( st ) ] ≤ hD . ( 3 ) We do not assume that the baseline policy satisfies the cost constraint . Hence we allow hD to be adjusted during the learning of π to allow for reward improvement and constraint satisfaction . Let µt ( ·|π ) denote the state distribution at time t under policy π . The discounted state distribution induced by π is defined to be dπ ( s ) .= ( 1− γ ) ∑∞ t=0 γ tµt ( s|π ) . Now bring in the reward advantage function ( Kakade & Langford , 2002 ) defined by AπR ( s , a ) . = QπR ( s , a ) − V πR ( s ) , where V πR ( s ) . = Eτ∼π [ ∑∞ t=0 γ tR ( st , at ) |s0 = s ] is the expected reward from state s under policy π , and QπR ( s , a ) . = Eτ∼π [ ∑∞ t=0 γ tR ( st , at ) |s0 = s , a0 = a ] is the expected reward from state s and initial action a , and thereafter following policy π . These definitions allow us to express the reward performance of one policy π′ in terms of another π : JR ( π ′ ) − JR ( π ) = 11−γEs∼dπ′ , a∼π′ [ A π R ( s , a ) ] . Similarly , we can define AπD ( s , a ) , Q π D ( s , a ) and V π D ( s ) for the divergence cost , and A π C ( s , a ) , QπC ( s , a ) and V π C ( s ) for the constraint cost .

Summary: The paper considers the constrained Markov Decision Process (CMDP) problem where the goal is to maximize cumulative reward while satisfying the safety constraint on cumulative cost. Solving CMDP is challenging, and this paper proposes a shortcut by utilizing a given baseline policy. The idea of the proposed SPACE algorithm is to have three steps for each iteration of policy optimization. The first step is a trust region optimization step to optimize reward, the second step projects the policy to a region close to the given baseline policy, and the third step projects the policy to the constraint set.