Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies

1 INTRODUCTION . Deep reinforcement learning ( RL ) has achieved superior performance in several domains such as games ( Mnih et al. , 2013 ; Silver et al. , 2016 ) and robotic control ( Levine et al. , 2016 ; Rajeswaran et al. , 2017 ) . However , in these complex applications , learning policies from scratch often requires tremendous amounts of time and computation power . To alleviate this issue , one would like to leverage a baseline policy available from a teacher or a previous task . However , the baseline policy may be sub-optimal for the new application and is not guaranteed to produce actions that satisfy given constraints on safety , fairness , or other costs . For instance , when you drive an unfamiliar vehicle , you do so cautiously to ensure safety , while at the same time you adapt your driving technique to the vehicle characteristics to improve your ‘ driving reward ’ . In effect , you ( as the agent ) gradually adapt a baseline policy ( i.e. , prior driving skill ) to avoid violating the constraints ( e.g. , safety ) while improving your driving reward ( e.g. , travel time , fuel efficiency ) . This problem is challenging because directly leveraging the baseline policy , as in DAGGER ( Ross et al. , 2011 ) or GAIL ( Ho & Ermon , 2016 ) , may result in policies that violate the constraints since the baseline is not guaranteed to satisfy them . To ensure constraint satisfaction , prior work either adds a hyper-parameter weighted copy of the imitation learning ( IL ) objective ( i.e. , imitating the baseline policy ) to the RL objective ( Rajeswaran et al. , 2017 ; Gao et al. , 2018 ; Hester et al. , 2018 ) , or pre-trains a policy with the baseline policy and then fine-tunes it through RL ( Mülling et al. , 2013 ; Chernova & Thomaz , 2014 ) . Both these approaches incur the cost of weight tuning to satisfy the cost constraint and do not ensure constraint satisfaction during training . In this work , to learn from the baseline policy while satisfying constraints , we propose an iterative algorithm that performs policy updates in three stages . The first step updates the policy to maximize expected reward using trust region policy optimization ( e.g. , TRPO ( Schulman et al. , 2015 ) ) . This can , however , result in a new intermediate policy that is too far from the baseline policy and one that may not satisfy the constraints . The second step performs a projection in policy space to control 1Code is available at : https : //sites.google.com/view/spacealgo . the distance between the current policy and the baseline policy . This distance is updated each episode depending on the reward improvement and constraint satisfaction , allowing the learning algorithm to explore without being overly restricted by the ( potentially constraint-violating ) baseline policy ( Rajeswaran et al. , 2017 ) . This also enables the baseline policy to influence the learning without the computational burden of learning a cost function for the baseline policy ( Kwon et al. , 2020 ) . The third step ensures constraint satisfaction at every iteration by performing a projection onto the set of policies that satisfy the given constraints . This ensures recovery from infeasible ( i.e. , constraint-violating ) states ( e.g. , due to approximation errors ) , and eliminates the need for tuning weights for auxiliary cost objective functions ( Tessler et al. , 2018 ) . We call our algorithm Safe Policy Adaptation with Constrained Exploration ( SPACE ) . This paper ’ s contributions are two-fold . We first analyze our proposed SPACE algorithm and provide a finite-time guarantee for its convergence . We also provide an analysis of controlling the distance between the learned policy at iteration k and the baseline policy to ensure both feasibility of the optimization problem and safe exploration by the learning agent . Second , we empirically compare SPACE with state-of-the-art algorithms on five different control tasks , including two Mujoco environments with safety constraints from Achiam et al . ( 2017 ) , two challenging traffic management tasks with fairness constraints from Vinitsky et al . ( 2018 ) , and one human demonstration driving task with safety constraints from Brockman et al . ( 2016 ) . In all tasks , SPACE outperforms the state-of-the art safe RL algorithm , projection-based constrained policy optimization ( PCPO ) in Yang et al . ( 2020 ) , averaging 40 % more reward with 10 times fewer cost constraint violations . This shows that SPACE leverages the baseline policy to achieve better learning efficiency while satisfying the cost constraint . 2 RELATED WORK . Policy optimization with constraints . Learning constraint-satisfying policies has been explored in the context of safe RL ( Garcia & Fernandez , 2015 ) , ( Hasanbeig et al. , 2020 ; Junges et al. , 2016 ; Jansen et al. , 2020 ) . Prior work either uses a conditional-gradient type of approach ( Achiam et al. , 2017 ) , adds a weighted copy of the cost objective in the reward function ( Tessler et al. , 2018 ; Chow et al. , 2019 ; Fujimoto et al. , 2019 ; Stooke et al. , 2020 ) , adds a safety layer to the policy ( Dalal et al. , 2018 ) ( Avni et al. , 2019 ) , or concerns about the chanced constraints ( Fu & Prashanth L , 2018 ; Zheng & Ratliff , 2020 ) . Perhaps the closest work to ours is Projection-based Constrained Policy Optimization ( PCPO ) ( Yang et al. , 2020 ) , which also uses projections in policy space to ensure constraint satisfaction . However , PCPO does not have the capability to safely exploit prior information ( through a baseline policy ) . The lack of using prior policies in PCPO makes it sampleinefficient . In addition , our SPACE algorithm ’ s update dynamically sets distances between policies while PCPO does not . This update is important to effectively and safely learn from the baseline policy . Furthermore , we provide a safety guarantee to ensure the feasibility of the optimization problem while PCPO does not . Merely adding an IL objective in the reward objective of PCPO can not learn efficiently , as shown in experiments . This analysis allows us to advance towards the practical use of RL in real applications , which PCPO and other algorithms have never done before . Policy optimization with the initial safe set . Wachi & Sui ( 2020 ) ; Sui et al . ( 2015 ) ; Turchetta et al . ( 2016 ) assume that the initial safe set is given , and the agent explores the environment and verifies the safety function from this initial safe set . There is no baseline policy here . In contrast , our assumption is to give a baseline policy to the agent . Both assumptions are reasonable as they provide an initial understanding of the environment . Leveraging baseline policies for RL . Prior work has used baseline policies to provide initial information to RL algorithms to reduce or avoid undesirable situations . This is done by either : initializing the policy with the baseline policy ( Driessens & Džeroski , 2004 ; Smart & Kaelbling , 2000 ; Koppejan & Whiteson , 2011 ; Abbeel et al. , 2010 ; Gao et al. , 2018 ; Le et al. , 2019 ; Vecerik et al. , 2017 ; Jaques et al. , 2019 ) ; or providing a teacher ’ s advice to the agent ( Garcia & Fernández , 2012 ; Quintı́a Vidal et al. , 2013 ; Abel et al. , 2017 ; Zhang et al. , 2019 ) . However , such works often assume that the baseline policy is constraint-satisfying ( Sun et al. , 2018 ; Balakrishna et al. , 2019 ) . In contrast , our SPACE algorithm safely leverages the baseline policy without requiring it to satisfy the specified constraints . Pathak et al . ( 2015 ) ; Bartocci et al . ( 2011 ) also modify the existing known models ( policies ) based on new conditions in the context of the formal methods . In contrast , we solve this problem using projections in the policy space . Learning from logged demonstration data . To effectively learn from demonstration data given by the baseline policy , Wu et al . ( 2019 ) ; Brown et al . ( 2019 ) ; Kwon et al . ( 2020 ) assess the demonstration data by either : predicting their cost in the new task using generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) ; or directly learning the cost function of the demonstration data . However , these approaches require a large number of training samples from the new task . In addition , the learned cost function is not guaranteed to recover the true one . This may result in driving the agent to undesirable situations . In contrast , SPACE controls the distance between the learned and baseline policies to ensure reward improvement and constraint satisfaction . Table 1 catalogs related work on safe RL and IL . We can differentiate these methods on several key axes : ( 1 ) whether they optimize reward and/or ensure cost satisfaction , ( 2 ) whether they use on-policy methods ( i.e. , query the baseline policy ) or off-policy methods ( i.e. , learn from a batch of demonstration data ) , ( 3 ) whether the baseline policy or demonstration data is optimal for the agent ’ s own objective , and ( 4 ) the technical approach of the work . 3 PROBLEM FORMULATION . We frame our problem as a constrained Markov Decision Process ( CMDP ) ( Altman , 1999 ) , defined as a tuple < S , A , T , R , C > . Here S is the set of states , A is the set of actions , and T specifies the conditional probability T ( s′|s , a ) that the next state is s′ given the current state s and action a . In addition , R : S × A → R is a reward function , and C : S × A → R is a constraint cost function . The reward function encodes the benefit of using action a in state s , while the cost function encodes the corresponding constraint violation penalty . A policy is a map from states to probability distributions on A . It specifies that in state s the selected action is drawn from the distribution π ( s ) . The state then transits from s to s′ according to the state transition distribution T ( s′|s , a ) . In doing so , a reward R ( s , a ) is received and a constraint cost C ( s , a ) is incurred , as outlined above . Let γ ∈ ( 0 , 1 ) denote a discount factor , and τ denote the trajectory τ = ( s0 , a0 , s1 , · · · ) induced by a policy π . Normally , we seek a policy π that maximizes a cumulative discounted reward JR ( π ) . = Eτ∼π [ ∑∞ t=0 γ tR ( st , at ) ] , ( 1 ) while keeping the cumulative discounted cost below hC JC ( π ) . = Eτ∼π [ ∑∞ t=0 γ tC ( st , at ) ] ≤ hC . ( 2 ) Here we consider an additional objective . We are provided with a baseline policy πB and at each state s we measure the divergence between π ( s ) and πB ( s ) . For example , this could be the KL-divergence D ( s ) . = DKL ( π ( s ) ‖πB ( s ) ) . We then seek a policy that maximizes Eq . ( 1 ) , satisfies Eq . ( 2 ) , and ensures the discounted divergence between the learned and baseline policies is below hD : JD ( π ) . = Eτ∼π [ ∑∞ t=0 γ tD ( st ) ] ≤ hD . ( 3 ) We do not assume that the baseline policy satisfies the cost constraint . Hence we allow hD to be adjusted during the learning of π to allow for reward improvement and constraint satisfaction . Let µt ( ·|π ) denote the state distribution at time t under policy π . The discounted state distribution induced by π is defined to be dπ ( s ) .= ( 1− γ ) ∑∞ t=0 γ tµt ( s|π ) . Now bring in the reward advantage function ( Kakade & Langford , 2002 ) defined by AπR ( s , a ) . = QπR ( s , a ) − V πR ( s ) , where V πR ( s ) . = Eτ∼π [ ∑∞ t=0 γ tR ( st , at ) |s0 = s ] is the expected reward from state s under policy π , and QπR ( s , a ) . = Eτ∼π [ ∑∞ t=0 γ tR ( st , at ) |s0 = s , a0 = a ] is the expected reward from state s and initial action a , and thereafter following policy π . These definitions allow us to express the reward performance of one policy π′ in terms of another π : JR ( π ′ ) − JR ( π ) = 11−γEs∼dπ′ , a∼π′ [ A π R ( s , a ) ] . Similarly , we can define AπD ( s , a ) , Q π D ( s , a ) and V π D ( s ) for the divergence cost , and A π C ( s , a ) , QπC ( s , a ) and V π C ( s ) for the constraint cost .

Summary: The authors propose SPACE, an RL algorithm for learning a policy that maximizes reward while satisfying given constraints in a setting where a baseline policy is provided. They design a three-step update rule for learning such a policy and provide a finite-sample analysis of the resulting method in a simplified setting. They report numerical results in several domains that show how the proposed approach outperforms competitive baselines.