Inverse Constrained Reinforcement Learning

1 INTRODUCTION . Reward functions are a critical component in reinforcement learning settings . As such , it is important that reward functions are designed accurately and are well-aligned with the intentions of the human designer . This is known as agent ( or value ) alignment ( see , e.g. , Leike et al . ( 2018 ; 2017 ) ; Amodei et al . ( 2016 ) ) . Misspecified rewards can lead to unwanted and unsafe situations ( see , e.g , Amodei & Clark ( 2016 ) ) . However , designing accurate reward functions remains a challenging task . Human designers , for example , tend to prefer simple reward functions that agree well with their intuition and are easily interpretable . For example , a human designer might choose a reward function that encourages an RL agent driving a car to minimize its traveling time to a certain destination . Clearly , such a reward function makes sense in the case of a human driver since inter-human communication is contextualized within a framework of unwritten and unspoken constraints , often colloquially termed as ‘ common-sense ’ . That is , while a human driver will try to minimize their traveling time , they will be careful not to break traffic rules , take actions that endanger passersby , and so on . However , we can not assume such behaviors from RL agents since they are are not imbued with common-sense constraints . Constrained reinforcement learning provides a natural framework for maximizing a reward function subject to some constraints ( we refer the reader to Ray et al . ( 2019 ) for a brief overview of the field ) . However , in many cases , these constraints are hard to specify explicitly in the form of mathematical functions . One way to address this issue is to automatically extract constraints by observing the behavior of a constraint-abiding agent . Consider , for example , the cartoon in Figure 1 . Agents start at the bottom-left corner and are rewarded according to how quickly they reach the goal at the bottom-right corner . However , what this reward scheme misses out is that in the real world the lower bridge is occupied by a lion which attacks any agents attempting to pass through it . Therefore , agents that are naı̈vely trained to maximize the reward function will end up performing poorly in the real world . If , on the other hand , the agent had observed that the expert ( in Figure 1 ( a ) ) actually performed suboptimally with respect to the stipulated reward scheme by taking a longer route to the goal , it could have concluded that ( for some unknown reason ) the lower bridge must be avoided and consequently would have not been eaten by the lion ! Scobee & Sastry ( 2020 ) formalizes this intuition by casting the problem of recovering constraints in the maximum entropy framework for inverse RL ( IRL ) ( Ziebart et al. , 2008 ) and proposes a greedy algorithm to infer the smallest number of constraints that best explain the expert behavior . However , Scobee & Sastry ( 2020 ) has two major limitations : it assumes ( 1 ) tabular ( discrete ) settings , and ( 2 ) the environment ’ s transition dynamics . In this work , we aim to address both of these issues by learning a constraint function instead through a sample-based approximation of the objective function of Scobee & Sastry . Consequently , our approach is model-free , admits continuous states and actions and can learn arbitrary Markovian constraints1 . Further , we empirically show that it scales well to high-dimensions . Typical inverse RL methods only make use of expert demonstrations and do not assume any knowledge about the reward function at all . However , most reward functions can be expressed in the form “ do this task while not doing these other things ” where other things are generally constraints that a designer wants to impose on an RL agent . The main task ( “ do this ” ) is often quite easy to encode in the form of a simple nominal reward function . In this work , we focus on learning the constraint part ( “ do not do that ” ) from provided expert demonstrations and using it in conjunction with the nominal reward function to train RL agents . In this perspective , our work can be seen as a principled way to inculcate prior knowledge about the agent ’ s task in IRL . This is a key advantage over other IRL methods which also often end up making assumptions about the agent ’ s task in the form of regularizers such as in Finn et al . ( 2016 ) . The main contributions of our work are as follows : • We formulate the problem of inferring constraints from a set of expert demonstrations as a learning problem which allows it to be used in continuous settings . To the best of our knowledge , this is the first work in this regard . • We eliminate the need to assume , as Scobee & Sastry do , the environment ’ s transition dynamics . • We demonstrate the ability of our method to train constraint-abiding agents in highdimensions and show that it can also be used to prevent reward hacking . 2 PRELIMINARIES . 2.1 UNCONSTRAINED RL . A finite-horizon Markov Decision Process ( MDP ) M is a tuple ( S , A , p , r , γ , T ) , where S ∈ R|S| is a set of states , A ∈ R|A| is a set of actions , p : S × A × S 7→ [ 0 , 1 ] is the transition probability function ( where p ( s′|s , a ) denotes the probability of transitioning to state s′ from state s by taking 1Markovian constraints are of the form c ( τ ) = ∏T t=1 c ( st , at ) i.e . constraint function is independent of the past states and actions in the trajectory . action a ) , r : S × A 7→ R is the reward function , γ is the discount factor and T is the timehorizon . A trajectory τ = { s1 , a1 , . . . , sT , aT } denotes a sequence of states-action pairs such that st+1 ∼ p ( ·|st , at ) . A policy π : S 7→ P ( A ) is a map from states to probability distributions over actions , with π ( a|s ) denoting the probability of taking action a in state s. We will sometimes abuse notation to write π ( s , a ) to mean the joint probability of visiting state s and taking action a under the policy π and similarly π ( τ ) to mean the probability of the trajectory τ under the policy π . Define r ( τ ) = ∑T t=1 γ tr ( st , at ) to be the total discounted reward of a trajectory . Forward RL algorithms try to find an optimal policy π∗ that maximizes the expected total discounted reward J ( π ) = Eτ∼π [ r ( τ ) ] . On the other hand , given a set of trajectories sampled from the optimal ( also referred to as expert ) policy π∗ , inverse RL ( IRL ) algorithms aim to recover the reward function r , which can then be used to learn the optimal policy π∗ via some forward RL algorithm . 2.2 CONSTRAINED RL . While normal ( unconstrained ) RL tries to find a policy that maximizes J ( π ) , constrained RL instead focuses on finding a policy that maximizes J ( π ) while respecting explicitly-defined constraints . A popular framework in this regard is the one presented in Altman ( 1999 ) which introduces the notion of a constrained MDP ( CMDP ) . A CMDP Mc is a simple MDP augmented with a cost function c : S ×A 7→ R and a budget α ≥ 0 . Define c ( τ ) = ∑T t=1 γ tc ( st , at ) to be the total discounted cost of the trajectory τ and Jc ( π ) = Eτ∼π [ c ( τ ) ] to be the expected total discounted cost . The forward constrained RL problem is to find the policy π∗c that maximizes J ( π ) subject to J c ( π ) ≤ α . In this work , given a set D of trajectories sampled from π∗c , the corresponding unconstrained MDP M ( i.e. , Mc without the cost function c ) and a budget α , we are interested in recovering a cost function which when augmented withM has an optimal policy that generates the same set of trajectories as in D. We call this as the inverse constrained reinforcement learning ( ICRL ) problem . If the budget α is strictly greater than 0 , then the cost function c defines soft constraints over all possible state-action pairs . In other words , a policy is allowed to visit states and take actions that have non-zero costs as long as the expected total discounted cost remains less than α . If , however , α is 0 then the cost function translates into hard constraints over all state-action pairs that have a non-zero cost associated with them . A policy can thus never visit these state-action pairs . In this work , we restrict ourselves to this hard constraint setting . Note that this is not particularly restrictive since , for example , safety constraints are often hard constraints as well are constraints imposed by physical laws . Since we restrict ourselves to hard constraints , we can rewrite the constrained RL problems as follows : define C = { ( s , a ) |c ( s , a ) 6= 0 } to be the constraint set induced by c. The forward constraint RL problem is to find the optimal constrained policy π∗C that maximizes J ( π ) subject to π∗C ( s , a ) = 0 ∀ ( s , a ) ∈ C. The inverse constrained RL problem is to recover the constraint set C from trajectories sampled from π∗C . Finally , we will refer to our unconstrained MDP as the nominal MDP hereinafter . The nominal MDP represents the nominal environment ( simulator ) in which we train our agent . 3 FORMULATION . 3.1 MAXIMUM LIKELIHOOD CONSTRAINT INFERENCE . We take Scobee & Sastry as our starting point . Suppose that we have a set of trajectories D = { τ ( i ) } Ni=1 sampled from an expert π∗C navigating in a constrained MDPMC ∗ where C∗ denotes the ( true ) constraint set . Furthermore , we are also given the corresponding nominal MDPM2 . Our goal is to recover a constraint set which when augmented withM results in a CMDP that has an optimal policy that respects the same set of constraints as π∗C does . Scobee & Sastry pose this as a maximum likelihood problem . That is , if we let pM denote probabilities given that we are considering MDP M and assume a uniform prior on all constraint sets , then we can choose C∗ according to C∗ ← arg max C pM ( D|C ) . ( 1 ) 2Availability of transition dynamics model of nominal MDP is not necessary . Under the maximum entropy ( MaxEnt ) model presented in Ziebart et al . ( 2008 ) , the probability of a trajectory under a deterministic MDPM can be modelled as πM ( τ ) = exp ( βr ( τ ) ) ZM 1 M ( τ ) , ( 2 ) where ZM = ∫ exp ( βr ( τ ) ) 1M ( τ ) dτ is the partition function , β ∈ [ 0 , ∞ ) is a parameter describing how close the agent is to the optimal distribution ( as β →∞ the agent becomes a perfect optimizer and as β → 0 the agent simply takes random actions ) and 1 is an indicator function that is 1 for trajectories feasible under the MDPM and 0 otherwise . Assume that all trajectories in D are i.i.d . and sampled from the MaxEnt distribution . We have p ( D|C ) = 1 ( ZMC ) N N∏ i=1 exp ( βr ( τ ( i ) ) ) 1M C ( τ ( i ) ) . ( 3 ) Note that 1M C ( τ ( i ) ) is 0 for all trajectories that contain any state-action pair that belongs to C. To maximize this , Scobee & Sastry propose a greedy strategy wherein they start with an empty constraint set and incrementally add state-action pairs that result in the maximal increase in p ( D|C ) .

Summary: The submission focuses on a variant of inverse reinforcement learning, where the learner knows the task reward but is unaware of hard constraints that need to be respected while completing the task. The authors provide an algorithm to recover these constraints from expert demonstrations. The proposed algorithm builds upon a recent technique (Scobee & Sastry 2020) and addresses problems with large and continuous state spaces.