Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning

1 INTRODUCTION . Advances in mechanical design and artificial intelligence continue to expand the horizons of robotic applications . In these new domains , it can be difficult to design a specific robot behavior by hand . Even manually specifying a task for a reinforcement-learning-enabled agent is notoriously difficult ( Ho et al. , 2015 ; Amodei et al. , 2016 ) . Inverse Reinforcement Learning ( IRL ) techniques can help alleviate this burden by automatically identifying the objectives driving certain behavior . Since first being introduced as Inverse Optimal Control by Kalman ( 1964 ) , much of the work on IRL has focused on learning environmental rewards to represent the task of interest ( Ng & Russell , 2000 ; Abbeel & Ng , 2004 ; Ratliff et al. , 2006 ; Ziebart et al. , 2008 ) . While these types of IRL algorithms have proven useful in a variety of situations ( Abbeel et al. , 2007 ; Vasquez et al. , 2014 ; Ziebart , 2010 ; Scobee et al. , 2018 ) , their basis in assuming that reward functions fully represent task specifications makes them ill suited to problem domains with hard constraints or non-Markovian objectives . Recent work has attempted to address these pitfalls by using demonstrations to learn a rich class of possible specifications that can represent a task ( Vazquez-Chanlatte et al. , 2018 ) . Others have focused specifically on learning constraints , that is , behaviors that are expressly forbidden or infeasible ( Pardowitz et al. , 2005 ; Pérez-D ’ Arpino & Shah , 2017 ; Subramani et al. , 2018 ; McPherson et al. , 2018 ; Chou et al. , 2018 ) . Such constraints arise in safety-critical systems , where requirements such as an autonomous vehicle avoiding collisions with pedestrians are more naturally expressed as hard constraints than as soft reward penalties . It is towards the problem of inferring such constraints that we turn our attention . In this work , we present a novel method for inferring constraints , drawing primarily from the Maximum Entropy approach to IRL described by Ziebart et al . ( 2008 ) . We use this framework to reason about the likelihood of observing a set of demonstrations given a nominal task description , as well as about their likelihood if we imposed additional constraints on the task . This knowledge allows us to select a constraint , or set of constraints , which maximizes the demonstrations ’ likelihood and best explains the differences between expected and demonstrated behavior . Our method improves on prior work by being able to simultaneously consider constraints on states , actions and features in a Markov Decision Process ( MDP ) to provide a principled ranking of all options according to their effect on demonstration likelihood . 2 RELATED WORK . 2.1 INVERSE REINFORCEMENT LEARNING . A formulation of the IRL problem was first proposed by Kalman ( 1964 ) as the Inverse problem of Optimal Control ( IOC ) . Given a dynamical system and a control law , the author sought to identify which function ( s ) the control law was designed to optimize . This problem was brought into the domain of MDPs and Reinforcement Learning ( RL ) by Ng & Russell ( 2000 ) , who proposed IRL as the task of , given an MDP and a policy ( or trajectories sampled according to that policy ) , find a reward function with respect to which that policy is optimal . One of the chief difficulties in the problem of IRL is the fact that a policy can be optimal with respect to a potentially infinite set of reward functions . The most trivial example of this is the fact that all policies are optimal with respect to a null reward function that always returns zero . Much of the work in IRL has been devoted to developing approaches that address this ambiguity by imposing additional structure to make the problem well-posed ( Abbeel & Ng , 2004 ; Ratliff et al. , 2006 ) . Ziebart et al . ( 2008 ) approach the problem by employing the principle of maximum entropy ( Jaynes , 1957 ) , which allows the authors to develop an IRL algorithm that produces a single stochastic policy that matches feature counts without adding any additional constraints to the produced behavior . This so called Maximum Entropy IRL ( MaxEnt ) provides a framework for reasoning about demonstrations from experts who are noisily optimal . The induced probability distribution over trajectories forms the basis for our efforts in identifying the most likely behavior-modifying constraints . 2.2 BEYOND REWARD FUNCTIONS . While Markovian rewards do often provide a succinct and expressive way to specify the objectives of a task , they can not capture all possible task specifications . Vazquez-Chanlatte et al . ( 2018 ) highlight the utility of non-Markovian Boolean specifications which can describe complex objectives ( e.g . do this before that ) and compose in an intuitive way ( e.g . avoid obstacles and reach the goal ) . The authors of that work draw inspiration from the MaxEnt framework to develop their technique for using demonstrations to calculate the posterior probability that an agent is attempting to satisfy a Boolean specification . A subset of these types of specifications that is of particular interest to us is the specification of constraints , which are states , actions , or features of the environment that must be avoided . Chou et al . ( 2018 ) explore how to infer trajectory feature constraints given a nominal model of the environment ( lacking the full set of constraints ) and a set of demonstrated trajectories . The core of their approach is to sample from the set of trajectories which have better performance than the demonstrated trajectories . They then infer that the set of possible constraints is the subset of the feature space that contains the higher-reward sampled trajectories , but not the demonstrated trajectories . Intuitively , they reason that if the demonstrator could have passed through those features to earn a higher reward , but did not , then there must have been a previously unknown constraint preventing that behavior . However , while their approach does allow for a cost function to rank elements from the set of possible constraints , the authors do not offer a mechanism for determining what cost function will best order these constraints . Our approach to constraint inference from demonstrations addresses this open question by providing a principled ranking of the likelihood of constraints . We adapt the MaxEnt framework to allow us to reason about how adding a constraint will affect the likelihood of demonstrated behaviors , and we can then select the constraints which maximize this likelihood . We consider feature-space constraints as in Chou et al . ( 2018 ) , and we explicitly augment the feature space with state- and action-specific features to directly compare the impacts of state- , action- , and feature-based constraints on demonstration likelihood . 3 MAXIMUM LIKELIHOOD CONSTRAINT INFERENCE . 3.1 PROBLEM FORMULATION . Following the formulation presented by Ziebart et al . ( 2008 ) , we base our work in the setting of a ( finite-state ) Markov Decision Process ( MDP ) . We define an MDP M as a tuple ( S , { As } , { Ps , a } , D0 , φ , R ) where S is a finite set of discrete states ; { As } is a set of the sets of actions available to be taken for each state s , such that As ⊆ A , where A is a finite set of discrete actions ; { Ps , a } is a set of state transition probability distributions such that Ps , a ( s′ ) = P ( s′|s , a ) is the probability of transitioning to state s′ after taking action a from state s ; D0 : S → [ 0 , 1 ] is an initial state distribution ; φ : S × A → Rk+ is a mapping to a k-dimensional space of non-negative features ; and R : S ×A→ R is a reward function . A trajectory ξ through this MDP is a sequence of states st and actions at such that s0 ∼ D0 and state si+1 ∼ Psi , ai . Actions are chosen by an agent navigating the MDP according to a , potentially time-varying , policy π such that π ( ·|s , t ) is a probability distribution over actions in As . We denote a finite-time trajectory of length T + 1 by ξ = { s0 : T , a0 : T } . At every time step t , a trajectory will accumulate features equal to φ ( st , at ) . We use the notation φi ( · , · ) to refer to the i-th element of the feature map , and we use the label φi to denote the i-th feature itself . We also introduce an augmented indicator feature mapping φ̃1 : S ×A→ { 0 , 1 } nφ , where nφ = k+ |S|+ |A| . This augmented feature map uses binary variables to indicate the presence of a feature and expands the feature space by adding binary features to track occurrences of each state and action , such that φ̃1φi ( s , a ) = { 1 if φi ( s , a ) > 0 0 otherwise , φ̃1si ( s , a ) = { 1 if s = si 0 otherwise , φ̃1ai ( s , a ) = { 1 if a = ai 0 otherwise . ( 1 ) Typically , agents are modeled as trying to maximize , or approximately maximize , the total reward earned for a trajectory ξ , given by R ( ξ ) = ∑T t=0 γ tR ( st , at ) , where γ ∈ ( 0 , 1 ] is a discount factor . Therefore , an agent ’ s policy π is closely tied to the form of the MDP ’ s reward function . Conventional IRL focuses on inferring a reward function that explains an agent ’ s policy , revealed through the behavior observed in a set of demonstrated trajectories D. However , our method for constraint inference poses a different challenge : given an MDPM , including a reward function , and a set of demonstrations D , find the most likely set of constraints C∗ that could modifyM to explain these demonstrations . We define our notion of constraints in the following section . 3.2 CONSTRAINTS FOR MDPS . Constraints are those behaviors that are not disallowed explicitly by the structure of the MDP , but which would be infeasible or prohibited for the underlying system being modeled by the MDP . This sort of discrepancy can occur when a generic or simplified MDP is designed without exact knowledge of specific constraints for the modeled system . For instance , for a generic MDP modeling the behavior of cars , we might want to include states for speeds up to 500km/h and actions for accelerations up to 12m/s2 . However , for a specific car on a specific roadway , the set of states where the vehicle travels above 100km/h may be prohibited because of a speed limit , and the set of actions where the vehicle accelerates above 4m/s2 may be infeasible because of the physical limitations of the vehicle ’ s engine . Therefore , any MDP trajectory of this specific car system would not contain a state-action pair which violates these legal and physical limits . Figure 1 shows an example of constraints driving behavior . We define a constraint set Ci ⊆ S ×A as a set of state-action pairs that violate some specification of the modeled system . We consider three general classes of constraints : state constraints , action constraints , and feature constraints . A state constraint set Csi = { ( s , a ) | s = si } includes all stateaction pairs such that the state component is si . An action constraint set Cai = { ( s , a ) | a = ai } includes all state-action pairs such that the action component is ai . A feature constraint set Cφi = { ( s , a ) | φi ( s , a ) > 0 } includes all state-action pairs that produce a non-zero value for feature φi . If we augment the set of features as described in ( 1 ) , it is straightforward to see that state and action constraints become special cases of feature constraints , with constraint sets given by Ci = { ( s , a ) | φ̃1i ( s , a ) = 1 } . It is also evident that we can obtain compound constraints , respecting two or more conditions , by taking the union of constraint sets Ci to obtain C = ⋃ i Ci . 3.2.1 ADDING CONSTRAINTS TO AN MDP We need to be able to reason about how adding a constraint to an MDP will influence the behavior of agents navigating that environment . If we impose a constraint on an MDP , then none of the state-action pairs in that constraint set may appear in a trajectory of the constrained MDP . To enforce this condition , we must restrict the actions available in each state so that it is not possible for an agent to produce one of the constrained state-action pairs . For a given constraint C , we can replace the set of available actions As in every state s with an alternative set ACs given by ACs = As \ { a ∈ As | ( s , a ) ∈ C } . ( 2 ) Performing such substitutions for an MDPM will lead to a modified MDPMC such thatMC = ( S , { ACs } , { Ps , a } , D0 , φ , R ) . The question then arises as to the how we should treat states with empty action sets ACs = ∅ . Since an agent arriving in such an empty state would have no valid action to select , any trajectory visiting an empty state must be deemed invalid . Indeed , such empty action sets will be produced for any state si such that Csi ⊆ C. For MDPs with deterministic transitions , agents know precisely which state they will arrive in following a certain action . Therefore , any agent respecting constraints will not take an action that leads to an empty state , since doing so will lead to constraint violations . If we consider the set of empty states Sempty , then for the purposes of reasoning about an agent ’ s behavior , we can impose an additional constraint set Cempty = { ( s , a ) | ∃ sempty ∈ Sempty : Ps , a ( sempty ) = 1 } . In this work , we will always implicitly add this constraint set , such thatMC will be equivalent toMC∪Cempty , and we recursively add these constraints until reaching a fixed point . For MDPs with stochastic transitions , the semantics of an empty state are less obvious and could lend themselves to multiple interpretations depending on the nature of the system being modeled . We offer a possible treatment in the appendix .

Summary: In this work, a novel inverse constraint learning method is proposed, where the goal is to find out the constraints over state-action pairs for given demonstration and MDP **including a reward function** (so different from inverse cost learning). The novelty of this work comes from introducing maximum entropy inverse reinforcement learning (MaxEntIRL) framework to previous works [1, 2], and this work mainly focused on the tabular setting. The objective of this work is to solve the optimization in (8), which tries to find out the constraint that maximizes the probability of trajectories that cannot be generated if that constraint is applied. (Such an objective minimizes the normalization constant in (5) and results in maximization of the demonstration likelihood under the constraint.) To solve this objective, the proposed algorithm first computes the feature occupancy (Algorithm 1), and then use those feature occupancy with greedy iterative constraint inference (Algorithm 2 that motivated by maximum coverage problem) to get constraints. Two experiments in the GridWorld show that the proposed method effectively works.