Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning
Authors: Dexter R.R. Scobee
S. Shankar Sastry
Venue:  (2020)
Keywords: learning from demonstration
inverse reinforcement learning
constraint inference

Abstract: While most approaches to the problem of Inverse Reinforcement Learning (IRL) focus on estimating a reward function that best explains an expert agent’s policy or demonstrated behavior on a control task, it is often the case that such behavior is more succinctly represented by a simple reward combined with a set of hard constraints. In this setting, the agent is attempting to maximize cumulative rewards subject to these given constraints on their behavior. We reformulate the problem of IRL on Markov Decision Processes (MDPs) such that, given a nominal model of the environment and a nominal reward function, we seek to estimate state, action, and feature constraints in the environment that motivate an agent’s behavior. Our approach is based on the Maximum Entropy IRL framework, which allows us to reason about the likelihood of an expert agent’s demonstrations given our knowledge of an MDP. Using our method, we can infer which constraints can be added to the MDP to most increase the likelihood of observing these demonstrations. We present an algorithm which iteratively infers the Maximum Likelihood Constraint to best explain observed behavior, and we evaluate its efficacy using both simulated behavior and recorded data of humans navigating around an obstacle.

Published as a conference paper at ICLR 2020
MAXIMUM LIKELIHOOD CONSTRAINT INFERENCE FOR INVERSE REINFORCEMENT LEARNING
Dexter R.R. Scobee & Shankar Sastry
Department of Electrical Engineering and Computer Sciences University of California, Berkeley {dscobee, sastry) e @eecs.
ABSTRACT
While most approaches to the problem of Inverse Reinforcement Learning (IRL.) focus on estimating a reward function that best explains an expert agent's policy demonstrated behavior control task, it is often the case that such behavior is more succinctly represented by a simple reward combined with a set of hard constraints. In this setting, the agent is attempting to maximize cumulative rewards subject to these given constraints on their behavior. We reformulate the problem IRL on Markov Decision Processes (MDPs) such that, given nominal model of the environment and a nominal reward function, we seek to estimate state, action, and feature constraints in the environment that motivate agent's behavior. Our approach is based on the Maximum Entropy IRL framework, which allows us to reason about the likelihood of an expert agent's s demonstrations given our knowl- edge of an MDP. Using our method, we can infer which constraints can be added to the MDP to most increase the likelihood of observing these demonstrations. We present algorithm which iteratively infers the Maximum Likelihood Constraint to best explain observed behavior, and we evaluate its efficacy using both simulated behavior and recorded data of humans navigating around an obstacle.
INTRODUCTION
Advances in mechanical design and artificial intelligence continue to expand the horizons of robotic applications. these new domains, can be difficult to design a specific robot behavior by hand. Even manually specifying a task for
agent is notoriously difficult (Ho et al., 2015 Amodei et al. 2016). Inverse Reinforcement Learning (IRL) techniques can help alleviate this burden by automatically identifying the objectives driving certain behavior. Since first being introduced as Inverse Optimal Control by Kalman (1964), much of the work on IRL has focused on learning environmental rewards to represent the task of interest (Ng & Russell (2000] Abbeel & Ng 2004 Ratliff et al.) 2006 Ziebart et al.] 2008). While these types of IRL algorithms have proven useful in a variety of situations (Abbeel et al.) [007)] Vasquez et al. 2014 Ziebart [2010) Scooee et al.)
2018), their basis in assuming that reward functions fully represent task specifications makes them ill
suited to problem domains with hard constraints or non- Mon-Maroovian objectives.
Recent work has attempted to address these pitfalls by using demonstrations to learn a rich class of possible specifications that can represent a task (Vazquez-Chanlatte et al., 22018) Others have focused specifically on learning constraints, that is, behaviors that are expressly forbidden or infeasible (Pardowitz al. 2005 Pérez-D' Pééez-D'Arrinoo & Shah, 2017: Subramani al. [0188] McPherson et al.) et al. 2018). Such constraints arise in safety-critical systems, where requirements such as an autonomous vehicle avoiding collisions with pedestrians are more naturally expressed hard constraints than as soft reward penalties. It is towards the problem of inferring such constraints that we turn our attention.
In this work, we present novel method for inferring constraints, drawing primarily from the Maximum Entropy approach to IRL described by Ziebart et al.] (2008). We use this framework to reason about the likelihood of observing set of demonstrations given nominal task description, as well as about their likelihood if we imposed additional constraints on the task. This knowledge allows select constraint, set of constraints, which maximizes the demonstrations' likelihood and

---

Published as a conference paper at ICLR 2020
best explains the differences between expected and demonstrated behavior. Our method improves prior work by being able to simultaneously consider constraints on states, actions and features in a Markov Decision Process (MDP) provide principled ranking of all options according to their effect on demonstration likelihood.
2
RELATED WORK
2.1 INVERSE REINFORCHMENT LEARNING
A formulation of the IRL problem was first proposed by Kalman (1964) the Inverse problem of Optimal Control (IOC). Given a dynamical system and a control law, the author sought to identify which function(s) the control law designed to optimize. This problem domain of MDPs and Reinforcement Learning (RI. by Ng & Russell (2000), who proposed IRI. the task of, given MDP and policy (or trajectories sampled according that policy), find reward function with respect to which that policy is optimal.
brought into the
One of the chief difficulties in the problem of IRL the fact that policy can be optimal with respect to potentially infinite set of reward functions. The most trivial example of this is the fact that all policies are optimal with respect in IRL has been devoted to developing approaches that address this ambiguity by imposing additional structure to make the problem well well-posed (Abbeel & Ng. 2004, Ratliff al.) 2006). Ziebart al.) (2008) approach the problem by employing the principle of maximum entropy (Jaynes) 1957), which allows the authors to develop an IRL. algorithm that produces a single stochastic policy that matches feature counts without adding any additional constraints to the produced behavior. This called Maximum Entropy IRL (MaxEnt) provides framework for reasoning about demonstrations from experts who noisily optimal. The induced probability distribution over trajectories forms the basis for our efforts in identifying the most likely behavior- maavvio-modiiyiigg constraints.
null reward function that always returns zero. Much of the work
2.2 BEYOND REWARD FUNCTIONS
While Markovian rewards do often provide succinet and expressive way specify the objectives of a task, they cannot capture all possible task specifications. Nazquez-Chanlatte et al. (2018) highlight the utility of Mon-Markovian Boolean specifications which can describe complex objectives (e.g. do this before that) and compose in an intuitive way (e.g. avoid obstacles and reach the goal). The authors of that work draw inspiration from the MaxEnt framework to develop their technique for using demonstrations to calculate the posterior probability that agent is attempting to satisfy Boolean specification.
A subset of these types of specifications that is of particular interest is the specification of constraints, which are states, actions, or features of the environment that must be avoided. [Chou et al.) (2018) explore how infer trajectory feature constraints given nominal model of the environment (lacking the full set of constraints) and a set of demonstrated trajectories. The core their approach is sample from the set of trajectories which have better performance than the demonstrated trajectories. They then infer that the set of possible constraints is the subset of the feature space that contains the higher-reward sampled trajectories, but not the demonstrated trajectories. Intuitively, they reason that if the demonstrator could have passed through those features to earn a higher reward, but did not, then there must have been a previously unknown constraint preventing that behavior. However, while their approach does allow for a cost function to rank elements from the set of possible constraints, the authors do not offer a mechanism for determining what cost function will best order these constraints.
Our approach to constraint inference from demonstrations addresses this open question by providing a principled ranking of the likelihood of constraints. We adapt the MaxEnt framework to allow us to reason about how adding constraint will affect the likelihood of demonstrated behaviors, and we can then select the constraints which maximize this likelihood. We consider feature-space constraints as in Chou et al. (2018), and explicitly augment the feature space with state- and action-specifie features to directly compare the impacts of state-, action-, and feature- based constraints on demonstration likelihood.
2

---

Published as a conference paper at ICLR 2020
3 MAXIMUM LIKELIHOOD CONSTRAINT INFERENCE
3.1 PROBLEM FORMULATION
Following the formulation presented by Ziebart et al. (2008), we base work in the set- ting of a (finite-state) Markov Decision Process (MDP). We define an MDP M as a tuple (S, {A₂), (Pa.a} Do. 0, R) where 5 is finite set of discrete states; {A,) is set of the sets of actions available to be taken for each state s, such that A, C A, where A is a finite set of discrete actions: } } is set of state transition probability distributions such that Pa.a(8 (s") P(s'|s, (ls[ss,ee a) is the probability of transitioning to state s' after taking action a from state Do : S - [0, 1] is an initial state distribution: Φ S A + I4, is mapping k-dimensional space of non non-negative features; and R: S A - IR is reward function. A trajectory & through this MDP is a sequence of states 8, and actions a such that 8q ~ Do and state 8141 Ps.aa, Actions are chosen by an agent navigating the MDP according to potentially time-varying, policy such that x (-|ss, () is probability distribution over actions in A.. We denote a finite-time trajectory of length T + 11++00 by { {So.T, ).
At every time step I, a trajectory will accumulate features equal to 0(5,, at). We use the notation (.(:.)) to refer to the i-th element of the feature map, and we use the label di denote the i-th feature itself. We also introduce an augmented indicator feature mapping Ø² SS - {0. 1)",, where n k |S| [A]. This augmented feature map uses binary variables to indicate the presence of feature and expands the feature space by adding binary features to track occurrences of each state and action, such that
71 0 (s,
{
[1
if (1(ss. 00 otherwise
(s,a) {
if = s: otherwise" : (s, f1 if a otherwise
Typically, agents are modeled as trying to maximize, or approximately maximize, the total reward eamedd for trajectory ç. given by R(() Σt= 7² R(st, at), where E (0, 1) is discount factor. Therefore, an agent's policy æ is closely tied to the form of the MDP's reward function.
Conventional IRL. focuses on inferring a reward function that explains an agent's policy, revealed through the behavior observed in a set of demonstrated trajectories D. However, our method for constraint inference poses a different challenge: given MDP M, including reward function, and a set of demonstrations D, find the most likely set of constraints C' that could modify M to explain these demonstrations. We define our notion of constraints in the following section.
3.2 CONSTRAINTS FOR MDPS
Constraints are those behaviors that are not disallowed explicitly by the structure of the MDP, but which would be infeasible or prohibited for the underlying system being modeled by the MDP. This sort of discrepancy can occur when a generic or simplified MDP is designed without exact knowledge of specific constraints for the modeled system. For instance, for generic MDP modeling the behavior of cars, we might want to include states for speeds up to 500km/ h and actions for accelerations up to 12m/s². However, for specific car on a specific roadway, the set of states where the vehicle travels above 100km// hh may be prohibited because of a speed limit, and the set of actions where the vehicle accelerates above 4m/s² may be infeasible because of the physical limitations of the vehicle's engine. Therefore, any MDP trajectory of this specific car system would not contain a state-action pair which violates these legal and physical limits. Figure shows an example of constraints driving behavior.
We define a constraint set C. C S x A as a set of state-action pairs that violate some specification of the modeled system. We consider three general classes of constraints: state constraints, action constraints, and feature constraints. A state constraint set C., {(ss a) = } includes all state- action pairs such that the state component is 5;. An action constraint set Ca, = {(s, a) | a = a.} includes all state-action pairs such that the action component is a;- A feature constraint set Co. = {(s, a) Φi(s, a) > 0} includes all state-action pairs that produce a non-zero value for feature Φ,.
If we augment the set of features as described in [.. is straightforward to see that state and action constraints become special cases of feature constraints, with constraint sets given by
3

---

Published as a conference paper at ICLR 2020
= ((s (s. 1}. It is also evident that obtain compound constraints, respecting two or more conditions, by taking the union of constraint sets C2 to obtain C' U, C.
3.2. ADDING CONSTRAINTS TO MDP
We need to be able to reason about how adding a constraint to an MDP will influence the behavior of agents navigating that envi- ronment. If we impose a constraint on an MDP, then none the state-action pairs in that constraint the constrained MDP. To enforce this condition, we must restrict the actions available in each state so that is not possible for an agent to produce one of the constrained state-action pairs. For a given constraint C, we can replace the set of available actions A, in every state s with an alternative set A given by
0
-1
1
-1
1
1
-1
4
may appear in trajectory of
11
-1
11
4
-1
-1
-1
1
4
1
-1
-1
1
1
-1
-1
-1
- = A. la (6s.) EC).
(2)
(a) Nominal MDP
Performing such substitutions for MDP M will lead modified
MDP M such that M = (S, { {4 ), (Ps,a), Do, d, R).
0
-1
1
-1
-1
-1
The question then arises to the how should treat states with empty action sets A = 0. Since an agent arriving in such an empty state would have valid action to select, any trajectory visiting an empty state must be deemed invalid. Indeed, such empty action sets will be produced for any state 8; such that C., C.
-1
11
-1
-1
-1
-1
11
1
-1
For MDPs with deterministic transitions, agents know precisely which state they will arrive in following a certain action. Therefore, any agent respecting constraints will not take an action that leads to an empty state, since doing so will lead to constraint violations. we consider the set of empty states Sempty. then for the purposes of rea- soning about an agent's ' behavior, we can impose an additional con- straint set Cempty {(s, 33 Semply Sempyy Ps.a Ps.a(Sempty) In this work, we will always implicitly add this constraint set, such that MC will be equivalent to MCUCsuwe and we recursively add these constraints until reaching a fixed point.
11
(b) Constrained MDP
Illustration of tra- jectories likely be produced by noisily optimal agents nav- igating an MDP. a Expected behavior
Figure 1:
1}.
generic, nom- inal MDP. (b) Demonstrated behavior from a specific, con- strained MDP. Numbers repre- sent based rewards, and the red-shaded tiles represent state constraints.
For MDPs with stochastic transitions, the semantics of empty state are less obvious and could lend themselves to multiple interpretations depending the nature of the system being modeled. We offer a possible treatment in the appendix.
3.2.2 NOMINAL MDPs
The nominal MDPs to which we will add constraints fall into two broad categories, which we denote generic and baseline. The car MDP described the beginning of Section 3.2 is an example of generic nominal model: its state and action spaces are broad enough to encompass wide range of car models, and we can use this nominal model infer constraints that specialize the MDP to specific car and task. For generic nominal MDP, the reward function may also come from generic, simplified task, such as "minimize time to the goal" or "minimize energy usage."
A baseline nominal MDP is a snapshot of system from point in time where it was well characterized. With a baseline nominal MDP, the constraints that we infer will represent changes to the system with respect to this baseline. In this case, the nominal reward function can be learned using existing IRL techniques with demonstrated behavior from the baseline model. We take this approach in our human obstacle avoidance example in Section 4.2 empty space to learn a nominal reward, then we can detect the presence a new obstacle in the space from subsequent demonstrations.
demonstrations of humans walking through the
4

---

Published as a conference paper at ICLR 2020
3.3 DEMONSTRATION LIKELIHOOD MAXIMIZATION
Our goal is to find the constraints C* which are most likely to have been added to a nominal MDP M. given set of demonstrations D from an agent navigating the constrained MDP. Let us define PM to denote probabilities given that we are considering MDP M. Our problem then becomes to select the constraints that maximize (M D). If we assume uniform prior over possible constraints, then we know from Bayes' Rule that PM(C D) & (M | C). Therefore, in order to find the constraints that maximize PM D), we can solve the equivalent problem of finding which constraints maximize the likelihood of the given demonstrations. In this section, we present our approach to solving maximum likelihood constraint inference via solving demonstration likelihood maximization.
Under the maximum entropy model presented by Ziebart et al. (2008), the probability of a cer- tain finite-length trajectory & being executed by an agent traversing a deterministic MDP M is exponentially proportional to the reward earned by that trajectory.
1 @ (E). 1/2 Z
(M(
(3)
where 7 is the partition function, (g) indicates if the trajectory is feasible for this MDP, and E [0, oo) is a parameter describing how closely an agent adheres to the task of optimizing the reward function (as 3 + the agent becomes perfect optimizer, and as 3 + O, the agent's 's actions become perfectly random). In the sequel, we assume that a given reward function will appropriately capture the role of B, so we omit 3 from notation without loss of generality.
In the case of finite horizon planning, the partition function will be the sum of the exponentially weighted rewards for all feasible trajectories MDP M of length greater than the planning horizon. We denote this set trajectories by EM- Because adding constraints C modifies the set of feasible trajectories, we express this dependence as
Z(C)=
1 M (E).
(4)
ξEEM
Assuming independence among demonstrated trajectories, the probability of observing a set D of N demonstrations is given by the product
Pmo (D)
II EED
R(E) M (E).
(5)
Z(C) N
Our goal is to maximize the demonstration probability given by (5). Because we take the reward function and demonstrations given, our only available decision variable in this maximization is the constraint set C which alters the indicator 1 M and partition function Z(C).
C arg max PM (D),
CEC
where C C 25x is the hypothesis space of possible constraints.
From the form of (5). clear that solve 66). we must choose a constraint set that does not invalidate any demonstrated trajectory while simultaneously minimizing the value of Z(C). Consider the set of trajectories that would be made infeasible by augmenting the MDP with constraint C,
which we denote as E MM = [g E 1M (E) 0}. The value of Z(C) is minimized when we maximize the sum of exponentiated rewards of these infeasible trajectories. Considering the form of the trajectory probability given by (3). we can see that this sum is proportional to the total probability of observing a trajectory from E Emyc on the original MDP M
Σ R(E) O E PM(E) = PMM(E
(7)
EEEMO
SEE
This insight leads us to the final form of the optimization
C
arg max PM((Mco)
Cec
(8)
s.l. D DNE DE =
5

---

Published as a conference paper at ICLR 2020
1
1
1
0
1
1
1
1
11
1
1
1
1
111
1
1
1
1
11
1
1
1-1
1-1
1
1
1
1-1
H1
1
1
1-1
1
-1
1
11
1
1
-1
-1
-1
-1
1-1
-1
-1
-1
(a) Nominal MDP
(b) Add constraint C1
(c) Add constraint C2
Figure 2: Selecting constraints to maximize demonstration likelihood. Trajectories that are likely to be observed given MDP shown dashed, angular arrows, and provided demonstration is shown as solid, curved arrow. Adding C₁ in (b) does little to align the expected trajectories with the demonstration. On the other hand, adding C2 (c) the original expected trajectories infeasible and causes the new expected trajectories to agree with the demonstration, greatly increasing the likelihood of the demonstration on this constrained MDP.
In order to solve X we must reason about the probability distribution of trajectories on the original MDP M, then find the constraint C' such that Eyc contains the most probability mass while containing any demonstrated trajectories. Figure 2 provides a graphical representation reasoning. We highlight here that the fact that the chosen C' must not conflict with any demonstration an important condition: all provided demonstrations must perfectly respect constraint in order for to be learned, otherwise a less restrictive set of constraints may be learned instead. Future work will look to relax this requirement in order to learn about constraints that are generally respected by of demonstrations, without needing to first isolate just the successful, constraint-respeccing demonstrations.
this
While equation g is derived for deterministic MDPs, if we assume, proposed by Ziebart et al. (2008). that for a given stochastic MDP, the stochastic outcomes have little effect on an agent's behavior and the partition function, then the solution to (8) will also approximate the optimal constraint selection for that MDP. However, in order to fully address the stochastic case, we would need reformulate our approach based on maximum causal entropy (Ziebart 2010). We save this extension for future work.
3.3. CONSTRAINT HYPOTHESIS SPACE
In order for the solutions hypothesis space C. For instance, if we let C 2SxA then the optimal solution will always be to choose the most restrictive C constrain all state-action pairs not observed in the demonstration set.
be meaningful, we must be careful with our choice of the constraint
One approach to avoid this trivial solution is to use domain knowledge of the modeled system to restrict C to a library of plausible common constraints. McPherson library by using reachability theory to calculate a family of likely unsale sets.
(2018) construct such
We could also potentially address this problem by adding regularization term to the optimization that would penalize constraints based on some notion of "size," which would encourage the selection of "smaller" constraints. The size of constraint set could be defined by the number of minimal constraints that it contains. These minimal constraint sets constrain single state, action, or feature, and were introduced in Section 3.22 as Cs Ca.: and Co. respectively. While penalizing this definition of size would effectively discourage overfitting, considering every possible combination of minimal constraints causes the hypothesis space to grow exponentially in the number of states, actions, and features of the MDP, which may make directly solving this formulation intractable.
Another approach, which avoids this combinatorial explosion, to use the minimal constraint sets themselves, not their combinations, our hypothesis space, and to select from among these constraints in an iterative, greedy manner. By iteratively selecting individual minimal constraint sets, and choosing proper stopping condition, it possible gradually grow the full estimated constraint

---

Published as a conference paper at ICLR 2020
set and avoid overfitting the demonstrations, It is this method that we will utilize in this work. Section 3.4 details our approach for selecting the most likely minimal constraint, and Section 3.5 details our approach for iteratively growing the estimated constraint set.
3.4 PROBABILITY MASS FOR MINIMAL CONSTRAINTS
As detailed in Section 3.3 the most likely constraint set the whose eliminated trajectories Eye have the highest probability of being demon- strated on the original, unconstrained MDP. Therefore, to find the most likely of the minimal constraints, we must find the expected proportion of trajectories which will contain any state or action, or accrue any feature. By using augmented indicator fea- ture map from (T) we can reduce this problem only examine feature cruals. Ziebart et al.] (2008) present their forward-backward algorithm for calculating expected feature counts for agent following policy in the maximum entropy setting. This al- gorithm nearly suffices for our pur- poses, but it computes the expectation of the total number of times feature will be accrued (i.e. how often will this feature be observed per trajec- tory), rather than the expectation of the number of trajectories that will ac- crue that feature any point in time. To address this problem, we present a modified form of the "forward" pass as Algorithm[] [ Our algorithm tracks state visitations well feature ac- cruals at each state, which allows us to produce the same maximum en- tropy distribution over trajectories as Ziebart et al. (2008) while not counting additional accruals for trajectories that have already accrued a feature.
Algorithm 1 Feature Accrual History Calculation
Input: an MDP M. policy π ( t), a time horizon T
Output: expected feature accrual history $1,21
1 Initialize state visitation and feature accrual history */
1: for 8 E S do
2:
D2.0 + Do(s)
3:
$s.0 f- 0ngX1
4: end for
1 Track feature accruals over the time horizon #/
5: for / E [0, T- do
6:
for E S do
7:
for E A, do
1* New feature accruals */
9:
1 "0" denotes element- wise multiplication */
10:
A Ad + (s, a) @ DD, 2..1
11:
end for end for
12:
13:
for s' E S do
14:
Dost1 + Σ E t) aESaEA
15:
2 f
Σ Σ
T A s x () ((('(ss.))
AESaEA,
16:
end for
17: Pe+1 +
Σ 22.111
SES
18: end for
19: Return $ (1.7]
The input of Algorithm inccuues the MDP itself, time horizon, and time-varying policy. This policy should capture the expected behavior of the demonstrator on the nominal MDP M, and can be computed via the "backward" part of the algorithm from Ziebart al. (2008). The output of Algorithm
T 11. an n4 T array such that the /-th column o, is vector whose i-th entry is the expected proportion of trajectories to have accrued the i-th feature by time I. In particular, the
i-th element of क. is equal to PM which allows us to now directly select the most likely constraint according to
3.5 MAXIMUM-COVERRAE BASED ITERATIVE CONSTRAINT INFERENCE
When using minimal constraint sets as the constraint hypothesis space, it is possible that the most likely constraint still does not provide a satisfactory explanation for the demonstrated behavior. In this case, it can be beneficial to combine minimal constraints. If the task of solving (8) is framed as finding the combination of constraint sets that "covers" the most probability mass, then the problem becomes a direct analog for the classic maximum coverage problem. While this problem is known to be NP-hard there exist a simple greedy algorithm with known suboptimality bounds (Hochbaum & Pathria 1998).
7

---

Published as a conference paper at ICLR 2020
We present Algorithm 2 as approach for adapting this greedy heuristic solve the problem of constraint inference. At each iteration, we grow our estimated constraint set by augmenting it with the constraint set in our hypothesis space that covers the most currently uncovered probability mass. By analogy to the maximum coverage problem (Hochbaum & Pathria 1998), we derive the following bound on the suboptimality of our approach.
Theorem 1. Let Cn. be the set of all constraints Cn, such that Cn. = Ur= C. foror Cs E C, and let Cr be the solution to (8) using Cn2 as the constraint hypothesis space. follows, then, that at the end of every iteration of Algorithm
i (11((1))) P(()) P(EMer)-
P P( -
MO.
This bound directly analogous to the suboptimality bound for the greedy solution to the maximum cov- erage problem proven by Hochbaum Pathria 1998). For space, the proof included in the appendix.
Algorithm 2 Greedy Iterative Constraint Inference
Input: MDP M, constraint hypothesis space C. empirical probability distribution Pp. threshold d
Output: estimated constraint set C*
1: Ō Ō++@ -
Rather than selecting the number of constraints ne to be used ahead of time, we check a stopping condition to decide if we should continue to add constraints. Because we are attempt-
2: for i E [1, [C]] do
C t solution to 8 using M C, and D
3:
4:
ADa. = DxI, (Po PMe.) DKr. (Po (emerr)
5:
if Ana. < dna then
6:
break
ing to maximize PM (D), it might seem natural to use a probability- based stopping condition. However,
7: end if
C + --CU
9: end for
choosing a stopping criterion based on probability is problematic because the probability of observing set of demonstrations is dependent on the number of demonstrations in the set, even if each individual demonstration has the same probability. We instead base our stopping condition not on the number of demonstrations. The quantity DKL (Pp (I provides a measure of how well the distribution over trajectories induced by inferred constraints, PMê-, agrees with the empirical probability distribution over trajectories observed in the demonstrations, Pp. Using this KL divergence in the stopping condition actually preserves a link the probability of the demonstration set, since the KI. divergence will decrease monotonically as PM (D) increases. The threshold parameter Dia is chosen avoid overfitting to the demonstrations, combating the tendency to select additional constraints that may only marginally better align our predictions with the demonstrations.
10: Return C*
KL divergence, which depends only on the distribution of trajectories in D and
4 EXAMPLES
4.1 SYNTHETIC GRID WORLD
We consider the grid world MDP presented in Figure 3 The environment consists of a 9-by-9 grid of states, and the actions are to move up, down, left, right, or diagonally by one cell. The objective is to move from the starting state in the bottom-left corner (so) to the goal state in the bottom-right corner (sc). Every -tate-action pair produces distance feature, and the MDP reward is negative distance, which encourages short trajectories. There are additionally two more features, denoted green and blue, which are produced by taking actions from certain states, as shown in Figure 3
The true MDP, from which agents generate trajectories, is shown in Figure 3a including its constraints. The nominal, more generic MDP shown in Figure 3b is what we take as M for applying the iterative maximum likelihood constraint inference in Algorithm 2 with feature accruals estimated using Algorithm [ While Figures 3c through 3e show the iteratively estimated constraints, which align with the true constraints, it is interesting to note that not all constraints present in the true MDP are identified. For instance, it is unlikely that an agent would ever select the up-left diagonal action, that the fact that demonstrated trajectories did not contain that action is unsurprising and does not make that action an estimated constraint.
8

---

Published as a conference paper at ICLR 2020
States
Accions
Scates
Actiosa
Accnal Prepardn
2
Ziee
(a) True MDP
(b) Nominal MDP
Meataces
(c) C², =1
(d) C* 2
(e) O
Figure 3: Algorithm performance on a synthetic grid world MDP. Each subfigure represents the MDP by showing (clockwise from left) its states, actions, and features. Each element shaded according the proportion of trajectories that are expected to accrue the respective augmented feature, computed via Algorithm [ Constraints marked with red "X," and bright bounding boxes mark the green and blue ceature-produciig states. The result here are shown for a set of 100 demonstrations sampled
according the expectation for the True MDP (a) We begin with the nominal MDP shown in (b) and produce (c) (d) and [c) by applying Algorithm
Note that (c)
(d) and
show the selections
of feature, action, and state constraints, respectively.
Figure 4 shows how the performance of approach varies based on the number of available demonstrations and the selection for the threshold d The false positive rate shown in Figure 4alis the proportion selected con- straints which not constraints of the true system. We can observe two trends in this data that we would expect. First, lower values of dx. lead greater false positive rates since they allow Algorithm 2 to continue iterating and accept constraints that do less to align expectations and demonstrations. Second, having more demonstrations available provides more information and reduces the rate of false positives. Further, Figure 4b shows that more demonstrations also allows the behavior predicted by con- straints to better align with the observations. It is interest- ing to note, however, that with fewer than demonstra- tions and very low d divergence, but at the cost of high false positive rate. This phenomenon highlights the role of selecting d'k.. to avoid over-fitting. The threshold d'p = 0.1 achieves a good balance of producing few false positives with sufficient examples while also producing lower KL divergences, and we used this threshold produce the results in Figures and 5
00
Base
Fale
03
(a) False positives
may produce very low KL
Fial
Nurther Descrneratioe
(b) DEL.
4.2 HUMAN OBSTACLE AVOIDANCE
Figure 4: Algorithm performance on the synthetic grid world. Each data point represents the mean result of 10 indepen- dent trajectory draws, and the margins show +1 standard error.
In our second example, we analyze trajectories from hu- mans they navigate around obstacle the floor. We map these continuous trajectories into trajectories through grid world where each cell represents lft-by-1ft area

---

Published as a conference paper at ICLR 2020
on the ground. The human agents are attempting to reach fixed goal state (sc) from given initial state (su). as shown in Figure We performed MaxEnt IRL. on human demonstrations of the task without the obstacle obtain the nominal distance- iissance-based reward function. We restrict ourselves to estimating only state constraints, as we do not supply our algorithm with knowledge of any additional features in the environment and assume that the humans' motion is unrestrained.
Demonstrations were collected from 16 volunteers, and the results of performing constraint inference are shown in Figure 5 Our method is able to successfully predict the existence of a central obstacle. While we do not estimate every constrained state, the constraints that we do estimate make all of the obstacle states unlikely to be visited. In order to identify those states as additional constraints, we would have to decrease our De, threshold, which could also lead to more spurious constraint selections, such as the three shown in Figure 5
Obstacle Avoidance Trajectories
X
XXXX
5 CONCLUSION AND FUTURE WORK
X
We have presented our novel technique for learning con- straints from demonstrations. We improve upon previous work in constraint-leariin framework for identifying the most likely constraint(s), and we do so in a way that explicitly makes state, action, and feature constraints all directly comparable to one another. We believe that the numerical results presented in Section 4 are promising and highlight the usefulness our approach.
IRI. by providing a principled
Figure 5: Human trajectories overlaid grid world MDP. The shaded re- gion represents an obstacle in the hu- man's 's environment, and the red "X"s represent learned constraints.
Despite its benefits, the formulation is based on (3) which only exactly holds for deterministic MDPs. As mentioned in Section 3.3] plan to investigate the use of a maximum causal entropy approach address this issue and fully handle stochastic MDPs. Additionally, the methods presented here require all demonstrations to contain no violations of the constraints we will estimate. We believe that softening this requirement, which would allow reasoning about the likelihood of constraints that are occasionally violated in the demonstration set, may be beneficial in cases where trajectory data is collected without explicit labels of success or failure. Finally, the structure of Algorithm | which tracks the expected features accruals of trajectories over time, suggests that we may be able to reason about mar-Markovinn constraints by using this historical information to our advantage.
drawback of approach is that
Overall, we believe that our formulation of maximum likelihood constraint inference for IRL shows promising results and presents attractive avenues for further investigation.
ACKNOWLEDGMENTS
This work is supported by the National Science Foundation through grant CNS-1545126 (VeHICaL). We would like to thank Jaime Fisac, Andrea Bajesy, Sylvia Herbert, and David Fridovich-Ked for their insightful discussions and for sharing their collected data.
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 1. ACM, 2004.
Pieter Abbeel, Adam Coates, Morgan Quigley, and Andrew Ng. An application of reinforcement learning aerobatic helicopter flight. In Advances in neural information processing systems, pp. 1-8, 2007.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in Al safety. arXiv, arXiv: (oXx::6066.000 606.06565, 2016.
10

---

Published as a conference paper at ICLR 2020
Glen Chou, Dmitry Berenson, and Necmiye Ozay. Learning constraints from demonstrations. In Springer Proceedings Advanced Robotics (SPAR) series on 2018 Workshop on the Algorithmic Foundations of Robotics (WAFR 2018). Springer, 2018.
Mark K Ho, Michael L Littman, Fiery Cushman, and Joseph L Austerweil. Teaching with rewards and punishments: Reinforcement or communication? In CogSci, 2015.
Dorit S Hochbaum and Anu Pathria. Analysis the greedy approach in problems of maximum k-coverage. Naval Research Logistics (NRL), 45(6))615--6)) 627, 1998.
Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
Rudolf Emil Kalman. When is a linear control system optimal? Journal of Basic Engineering, 86(1): 51 51-60, 1964.
David L. McPherson, Dexter R.R. Scobee, Joseph Menke, Allen Yang, and S. Shankar Sastry. Modeling supervisor safe sets for improving collaboration in human-robot teams. In 2018 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 861-86$ Oct 2018. doi:
Andrew Y Ng and Stuart J Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, pp. 663-670. Morgan Kaufmann Publishers Inc., 2000.
Michael Pardowitz, Raoul Zollner, and Rüdiger Dillmann. Learning sequential constraints of tasks from user demonstrations. In 5th IEEE-RAS International Conference on Humanoid Robots, 2005., pp. 424-429. IEEE, 2005.
Claudia Pérez-D' Arpino and Julie A Shah. C-learn: Learning geometric constraints from demonstra- tions for multi-step manipulation in shared autonomy. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 4058-4065, IEEE, 2017.
Nathan D Ratliff, Andrew Bagnell, and Martin A Zinkevich. Maximum margin planning. In Proceedings of the 23rd international conference on Machine learning, pp. 729-736. ACM, 2006.
Dexter R R Scobee, Vicenc Rubies Royo, Claire Tomlin, and S Shankar Sastry. Haptic assistance via inverse reinforcement learning. In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 1510-1517. IEEE, 2018.
Guru Subramani, Michael Zinn, and Michael Gleicher. Inferring geometric constraints in human demonstrations. In Conference on Robot Learning, pp. 223-236 2018.
Dizan Vasquez, Billy Okal, and Kai O Arras. Inverse reinforcement learning algorithms and features for robot navigation in crowds: an experimental comparison. In 2014 IEEE/RSI International Conference on Intelligent Robots and Systems, pp. 1341-1346. IEFE, 2014.
Marcell Vazquez-Chanlatte Chanlatte, Susmit Jha, Ashish Tiwari, Mark K Ho, and Sanjit Seshia. Learning task specifications from demonstrations. Advances Neural Information Processing Systems, pp- 5367-5377, 2018.
Brian D Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. PhD thesis, Carnegie Mellon University, 2010.
Brian Ziebart, Andrew Maas, Andrew Bagnell, and Anind Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp.
Chicago, IL, USA, 2008
11

---

Published as a conference paper at ICLR 2020
APPENDIX
A ADDING CONSTRAINTS TO STOCHASTIC MDPs
For MDPs with stochastic transitions, the semantics of empty state are less obvious and could lend themselves to multiple interpretations depending on the nature of the system being modeled. In our context, we use constraints describe how observed behaviors from demonstrations differ from pos- sible behaviors allowed by the nominal MDP structure. We therefore assume that any demonstrations provided are, by the fact that they were selected to provided, consistent with the system's 's con- straints, including avoiding empty states. This assumption implies that any stochastic state transitions that would have led to an empty state will not be observed in trajectories from the demonstration set. The omission of these transitions means that, for a given (s, a), if P. (Seppy)) p, then proportion p of these (s, a) pairs which occur as an agent navigates the environment will be excluded from demonstrations. Therefore, updated transition probabilities which eliminate the probability mass of transitioning to empty states, an event which will never be observed demonstration, Such modified probabilities be given as
modify the MDP to reason about demonstrated behavior, need
[10 P-PasaaSSane
if E Sempay otherwise
P..
Ps.a(s))
(9)
We must also capture the change to observed state-action pair frequencies by understanding that any observed policy TC will be related to agent's actual policy иT according
T(als, ()(1 Pa.a (Sempty))
n (als,
(10)
Σ (a'ls, (1 Ps (Sempt))
a'EAÇ
is important to note that the modifications presented in (9) and (10) for non-deterministic MDPs are not meant to directly reflect the reality of the underlying system (we wouldn't expect the actual transition dynamics to change, for instance), but to reflect the apparent behavior that we would expect to observe in the subset of trajectories that would be selected as demonstrations. We further note that applying these modifications deterministic MDPs will result in the same expected behavior as augmenting the constraint set with Cempty.
B PROOF FOR THEOREM
Theorem 1. Let Co, be the set of all constraints Cn, such that C,,, - for Cs E C, and let C be the solution to (8) using Cn, as the constraint hypothesis space. follows, then, that at the end of every iteration of Algorithm
P (EMMe)) ( (1-))==211)))))))))1)1(11(11)( P ((E
Proof. The problem of finding C is analogous to solving the maximum coverage problem, dis- cussed by Hychbaum & Pathria (1998), where the set of elements to be covered is the set of trajectories {E C E Mr and D EE Me = 0} and the weight of each element & is (M)(E). Because 2 constructs Ō+ iteratively by taking the union of the previous value of Ō and the set C. E which solves (8), the value of Ō* at the end of the i-th iteration analogous to the greedy solution of the maximum coverage problem with ne = v. Therefore, we can directly apply the suboptimality bound for the greedy solution proven in Hochbaum & Pathria (1998) to arrive at our given bound on climinated probability mass.
Algorithm
12


Reviews:
[{"review_id": "rJlIrVJk9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers learning of constraints in MDPs in an IRL setting with the goal of maximizing the likelihood of demonstrations (in the constrained MDP). The constraints come in the form of avoiding certain states, actions or features. The authors propose an algorithm for learning the constraints and evaluate their approach in synthetic and a real-world experiment.\n\nThe paper is mainly well written and considers a relevant problem. However, at a conceptual level, I am missing a more precise problem formulation and an extended discussion of the possible failure cases/limitations of the approach. Also more real-world experiments would be welcome (the presented real-world problem is relatively easy, expert's trajectories are likely to agree and not make any \"mistakes\").\n\nIn more detail, I think the paper can be improved by spelling out the problem formulation more precisely (Section 3.1) -- in its current form I think it somewhat fuzzy. What I mean by this is that there is a nominal MDP (which is at least as \"large\" as the true MDP) and there are demonstrations. The goal seems to be to identify constraints from the demonstrations and the MDP such that if added to the nominal MDP it \"shrinks\" to the true MDP. Is that the actual underlying problem? (I understand the current formulation in the terms of the likelihood of demonstrations). In this formulation, I think the definition of over-fitting arises naturally. In the current formulation though, over-fitting seems somewhat disconnected in the sense that even if we identify a solution which generalizes perfectly to new demonstrations, we would talk about over-fitting (the optimization of the likelihood maximization is not the problem that we actually want to solve). \n\nFurther, as also mentioned by the authors, there is a problem with sub-optimal demonstrators. Considering the example of cars given in the paper, there might be drivers that do violate speed limits. In that case, the proposed approach will fail to identify some constraints. On the other hand, if all \"optimal\" demonstrations are to go fast, the approach would constraint the possibility to drive slowly. All this is fine, but I think it warrants a broader discussion which is not deferred to the Conclusion/Future Work section as it is very crucial regarding the applicability of the proposed approach.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkxBRWYCFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission considers estimating the constraints on the state, action and feature in the provided demonstrations, instead of learning rewards. The authors use the likelihood as MaxEnt IRL methods to evaluate the \"correctness\" of the constraints, and find the most likely constraints given the demonstrations. While the problem is challenging (NP-hard), suboptimality of the proposed algorithm is analyzed. Experiments are provided to demonstrate the performance of the proposed method. \n\nThe problem considered is interesting, and the authors provide a straightforward but empirically effective method. However, the motivation is a little unclear to me. Specifically, what will be the practical cases, where the learning the constraints is important and necessary? Can authors further motivate this topic by providing more real-world applications? \n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1xYIh4v9r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper aims to address a new method for inverse reinforcement learning based on maximum likelihood constrained inference. In general, I find the problem very interesting and the motivation of the work is quite reasonable. However, I have two major comments:\n\n(i) Does the constraint semantically similar to the domain of the MDP? In my intuition, one can create a convex hull over the state and action representations to  actually estimate the constraint.\n\n(ii) Suppose, the reward function is unknown, how your method will fare in this case?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HkeSgyVa9r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, a novel inverse constraint learning method is proposed, where the goal is to find out the constraints over state-action pairs for given demonstration and MDP **including a reward function** (so different from inverse cost learning). The novelty of this work comes from introducing maximum entropy inverse reinforcement learning (MaxEntIRL) framework to previous works [1, 2], and this work mainly focused on the tabular setting. The objective of this work is to solve the optimization in (8), which tries to find out the constraint that maximizes the probability of trajectories that cannot be generated if that constraint is applied. (Such an objective minimizes the normalization constant in (5) and results in maximization of the demonstration likelihood under the constraint.) To solve this objective, the proposed algorithm first computes the feature occupancy (Algorithm 1), and then use those feature occupancy with greedy iterative constraint inference (Algorithm 2 that motivated by maximum coverage problem) to get constraints. Two experiments in the GridWorld show that the proposed method effectively works. \n\nI think this work is quite fundamental, impactful to be accepted at the conference and is possibly extended to practical scenarios (like explainable and safety RL and imitation learning) in the future. One thing I’d like to point out is to enhance the readability by reordering contents and adding some additional explanations to clarify their arguments. There are a few comments and questions I have:\n\n- The exact definition and usage of nominal environments and rewards are still unclear to me. For example in Figure 2 (b), how did you define and get nominal MDP?\n- Since Figure 1 is related to the second experiment, I recommend moving it to the experiment section. \n- At 3.2.1., “Because constraints are sets of state-action pairs, imposing a\nconstraint within an MDP means restricting the set of actions that can be taken from certain states.” -> Need clarification\n- At 3.2.1., “For MDPs with deterministic transitions, it is clear that any agent respecting these constraints will not visit an empty state.” -> Why?\n- At (8), $\\{\\}$ to $\\emptyset$\n- In Figure 3, I was a bit confused about the relationship between the threshold and the false positive rate at first glance. What I understood is that a small threshold leads to lots of iteration for constraint selection, which increases the false positive. I want authors to add some comments on that.\n\n Reference\n[1] Chou, Bereson, Ozay, “Learning Constraints from Demonstrations,” arXiv 2019\n[2] Chou, Bereson, Ozay, “Learning Parametric Constraints in High Dimensions from Demonstrations,” CoRL 2019", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel way of doing IRL based on learning constraints. The topic of IRL is an important one in RL and the approach introduced is interesting and forms a fundamental contribution that could lead to relevant follow-up work.
Decision: Accept (Spotlight)