Inverse Constrained Reinforcement Learning
Authors: Shehryar Malik
Usman Anwar
Alireza Aghasi
Ali Ahmed
Venue:  (2021)
Keywords: Constrained reinforcement learning
constrain inference
safe reinforcement learning

Abstract: Standard reinforcement learning (RL) algorithms train agents to maximize given reward functions. However, many real-world applications of RL require agents to also satisfy certain constraints which may, for example, be motivated by safety concerns. Constrained RL algorithms approach this problem by training agents to maximize given reward functions while respecting \textit{explicitly} defined constraints. However, in many cases, manually designing accurate constraints is a challenging task. In this work, given a reward function and a set of demonstrations from an expert that maximizes this reward function while respecting \textit{unknown} constraints, we propose a framework to learn the most likely constraints that the expert respects. We then train agents to maximize the given reward function subject to the learned constraints. Previous works in this regard have either mainly been restricted to tabular settings or specific types of constraints or assume knowledge of transition dynamics of the environment. In contrast, we empirically show that our framework is able to learn arbitrary \textit{Markovian} constraints in high-dimensions in a model-free setting.

Under review as a conference paper at ICLR 2021
INVERSE CONSTRAINED REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Standard reinforcement learning (RL) algorithms train agents to maximize given reward functions. However, many real-world applications of RL require agents also satisfy certain constraints which may. for example, be motivated by safety concerns. Constrained RL algorithms approach this problem by training agents to maximize given reward functions while respecting explicitly defined constraints. However, in many cases, manually designing accurate constraints is challenging task. In this work, given a reward function and a set demonstrations from an expert that maximizes this reward function while respecting unknown constraints, we propose a framework to learn the most likely constraints that the expert re- spects. We then train agents to maximize the given reward function subject the learned constraints. Previous works in this regard have either mainly been re- stricted tabular settings specific types of constraints or assume knowledge of transition dynamics of the environment. In contrast, we empirically show that our framework is able learn arbitrary Markovian constraints in high-dimensions a model-fin setting.
INTRODUCTION
Reward functions critical component in reinforcement learning settings. As such, it is impor- tant that reward functions are designed accurately and are well-aligned with the intentions of the human designer. This is known as agent (or value) alignment (see, e.g., Leike al. (2018) 2017); Amodei et al. (2016)). Misspecified rewards can lead to unwanted and unsafe situations (see, e.g, Amodei & Clark] (2016)) However, designing accurate reward functions remains challenging task. Human designers, for example, tend to prefer simple reward functions that agree well with their intuition and are easily interpretable. For example, human designer might choose a reward function that encourages an RI. agent driving car to minimize its traveling time to a certain desti- nation. Clearly, such reward function makes sense in the case of a human driver since inter-human communication is contextualized within a framework of unwritten and unspoken constraints, often colloquially termed as 'common-sens" traveling time, they will be careful not to break traffic rules, take actions that endanger passersby, and so on. However, we cannot assume such behaviors from RL agents since they are are not imbued with common-sense constraints.
That is, while a human driver will try to minimize their
Constrained reinforcement learning provides a natural framework for maximizing a reward function subject some constraints (we refer the reader to Ray et al. (2019) for a brief overview of the field), However, in many cases, these constraints are hard to specify explicitly in the form of mathematical functions. One way address this issue is automatically extract constraints by observing the behavior of constraint-bbiding agent. Consider, for example, the cartoon in Figure I Agents start at the bottom-left corner and are rewarded according how quickly they reach the goal at the bottom-right corner. However, what this reward scheme misses out is that in the real world the lower bridge occupied by a lion which attacks any agents attempting to pass through Therefore, agents that naively trained to maximize the reward function will end up performing poorly in the real world. If, on the other hand, the agent had observed that the expert (in Figure ((a)) actually performed suboptimally with respect to the stipulated reward scheme by taking longer route to the goal, could have concluded that (for some unknown reason) the lower bridge must be avoided and consequently would have not been eaten by the lion!
Scobee & Sastry (2020) formalizes this intuition by casting the problem of recovering constraints in the maximum entropy framework for inverse RL (IRI.) (Ziebart al. 2008) and proposes a greedy

---

Under review as a conference paper at ICLR 2021
(a) Expert policy.
(b) Nominal policy.
(c) Recovered policy.
Figure 1: The TwoBridges environment. (a) The expert avoids the lion and takes the upper bridge.
bb) Since the nominal policy is simply trained to get to the goal as quickly as possible, it instead takes the lower bridge. [c) Our method, on the other hand, is able to learn that the lower bridge
should be avoided, and consequently our policy takes the upper bridge.
algorithm to infer the smallest number of constraints that best explain the expert behavior. However, Scobee & Sastry (2020) has two major limitations: it assumes (1) tabular (discrete) settings, and (2) the environment's 's transition dynamics. In this work, we aim to address both these issues by learning constraint function instead through sample-based approximation of the objective function of |Scobee & Sastry) Consequently, our approach is model-free, admits continuous states and actions and can learn arbitrary Markovian constraints" Further, we empirically show that scales well to high-dimension
Typical inverse RL methods only make use of expert demonstrations and do not assume any knowl- edge about the reward function at all. However, most reward functions can be expressed in the form "do this task while not doing these other things" where other things are generally constraints that designer wants to impose on an RI agent. The main task ("do this"" is often quite easy to encode in the form of a simple nominal reward function. this work, we focus on learning the constraint part ("do not do that") from provided expert demonstrations and using it in conjunction with the nominal reward function to train RL agents. In this perspective, our work can be seen as a principled way to inculcate prior knowledge about the agent's 's task in IRL. This is key advantage over other IRL methods which also often end up making assumptions about the agent's task in the form of regularizers such in Finn al. 2016).
The main contributions of our work are as follows:
We formulate the problem of inferring constraints from a set of expert demonstrations as learning problem which allows it be used in continuous settings. To the best of our knowledge, this is the first work in this regard.
We eliminate the need assume, as Scobee Sastry do, the environment's transition dynamics.
We demonstrate the ability of our method to train constraint-abiding agents in high- dimensions and show that it can also used prevent reward hacking.
PRELIMINARIES
2.1 UNCONSTRAINED RL
A finite-horizon Markov Decision Process (MDP) M is a tuple (,,,,,,,, where S E RISI is set of states, A R is set of actions, S x A S + [0. 1] is the transition probability function (where p(s'ls, a) denotes the probability of transitioning to state s' from state by taking
Markovian constraints are of the form c(7) II c(st, ax) i.c. constraint function is independent of the past states and actions in the trajectory.
2

---

Under review as a conference paper at ICLR 2021
action a), S x A + R is the reward function, y is the discount factor and T is the time- a1...... ST, ar) denotes a sequence of states-action pairs such that horizon. A trajectory T T= {s1,
8141 ~ p(-|s,, az). A policy : S + P(A) is map from states to probability distributions over actions, with T (aas)) denoting the probability of taking action a in state 8. We will sometimes abuse notation to write It (ss,a) a) mean the joint probability of visiting state and taking action a under the policy π and similarly (T) to mean the probability of the trajectory under the policy π.
Define r(r) = EE= 7"1(s1, az) to be the total discounted reward of a trajectory. Forward RI. algorithms try to find an optimal policy x² that maximizes the expected total discounted reward (TT) Ey [r() On the other hand, given set of trajectories sampled from the optimal (also referred to as expert) policy x², inverse RL (IRL) algorithms aim to recover the reward function r, which can then be used to learn the optimal policy x' via some forward RL algorithm.
2.2 CONSTRAINED RL
While normal (unconstrained) RL tries to find a policy that maximizes J (T), constrained RL instead focuses finding a policy that maximizes (TT) while respecting explicitly-define constraints. A popular framework in this regard is the one presented in Altman (1999) which introduces the notion of a constrained MDP (CMDP). A CMDP M is simple MDP augmented with cost function e: S A + I and budget a 2 0. Define c(T) Σ Y cs( ae) be the total discounted cost of the trajectory 7 and (Tr) = [e(r)(((T) to be the expected total discounted cost. The forward constrained RI. problem to find the policy π. that maximizes J(T)) subject to (x) S a.
In this work, given set D of trajectories sampled from n.. the corresponding unconstrained MDP M (i.e., MMS without the cost function c) and a budget a, we are interested in recovering a cost func- tion which when augmented with M has optimal policy that generates the same set of trajectories as in D. We call this as the inverse constrained reinforcement learning (ICRL) problem.
If the budget is strictly greater than 0, then the cost function defines soft constraints over all possible state-action pairs. In other words, a policy is allowed to visit states and take actions that have non-zero costs as long the expected total discounted cost remains less than a. If, however, a is 0 then the cost function translates into hard constraints over all state-action pairs that have a non-zero cost associated with them. A policy work, we restrict ourselves to this hard constraint setting. Note that this is not particularly restrictive since, for example, safety constraints are often hard constraints as well are constraints imposed by physical laws.
thus never visit these state-action pairs. In this
Since we restrict ourselves to hard constraints, we can rewrite the constrained RL problems as fol- lows: define C = {(ss, [[(ss a) # 0} to be the constraint set induced by c. The forward con- straint RL problem is to find the optimal constrained policy à that maximizes J(T) subject to (s, a) = 0 V(s, a) E C. The inverse constrained RL problem is to recover the constraint set C from trajectories sampled from हe.
Finally, we will refer to our unconstrained MDP as the nominal MDP hereinafter. The nominal MDP represents the nominal environment (simulator) in which we train our agent.
FORMULATION
3.1 MAXIMUM LIKELIHOOD CONSTRAINT INFERENCE
We take Scobee & Sastry as our starting point. Suppose that we have a set of trajectories D 1, sampled from an expert गह navigating in a constrained MDP M where C* denotes the (true) constraint set. Furthermore, we are also given the corresponding nominal MDP M Our goal is to recover a constraint set which when augmented with M results in CMDP that has an optimal policy that respects the same set of constraints as हेे does. Scobee & Sastry pose this as a maximum likelihood problem. That is, if we let PM denote probabilities given that we are considering MDP M and assume uniform prior on all constraint sets, then we can choose C* according to
C* +- arg max (DIC).
(1)
2 AAaillability of transition dynamics model of nominal MDP is not necessary.
3

---

Under review as a conference paper at ICLR 2021
Under the maximum entropy (MaxEnt) model presented in Ziebart et al. (2008), the probability of trajectory under deterministic MDP M can be modelled as
(M exp(3r(T)) (M). ZM
where ZM exp(3r(7)) (001MM(rr (r)dr is the partition function, / E [0, x:) is a parameter describing how close the agent to the optimal distribution (as -> oo the agent becomes a perfect optimizer and 3 + the agent simply takes random actions) and is an indicator function that is 1 for trajectories feasible under the MDP M and 0 otherwise.
Assume that all trajectories in D are i..dd and sampled from the MaxEnt distribution. We have
N
(DDCC (Zme)
II exp(3r(r())) (7(0).
(3)
Note that 1 ()(0) is O for all trajectories that contain any state-action pair that belongs to C. To maximize this, Scobee & Sastry) propose greedy strategy wherein they start with empty constraint set and incrementally add state-action pairs that result in the maximal increase in p p(D(C).
3.2 SAMPLE-BASE APPROXIMATION
Since we are interested in more realistic settings where the state and action spaces can be continuous, considering all possible state-action pairs individually usually becomes intractable. Instead, we propose learning-based approach wherein try approximate 1 (r) using neural network. Consider the loglikelihood
1 (CC) = N
log logp(D(C)
= Σ
(r(( + log M (+(0)) -log ZM
N
1=1
|r((((0) + log 1 Mr 1 (r(00) log / exp(Br(T))1 Me (r)dr.
N
(4)
Note that M (7) (7) = II" M (st, at) merely tells us whether the trajectory T is feasible under the constraint set C or not. Let us have binary classifier Se parameterized with 0 do this for us instead, i.c., we want Ço (st, az) to be (st. a.) is not in C* and otherwise. Using Ço (r) as a short hand for IT Co az), we have
C(C)= = C(0)
Σ
(r(r(0) + log (o(T log / exp(3r(F))GeT))d.
(5)
Let M5a denote MDP obtained after augmenting M with the cost function Eo := 1 - G and Mhã denote the corresponding MaxEnt policy. Taking gradients of (5) with respect to @ gives us (see Appendix(A. A.I for derivation)
VoL(0) = N
Σ
Vo log lo (T)) - Erax mno
[V log (o(())
(6)
Using bample-based approximation for the right-hand term rewrite the gradient
M
VoL(O)
1
V log (0(
1
Σ
v log (o(())).
N
M
where & are sampled from 7 n (discussed in the next section). Notice that making VoL(O) zero essentially requires matching the expected gradient of log Ço under the expert (left hand term) and
Note that since assuming that is 0, we assign any non-zero (positive) cost to state-action pairs that constrain. Here Ço assigns of to all such pairs.
4

---

Under review as a conference paper at ICLR 2021
nominal (right hand term) trajectories. For brevity, we will write M no from now onwards. We can choose (o to be a neural network with parameters and a sigmoid at the output. We train neural network via gradient descent by using the expression for the gradient given above.
In practice, since we have a limited amount of data, Ço. parameterized as a neural network, will tend to overfit. To mitigate this, incorporate the following regularizer into objective function.
R(O) = Σ (<o(T) -1] (D(D,))
(8)
where S denotes the set of trajectories sampled from Te and ā E 10 a fixed constant. R incentivizes Ço to predict values close to thus encouraging Ço to choose the smallest number of constraints that best explain the expert data.
3.3 FORWARD STEP
To evaluate we need to sample from Te. Recall that Ti needs to maximize ( subject to To(s,a) (s, a) Z where Z is the constraint set induced by Ço- However, since outputs continuous values the range (0, 1). we instead solve the soft constrained RL version, wherein we try to find policy that maximizes J (T) subject Er [[o(T]] S
In experiments, set to very small value. Note that if a is strictly set to our optimization program will have an empty feasible Please refer to Appendix A.5 for some
discussion a.
We represent our policy as a neural network with parameters & and train it by solving the following equivalent unconstrained min-max problem on the Lagrangian of our objective function
min max (o = J(TT) + - (((=-=)))) X20
(9)
by gradient ascent on (via the Proximal Policy Optimization (PPO) algorithm (Schulman et al,, 2017)) and gradient descent on the Lagrange multiplier X. Note that we also add the entropy H(TT) = -E [log (T)] of TO to our objective function. Maximizing the entropy ensures that we recover the MaxEnt distribution in convergence (see Appendix A.3 for proof).
3.4 INCORPORATING IMPORTANCE SAMPLING
Running the forward step in each iteration is typically very time consuming. To mitigate this prob- lem, instead of approximating the expectation in (6) with samples from Te, we approximate it with samples from older policy тy where 0 were the parameters of Ç at some previous iteration. We, therefore, only need to learn a new policy after a fixed number of iterations. To correct for the bias introduced in the approximation because of using different sampling distribution, we add important sampling weights to our expression for the gradient. In this case, the importance sampling weights can be shown to be given by (see Appendix A.2 for the derivation)
(e(T)
s(T)
(10)
(p(T)
The gradient can thus be rewritten
N Vo log Co (T))
VaL(O) ~
Σ v log ca(F)
(1I)
N
Σ Eirno s(f)
Foum
Algorithm summarizes our training procedure.
4 EXPERIMENTS
Learning a single constraint: Consider the TwoBrdiges environment in Figure] I The agent starts at the bottom-left corner and take of the following 4 actions each step: right, left, up, down. The reward in the left half is negative and the right half is positive (and proportional to how close the agent is to the goal). This incentivizes the agent to cross over into the right half as
5

---

Under review as a conference paper at ICLR 2021
Algorithm 1: ICRL with Importance Sampling
Input: Expert trajectories D, iterations N, number of backward iterations B. Initialize and randomly.
for ,,..., do
Learn 7 by solving (9) using current Ço.
for j 1, B do
Sample a set of trajectories S = {T((k) j==
k=1 using T..
Compute importance sampling weights (k) using (10) for k = 1, .,.., M.
Use S and to update 0 via SGD by using the gradient in (11).
end end
return π
5
ThreeBridges
LapGridWorld
(c) HallCheetah
(d) Walker
Figure 2: The environments used in the experiments. Note that nominal agents are not aware of the constraints shown.
quickly as possible, which obviously via the lower bridge. However, since the lower bridge is occupied by lion, the expert agent avoids it, whereas the nominal agent does not.
Learning multiple constraints:
For this experiment we design the ThreeBridges environment shown in Figure 2(a)] The agent starts either the bottom top-left corner with equal proba- bility and, as in the TwoBridges case, is incentivized to cross over into the right half as quickly as possible. The expert the other hand always takes the middle bridge, since both the upper and lower bridges are actually constrained.
Preventing reward hacking: Figure 2(b) shows the LapGridWorld environment" which we use to test if our method can prevent reward hacking. The agent is intended to sail clockwise around the track. Each time it drives onto golden dollar tile, gets reward of However, the nominal agent "cheats" by stepping back and forth on one dollar tile, rather than going around the track, and ends up getting more reward than the expert (which goes around the track, as intended).
Scaling to high dimensions: For this experiment, we use simulated robot called HalfCheetah from OpenAl Gym (Brockman et al. 2016). The state and action spaces are of 18 and dimensions respectively. The robot move both in forward and backward directions and is rewarded propor- tionally to the distance it covers. For the constrained environment, shown in Figure 2 we add a solid wall at distance of 5 units from the origin to prevent the robot from moving forwards. Conse- quently, the expert always moves backwards, whereas the nominal agent moves in both directions.
Transferring constraints: In many cases, constraints actually part of the environment and the same for different agents (for example, all vehicles must adhere to the same speed limit). In such instances, it useful first learn the constraints using only agent and then transfer them onto other agents. As a proof of concept, we transfer the constraints learnt on the HalfCheetah agent from the previous paragraph on a Walker2d agent. Note that in this case Ça must only be fed a subset of the state and action spaces that are common across all agents. As such, we only train Ço on the
*This environment is similar the hoat environment in Leike 2017

---

Under review as a conference paper at ICLR 2021
ICRL
nominal
expert
20K
15K
40
15K
1900
20
1000
5K
500
-40
0K
60
100
250
200
20
50
20
30
10
20
30
2
4
0
20
40
60
Reration
Iteration
Iteration
Reeation
TwoBridges
ThreeBridges
LapGridWorld
(d) HalfCheetah
Figure 3:
Performance of agents during training on their respective constrained (true) environments. All plots were smoothed and averaged over seeds.
(I, 3) position coordinates of the HalfCheetah agent, since the rest of elements in the state space are specific to agent.
Figures 3 and 4 show the results these experiments. The re- wards shown are the actual rewards that the agent gets in the constrained (true) environment. In the case of TwoBridges and TrreeBridges, add solid obstacles the constrained bridges to prevent the agent from passing through them. In the con- strained LapGridWorld environment, we award the agent 12 points whenever it completes a lap (rather than awarding 3 points each time it lands on golden dollar tile, in the nominal en- vironment). Finally, in the case of HalfCheetah and Walker2d, we terminate the episode whenever the agent moves beyond the wall. The bottom row in Figure 3 shows Shows the average number of constraints that the agent violates per trajectory when rolled out in the nominal environment. (For the LapGridWorld this is the number of times the agent attempts to move in the anti-clockwise direction.) For the Walker2d experiment, we observed 0 constraint violations throughout training. This is because, in practice, Ço usually acts conservatively compared to the true constraint function (by also constraining state-action pairs that are close to the true con- strained ones). Additional details these experiments, including hyperparameters, Appendix A.4 As can be seen, over the course of training, the agent's 's true reward increases and its number constraint violations go down.
3000
2000
1000
0
20
40
60
80
Iteration
Figure 4: Walker2d
be found in
ABLATION STUDIES
We conduct experiments the TwoBridges environment answer the following questions:
(a) Can we learn constraints even when we have only one expert rollout?
(b) Does importance sampling speedup convergence?
(c) Does the regularization term in (8) encourage Ço choose minimal set of constraints?
To answer (a) we run our algorithm for different number of expert rollouts. As shown in Figure ((a) are able to achieve expert performance
with only expert rollout.
For (b) we run experiments with and without importance sampling for different values of B (the number of backward iterations). Figures (bb) and [(c) show the results. Using B > without

---

Under review as a conference paper at ICLR 2021
20K
Reward
Reward
Reward
20K-
10K
5K
5K
5K
0K
0K
10
20
30
40
10
20
40
20
30
40
Iteration
iteration
iteration
(a) Number of expert rollouts.
(b) With importance sampling.
(c) Without importance sampling.
Figure 5: Ablation study results. All plots were smoothed and (except in (c) averaged over 3 seeds. Incomplete plots indicate failure the training procedure.
importance sampling results in failure in the training procedure (usually resulting in NaNs) or low reward whereas using 1 with importance sampling accelerates convergence.
For (c) we run experiments with different values of 8 which controls the extent of regularization (see Figure shows the results. Note that increases, Ço constrains fewer constraints. Also note that when 1, Co fails to constrain any state.
0.8
0.6
0.2
0.0
(a) 6 0
(b) 8 = 0.1
(c) 5 = 0.2
(d) 8 = 0.5
(c) 8 =
Figure 6: Heatmap of Ço for each of the four possible actions TwoBridges (in clockwise direction from top-left): right, left, down, up.
RELATED WORK
Forward Constrained RL: Several approaches have been proposed in literature solve the forward constrained RL problem the context of CMDPs (Aluman 1999). Achiam et al. (2017) analytically solves trust region policy optimization problems each policy update enforce the constraints. Chow et al. (2018) uses a Lyapnuov approach and also provide theoretical guarantees. Le et al.) (2019) proposes algorithm for cases when there multiple constraints. Finally, well-known approach centers around rewriting the constrained RL problem as an equivalent unconstrained min- max problem by using Lagrange multipliers (Zhang al. 2019 Tessler al. 2019 Bhatnagar 2010 (see Section
3.3 for further details).
Previous work done on inferring constraints from expert demonstrations has either focussed on either inferring specific type of constraints such as geometric D'AAriioo & Shah 2017 Subramani al. 2018). sequential (Pardowitz al. (005) convex (Miryoosefi al. (019) constraints or is restricted to tabular settings (Scobee & Sastry 2020)
Constraint Inference:
Chou et al. 2018) or assume
transition dynamics (Chou al. 2020).
Preference Learning: Constraint inference also links to preference learning which aims extract user preferences (constraints imposed by an expert on itself, in our case) from different sources such as ratings (Daniel et al.) 2014), comparisions (Christiano et al. 2017 Sadigh et al. 2017), human reinforcement signals (MacGlashan et al. 2017) or the initial state of the agent's 's environment (Shah) et al. 2019). Preference learning also includes inverse RL, which aims recover agent's 's reward function by using its trajectories. To deal with the inherent ill poseddeessss of this problem, inverse RL algorithms often incorporate regularizers (Ho & Ermon 2016: Finn al. 2016) assume prior
8

---

Under review as a conference paper at ICLR 2021
distribution over the reward function (Jeon et al. 2018 Michini & How 2012
Ramachandran &
Amir 2007
7 CONCLUSION AND FUTURE WORK
We have presented a method to learn constraints from an expert's demonstrations. Unlike previous works, our method both learns arbirtrary constraints and can be used in continuous settings.
While we consider our method to be an important first step towards learning arbitrary constraints in real-world continuous settings, there is still considerable room for improvement. For example, as is the case with Scobee & Sastry) our formulation is also based on (2) which only holds for deterministic MDPs. Secondly, we only consider hard constraints. Lastly, one very interesting extension of this method can be to learn constraints only from logs data in an offline way to facilitate safe-RL in settings where it is difficult even build nominal simulators such as the case for plant controllers.
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning, 2017.
Eitan Altman. Constrained Markov Decision Processes. Chapman and Hall, 1999.
Dario Amodei and Jack Clark. Faulty reward functions in the wild. https:/ openai.en....... com/ 2016. Accessed: 2020-07-20.
blog/ blog/ wuu.ty--wwwrr---uttt.ooo
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in Al safety, 2016.
Shalabh Bhatnagar. An actor-critic algorithm with function approximation for discounted cost strained markov decision processes. Systems and Control Letters, 59(12);700-76,, 2010.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL ht / www.. wandb. com/ Software available from wandb.com.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym, 2016. arXiv:1606.01540.
Glen Chou, Dmitry Berenson, and Neemiye Ozay. Learning constraints from demonstrations, 2018. arXiv:
Glen Chou, Necmiye Ozay, and Dmitry Berenson. Learning constraints from locally-optimal demonstrations under cost function uncertainty. IEEE Robotics Autom. Lett., 5(2):3682-3690, 2020.
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Iyapunov- based approach to safe reinforcement learning. In Advances in Neural Information Processing Systems. 2018.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017.
Christian Daniel, Malte Viering, Jan Metz, Oliver Kroemer, and Jan Peters. Active reward learning. In Proceedings of Robotics: Science and Systems, 2014.
Claudia Pérez D' D'Arpino and Julie A. Shah. C-learn: Learning geometric constraints from demon- strations for multi-step manipulation in shared autonomy. In IEEE International Conference on Robotics and Automation (ICRA), 2017.
Chelsea Finn. Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference Machine Learning, 2016.

---

Under review as a conference paper at ICLR 2021
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems. 2016.
Wonseok Jeon, Seokin Seo, and Kee-Eung Kim. A bayesian approach to generative adversarial imitation learning. In Advances in Neural Information Processing Systems, 2018
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, ICLR, 2015.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Inter- national Confernce on Machine Learning, ICML, 2019.
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A. Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. Al safety gridworlds. 2017. arXiv:1711 1ttv:1711.09960 1.0883...
Jan Leike, David Krueger, Tom Everitt, Miljan Martie, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: A research direction, 2018, arXiv:1811.0788
James MacGlashan, Mark K. Ho, Robert Loftin, Bei Peng, Guan Wang, David L. Roberts, Matthew E. Taylor, and Michael L. Littman. Interactive learning from policy-dependent human feedback. In International Conference Machine Learning, 2017.
Bernard Michini and Jonathan P. How. Bayesian nonparametric inverse reinforcement learning. In Machine Learning and Knowledge Discovery in Databases, 2012.
Sobhan Miryoosefi, Kianté Brantley, Hal Daume III, Miro Dudik, and Robert E Schapire. Reinforce- ment learning with convex constraints. In Advances in Neural Information Processing Systems. 2019.
Michael Pardowitz, Raoul Zöllner, and Rüdiger Dillmann. Learning sequential constraints of tasks from user demonstrations. In 5th IEEE-RAS International Conference on Humanoid Robots, 2005.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In 20th Interna- fional Joint Conference on Artifical Intelligence, 2007.
Alex Ray, Joshua Achiam, and Dario Amodei. Bewchmarkingg safe exploration in deep reinforcement learning. 2019. https: //cdn. openai. cww httpm//aaeepp-pssaopee påf
Dorsa Sadigh, Anca D. Dragan, Shankar Sastry, and Sanjit A. Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems XIII, 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. arxiv:1707.063477
Dexter R.R. Scobee and S. Shankar Sastry. Maximum likelihood constraint inference for inverse reinforcement learning. In International Conference on Learning Representations, 2020.
Rohin Shah, Dmitrii Krasheninnikov, Jordan Alexander, Pieter Abbeel, and Anca D. Dragan. Prefer- ences implicit in the state of the world. In International Conference on Learning Representationss 2019.
Guru Subramani, Michael Zinn, and Michael Gleicher. Inferring geometric constraints in human demonstrations. In 2nd Conference Robot Learning (CoRL), 2018.
Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In International Conference on Learning Representations, 2019.
Ruiyi Zhang, Tong Yu, Yilin Shen, Hongxia Jin, and Changyou Chen. Text-based interactive recom- mendation via constraint-aut auggenttd reinforcement learning. Advances in Neural Information Processing Systems. 2019.
Brian D. Ziebart, Andrew Maas. J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In 23r National Conference on Artificial Intelligence (AAAI), 2008.
10

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1
GRADIENT OF LOG LIKELIHOOD
The gradient of (5) is
VoL(O)=
1
Σ
0 Vo log co((())
I (xxp(8((+))((()4) / cxp((rtt)))ooo((()t
N
N
Ve log So (T( /
exp(3r(F))so(T)
[xxp(B((7"))o( Vo log (T(
N
v log Co( - /PMIs log log
N
V log () ETNTAN (oolog
(12)
where the second line follows from the identity Voco(T) = co(T)Vo logco(t) and the fourth line from the MaxEnt assumption.
A.2 DERIVING THE IMPORTANCE SAMPLING WEIGHTS
Suppose that gradient the log of the partition function Vo log Ze (where 0 are the current parameters of our classifier) using older policy TC, (where 0 were the parameters of the classifier which induced the constraint set that this policy respects). We can do so by noting that
iteration of training procedure
interested in approximating the
/ exp(r(r))Ge(Tddr
Zo
-f (T)(TT)
exp(r(T))co(T) dr,
Tco (T) (exp(r(T))co(T)
= (ywnteg(7)
Tco (T)
(13)
(o(TT)
Zğ Bynwteg (r)
(T(TT)
So (7()
Zā
M
Ca(r(())
where the fourth lines follows from our MaxEnt assumption, i.e., (T) cxp(r(+))GG(7)/2z
Therefore
V log Zg
VoZo.
Zo
VoGa(T)
Z
ZB M Exeg(T)
Co(r) Co(r)
M Tmo((T)
SØ(T)
(14)
(o(T)
Ve log Co(T)
Ca(7) Σ ET)((T) (p(7)
(o(T)
(ru((T)

---

Under review as a conference paper at ICLR 2021
A.3 RATIONALE FOR 9
Consider a constrained MDP M as defined in Section 2.2 following policy
We are interested in recovering the
(T) exp(3r(T)) (T),
(15)
ZM
where Me (xxp(8((r)) (r)d7 is the partition function and 1 is an indicator function that T C and otherwise.
Lemma: The Boltzmann policy n (T) exp(Br(7))/2 maximizes [(T) = B (r())) H
H()).
where H (अ) denotes the entropy of m.
Proof: Note that the KL-divergence, D between a policy and π can be written as
DxL(=||=0) B) = E Plog (T) log x B (r)]
E olo π (T) (B((r) + log Z].
(16)
= -, [3r(T)] H((T) + log Z,
-B((T) + log Z.
Since log Z is constant, minimizing ( |T B) is equivalent to maximizing ((T). Also, we know Therefore, N maximizes C.
that Pxz(=|]=* is minimized when
Proposition: The policy in (15) is solution of
minimize max on [r(r] + } [((T)) (
(17)
A20
Proof: Let us rewrite the inner optimization problem as
max [r~-/r(T) ((o(r))-a)) + H(T))
(18)
From the Lemma we know that the solution to this is
g(T, A)
T(T, 2) [g(T', X)dr
(19)
where g(T, A) exp(3(((p)) - A(Co(T) - a))) To find (T() = mInx x (T, A), note that:
1. When (o(T) < a, then minimizes æ. In this case g(T, >²) = exp(Br(r)).
2. When (e(r) a, then co minimizes æ. In this case g(T, A+)
We can combine both of these cases by writing
exp(r(r)) 1 (T). T (T) f (ex((((7)) (7))T'
(20)
where 1s < a and ( otherwise. (Note that the denominator is greater than 0 as long as we have at least one for which (o(T) S a, i.e, we have least one feasible solution.) QED.
A.4 EXPERIMENTAL SETTINGS
manage experiments and conduct sweeps hyperparameters. We used Adam Kingma & Ba 2015) to optimize all of our networks. All important hyperparameters listed in Table [
We used W&B (Biewald 2020)
Details on the environments can be found below.
A.4.1 TWOBRIDGES
In this environment, the agent's 's state consists of its (z, v) position coordinates. Agents start (0, 0) and the goal is at (20, 0). Episdoes terminate when the agent is within one unit circle the goal or when the number of timesteps exceeds 200. The bottom left comerrs of the bridges at (4, 5) and
12

---

Under review as a conference paper at ICLR 2021
(4, 14). Each bridge is units long and unit wide. Agents can take of the following actions: right, left, up and down. Each action moves the agent 0.7 units in the respective direction. Agents attempting move outside the 20 20 simulator into the water (in between the bridges) end up in the same position and receive a reward -2. The reward in the regions left of the bridge is fixed 1 and on and the right of the bridges is equal 10/d where d is the Euclidean distance of the agent to the goal. Additionally, the agent's reward is scaled by 20 if it is to the right of bridges and or below the lower bridge (i.e. y 6). Finally, the reward within one unit circle of the goal is fixed to 250.
A.4.2 THRERBRIDGES
This is similar to the ThreeBridges environment except that we now have three bridges. The bottom- left corners of each of the bridges and 2 units wide while the other two bridges are 4 units long and 1.5 units wide. Agents attempting move outside the simulator into water receive a reward of -2. The reward in regions left of the bridges is fixed to -5 and on and to the right of the bridges is 200/d where is Euclidean distance
(4, 1), (4, and (4, 17.5). The middle bridge is 4 units long
of the agent the goal. The reward within one unit eircle of the goal is fixed to 250. Finally, agents randomly start at either the bottom-left of top-left corners with equal probability.
A.4.3 LAPGRIDWORLD
Here, agents move on 11 x 11 grid by taking either clockwise or anti-clockwise actions. The agent awarded reward each time it moves onto bridge with a dollar (see Figure 2). The agent's state is the number of the grid it is on.
A.4.4 HALFCHEETAI AND WALKER2D
The original reward schemes for HalfCheetah and Walker2d in OpenAl Gym (Brockman et al., 2016), reward the agents proportional the distance they cover the forward direction. We modify this and instead simply reward the agents according to the amount of distance they cover (irrespective of the direction they move in).
A.5 ON THE BUDGET SIZE
As noted in Section 3.3 we set the budget size to very small value, typically around 0.01. controls the extent to which the agent respects the constraints imposed by Ço- this section, we study the effect of a the agent's performance. We train agent using our algorithm TwoBridges environment for different values of a. Figure shows the results. As can be seen, the agent's 's performance drops higher values of a. For example when is 1, the agent fails to achieve any meaningful reward.
the
20K
eeport
nominoll
15K
Reward
10K
5K
OK
20
30
Iteration
Figure 7: Performance of agent during training on the constrained (true) environment for different values for
13

---

Under review as a conference paper at ICLR 2021
Table 1: Hyperparameters for various environments
Parameter
TwoBridges
ThreeBridges
LapGridWorld
HalfCheetah
Rollout length
200
200
200
500
Batch size
6000
10000
6000
20000
Expert rollouts
20
2
20
20
Learning of policy
1.23e-3
3.7e-3
1.23e-3
2.4e-2
Learning rate of Co Learning rate a Learning rate of A Initial value of Budget, a Entropy bonus, 1/3 GAE-y GAE-À
0.001
0.005
0.001
0.001
0.00124
0.00378
0.00124
0.0240
10
10
10
0.01
0.01
0.01
0.01
0.01
0.0871
0.1425
0,21
0.01
0.92
0.96
0.92
0.94
0.89
0.74
0.89
0.94
Architecture of Architecture of n
64,64
64,64
64,64
10
64,64
64,64
64,64
64,64
PPO clip ratio
0.13
0.07
0.13
0.21
PPO target kl
0.01
0.01
0.01
0.01
14


Reviews:
[{"review_id": "TceFVSQQX7n", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n\nThe submission focuses on a variant of inverse reinforcement learning, where the learner knows the task reward but is unaware of hard constraints that need to be respected while completing the task. The authors provide an algorithm to recover these constraints from expert demonstrations. The proposed algorithm builds upon a recent technique (Scobee & Sastry 2020) and addresses problems with large and continuous state spaces.\n\n++++++++++++++++++++++++++++++++++\n#### Reasons for score\n\nStrengths:\n* The problem considered is interesting and relevant to the ICLR community.\n* The technique developed (Algorithm 1) is novel and well motivated.\n* The experiments provide adequate evidence to back the claims.\n* The paper is very well written and organized.\n\nWeaknesses:\n* Justification for the policy loss function (Equation 9) is unclear.\n* Comparison with prior art is lacking.\n* Discussion of related work is sparse and can be more detailed.\n\nBased on the above-mentioned strengths, I vote for accepting. My concerns (further detailed below) potentially can be addressed during the rebuttal phase.\n\n++++++++++++++++++++++++++++++++++\n#### Major Comments\n\n1. (page 2) The requirement of ‘ability to modify the environment’ is listed as a limitation of prior art (Scobee & Sastry 2020). However, like the current approach, the prior art adds the constraints / modifies the environments only conceptually (and not physically). Further, both the current and prior work focus on the case of hard constraints. Please clarify this limitation of the prior art vis-à-vis proposed approach.\n\n2. (page 2) The rationale behind the objective (Equation 7) of the prior art and the proposed approach is identical. Please clarify, then, if the current algorithm is also greedy.\n\n3. (Equation 9) Please provide additional details for the inclusion of the entropy term in the policy loss function. \n  - The principle of maximum entropy is used to arrive at Eq. 4, the loss function of theta (since Eq. 4 uses the term derived in Eq. 2, which in turn is obtained from the maximum entropy principle). Given this, it is unclear why the entropy term is also included in Eq. 9. Is it used as a regularizer?\n  - Alternatively put, consider the unconstrained version of Equation 9. In this unconstrained case, the problem is analogous to MaxEnt IRL (Ziebart et al.). In MaxEnt IRL, given the reward $\\theta$, the policy $\\phi$ is computed by value / policy iteration and without the extra entropy term.\n  - Further, adding both $J$ and $H$ in the loss seem counterintuitive as they have different ‘units’. J is cumulative reward, while H is dimensionless entropy. Why is the entropy term normalized by $\\beta$? How is the normalization constant chosen?\n\n4. (Section 4) While not all domains considered in the Experiments can be captured by the prior art (Scobee & Sastry 2020), the first three can be (as they have discrete state, action spaces). Please benchmark the proposed approach with prior art for these three domains. Time permitting, also consider utilizing one of the recent high-dimensional techniques (see below) as another baseline.\n\n5. (Section 6) Space permitting, please include a discussion of following related works.\n\n  - Constrained IRL for high-dimensional problems:\n\n    * Chou, Glen, Necmiye Ozay, and Dmitry Berenson. \"Learning parametric constraints in high dimensions from demonstrations.\" Conference on Robot Learning. PMLR, 2020. \n\n    * Park, Daehyung, et al. \"Inferring Task Goals and Constraints using Bayesian Nonparametric Inverse Reinforcement Learning.\" Conference on Robot Learning. PMLR, 2020. Notes: Extends beyond the proposed approach to consider constraints which may not be global (i.e., locally active constraints).\n\n    * Chou, Glen, Necmiye Ozay, and Dmitry Berenson. \"Learning constraints from locally-optimal demonstrations under cost function uncertainty.\" IEEE Robotics and Automation Letters 5.2 (2020): 3682-3690.\n\n  - Inverse reward / policy learning frameworks that incorporate prior knowledge of reward / policy:\n\n    * Ramachandran, Deepak, and Eyal Amir. \"Bayesian Inverse Reinforcement Learning.\" IJCAI. Vol. 7. 2007. \n\n    * Michini, Bernard, and Jonathan P. How. \"Bayesian nonparametric inverse reinforcement learning.\" Joint European conference on machine learning and knowledge discovery in databases. Springer, Berlin, Heidelberg, 2012.\n\n    * Unhelkar, Vaibhav V., and Julie A. Shah. \"Learning models of sequential decision-making with partial specification of agent behavior.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.\n\n    * Jeon, Wonseok, Seokin Seo, and Kee-Eung Kim. \"A bayesian approach to generative adversarial imitation learning.\" Advances in Neural Information Processing Systems. 2018.\n\n  - Learning features (which can be in the form of logical constraints) for IRL:\n\n    * Choi, Jaedeug, and Kee-Eung Kim. \"Bayesian nonparametric feature construction for inverse reinforcement learning.\" Twenty-Third International Joint Conference on Artificial Intelligence. 2013.\n\n++++++++++++++++++++++++++++++++++\n#### Questions for Rebuttal Phase\n\nPlease address comments 1-4.\n\n++++++++++++++++++++++++++++++++++\n#### Minor Comments\n\n- (typo) In the Introduction, Scobee & Sastry is used as singular noun, where in fact it is plural.\n\n- (Equation 5) Beta is missing in the log exponential term.\n\n- (Page 4, below Equation 7) The statement ‘Notice that … essentially tries to match’ is ambiguous, since the gradient by itself does not try to match the two values. Please consider rephrasing to say that this matching occurs at the minima (where the gradient is zero).\n\n- (Page 5, Section 3.3) Please denote the range $[0,1]$ as $(0,1)$, since 0 and 1 are not in the range of $\\zeta$.\n\n- (Equation 9) Consider distinguishing the loss functions in Equation 5 and 9 (say through superscript or subscript). Due to $L$ being overloaded, at first glance, I misunderstood the loss function in Eq 9 as a continued derivation of Eq 5.\n\n- (Section 7, typo) (2) -> Eq. (2)\n\n++++++++++++++++++++++++++++++++++\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "I5twSL251x", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "POST-REBUTTAL COMMENTS:\n=========================\n\nSeveral of the original concerns, mainly about the scope of the contribution, have been addressed by the rebuttal, so I've increased the overall score. See my responses to the rebuttal for the concerns that remain outstanding.\n\n=========================\n\n\nSUMMARY\n\nThis submission extends Scobee and Sastry's ICML-2020 work, which formulates the problem of learning (hard) constrains in RL as inverse constrained RL setting, by generalizing it from tabular to continuous settings and, empirically, increasing the latter's scalability.\n\n\nHIGH-LEVEL REMARKS\n\nI believe that the paper, while incremental, might be making a valuable contribution. At the same time, the magnitude of this contribution is difficult to assess, because paper makes a number of unsubstantiated claims about them, doesn't thoroughly analyze the limitations of its approach, doesn't provide an empirical comparison to existing alternatives, and has several other presentation issues as detailed below.\n\n\nDETAILED COMMENTS\n\n-- The paper claims, \"[Scobee and Sastry] assume ability to modify the environment (specifically, to add arbitrary constraints to it).\" I'm not sure what is meant here -- adding a constraint doesn't mean modifying an environment. The constraints apply to a policy.\n\n-- The paper also claims, \"our approach [...] does not suffer from the curse of dimensionality\". This claim is unsubstantiated and doesn't sound true. Is the approach really oblivious to observation space dimensionality? Do you have a theorem showing this?\n\n-- The problems in the experiments are too small to support the above claim about the approach being impervious to the curse of dimensionality empirically, if that was the intent. In particular, the paper doesn't experiment with environments involving visual observations.\n\n-- How many expert trajectories were used in the experiments in Figures 3 and 4? \n\n-- The paper claims that its approach can handle arbitrary constraints. This is a misleading claim, at best. There are rich temporal-logic based constraint languages such as LTL, CTL, etc. The paper doesn't discuss whether they are subsumed but CMDP's constraint form even theoretically. In the meantime, even the fragments of these constraint languages that can be compiled into CMDPs' constraints can cause exponential state space size explosion -- consider, e.g., constraints that specify that every trajectory must visit state A before state B before state C, etc. So in practice the paper's claim can't be true.\n\n-- The paper's title, \"CONSTRAINED REINFORCEMENT LEARNING WITH LEARNED CONSTRAINTS\" isn't descriptive of the proposed technique. The paper assumes that the entire nominal MDP (the MDP without constraints) is given, along with a set of demonstration trajectories, and the agent tries to learn *only* the constraints. This means that once the constraints are learned, there is no more learning going on, only optimization. Thus, a more appropriate title for the paper would be something like \"constraint learning via maxent inverse RL\".\n\n-- The paper's assumption of the nominal MDP being fully known, including the transition and reward function, is steep, stronger than IRL's. It also seems to require that the environment should be able to indicate constraint violations during training. All of this raises the question: why not solve the constrained MDP using one of existing forward methods for doing so (the paper mentions a few in the related work) without explicitly learning the constraints? Note that the paper's experiments don't just evaluate constraint learning, they evaluate solutions to the MDPs at hand, so a comparison to forward CMDP RL solvers is natural to ask for.\n\n-- One answer to the above question can be that the goal of learning constraints is transferring them to other settings. But if so, the paper ought to analyze the effect of expert quality on the quality of constraints. Note that this effect can be drastic: if the expert is suboptimal, then its trajectories don't necessarily imply constraints, which means that the proposed methods might learn fictitious constraints that may make other environments where they are applied unsolvable. \n\n-- In general, the paper doesn't compare its approach even to alternatives that also try to learn constraints explicitly. They may try to learn more constrained constraint types, but it would still be very useful to see a comparison to one of them (see the \"constraint inference\" part of the related work section).\n\n-- The paper proposes to use a sigmoid at the output. I'm wondering whether this will lead to vanishing gradient for DNNs.\n\n-- Related to the above point, note that while the paper claims to list all relevant hyperparameters for the experiments, it doesn't describe the network architecture (and the number of experts used in some of the experiments, as mentioned above) either in the main body or in the appendix, which makes its results unreproducible.\n\n-- The use of J(.), with or without superscripts, for a value function is non-standard and confusing. The RL and controls convention is to use J(.) for a cost function to be minimized and V(.) for a value function to be maximized. The paper uses J both for a value function (the paragraph right before section 2.2) and, with a superscript, for a cost function (the first paragraph of 2.2). I strongly suggest bringing the paper in agreement with the standard convention, which will also remove the need to use J with a superscript.\n\n-- In the related work is reasonably complete. A notable omission is Le, Voloshin, Yue, \"Batch Policy Learning under Constraints\", ICML-2019). Also, note that Tessler et al's and Subramani et al's papers' venue is listed in arXiv, but both have actually been published in refereed venues (ICLR-2019 for the former, CoRL-2018 for the latter). Please cite them as such. \n\n-- Figure 5's legends are barely legible, please increase their font.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Cx7sjrc7bo", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper extends a tabular method for constraint inference to work in high-dimensional environments, and demonstrates it on a few environments.\n\nOverall I liked the derivation of the algorithm (modulo some nitpicks described later). However, the experimental evaluation is sorely lacking: there are no baselines compared against, even though the obvious candidate to compare against is preference learning algorithms, e.g. imitation learning (such as GAIL [1]) and inverse reinforcement learning (such as AIRL [2]).\n\nIndeed, it is not clear even theoretically what benefit constraint inference gives over the preference inference framework. While it is not common for preference inference to learn what _not_ to do (as this paper highlights), it is possible -- [3] is an example that creates some toy environments where the agent must learn what not to do, and then solves it using a single-state version of Maximum Causal Entropy Inverse Reinforcement Learning. I would expect that high-dimensional preference learning algorithms like GAIL and AIRL would also be able to do this, and thus are important to compare against.\n\nIn fact, it seems to me that the proposed algorithm can be cast as a special case of preference inference. This is most easily seen from the gradient expression in [4], where the MaxEnt IRL gradient in high-dimensional environments is written as\n\n$$\\frac{\\delta r_{\\theta}}{\\delta \\theta} [\\mathbb{E} [ \\mu ] - \\mu_{D_{\\tau}}].$$\n\nThis is the negative of what you might expect because they are defining a loss function to be minimized while you are defining a log likelihood to be maximized. (Note that $\\mu$ is the state-occupancy measure.) If you set\n\n$$r_{\\theta}(\\tau) = \\log \\zeta_{\\theta}(\\tau)$$\n\nI believe you recover the proposed algorithm. So it seems that the main contribution relative to existing algorithms is to propose this particular structure on the reward function (which corresponds to a hard constraint because when $\\zeta$ is zero the reward is negative infinity). So it becomes even more important to show why this particular structure is an improvement over e.g. the GAN-like structure in AIRL.\n\n(Other minor contributions include the use of importance sampling and the regularization of $\\zeta$, though it is also hard to tell how important these contributions are.)\n\nQuality: As explained above, I have serious concerns about the experimental section.\n\nClarity: The paper was quite clear. I’ve listed a few minor cases below for improvement.\n\nOriginality: To my knowledge no other paper has inferred constraints based on the maximum entropy assumption in high-dimensional environments. However, there are both preference learning algorithms (discussed above) and algorithms that learn logic specifications [5], which seem similar in spirit to constraints (though are not the same).\n\nSignificance: Hard to judge without better experiments.\n\nNitpicks / typos:\n\n> Note that this is not particularly restrictive since, for example, safety constraints are often hard constraints as well are constraints imposed by physical laws.\n\nShouldn’t physical laws (things like F = ma) already be encoded in the transition dynamics T? Why do we need constraints for this?\n\nEquation (5) seems to have dropped a $\\beta$ compared to Equation (4).\n\nShould Equation (8) have an absolute value? Otherwise I don’t see how it incentivizes the classifier to predict values close to 1. Actually, I see, the network is constrained by the sigmoid to output values in [0, 1]. It could be worth clarifying this.\n\nWhen going from Equation (5) to Equation (6), a policy π has suddenly appeared -- please explain what the policy is; the reader should not have to look at the appendix to understand the notation. (I assume it is a Boltzmann rational policy since you are using the MaxEnt framework.)\n\nFirst line of Section 4: “TwoBrdiges” -> “TwoBridges”\n\nReferences:\n\n[1] Generative Adversarial Imitation Learning, https://arxiv.org/abs/1606.03476\n[2] Learning Robust Rewards with Adversarial Inverse Reinforcement Learning, https://arxiv.org/abs/1710.11248\n[3] Preferences Implicit in the State of the World, https://arxiv.org/abs/1902.04198\n[4] Learning a Prior over Intent via Meta-Inverse Reinforcement Learning, https://arxiv.org/abs/1805.12573\n[5] Maximum Causal Entropy Specification Inference from Demonstrations, https://arxiv.org/abs/1907.11792", "extra_scores": {}, "extra_text": {}}, {"review_id": "4k45kv-iT6q", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of review:\nInteresting work, clearly described and well executed. But weak support for the overall motivation, and incomplete baselines to confirm the approach is useful.\n\nDescription:\nThis paper describes a new approach for inverse constraint learning, whereby an RL agent infers constraints (on state/action pairs) from expert trajectories, and uses this in combination for a given reward function to optimize an agent’s behavior.\n\nThe main novelty in the work is the ability to infer constraints with fewer assumptions than previous (incl. handling continuous state/action spaces). The work is still limited to deterministic domains.\n\nThe paper is clearly written, well organized, easy to follow.  The authors are transparent about their assumptions throughout.  Interesting ablation studies are included in the experiments. The characterization of the results & discussion are fair and not over-inflated.\n\nMy main concerns with the work are: (1) the proposed setting itself, and (2) some missing baselines in the experiments.  Regarding the setting, I don’t really buy the motivation.  There are claims (p.2) that rewards are easier to express than constraints, and therefore it is reasonable to assume that the reward is given but the constraints are inferred. But I don’t see the support for this (experimental, theoretical or from the literature) either in people or in AI.  Furthermore, it is not obvious to me that this is a better strategy than straight-forward IRL.  Under some (reasonably flexible) assumptions, constraints can be transformed into (negative) rewards.  So why not just apply IRL to infer a reward function that includes both the positive (goal) and the negative (constraints)?  At the very least, this should be considered as a baseline in the experiments, applying one of the recent IRL methods, such as GAIL, to see whether there is any advantage to this formulation.  If anything, the proposed method makes a stronger assumption on knowledge (having the reward function) than most IRL approaches, so those should perform worse in practice.\n\nAdditional questions:\n-\tSec.3.1: Is it necessary for D (the observed trajectories) to be drawn from \\pi*_C?  Could the trajectories be drawn from other domains?  The results near the end on transfer suggest so.  Why not embrace this in the initial formulation?\n-\tSec.3.1 / 3.2: Can you clarify how you use the given reward function?  I assume this is r(\\tau) in Eqn 2, but if that’s the case, it may not be clear why you need to know the reward function a priori, rather than be able to observe it during trajectories.\n-\tSec.3.2: How do you draw negative examples for your binary classifier?  It seems all the trajectory data would be considered as positive examples.\n-\tSec.3.4:  I’m somewhat surprised that the importance sampling helps that much. In many other instances, the variance is too high to see an improvement in practice, especially when the importance weights are calculated over a full trajectory (not one-step).  Any idea why it works so well (as per Fig.5) in this setting?\n-\tFig.3:  Is the x-axis the number of trajectories of constraint data (i.e. D)?   In Fig.3(b),  Why do you break the constraints here?  Is this indicative of not enough data?  Why don’t you show the constraints for Walk2d (Fig.4)?  I strong recommend some analysis of the results in the main text to make sure to explicitly state the findings, not leave it to the reader’s reading of the plots.\n-\tWhy do you use only 3 seeds?  That seems insufficient, as per recent discussion on reproducibility in RL.  How were those chosen?\n-\tYou briefly mention some directions for future work in the last paragraph of Sec.7.  Can you also comment on what is the difficulty in each?  Why is the current work limited to deterministic MDPs, or hard constraints, or on-policy trajectories?\n\n\nMinor comments:\n-\tIntro: “This is known as agent alignment” -> It is also known as value alignment, and there are many more references.\n-\tIf you need additional space, some (or all) of Fig.2 and Fig.6 could be moved to appendix.\n-\tThe references need to be cleaned up. Stick to a consistent style (full first name, or initial, not a mix).  A few references don’t say “where” the work was published. A few references have the arxiv link, even though the work since has been accepted at a peer-reviewed venue.  This type of sloppiness gives a bad impression (suggesting the authors might be as sloppy with their scientific work.)\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces ICRL, where the RL agent is supposed to maximize the reward under unknown constraints, which should be inferred from the expert demonstration. Reviewers generally agreed that this is an interesting work, and potentially make RL to be applied to more general settings. However, they also would like to see more experimental results with baselines (e.g. agents based on IRL and also related constrained learning approaches) to make the motivation behind the approach more convincing. I hope these concerns are addressed in the future work.
Decision: Reject