Contextual Inverse Reinforcement Learning
Authors: Philip Korsunsky
Stav Belogolovsky
Tom Zahavy
Chen Tessler
Shie Mannor
Venue:  (2020)
Keywords: Contextual MDP
Inverse Reinforcement Learning
Reinforcement Learning
Mirror Descent

Abstract: We consider the Inverse Reinforcement Learning problem in Contextual Markov
Decision Processes. In this setting, the reward, which is unknown to the agent, is a
function of a static parameter referred to as the context. There is also an “expert”
who knows this mapping and acts according to the optimal policy for each context.
The goal of the agent is to learn the expert’s mapping by observing demonstrations.
We define an optimization problem for finding this mapping and show that when
it is linear, the problem is convex. We present and analyze the sample complexity
of three algorithms for solving this problem: the mirrored descent algorithm,
evolution strategies, and the ellipsoid method. We also extend the first two methods
to work with general reward functions, e.g., deep neural networks, but without the
theoretical guarantees. Finally, we compare the different techniques empirically in
driving simulation and a medical treatment regime.

Under review as a conference paper at ICLR 2020
CONTEXTUAL INVERSE REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
We consider the Inverse Reinforcement Learning problem Contextual Markov Decision Processes. In this setting, the reward, which is unknown to the agent, is a function of a static parameter referred as the context. There also an "expert" who knows this mapping and acts according to the optimal policy for each context. The goal of the agent is to learn the expert's mapping by observing demonstrations. We define an optimization problem for finding this mapping and show that when it is linear, the problem is convex. We present and analyze the sample complexity of three algorithms for solving this problem: the mirrored descent algorithm, evolution strategies, and the ellipsoid method. We also extend the first two methods to work with general reward functions, c.g., deep neural networks, but without the theoretical guarantees. Finally, we compare the different techniques empirically driving simulation and medical treatment regime.
INTRODUCTION
We study sequential decision-making in a Contextual Markov Decision Process (CMDP, Hallak et al. (2015)), where the reward, while unknown to the agent, depends static parameter referred to as the context. For a concrete example, consider the dynamic treatment regime (Chakraborty & Murphy, 2014). Here, there is a patient and clinician which acts to improve the patient's health. The context is composed of static information of the patient (such as age and weight); the state is composed of the patient's dynamic measurements (such heart rate and blood pressure); and the clinician's 's actions are a set of intervention categories (e.g., infusion). The reward is different for each patient (context), and there is mapping from the context the reward.
Recent trends in personalized medicine motivate this model - instead of treating the "average patient", patients are separated into different groups for which the medical decisions are tailored (Fig. 1b). For example, in Wesselink et al. (2018), the authors study organ injury, which may occur when a specific measurement (mean arterial pressure) decreases below certain threshold. They found that this threshold varies across different patient groups (context). In other examples, clinicians set treatment goals for the patients, i.e., they take actions to make the patient measurements reach some pre-determined values. For instance, in acute respiratory distress syndrome (ARDS), clinicians argue that these treatment goals should depend on the static patient information (the context) (Berngard et al., 2016).
There are serious issues when trying to manually define a reward signal in real-world tasks. When treating patients with sepsis, for example, the only available signal is the mortality of the patient at the end of the treatment (Komorowski et al., 2018). This signal is sparse, and it is unclear how to manually tweak the reward to maximize the patient's 's health condition (Leike al., 2017; Raghu et al., 2017; Lee et al., 2019).
To address these issues, we propose the Contextual Inverse Reinforcement Learning (COIRL) framework. Similarly to Inverse Reinforcement Learning (Ng & Russell, 2000, IRL), we focus on trying infer the mapping from contexts to rewards by observing experts. The main challenge in our problem is that for each context there is a different reward, hence, a different optimal policy for each context. Therefore, Apprenticeship Learning algorithms (Abbeel & Ng, 2004; Syed & Schapire, 2008) that try to mimic the expert cannot be used and, instead, we focus on directly learning the mapping.

---

Under review as a conference paper at ICLR 2020
In partieular, our main contributions are:
1. We formulate COIRL with a linear mapping as a convex optimization problem.
2. We propose and analyze the sample complexity of three algorithms for COIRL: the mirrored descent alg. (MDA), evolution strategies (ES), and the ellipsoid method.
3. For nonlinear mappings, we implement deep learning version for MDA and ES (without theoretical guarantees).
4. We compare these methods empirically on two frameworks: an autonomous driving simulator (Abbeel & Ng, 2004) and dynamic treatment regime (Komorowski et al., 2018).
pafromee antive call you
Context
Action
Action
Expert
Agent
Current sepei
uhild dro
grgeedy
Environment
grondy
State
State
Demonstrations
lppurt in vamber at cdeeusshy
(a) COIRL diagram
(b) Personalized medicine (Itenov et al., 2018)
The COIRL framework (left): a context vector parametrizes the environment. For each context, the expert uses the true mapping from contexts to rewards, W and provides demonstrations. The agent learns an estimation of this mapping W and acts optimally with respect to it.
Figure 1:
PRELIMINARIES
Contextual MDPs: Markov Decision Process (Puterman, 1994, MDP) is defined by the tuple
(S,A, P,E, R 7) where S is finite state space, AA finite action space, P x S x A - [0. 1] the transition kernel, & the initial state distribution, R -> TR the reward function and y E [o. 1) the discount factor. A Contextual MDP (Hallak et al., 2015, CMDP) is extension of an MDP, and is defined by (C, A, MM, 7) where C is the context space, and M is a mapping from contexts c to MDPs: M(c) (S, A, P. R {. In addition, each state is associated with feature vector S S [0, 1] Note that P and { are not context dependent.
We consider setting in which the reward for context c is a linear combination of the state features: R (s) = (c) (s)). The goal is to approximate f'(c) using a function fw (c). with parameters W. This notation allows to present particular, a deep neural network (DNN). For the theoretical analysis, we will further assume a linear setting, where f (c) d" W*, (c) c" W and that W* is in some convex set W.
algorithms for any function approximator fw (c), and in
We assume that c E C Aa-1. the standard d - dimensional simplex. This assumption makes the contexts bounded (which we use in our proofs), and it also allows Straight-forward expansion to model in which the transitions are also linear mapping of the context (Modi et al., 2018). One way of viewing this model is that each row in the mapping W is base rewards coefficient vector, and the reward for a specific context is a convex combination of these base rewards.
We consider deterministic policies æ The value of policy for reward coefficients vector r is: V.- = Eg.r.r I x R(s)) = r (ux)) where ((T) : Pg. (Eee (())) E R& is called the feature expeciations of n. For the optimal policy with respect to (w.r.t.) reward coefficients vector r. we denote the value by V,* For any context c, nh denotes the optimal policy w.r.t reward (s)s) = (c)) d(s) and (W) denotes the
which dictate the agent's behaviour at each state.
optimal policy w.r.t. reward (s) (c) (ss)).
Inverse Reinforcement Learning in CMDPs: In standard IRL, the goal is to learn a reward which best explains the behavior of observed expert. The model describing this scenario is the MDP
2

---

Under review as a conference paper at ICLR 2020
an MDP without reward function (also commonly called a controlled Markov chain). Similarly, denote CMDP without a mapping of context to reward by CMDPAM, The goal in Contextual IRL is to approximate the mapping (c) by observing expert. The expert knows (c), and for each context c, can provide a demonstration from T.-
Contextual dynamics: Learning transition kernel and an initial state distribution that is parametrized by the context is an orthogonal problem to COIRL. Therefore, we focus only on a contextual reward which simplifies our analysis. Existing methods, such as in Modi al. (2018), can be used to learn the mappings for the transition kernel and initial distribution in a contextual model. In conjunction with the simulation lemma (Kearns & Singh, 2002), these methods can extend our results to the more general CMDP setting.
3 OPTIMIZATION METHODS FOR COIRL
In this section, we propose and analyze optimization algorithms for minimizing the following loss function; Lemma below justifies its use for COIRL.
Loss (W) E. max (fw(e)- (p(T) - -imm)mmm)]] Ie (
(u(+t(W)) - [(
(1)
Lemma 1. Los)(W) satisfies the following properties: (1) VW, Los)(W) 2 O, and Loss (os)(W") O. (2) If LW) = then Vc C, the expert policy n; is the optimal policy w.z.. reward c. W.
To evaluate the loss, the optimal policy (W) and its features expectations fufite (W)) must be computed for all contexts. For a specific context, finding n. (W) can be solved with standard RL methods such as Value or Policy Iteration. Computing μ ((a.((') (W)) is equivalent to policy evaluation (solving linear equations).
The challenge is that Eq. (1) is is not differentiable in W. We tackle this problem using two methods for computing descent directions that do not involve differentiation: (i) computing subgradients and (ii) randomly perturbing the loss function. In addition, as the loss is defined in expectation over the contexts, computing it requires to calculate the optimal policy for all contexts. We deal with this issue at the end of Section 1. In the special case that fw (c) is a linear function, Eq. (1) is convex. The following Lemma characterizes Eq. (1) in this case.
[ cW (u(#t(W)) - [(TTt) We have that: (1) Lise (W) is convex function. (2) 9(W)= E. [c O (m)) u(x:))] is sub gradient of Lyn (W). (3) Lum is 2v0k
Lemma 2. Let (W)) E.
Lipschitz continuous functtio, with Lipschitz constant L - w.nt. I1-lloo and L T-7 w.n.t. 11:112-
A technical proof (by definition) is provided in the supplementary material. Note that g(W) Rd we will sometimes refer to it as a matrix and sometimes
flattened vector, no confusion will arise.
Remark 1. The Lipschitz of L (W) is related to the simulation lemma (Kearns & Singh, 2002); small change in the reward results in a small change in the optimal value.
Remark As gW) is a subgradient of Loss(W), ) it can be used to back- mack-propagate DNNs. Clearly, we cannot guarantee convexity (hence no theoretical guarantees). but we can design Loss(W) to be Lipschitz continuous in W using the methods presented in Cisse et al. (2017); Arjovsky el al. (2017).
Remark 3. The subgradient gWW is given in expectation over contexts, and in expectation over trajectories (feature expectations). We will later see how to replace it with an unbiased estimate, which can be computed by observing a single expert trajectory for a single context.
3.1 MIRRORED DESCENT FOR COIRL
Lemma 2 identifies Liin (W) as a convex function and provides a method to compute its subgradients. standard method for minimizing convex function over a convex set is the subgradient projection algorithm (Bertsekas, 1997): Wt+1 Projw(w, agg( w.),, where f f(w,) is a convex function, g g(w)) is a subgradient of fww)). and a the learning rate. W is a convex set, and specifically, we consider the lz ball (Abbeel & Ng. 2004) and the simplex (Syed & Schapire, 2008). We focus
Scaling of the reward by constant does not affect the resulting policy, thus, these are not restricting.
3

---

Under review as a conference paper at ICLR 2020
Algorithm 1 MDA for COIRL
Algorithm 2 ES for COIRL
input: a convex set W,, 7 number of itera- tions
input:
step sizes {artT= 1s, noise STD o, num- ber of evaluations m and smoothing parameter
initialize W1 E W
for = 1,..., do
initialize: W E R
Observe c. (((TT))
for t == 1...., T do
Compute (W) (WW),((F.. (W))
Observe e, (((T²)) for j = l,... mi do
Compute Sr according to Lemma 2 if PSGD then
U (0, o2)
0t = (1 - 7) 2dkt
Loss, = Loss (W + Tuus
W1+1 Wt- Ogfi
d ([)s = -,== Loss, (W)
uj
|W1+1|| > 1 then
||
W++1 w 1
If Loss (Ldsss()) (W) Loss(W) at
else Exponential weights then
then W = -W
mo Loss(W)
return W
a: = 12==1-10 (1-1) -) 7) log(dk)
for i = 1,..., dk do (1) = we (1) exp (-aigi(1)
1/++1 1/+11 / E; (1)
return -/=1 Wt
a generalization of the subgradient projection algorithm that called the mirror descent algorithm (Nemirovsky & Yudin, 1983, MDA): w++1 arg min w V 1
Da (w, wx) where
ax
(w wi) is Bregman distance? associated with a strongly convex function vb The following theorem characterizes the convergence rate of MDA.
Theorem (Convergence rate of MDA), Let yb be a o-strongly convex function on W w.rff ||-]], and let 2 s11Prr.we WIEW Dw(w1, W2)- Let f be convex and I.-Lipschitz continuous w.r.f. ||- Then,
MDA with oz = LV 2a T satisfies: f(+ ΣT= xt) f(xx)((+=)))))){())) < DLV of
We refer the reader to Beck & Teboulle (2003) and Bubeck (2015) for the proof. Next, we provide two MDA instances (sce, for example Beck & Teboulle (2003) for derivation) and analyze them for COIRL.
Projected subgradient descent (PSGD): Let W be 82 ball with radius 1. Fix 12, and vww) 1. 3b is strongly convex w.r.t.
with o = 1. The associated Bregman divergence is given by (w,, W2) = 0.5| ||| - w2 Thus, mirror descent is equivalent to PSGD. D = max WEEW (o w22 N 1, and according to Lemma 2, I. 2/d2 Thus, we have that after T iterations Lio ( ΣT= w.) - Lun(w") < O (1-1)T))
Exponential Weights (EW): Let W be the standard dk dimensional simplex. Let (ww) Σ, w(i) log(w(i)). & is strongly convex w.r.t. ||- I|1 with a = 1. We get that the associated Bregman divergence is given by (w w2) = E. (i) (1()) also known as the Kullback-Leibler
(1)
divergence. In addition, D max (wi, w2) S log(dk) and according to Lemma 2,
I. = 2. Furthermore, the projection onto the simplex w.r.t. to this distance amounts to a simple renormalization w/llwwl /||||//|l.-110 Thus, we get that MDA is equivalent to the exponential weights algorithm and Lin (+ Σ wx) - Lin(w" S O( y Mlogdkkj [1~))V]]
Practical MDA: One of the "miracles" of MDA is its robustness noise. If we replace ş with an unbiased estimate 9r. such that Eğ. 9: and E || || < L, we obtain the same convergence results as in Lemma 2 (Robbins & Monro, 1951) (see, for example, (Bubeck, 2015, Theorem 6. 1)). Such an unbiased estimate can be obtained in the following manner: (i) sample context ct (ii) compute
We refer the reader Appendix C for definitions of the Bregman distance, the dual norm, etc.
4

---

Under review as a conference paper at ICLR 2020
(( w..) ). (iii) observe a single expert demonstration 7." {s) ag. si. ai....... }, where a; is chosen by the expert policy "Tr-- (iv) let f [=.(00...17_11111 7 "((;)) be the accumulated discounted features across the trajectory such that Ei,; = (((T))
w The challenge is, that for iu to be unbiased estimate of (T ) T. needs to be of infinite length.
There are two ways in which we can tackle this issue. We can either (1) execute the expert trajectory online, and terminate it at each time step with probability 1 1- (as in (Kakade & Langford, 2002)),
or (2) execute a trajectory of length H = I- log(1/eH)) The issue with the first approach that since the trajectory length is unbounded, the estimate Tu cannot be shown to concentrate to (u(=²)) via Hoeffding type inequalities. Nevertheless, it is possible to obtain a concentration inequality using the fact that the length of cach trajectory is bounded in high probability (similar to Zahavy et al.). The second approach can only guarantee that ||g. Eğell S Cи (Syed & Schapire, 2008). Therefore, using the robustness of MDA to adversarial noise (Zinkevich, 2003), we get that MDA converges with
an additional error of CH. i.e., Lin + Et= We/ Lin(w") (w') I O
VT + CH- While this sampling
1
mechanism comes with the cost of controlled bias, usually it is more practical, in particular when the trajectories are given as a set demonstrations (offline data).
3.2 EVOLUTION STRATEGIES FOR COIRL
To minimize Eq. (I), we also design a derivative free algorithm (Algorithm 2) that is based on Evolution Strategies (Salimans et al., 2017, ES). For convex optimization problems, ES is a gradient- free descent method that is based on computing finite differences (Nesterov & Spokoiny, 2017), whose sample complexity is provided below in Theorem 2. The Theorem is given in terms of the 2v/i (Section 3.1). While this approach has looser
Lipschitz constant, which is upper bounded by T-y upper-bound guarantees compared to MDA (Theorem 1). Nesterov & Spokoiny (2017) observed that in practice, it often outperforms subgradient based methods. Thus, test this method empirically and compare with the subgradient method (Section 3.1). ES is also known to perform well in practice, even with nonconvex objectives. Specifically, Salimans al. (2017) has shown that ES can be used to optimize the parameters of a DNN to solve challenging high dimensional RL tasks like playing Atari.
Theorem (ES Convergence Rate (Nesterov & Spokoiny, 2017)). Let Lune (W) be a non-smooth convex function with Lipschitz constant L, such that | xx S D, step size of oo (dk+H)/TILE and v S 2L then in T = 4fdk+4f" D² ES finds solution which is bounded by EUT-, [+x)] Liie (xx) S c, where Ur (uo...., wr denotes the random variables of the algorithm up to time T and ZT arg min L (21).
3.3 ELLIPSOID ALGORITHM FOR COIRL
The final algorithm we consider is an el- lipsoid method, introduced to the IRL set- ting by Amin et al. (2017). In this sec- tion we extend it the contextual set- ting, specifically, we focus on finding a linear mapping W and further assume that W = WW || < 1}, and that W W.
Algorithm 3 Ellipsoid algorithm for COIRL
Initialize: O₂- Bec (0, 1) =(x æ ||ello $ 1) O + MVEE(Oe) for f 1 do
Observe c,, let W, be the center of O Play episode using म. = arg max, o W.
if of w. Vhg w- > c then
The algorithm maintains an ellipsoid- shaped feasibility set that contains W At any step, the current estimation W; of W' defined as the center of the ellip- soid, and the agent acts optimally w.r.t. this estimation. If the agent performs sub- optimally. the expert provides a demonstration in the form of the optimal feature expectations for ca ((TT))) ). The feature expectations are used to generate a linear constraint (hyperplane) on the ellipsoid that is crossing its center. Under this constraint, construct new feasibility set that is half of the
m( is revealed
Let a; = a @ (u(T²,) ) - (((tt))
0 + MVEE ((0 E O, 0T aa 2 W. WFa:))
else
O <- Or
5

---

Under review as a conference paper at ICLR 2020
previous ellipsoid, and still contains W For the algorithm to proceed, compute new ellipsoid that is the minimum volume enclosing ellipsoid (MVEE) around this "halffellipsoid" 3, These updates guaranteed to gradually reduce the volume of the ellipsoid (a well-known result (Boyd & Barratt, 1991)) until its center is a mapping which induces c-optimal policies. Theorem 3 shows that this algorithm achieves polynomial upper bound on the number of sub-optimal time-steps. Finally, note that in Algorithm 3 we use an underline notation to denote a "flattening" operator for matrices, and @ denote composition of outer product and the flattening operator. The proofs in this section are provided in the supplementary material, and are adapted from (Amin et al., 2017).
Theorem 3. In the linear setting where R2 Algorithm 1, the number of rounds in which the agent is not c-optimal is O 1.2 log( (1-)()))))))
e""'' for an agent acting according to dk
Remark 4. Note that the ellipsoid method presents new learning framework, where demonstrations are only provided when the agent performs sub-optti uub-optimally. Thus, the theoretical results in this section cannot be directly compared with those of the descent methods. We further discuss this in the experiments and discussion sections.
Remark 5. The ellipsoid method does not require distribution over contexts adversary may choose them. MDA can also be easily extended to the adversarial setting via known regret bounds on online MDA (Hazan, 2016).
Practical ellipsoid algorithm: many real-world scenarios, the expert cannot evaluate the value of the agent's 's policy and cannot provide its policy or feature expectations. To address these issues, we follow Amin et al. (2017) and consider a relaxed approach, in which the expert evaluates each of the individual actions performed by the agent rather than its policy, and provides finite rollouts instead of a policy or feature expectations (see the supplementary material (Algorithm We define the expert criterion for providing a demonstration to be Q'TW- (8, a) + e < V w (s) for
4) for pseudo code).
each state-action pair (s, a) in the agent's trajectory.
relax the optimality requirement of the expert and instead assume that, for each context ct the expert acts optimally w.r.t W.' which is close to W*; the expert also evaluates the agent w.r.t. this mapping. This allows the agent learn from different experts, and from non-stationary experts whose judgment and performance slightly vary over time. If a sub-optimal action w.r.t. W," is played at state s, the expert provides roll-out of /l steps from to the agent. As this roll-out is a sample of the optimal policy w.r.t. Wi,, we aggregate n examples assure that with high probability, the linear constraint that we use in the ellipsoid algorithm does not exclude W' from the feasibility set. Note that these batches may be constructed across different contexts, different experts, and different states from which the demonstrations start. Theorem 4 below upper bounds the number of sub-optimal actions that Algorithm 4 chooses.*
Near-optimal experts: In addition,
Theorem 4. For an agent acting according to Algorithm 4, with probability of at least 8, (gddkk(dk + log( ( )/5) if Vt n = 21- - 512k22
for H = _ log(
(og((1=h))) (T-77e) and
W E BoO (1-7)e) n Oo the number of rounds in which a sub-optimal action is played is Pb' 1--) log (II-))ō log( dk
dk
()))
EXPERIMENTS
The simulations in this section include two domains: (1) an autonomous driving simulation (Abbeel & Ng, 2004), that we adapted to the contextual setup and (2) medical treatment regime, constructed from data set of expert (clinician) trajectories for treating patients with sepsis". In each of these domains we compare the algorithms in two setups: the ellipsoid learning framework and an offline framework. All the results are averaged across 10 random seeds in Section 4.1 and 5 seeds in Section 4.2 (we report the mean and the standard deviation). Due to space considerations we present the simulations in the ellipsoid framework only for the car domain, and the simulations in the offline framework only in the dynamic treatment regime. Complementary simulations can be found in the supplementary material.
This procedure follows sequence of linear algebra operations which explain in the appendix. 4 'MDA also works with near optimal experts due to the robustness of MDA. The analysis of this case is identical to the analysis of biased trajectorics, as we discuss in the end of Section 3.1.
'The data, code and implementation of our algorithms can he found in

---

Under review as a conference paper at ICLR 2020
4.1 DRIVING SIMULATION THE ELLIPSOID FRAMEWORK
In the ellipsoid framework, an expert evaluates the agent policy. If the agent's policy is sub-optimal, the expert provides the agent its feature expectations; otherwise, no demonstration given. The algorithm performs learning in between demonstrations. This setup enables a proper comparison with the ellipsoid algorithm, which requires the additional expert supervision. We measure performance w.r.t. the following criteria: (1) # demonstrations the amount of contexts on which each algorithm requested an expert demonstration (y-axis) as a function of time, i.e., the total number of contexts (x-axis). (2) Value the difference in value, between the agent policy and the expert policy w.r.t. the true reward mapping, i.e., E fw- (c) (u(+ (W)) - {u(T²)), where Creat is a holdout (test) set of contexts. The x-axis measures the amount of demonstrations given.
Setup. This domain simulates a three-lane highway with two visible cars cars A and B (illustration provided in the appendix). The agent, controlling car A, can drive both on the highway and off-road. Car B drives on a fixed lane, at slower speed than car A. Upon leaving the frame, car B is replaced by a new car, appearing in a random lane the top of the screen. The feature vector ofs is composed of 3 features: (1) speed feature, (2) a collision feature, which is valued 0 in case of a collision and 0.5 otherwise, and (3) an off-road feature, which is 0.5 if the car is on the road and otherwise.
In this task, the context vector implies different priorities for the agent; should it prefer speed or safety? going off-road avoid collisions a valid option? For example, an ambulance will prioritize speed and may allow going off-road as long as it goes fast and avoids collisions, while bus will prioritize avoiding both collisions and off-road driving as safety is its primary concern. To demonstrate the effectiveness of our solutions, the mapping f -+ [], is constructed in way that induces different behaviors for different contexts, making generalization a challenging task. We provide additional details on the domain as well as the hyper parameter selection in the appendix.
Linear: The optimal behavior is defined using a linear mapping W In this setting, all three approaches obtain competitive results, in terms of generalization, although the ES is capable of obtaining these results faster, as is seen through the regret and number of required demonstrations.
good
300
Blgood
0:5
2.44
232
102
PSDD
2.00
0.65
P500
0.00
200
20
2
150
100
Time
(a) # demonstrations
(b) Value
(c) # demonstrations
(d) Value
Figure 2: Experimental results in the autonomous driving simulation with a linear mapping (a & b) and nonlinear mapping (c & d)
Nonlinear: For the nonlinear task, consider two reward coefficient vectors i11 and r2, and define the mapping by (c) = Tı if ||c||>> 2 0.55, and r2 otherwise- an illustration is provided in the appendix. In order to learn the nonlinear mapping, we represent fw (c) using DNN, a multi-layered perceptron, which maps from context to reward vector. DNNs have proven to be capable of extracting meaningful features from complex assumption no longer holds, yet DNNs often overcome such issues. In this setting, the superiority of the descent methods rises; as the linear assumption in the ellipsoid algorithm not met, it fails to generalize and keeps requiring new demonstrations. We believe these results to be crucial when considering real-life applications, in which the problem is not necessarily linear. Such cases highlight the strength of the descent methods, which, as Fig. 2 shows, are capable of scaling to nonlinear high dimensional mappings.
dimensioonnl data, e.g., images in these scenarios, the linear
4.2 DYNAMIC TREATMENT REGIME THE OFFLINE FRAMEWORK
In the offline framework, we focus the ability to learn from previously collected data. A data set of previously collected trajectories is given, such that a single trajectory of finite length is observed for each context and no context is observed more than once. We measure performance w.r.t. the following
7

---

Under review as a conference paper at ICLR 2020
criteria: (1) Value as in the ellipsoid framework above, but here the x-axis corresponds to the amount of iterations. Each iteration corresponds to a single subgradient step, where the subgradient is computed from mini batch of 10 contexts. (2) Loss as in Eq. (I). (3) Accuracy % the percent of actions on which the expert and the agent agree on. All these criteria are evaluated on a holdout set.
Setup. In the dynamic treatment regime, there is clinician which acts to improve sick patient's medical condition. The context (static information) represents patient features, which do not change during treatment, such as age and gender. The state summarizes the dynamic measurements of the patient, e.g., blood pressure and EEG readouts. The actions are the forms of intervention a clinician may take, including combinations of various treatments provided in parallel. Dynamic treatment regimes are particularly useful for managing chronic disorders and fit well into the broader paradigm of personalized medicine (Komorowski et al., 2018; Prasad et al., 2017).
The agent needs to choose the right treatment for a patient that is diagnosed with sepsis. We use the MIMIC-III data set (Johnson et al., 2016) and follow the data processing steps that were taken in Jeter et al. (2019). As performing off off-policy evaluation is not possible using this data set, due to it not satisfying basic requirements (Gottesman et al., 2018: 2019), we designed a simulator of a CMDP The simulator is based on this data set; a complete overview and explanation on how it was created is provided in the appendix. The mapping W* is linear, W' E R8x42 which we constructed from the data. In the simulator, the expert acts optimally w.r.t. this W
Specifically, when treating a sepsis patient, the clinician has several decisions to make, such as whether not to provide a patient with vasopressors, drugs which commonly provided to restore and maintain blood pressure in patients with sepsis. However, what is regarded as healthy blood pressure differs based on the age and weight of the patient (Wesselink al., 2018). In our setting, W captures this information as it maps from contextual (e.g., age) and dynamic information (e.g.. blood pressure) reward.
Results. Fig. presents the ability of the descent methods to generalize to unseen contexts by learning from offline data (without supervision). The data composed of of trajectories, i.e., offline data, that were collected from experts (clinicians treating patients). In each iteration, we sample a mini- batch of 10 contexts, i.i.d, from the context distribution. For each context, there a corresponding expert trajectory of length H = 40. Performance is measured on a holdout set of 300 contexts (that are sampled from the same context distribution) according to Theorem 1. We can see that both ES and PSGD attain near-optimal performance using only previously collected expert trajectories.
0.0203
000000
2.0032
PSGD
0.0202
0.0024
LW
0016
Value
0.020
60-
0.0000
Expert
50
0.0199
P5G0
PSGD
EW
20
40
60
100
0.0108
20
60
80
100
0
20
40
60
80
100
Zorations
Iterations
Morations
(a) Loss
(b) Value
(c) Accuracy %
Figure 3: Experimental results in the dynamic treatment regime with a linear mapping
Looking Fig. 3a, we can see that all the algorithms manage minimize the loss to roughly the same eиror. The small bias is explained by the fact that we use truncated trajectories (as we discussed in the practical MDA paragraph) where in the ellipsoid framework experiments we used feature expectations. We can also see that minimizing the loss leads to policies that attain e e-optimal value w.r.t. the true reward Fig. 3b. Finally, Fig. we can see that all the algorithms reach around 70% accuracy with the expert policy. We emphasize here that 100% accuracy should not be expected for two reasons: (i) different policies may have the same feature expectations (hence the same value) but make different decisions (ii) there exists reward for which there is more than optimal policy. Nevertheless, Fig. 3 suggests that accuracy is correlated with minimizing the COIRL loss (Eq. (1)).
Finally, present results in the nonlinear setting. Here, there is non-linear function of the context that determines which one of the two reward coefficient vectors is used, i.c., fc( = r, if age > 0.1, and r2s otherwise. Where age refers to the normalized age of the patient, which is element
8

---

Under review as a conference paper at ICLR 2020
0.12
PSGD
60
0.09
0.4
0.05
0.29
Hit
0.5
012
30
Expert PSGO
10-1
0
50
100
150
150
Rerations
Ite Perrations
Iterations
(a) Loss
(b) Value
(c) Accuracy %
Figure 4: Experimental results in the dynamic treatment regime with non-linear mapping
of the context vector. We use DNN to learn the mapping and follow Section 3.1 (PSGD). As seen in Fig. 4, the PSGD algorithm minmmizes the loss and achieves a value that is close to that the expert. In addition, similarly to Fig. 7, accuracy and performance do not necessarily correlate one to another.
RELATED WORK
We begin with a short discussion on contextual policies, i.e., a policy that is function of both the state and the context. While there is empirical evidence that learning such a policy may perform well in practice (e.g.. Xu al. (2018); Fu et al. (2019)), from a theoretical point of view, there exist hardness results in this setting. Specifically, given an MDP with & + states, there is a reduction from the training problem of the union of hyperplanes to the policy (see appendix E for proof).
Alternatieel,, one may consider applying an AL algorithm on a single, large MDP that includes all the states and the contexts. For a concrete example, consider reduction from the CMDP model to a large MDP where each state is expanded by concatenating the context to it. The new states are = (s, c), and the new features are ((8') = e (()) Generally speaking. applying an AL. algorithm to this large MDP will give the same scalability and sample complexity COIRL. However, the
large MDP has S = [C||8] states, computing the optimal policy in each iteration of the algorithm will require at least |C| times more time.
To illustrate this problem we conducted a simple grid world experiment on a 3 x grid world MDP (with 12 states) and one-hot features (ó(s;) e, E IR 1). The dynamics are deterministic, and the actions correspond to going up, down, left and right (with cyclic transitions on the borders). The contexts correspond to "preferences"
the grid; mathematically, each context is sampled from uniform distribution over the simplex. The mapping W* is set to be 112-12, and for AL, we let w" to be a flattened version of W*.
7.5
(Abbeel
100
100
COIRL (PSGC)
Valuee of Expert
90-
4.5
96
3.0
94
70
1.5
92
AL (Abbeel NG)
Al (Abbeel & NG)
COIRL (PSGD)
COIRL (PSGD)
0.0
10
20
30
40
50
10
20
30
40
Contexts
Contexts
Co ttesits
(a) Loss
(b) Value
(c) Accuracy %
Figure 5: Comparison between COIRL and AL on a large MDP
We compare the performance of PSGD with the projection algorithm of (Abbeel & Ng, 2004) in Fig. 5. We measure performance by three metrics: run-time, value, and accuracy. Inspecting the results, we can see that AL in the large MDP requires significantly more time to run the number of contexts grows, while the run time of PSGD (COIRL) is not affected by the number of contexts. We can also see that both methods achieve roughly the same performance: COIRL performs slightly better in terms of value while AL performs slightly better in terms of accuracy. To conclude, AL on large MDP does not scale to problems with large context spaces. In addition, this construction is
9

---

Under review as a conference paper at ICLR 2020
only possible when there is finite number of context and does not provide generalization results. We avoided all of these issues in the COIRL framework.
SUMMARY AND DISCUSSION
In this work, we formulated and studied the COIRL problem. We presented two types of algorithmss to solve it: (1) cutting plane methods (ellipsoid) and (2) iterative descent approaches (MDA and ES). We summarize the theoretical guarantees of the different algorithms in Table 1.
that the iterative descent approaches have better dependence in dk than the ellipsoid method, i.e., they scale better with the dimensions of the problem. In particular, the EW algorithm has logarithmic dependence in dk: which makes it computationally comparable to standard IRL/AL (on a single, noncontextual MDP). In addition, the iterative methods extend naturally to the more general scenario where the mapping from contexts to rewards is not linear, and fw modeled as DNN. As Sutton (2019) puts it: "The biggest lesson that can be read from 70 years of Al research that general methods that leverage computation are ultimately the most effective, and by large margin".
We
The ellipsoid method has better sample complexity (as a function c) than the descent methods in the deterministic setting. However, both methods attain the same complexity in the more realistic, stochastic setting. Our empirical findings suggest that the iterative methods always outperform the ellipsoid algorithm. Among these methods, we found the ES method to perform better than the MDA method. Similar findings were reported in (Nesterov & Spokoiny, 2017) for other convex problems.
The iterative methods have another advantage over the ellipsoid method they can learn from previously collected demonstrations (i.e., offline learning). The ellipsoid framework, on the other hand, requires expert supervision throughout the entire learning process.
Finally, an attractive property of the ellipsoid learning framework its safety, i..., an IRL algorithm that is being supervised by expert will never perform sub-optimally. In each step, either that the agent performs c-optimally or that the expert acts on its behalf (provides a demonstration), This property is appealing in mission-critical domains where errors have high cost; for instance, in health-care a failure may result in a loss of lives. In the experimental section, we have seen that we can use this learning framework for the iterative methods as well while enjoying improved efficiency.
Scalability
Sample Complexity
Extension to DNNs
Deterministic
Stochastic
Deterministic
Stochastic
PSGD
(dk)
MDA
EW
(log dk)
X
ES
(dd))
(
d
Ellipsoid
(d2k2)
(dP14)
x
Tog(1/e)
Table 1: Comparison between various approaches.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 1. ACM, 2004.
Kareem Amin, Nan Jiang, and Satinder Singh. Repeated inverse reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1815-1824, 2017.
Martin Arjovsky, Soumith Chintala, and Léon 1rX1::/701.07875. 2017.
Bottou.
Wasserstein gan. arXiv preprint
Amir Beck and Mare Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31:167-175, 2003.
Clark Berngard, Jeremy Beitler, and Atul Malhotra. Personalizing mechanical ventilation for acute respiratory distress syndrome. Journal of thoracic disease, 8(3):E17,, 2016.
Dimitri Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3): 334-334, 1997.
Stephen Boyd and Craig Barratt, Linear controller design: limits of performance. Prentice Hall Englewood Cliffs, NJ, 1991.
Sébastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends()) in Machine Learning, 8(3-4):231-357 2015.
Bibhas Chakraborty and Susan A Murphy. Dynamic treatment regimes. Annual review of statisties and its application, 1:447-464, 2014.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 854 863, JMLR. org, 2017.
Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision-based instruction following. arXiv preprint arXiv: 1rXiv:1902.07744 2019.
Omer Gottesman, Fredrik Johansson, Joshua Meier, Jack Dent, Donghun Lee, Srivatsan Srinivasan, Linying Zhang, Ding, David Wihl, Xuefeng Peng, et al. Evaluating reinforcement learning algorithms in observational health settings. arXiv preprint arXiv: 12298, 2018.
Omer Gottesman, Fredrik Johansson, Matthieu Komorowski, Aldo Faisal, David Sontag, Finale Doshi-Velez, and Leo Anthony Celi. Guidelines for reinforcement learning in healthcare. Nature medicine, 25(1):16-18, 2019.
Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv:1502.0223 2015.
Elad Hazan. Introduction to online convex optimization. Foundations and Trends in Optimization, 23555323 2016.
Theis Itenov, Daniel Murray, and Jens Jensen. Sepsis: Personalized medicine utilizing tonooottchnologiess-aaa paradigm shift? In Healthcare, pp. 111. Multidisciplinary Digital Publishing Institute, 2018.
Russell Jeter, Christopher Josef, Supreeth Shashikumar, and Shamim Nemati. Does the "artificial intelligence clinician" " learn optimal treatment strategies for sepsis in intensive care?, 2019. URL httpc///oooiiio///leeleeeeeeeeee
Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. Mimic-iii, freely accessible critical care database. Scientific Data, 3:160035, May 2016. ISSN 2052-4463.
doi: 10. 10.1003//daa... 2016.35. URL http:///k..bb..ogg/100888vsd...

---

Under review as a conference paper at ICLR 2020
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International conference on Machine learning, pp. 267-274, 2002.
Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time.
Machine learning, 49(2-3))209-22 2002.
Matthieu Komorowski, Leo A Celi, Omar Badawi, Anthony C Gordon, and A Aldo Faisal. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature Medicine, 24(1 1):1716, 2018.
Donghun Lee, Srivatsan Srinivasan, and Finale Deshi-Velez. Truly batch apprenticeship learning with deep successor features. arXiv preprint veXiv:1903.10077 2019.
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. Ai safety gridworlds. arXiv preprint trXiv:1711.098.0 2017.
James MacQueen et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statisties and probability, pp. 281-297. Oakland, CA. USA, 1967.
Aditya Modi, Nan Jiang, Satinder Singh, and Ambuj Tewari. Markov decision processes with continuous side information. In Algorithmic Learning Theory, pp. 597-618, 2018.
Arkadii Semenovich Nemirovsky and David Borisovich Yudin. Problem complexity and method efficiency in optimization. Wiley, New York, 1983.
Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization convex functions. Foundations of Computational Mathematics, 17(2):277-566, 2017.
Andrew Y Ng and Stuart J Russell. Algorithms for inverse reinforcement learning. In Icml, volume 1, pp. 2. 2000.
Niranjani Prasad, Li-Fang Cheng, Corey Chivers, Michael Draugelis, and Barbara E Engelhardt. A reinforcement learning approach to weaning of mechanical ventilation intensive care units. UAI, 2017.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 1994.
Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzyeh Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv preprint arXiv:1711.090 2017.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies scalable alternative to reinforcement learning. arXiv preprint arXiv: 2017.
Richard Sutton. The bitter lesson, March 2019, URL http://wwwwiiccoompeeeeeeeeseee/ Bister.esson.hmml.
Umar Syed and Robert Schapire. game-the approach to apprenticeship learning. In Advances in neural information processing systems, pp. 1449-1456, 2008.
EM Wesselink, TH Kappen, HM Torn, AJC Slooter, and WA van Klei. Intraoperative hypotension and the risk of postoperative adverse outcomes: a systematic review. British journal of anaesthesia, 2018.
Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, and Chelsea Finn. Learning prior over intent via meta-inverse reinforcement learning. arXiv preprint 1rXiv:1805.1257. 2018
Tom Zahavy, Alon Cohen, Haim Kaplan, and Yishay Mansour. Average reward reinforcement learning with unknown mixing times. arXiv preprint 10Xxv:19050097000 2019.
Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 928-936, 2003.
12

---

Under review as a conference paper at ICLR 2020
Appendices
CONTENTS
A Complementary simulations
14
A.1 Autonomous driving simulation
14
A.2 Dynamic treatment regime
14
Experimental details
15
B.1 Autonomous driving simulation
15
B.2 Dynamic treatment regime
16
C Proofs for Section 3
18
D Proofs & pseudo code for Section 3.3
20
D.I Ellipsoid Algorithm for trajectories
20
D.2 MVEE computation
20
D.3 Proof of Theorem
20
D.4 Proof of Theorem 4
Hardness learning contextual policies
23
13

---

Under review as a conference paper at ICLR 2020
A COMPLEMENTARY SIMULATIONS
A.1 AUTONOMOUS DRIVING SIMULATION
Similar to Section 4.2, compare the various methods the offline framework. We can see that all the algorithms manage to minimize the loss, and achieve near optimal value. We can also see that they achieve high accuracy with the expert policy but not 100%.
23
23:00'
10
30
30
(a) Loss
(b) Value
(c) Accuracy %
Figure 6: Offline framework, autonomous driving simulation, linear mapping
A.2 DYNAMIC TREATMENT REGIME
Similar to Section 4.1, we compare the various methods in the ellipsoid framework. We observe that ES outperforms the ellipsoid method. Additionally, we compare the accuracy, i.e., how often does the policy derived from W match the expert's ' policy, which is derived from W As IRL is not supervised learning problem, we observe that while there correlation between the success in the task and the ability to act similarly to the experts policy this correlation is not strict, in the sense that the agent capable of finding near-optimal policies with a relatively high miss rate (accuracy of approximately 80%). For more intuition, see the proof of Lemma 1.
250
Ellipsoid
0.020
Demonstrations
200
ES
80
Expert value
0.018
Ellipsoid value ES value
150
Value
100
Ellipsoid miss
40
miss
0.014
0.012
200
150
200
Time
Demonstrations
(a) demonstrations
(b) Value and miss
Figure 7: Ellipsoid framework, dynamic treatment regime, linear mapping
14

---

Under review as a conference paper at ICLR 2020
B EXPERIMENTAL DETAILS
In this section, we describe the technical details of our experiments, including the hyper-parameters used. To solve MDPs, we use value iteration. Our implementation is based on stopping condition with tolerance threshold, such that the algorithm stops if V4 V--1l < In the driving simulation we used 10-4 and in the sepsis treatment we use
10-3
B. AUTONOMOUS DRIVING SIMULATION
Colisions = Off-roads
Speed = Med.
Figure 8: Driving simulator
The environment is modeled tabular MDP that consists of 1531 states. The speed is selected once, at the initial state, and is kept constant afterward. The other 1530 states are generated by 17 X-axis positions for the agent's car, available speed values, lanes and 10 Y-axis positions which car B may reside. During the simulation, the agent controls the steering direction of the car, moving left or right, i.e., two actions.
In these experiments, we define our mappings in way that induces different behaviours for different contexts, making generalization a more challenging task. Specifically, for the linear setting we before normalization. For our nonlinear mapping, contexts with
0.75 0.75
W
0.5 0.75
||e|l>c 0.55 are mapped reward coefficients vector (1, 11,-11-0.055, otherwise they mapped 1, -)), which induce the feature expectations (9.75, 3.655, 5), (5.25, 5, 2.343) respectively. The decision regions for the nonlinear mapping are visualized in Appendix B.1. The contexts are sampled uniformly in the 2-dimensional simplex. We evaluate all algorithms on the same sequences of contexts, and average the results over 20 such sequences. The algorithms were modified to fit the Ellipsoid framework; instead of iterating over the whole data, the algorithms iterate over the given expert feature expectations, one at time, until convergence, i.c. every timestep a new demonstration presented.
1.0
0.8
0.6
0.4
0.2
0.0
0.0
0.
7.
4..
0.6
0.4
P.20
0.0
0.8,
0.8
Figure 9: Visualization of nonlinear decision boundaries
Hyper-parameter selection and adjustments:
For the linear model the algorithms maintained x 3 matrix estimate W' Ellipsoid: By definition, the ellipsoid algorithm is hyper-paramette free and does not require tuning.
Ellipsoid Framework:
PSGD: The algorithm was executed with with the parameters: ag = 0.3, a; = a. and iterated for 40 epochs. An outer decay on the step size was added for faster convergence, the initial oo
15

---

Under review as a conference paper at ICLR 2020
becomes 0.94 og every time demonstration is presented. The gradient, 9t is normalized to be 2t It [oll and the calculated step is taken if: (u(#)) - -m(=)) > cWe+1 (u(+t+')) - -=())))
9r =
where #2 denotes the optimal policy for a context according to Wt-
ES: The algorithm was executed with the parameters: o = 10-3 3 m = 250, a = 0.1 with decay rate of 0.95, for 50 iterations which didn't 't iterate randomly over one the contexts, but rather used the entire training set (all of the observed contexts and expert demonstrations up to the current time-step) for each step. The matrix was normalized according to || ll2. and was the step calculated by the ES algorithm, before it was multiplied by a and applied.
For the nonlinear setting, the model used for the ES method was fully connected DNN, with layers of sizes 15, 10, 5, The activation function used was the leaky ReLU function, with a parameter of = 0.1. Note that we can't 't normalize the parameters here as in the linear case; therefore an L2- 12-normalization layer is added to the output. The same parameters were used as in the linear case, except with 120 iterations over the entire training set. They were originally optimized for this model and setting and worked as-is for the linear environment. As we aim to estimate the gradient, a small o = 10-3 was used and performed best. The number of points, m = 250, was selected as fewer points produced noisy results. The step size, decay rate and the number of iterations were 0.1, 0.96, 120 respectively, and were selected in a way that produced fast yet accurate convergence of the loss. Note that here the steps were also normalized before application, and the normalization was applied per layer. For PSGD, a similar network was used. Specifically, it had layer sizes 14, 10, 6, 3, and the same leaky ReLU activation function was used in this network. In parallel to the normalization used for the ES model, here we used batch normalization and gradient clipping. The learning rate was set to 0.1 0.98* and 120 iterations were performed. For this result, as with the ES method, the batch consisted of all available training data.
Offiine Framework: In the offline framework we compute the subgradients using expert trajectories of length 40, instead of the feature expectations. In this framework at every iteration we sample a mini batch of 10 contexts (from a finite set) and their corresponding trajectories (sampled from the expert policy and dy dynamics) then taking one descent step according to them. The generalization measured over 80 holdout contexts that are referred as the test set, where the W that is used to calculate the feature expectations of the agent is fitted to the EW algorithm requirement to be in the e dk - simplex. The PSGD and the EW algorithms are configured as the theory specifics, where each descent step is calculated from the whole batch. The ES algorithm is applied with the parameters σ 10 3 mmm = 500, a = 0.1 with decay rate 0.95, for every iteration.
B.2 DYNAMIC TREATMENT REGIME
The environment we describe in 4.2 simulates a decision makonn process for treating sepsis. Sepsis is a life-threattning severe infection, where the treatment applied to sepsis patient is crucial for saving his life. To create a sepsis treating simulator, we leverage the MIMIC-III data set (Johnson et al., 2016). This data set includes data from hospital electronic databases, social security, and archives from critical care information systems, that had been acquired during routine hospital care. We follow the data processing steps that were taken in Jeter et al. (2019) extract the relevant data in form of normalized measurements of sepsis patients during their hospital admission and the treatments that were given to each patient. The measurements include dynamic measures, c.g., heart rate, blood pressure, weight, body temperature, blood analysis standard measures (glucose, albumin, platelets count, minerals, etc.), as well as static measures such as age, gender, re-admission (of the patient), and more.
The processed data from Jeter et al. (2019) consists of 5366 trajectories, each representing the sequential treatment provided by clinician to a patient. At each time-step, the available information for each patient consists of 8 static measurements and 41 dynamic measurements. In addition, each trajectory contains the reported actions performed by the clinician (the number of fluids and vasopressors given to a patient at each time-step and binned to 25 different values), and there is a mortality signal which indicates whether the patient was alive 90 days after his hospital admission.
In order to create a tabular CMDP from the processed data, we separate the static measurements of each patient and keep them the context. We cluster the dynamic measurements using K-means (MacQueen et al., 1967). Each cluster is considered a state and the coordinates of the cluster centroids taken as its features ((s). We construct the transition kernel between the clusters using the
16

---

Under review as a conference paper at ICLR 2020
empirical transitions in the data given the state and the performed actions. Two states added the MDP and the feature vector is extended by 1 element, corresponding to whether or not the patient died within the 90 days following hospital release. This added feature receives value of O on all non-terminal states, a value of -0.5 for the state representing the patient's 's death and 0.5 for the one representing survival. In addition, the data is limited, not all state-action pairs In order to ensure the agent does not attempt to perform such an action for which the outcome is unknown, add an additional terminal state. this state, all features are to -1 to make it clearly distinguishable from all other states in the CMDP.
available.
In our simulator, we used the same structure as the raw data, i.e., we used the same contexts prevalent in the data and the same imitial state distribution. Each context is projected onto the simplex and the expert's 's feature expectations for each context are attained by solving the CMDP. While we focus on simulator, as it allows us to analyze the performance of the algorithms, our goal is to have reward structure which influenced by the data. Hence, we produce W by running the ellipsoid algorithm on trajectories obtained from the data. As done in the Autonomous Driving Simulation, the algorithms were modified to fit the Ellipsoid framework.
Hyper-paramettr selection and adjustments:
Ellipsoid Framework: ES algorithm: the same method as in the autonomous driving is applied, with the parameters 10-4, 1000, 0.25 with decay rate 0.95, for 80 iterations over the entire training set.
In the offline framework for the linear setting, we compute the subgradients using expert trajectories of length 40. instead of the exact feature expectations. At every iteration, we sample mini batch of 10 contexts (from finite set) and their corresponding trajectories (sampled from the expert policy and dynamics) and perform single descent step. Generalization is measured of 300 holdout contexts, referred to the test set, where W is fitted be in the dk simplex. The PSGD and the EW algorithms are configured as specified by the theory, where each descent step calculated from the entire batch. The ES algorithm is applied with the parameters 10 4 m = 1000, a = 0.3 with decay rate 0.95, for every iteration.
Offiine Framework:
For the nonlinear setting. the model used for the ES method was fully connected DNN, with layers of sizes 24, 42, 42 which include a bias term. The activation function used was the leaky ReLU function, with parameter
0.1. In this setting we use trajectories of length 80, mini batch of 32 contexts. The PSGD algorithm uses the gradient computed by pack-propagation of the loss function value on the DNN, where each descent step calculated from the entire batch. The algorithm is applied with modified parameters; 0.1 with decay rate 0.98, for every iteration.
17

---

Under review as a conference paper at ICLR 2020
C PROOFS FOR SECTION 3
Definition (Bregman distance). Let bb W - R be strongly convex and continuously differential in the interior W. The Bregman distance defined by De(x,y) = (())-((())-- -VVVv)() Vvy(y, where Vb is strongly convex with parameter a.
Definition (Conjugate function). The conjugate of a function ((y), denoted by 30" (v) is
m {ar y (xx)).
Example: let
|| be a norm on R". The associated dual norm, denoted || ||+. is defined as || = sup (2Tx || S 1}. The dual norm of 2l2 is || 112, and the dual norm of || ||1 || llse-
Before we begin with the proof of Lemma we make the following observation. By definition, (W) is the optimal policy w.r.t. cf W; i.e., for any policy we have that
c² W ((F((W)) c W ((T))
Proof of Lemma 1.
Fix W. For any context C, we have that ((ffe) is the optimal policy w.r.t reward fw(c), thus, fw(() (u(++((W) p((=)) 0. Therefore get that Loss(W) 2 O. For W have that u(+o(WW) = p((,)), thus Loss(W")) O.
For the second statement, note that Los)(W) - 0 implies that Ve, fw (c) (u(ff -)(==)(())))))))))))))))))))-- - 0. This can happen in one of two cases. (1) μ (W)) = ((Tz), in this case mē: 4, (W) have the same feature expectations. Therefore, they equivalent in terms of value. (2) j= (W)) # u(m²),, but fw (p)) ((+²)) 0. In this case, n.. fte (W) have different feature expectations, but still achieve the same value w.r.t. reward fw(c). Since (W) is an optimal policy w.r.t. this reward, so is T.
Proof of Lemma 2.
1. We need show that WW,, W2 E W, [0, have that
Lin (AW; + (1 - 2) - XLin + (1- A) (W:) (W2)
Luin (AW, + (1
- -1-
2)W2)
= E. (AWW), + (1 A)WW))
-
m(m:))
AB. [TT (a(E + (1- (1=))W3)) m(T2a)]
+ +11- +11A))E, [c (u(#c(AW) + (1- - m((TT))
S AE ( (M(=. m(TT)) +1 (1)-))B I (Wa) -
= (W1) + (1 Li (W2),
where the inequality follows from Eq. (2).
2. Fix E W. We have that
(2n = Be (T (((f(())-<(TT)))
2 Ee [Tz (u(#t(W))-wi -
((({))))m)))) m(x2))]
Linn(WW) + (2-W), E, ooo -
where the inequality follows from Eq. (2).
Recall that a bound on the dual norn of the subgradients implies Lipschitz conti- nuity for convex functions. Thus it is enough to show that VW E W,llo(W)ll,
18

---

Under review as a conference paper at ICLR 2020
E |oo (u(ao(W)) - ((<=:) - 1| - L. Let = oc, we have that
llx = FOcO (m(#t((W)) m(x=))
S Eellco (u(+t(W)) m((T,)) 1
(Jensen inequality)
- Eellcloo (A(T,) ((TT) - 1 1-7
(3)
where in Eq. (3) we used the fact that VT we have that [|((
I r thus, for any Te. Kj+
[Im( - olo I I-7
2 Therefore, L I-y w.r.t I1 Since 11. S vol 11: we get that L 2y T-7 w.r.t. 11-112
19

---

Under review as a conference paper at ICLR 2020
D PROOFS & PSEUDO CODE FOR SECTION 3.3
D.1 ELLIPSOID ALGORITIM FOR TRAJECTORIES
Algorithm 4 Batch ellipsoid algorithm for COIRL
Initialize: Oo t Bo (0, 1) {x Rd* |ko S 1 θ < MVEE(
+ 0. < - 0
=, do
G3 is revealed, Let W, be the center of O Play episode using मं arg max of W.
Vi
O(+1 f O,
if a sub-optimal action a is played at state then
Expert provides H-step trajectory (sg of ..... R Let i i, be the H-step sample of the expert's feature expectations for E 1,: 2 = H (ss Let x; be the agent's 's feature expectations for E Pa, >
Denote 2, C. C Iz. 2i H.
4 2 a7-
i f i 1, Z Z + 2z; 7" t 2 + #/ #
if = then
{ |EE+: (o-W: W,
1 - 5 2)20))
0.+1 MVEE
i - 0, + 0, 0 0-
D.2 MVEE COMPUTATION
This computation is commonly found in optimization lecture notes and textbooks. First, define an ellipsoid by {x (x - c)Q (I - c) I 1} for a vector c, the center of the ellipsoid, and an invertible matrix Q. Our first task is computing θ the MVEE for the initial feasibility set Og = Bo (0, 1) = (a E Rddkk ||0||00 1). The result is of course a sphere around 0: c1 = 0, Q1 = dk.T.
For the update O.+1 + MVEE {oE O.: (o- W.)
T
ae 2 0 of)) we define ãr Vo7 VaiQiar Qiar at
and calculate the ellipsoid by c C. J6+I Qā, Q++1
J -x=II(Q: (Q:
2 Q,ā,aTQQ. Q.).
Jh+I
D.3 PROOF OF THEOREM 3
For simpler analysis, we define a "flattening" operator, converting a matrix to a vector: Rdx + Rd by W = [......... wi,kr..., Wd. W.,k]. We also define the operator C to be the composition of the flattening operator and the outer product: u C v = [uIV1,.... U1 1i.k..... Uddt,..... ... UdUa).
Therefore, the value of policy æ for context c is given by V = W u(=)) =
((T)),
where | 1, ||e ((T)]li S
&
Lemma 3 (Boyd& & Barratt (1991)). If R is ellipsoid with center w, and RPM
define B MV E B (0 - w)TT r 2 0}), then: Vol(B=)
Vol(B) < e -x0+1)
Proof of Theorem 3. We prove the theorem by showing that the volume of the ellipsoids θ for 1= 1, is bounded from below. In conjunction with Lemma 3, which claims there is a minimal rate of decay in the ellipsoid volume, this shows that the number of times the ellipsoid is updated is polynomially bounded.
We begin by showing that W always remains in the ellipsoid. We note that in rounds where VT V <-w > e, we have W-T ca c (u(TZ)= - m(+t))) > e. In addition, as the agent acts
optimally w.r.t. the reward ri = of Wr, we have that W ② (u(=ā) - u(ft)) < 0. Combining
20

---

Under review as a conference paper at ICLR 2020
these observations yield:
W W.) "T (m(rt) 2020.
(4)
This shows that W is never disqualified when updating Q Since W E Oo this implies that Vt W E Oe Now we show that not only W remains in the ellipsoid, but also a small ball
surrounding it. If 0 is disqualified by the algorithm: (0 (0-,)) -T, (ce @ (u(Tā) ) su(fte) 0.
Multiplying this inequality by and adding to (4) yields: (W" - 0)" 100-0)0-(0 (9o (go(((Ta)) - -m((tt)) > 6.
We apply Hölder inequality to LHS: c < LHS & I[ A 11 [1 1 (m(Taa) (u 111 5
2k W - Oz Therefore for any disqualified 0: || - 011s > (1-2le thus Bo ( (1-28
2h
is never disqualified. This implies that Vt vol(O,) 2 vol(@u n B (W", (2-2)e)
))
2
2h
(x (W", )). Finally, let Mr be the number of rounds by T in which V.rw
4k
Vēe w > Using Lemma 3 we get that: 2[dk+IJ
My S log (vol(@1)) log (vol(Or+1)) S
log (vol (MVEE( B (0, 1))) -log (oo ))) I log (vol vdk))))
log
- ))) s log (( joh) $ dk log 4ky/k (1-7)e
Therefore Mr S 2dk(dk
1) log 1kvdk
[1-7)c = (d k2 log( (1-7)e)
dk
D.4 PROOF OF THROREM 4
Lemma 4 (Azuma's inequality). For a martingale { if Si-1/ $ bas. for i = -,....
P Sn - Sol > by 2n log( 3)
< 6
| W1 2. If & Oo. update the ellipsoid by θ, MVEE { (0 O : (o (0-W; W. Eg S where ey is the indicator vector of coordinate in which W, exceeds 1, and the inequality direction depends on the sign of vy If W, & Og still, this process can be repeated for a finite number of steps until W, E Og. as the volume of the ellipsoid is bounded from below and each update reduces the volume (Lemma 3). Now we have W, E O,, implying I|W* Willoo I As no points of Oo are removed this way, this does not affect the correctness of the proof. Similarly, we may assume ||W;- W₁ lloo 2 2 as Wt E Og.
Proof of Theurem 4. We first note that we may assume that for any L:
We denote W; which remains constant for each update in the batch by W. We define f(i) the time- steps corresponding to the demonstrations in the batch for i 1, 1,... n. We define z-- to be the expected value of 2x Ht. and 2 to be the outer product of Cnj and the feature expectations of the expert policy for Wis C((e)) S1g3 We also denote W ((s) by W., We bound the following term from below, as in Theorem 3:
Z
Z
W
WTT
W
W)T,
(2 zz))
n
n1
11
W
W)T T
(#f
2,)
+
W W
(2,"1 -2,)+
1=1
1=1
(W*
WTT (8)
-2,
H
21

---

Under review as a conference paper at ICLR 2020
1
W
Wy
(2f
2)
W
W
(2f 21) +
i=1
(1)
(2)
(W" w)
(2] x.
27 ( w) (2) -, x.H
(3)
(4)
(1): Since the sub-optimality criterion implies difference in value of at least for the initial distribution which assigns to the state where the agent errs, we may use identical arguments to the previous proof. Therefore, the term is bounded from below by 6.
(2): By assumption ||W* Wil S the term is bounded by 1
(1-7)s thus since I1(2; - -2)1l1 S I-y by Hölder's inequality 8k
(3): We have || 2ill1 - I by"
by 1-7 from definitions, thus || 2/ I 1-7 since e E Ad-1- As mentioned previously we may assume I| Welloo I 2, therefore by Hölder's inequality the term bounded by & due to our choice of H: y
(1 7)) S ((1 (1 7)) - (1
)
sk {1-7)s ((1 7)) =-r) los( (1-TTT) Se logl (
sk
(4): The partial sums Σ (W* (W*-W)"
W)T (2, +H 21 H for N 0, ..., form a martingale sequence.
Note that ||2,"*11 [la I I-y; and || 11 S I-y Also, we have that || Wello S 2, thus, can apply Azuma's inequality (Lemma 4) with (1-7) and with our chosen this yields:
k
k.
E (W" WYT. (2 5x sa.H) S )
with probability of at least (dk 1) log( 10kw
& and as in Theorem 3 this shows Bo (W", (1-7)e 7)
Thus (W w/ -
is never
disqualified, and the number of updates is bounded by 2dk(dk + 1) log( 16kväk ), and multiplied by this yields the upper bound on the number of rounds in which sub-optimal action is chosen. By amion-bounde the required bound for term (4) holds in all updates with probability of at least 1 6.
22

---

Under review as a conference paper at ICLR 2020
E CONTEXTUAL POLICIES
Consider the following problem. Given a complete specification an MDP, and a hypothesis class H. for each state assign hypothesis ha - A such that the return is maximized.
The following theorem shows that it is NP-complete to find such policy. We will use the class of linear separators. We define the following contextual MDP training problem. We are given MDP with only transitions and sample of m contexts and their reward (namely for each context we specify the rewards for each state and action).
Theorem 5. There is reduction from training problem the the union of hyperplanes to the contextual
MDP training problem of le + states.
Proof. Consider the following MDP, which has two parameters ro and ri which will define the rewards for each context. The MDP composed from a line of A: internal states i, and one sink state. Each internal state E [1, k 1] has two actions: action leads to the sink state with reward rı and the action 0 lead to the next internal state i + 1 with reward O. In internal state & action leads to the sink state with reward r1 and action leads to the sink state with reward ro. In the sink state there is singe action that stays in the sink state and has reward O.
The context is C' = Rd The hypotheses class includes all hyperplanes, each hyperplane characterize by weight vector
Rd. and if w 2 0 then the action is 1 and otherwise is O.
Given a sample of size m to the training problem (c,, y4). we generate rewards for the contextual MDP training problem by having the context 6; and specifying the parameters ro and rı. Specifically, we will set ro = 1 y/1 and rı yi-
Given l hyperplanes wi:-... Wk: and example, assume that they make errors. By using those k hyperplanes in the & internal nodes for each context e; we have reward iff the union of the hyperplanes classify it correctly. This implies that we have reward m This is since each example that is labeled correctly will get reward of and each incorrect example will get a reward of 0.
Given an assignment of k hyperplanes to the internal nodes, which gets return of m e. we output as hypothesis the union of the k hyperplanes, Again, the number of errors on the sample is exactly errors.
23


Reviews:
[{"review_id": "r1euwGVP9B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a formulation for the contextual inverse reinforcement learning (COIRL) problem and proposed three algorithms for solving the proposed problem. Theoretical analysis of scalability and sample complexity are conducted for cases where both the feature function and the context-to-reward mapping function are linear. Experiments were conducted in both a simulated driving domain and a medical treatment domain to compare the three proposed algorithms empirically. Empirical results for using a deep network as the contextual mapping function is also provided.\n\nAs a special case of IRL for POMDPs, the contextual IRL problem (with latent contexts) is an interesting research topic that is of interest to researchers working on generalizing IRL to a greater range of real-world applications. This paper was written with clarity and detailed descriptions of experiments. The authors presented their algorithms with thorough theoretical analysis. However, further ablation study and proofs are needed to demonstrate the proposed problem formulation and algorithms outperform existing IRL frameworks.\n\nThe authors motivate the problem of COIRL from the formulation of contextual MDP by Hallak et al. (2015). However, the referenced work of Hallak et al. (2015) (published as a preprint on arXiv) does not provide a sufficient argument/evidence for why or in what cases this particular formulation, especially when the context is observed (as assumed by this paper), is better than alternative ones, such as directly modeling the context as part of the state. When a discrete number of contexts exist (as in the simulated driving domain: average vehicle v.s. ambulance), the proposed problem can be reduced to solving a set of normal IRL problems, i.e. learning a reward function under each observed context. For continuous context variables (such as age and weight for patients in the medical treatment domain), an intuitive solution is to model them as part of the state/feature and run normal IRL algorithms. However, the authors did not analyze or show in either of their experiments how existing IRL algorithms compare with the proposed algorithms. Without further explanation, the authors claim, in the fourth paragraph of section 1, that “..Apprenticeship Learning algorithms that try to mimic the expert cannot be used..”, yet it is not trivial to understand why the proposed formulation and algorithms would outperform existing methods/baselines. \n\nLastly, it would be more insightful to the readers if the authors can provide some discussion on how they would extend their algorithms to the more interesting/practical case where the context is not directly observed and analyze how latent contexts would affect the performance/complexity of their proposed algorithms.\n\nOverall, this is a well-written paper on an exciting research topic but lacks sufficient analysis and experimental results to support the significance of the intended contributions.\n\n\nReference (from the paper): \n\nHallak, A., Di Castro, D., & Mannor, S. (2015). Contextual Markov decision processes. arXiv preprint arXiv:1502.02259.\n\n\n==================================\n\n\nEdit after rebuttal period: \n\n- The updated version of the paper includes preliminary results of comparisons with alternative methods and shows that COIRL does have better scalability for modeling problems with large number of observable contexts. It would be great if the authors can show how the baseline performs in the two original experiments as well to understand the improvements gained from adopting COIRL for these problems.\n\n- As stated before, I think COIRL is an exciting research topic on its own. While many recent works on IRL focuses on generalizing to complex problem with deep neural nets as function approximation tools, it is important to analyze the underlying problems from bottom up (in smaller domains and with assumptions that may not always hold) in order to better understand the limits of different solutions. I'd certainly love to see more follow-up works on this topic and how it can be extended to more complex problems.\n\n- Hence, after carefully reading the authors responses and the updated version of the paper, I changed my decision to 'Weak Accept'.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlQiXlTYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work focuses on the problem of 'contextual' inverse reinforcement learning, where the reward is a function of the current state of the MDP, and a set of context features, which remain constant within each episode.  The primary contribution of this work is the formulation of inverse reinforcement learning (for restricted spaces of context-dependent reward functions) as a convex optimization problem.  Based on this formulation, the paper describes several IRL algorithms based on approaches to solving convex and non-convex optimization problems, including variations of mirror descent, and evolution strategies (in principle allowing for the optimization of reward functions with arbitrary parametric representations).  The algorithms presented in this work all assume that computing an optimal policy for a specific reward function is a relatively inexpensive subroutine, which limits their applicability to domains where such planning is straightforward.  Experimental results are presented for a simple highway driving domain, as well as a simulated patient treatment domain constructed from real-world clinical data.\n\nWhile the paper focuses one contextual IRL as its key contribution, the paper fails to sufficiently motivate the contextual IRL problem as a useful specialization of inverse reinforcement learning.  While there are clearly problems where a distinct 'context' can be identified, it isn't clear what advantage is gained by considering the context as a separate set of features, as opposed to treating the context as an additional set of state features which do not change within an episode.  With the latter formulation, many existing approaches to inverse reinforcement learning or imitation learning would be applicable.  The specific formulation of the reward as the product of the context and state features is somewhat more flexible than limiting the reward to linear functions of both, but it would be straightforward to apply the proposed algorithms to the case where the state and context are concatenated, and the reward is a quadratic function of state-context vector.\n\nThe experimental results are not particularly useful in evaluating the proposed algorithms, as the tasks involved are relatively simple, discrete-state MDPs (with continuous state features), and more importantly, no comparisons with existing IRL approaches are provided.  The potential scalability of evolution strategy optimization to more complex, non-convex reward representations such as deep networks is mentioned, but never empirically demonstrated.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BklJpy7Jqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors consider the problem of inverse reinforcement learning for CMDPs in which the reward function is a function of the context. They  propose different algorithms for learning the function and evaluate their algorithms on a driving simulator and a sepsis treatment problem (based on real data from the MIMIC corpus).\n\nI think this is a good paper, studying an interesting problem and proposing useful solutions, so it should be accepted. The paper could be improved by putting more focus on the presentation of the studied problem and its variants (also the ones the authors mention their approach is easy to generalize to) and reducing a little the focus on the algorithm. In total they propose three algorithms and give convergence guarantees for all of them. Of course this analysis is important but I found it somewhat distracting from the main flow of the paper. Maybe these guarantees could be moved into a separate section.\n\nA few more points:\n* Please comment on why you see estimating contextual transition dynamics as an orthogonal problem.\n* In the last paragraph of the conclusion you talk about a safety critical application. For which applications do you think this is practical? I would assume that constantly reviewing of an AI systems' actions is very impractical. (But I agree on these aspects being important.)\n* One of the main limitations of this work seems to be that the CMDP\\M has to be known. Please comment on how one could expand the analysis/applications/experiments to extend to the case where the CMDP is not known.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce a framework for inverse reinforcement learning tasks whose reward functions are dependent on context variables and provide a solution by formulating it as a convex optimization problem.  Overall, the authors agreed that the method appears to be sound.  However, after discussion there were lingering concerns about (1) in what situations this framework is useful or advantageous, (2) how it compares to existing, modern IRL algorithms that take context into account, and (3) if the theoretical and experimental results were truly useful in evaluating the algorithm.  Given that these issues were not able to be fully resolved, I recommend that this paper be rejected at this time.
Decision: Reject