Causal Imitation Learning via Inverse Reinforcement Learning
Authors: Kangrui Ruan
Junzhe Zhang
Xuan Di
Elias Bareinboim
Venue: ICLR 2023 poster (2023)
Keywords: Causal Inference
Graphical Models

Abstract: One of the most common ways children learn when unfamiliar with the environment is by mimicking adults. Imitation learning concerns an imitator learning to behave in an unknown environment from an expert's demonstration; reward signals remain latent to the imitator. This paper studies imitation learning through causal lenses and extends the analysis and tools developed for behavior cloning (Zhang, Kumor, Bareinboim, 2020) to inverse reinforcement learning. First, we propose novel graphical conditions that allow the imitator to learn a policy performing as well as the expert's behavior policy, even when the imitator and the expert's state-action space disagree, and unobserved confounders (UCs) are present. When provided with parametric knowledge about the unknown reward function, such a policy may outperform the expert's. Also, our method is easily extensible and allows one to leverage existing IRL algorithms even when UCs are present, including the multiplicative-weights algorithm (MWAL) (Syed & Schapire, 2008) and the generative adversarial imitation learning (GAIL) (Ho & Ermon, 2016). Finally, we validate our framework by simulations using real-world and synthetic data.

Published as a conference paper at ICLR 2023
CAUSAL IMITATION LEARNING VIA INVERSE REINFORCEMENT LEARNING
Kangrui Ruan", Junzhe Zhang", Xuan Di, and Elias Bareinboim Columbia University, New York, NY 10027, USA (kr2910, junzhez, sharon. el a @co umbi a eddu
ABSTRACT
One of the most common ways children learn when unfamiliar with the environment is by mimicking adults. Imitation learning concerns an imitator learning to behave in an unknown environment from an expert's 's demonstration; reward signals remain latent to the imitator. This paper studies imitation learning through causal lenses and extends the analysis and tools developed for behavior cloning (Zhang, Kumor, Bareinboim, 2020) to inverse reinforcement learning. First, we propose novel graphical conditions that allow the imitator to learn a policy performing as well as the expert's 's behavior policy, even when the imitator and the expert's state-action space disagree, and unobserved confounders (UCs) are present. When provided with parametric knowledge about the unknown reward function, such a policy may outperform the expert's. Also, our method easily extensible and allows to leverage existing IRL algorithms even when UCs are present, including the multiplicativeweiihhh algorithm (MWAL) (Syed & Schapire, 2008) and the generative adversarial imitation learning (GAIL) (Ho & Ermon, 2016). Finally, we validate our framework by simulations using real-world and synthetic data.
INTRODUCTION
Reinforcement Learning (RL) has been deployed and shown to perform extremely well in highly complex environments in the past decades (Sutton & Barto 1998; Mnih al. 2013 Silver et al.) 2016, Berner et al. 2019). One of the critical assumptions behind many of the classical RI. algorithms that the reward signal is fully observed, and the reward function could be well-specified. In many real-world applications, however, it might be impractical to design a suitable reward function that evaluates each and every scenario (Randløv & Alstrom 1998 Ng et al. 1999). For example, in the context of human driving, it is challenging to design precise reward function, and experimenting in the environment could be ill ill-advised; still, watching expert drivers operating is usually feasible.
In machine learning, the imitation learning paradigm investigates the problem of how an agent should behave and learn in an environment with an unknown reward function by observing demonstrations
from a human expert (Argall et al.) 2009 Billard et al.; 2008 Hussein et al.) 2017] Osa et al. (2018) 8).
There are two major learning modalities that implements IL- behavioral cloning (BC) (Widrow
1964 Pomerleau 1989 Muller et al.) 2006; Mülling et al. 2013; Mahler & Goldberg 2017) and inverse reinforcement learning (IRL) Ng et al. (2000); Ziebart et al. (2008); Ho & Ermon (2016);
Fu et al. (2017). BC methods directly mimic the expert's behavior policy by learning mapping from observed states to the expert's action via supervised learning. Alternatively, IRL methods first learn potential reward function under which the expert's behavior policy is optimal. The imitator then obtains a policy by employing standard RI. methods to maximize the learned reward function. Under some common assumptions, both BC and IRL are able to obtain policies that achieve the expert's 'ss performance (Kumor et al. 2021; 202I:SSammy Swamy et al. 202T). Moreover, when additional parametric knowledge about the reward function is provided, IRL may produce a policy that outperforms the expert's 's in the underlying environment (Syed & Schapire 2008 Li et al. 2017 Yu et al., 2020).
For concreteness, consider a learning scenario depicted in Fig. Ia describing trajectories of human- driven cars collected by drones flying over highways (Krajewski et al. 2018 Etesami & Geiger| 2020). Using such data, we want to learn a policy X <- x(Z) deciding on the acceleration (action) X E
Equal contribution.

---

Published as a conference paper at ICLR 2023
UL
(a)
(b)
(d)
Figure 1: Causal diagrams where X represents action (shaded red) and Y represents latent reward (shaded blue). Input covariates of the policy scope S are shaded in light red.
{0, 13 of the demonstrator car based on velocities and locations of surrounding cars. The driving performance is measured by latent reward signal Y. Consider an instance where Y + (1 X)
( 2) and values of Z are drawn uniformly over {0. 1). A human expert generates demonstrations
following behavior policy such that P(X = 1 Z = 0) 0.6 and P(X = 0 Z =
0.4.
Evaluating the expert's 's performance gives E[[] = P(X = 1, Z = + P =
1) P
0,
0.5.
Now we apply standard IRL. algorithms learn a policy X +- T()) that the imitator's driving performance, denoted by E|Y do(z)). is at least as good as the expert's performance E|y]. Detailed derivations of IRL policy are shown in (Ruan et al. 2023 Appendix A). Note that B |2, x]
+ 2 2rz belongs to a family of reward functionn Jr 2) = Q.T - 82 - YIz, where 0 < a a<< typical IRL. imitator solves a minimax problem min, max fr E [X,, 2) -E [fx 2) do(x)) The inner step "guesses" a reward function being optimized by the expert; while the outer step learns policy maximizing the learned reward function. Applying these steps leads to a policy x² : X + -Z with the expected reward E do(TT)] = 1, which outperforms the sub-optimal expert.
Despite the performance guarantees provided by existing imitation methods, both BC and IRL rely on the assumption that the expert's ' input observations match those available to the imitator. More recently, there exists an emerging line of research under the rubric of causal imitation learning that augments the imitation paradigm to account for environments consisting of arbitrary causal mechanisms and the aforementioned mismatch between expert and imitator's 's sensory capabilities (de Haan al., [019 Zhhngg et al. [2020] Etesami & Geiger 2020 Kumor al., (021). Closest to
work, /Zhng et al. (2020): Kumor el al. (2021) derived graphical criteria that completely characterize when and how BC could lead successful imitation even when the agents perceive reality differently.
Still, it is unclear how to perform IRL-type training if some expert's 's observed states remain latent to the imitator, which leads to the presence of unobserved confounding (UCs) in expert's demonstrations. Perhaps surprisingly, naively applying IRL. methods when UCs are present does not necessarily lead to satisfactory performance, even when the expert itself behaves optimally.
To witness, we now modify the previous highway driving scenario to demonstrate the challenges of UCs. In reality, covariates Z (i.e., velocities and location) are also affected by the car horn U, of surrounding vehicles and the wind condition U2. However, due to the different perspectives of drones (recording from the top), such critical information (i.e, U1, U2) ) is not recorded by the camera and thus remains unobserved. Fig. [b graphically describes this modified learning setting. More specifically, consider an instance where Z + U1 @ U2. Y < -X 4 Z @ U2; O is the exclusive-or operator; and values of U1 and U2 are drawn uniformly over (0. An expert driver, being able to hear the car horn U1, follows a behavior policy X +- Un and achieves the optimal performance E[Y] = 1. belongs to family of reward functions fy (z., 2) = a (where > 0). Solving min, max L [X, X 2)] - IE [X, 2) do(TT)] leads to an IRL policy πt with expected reward EE (d( = 0.5, which far from the expert's optimal performance [E[]] = 1.
Meanwhile, observe that B| [2,
After all, a question that naturally arises is, under what conditions an IRL imitator procedure can perform well when UCs are present, and there a mismatch between the perception of the two agents? In this paper, we answer this question and, more broadly, investigate the challenge of performing IRL through causal lenses. In particular, our contributions are summarized as follows. (1) We provide a novel, causal formulation of the inverse reinforcement learning problem. This formulation allows one to formally study and understand the conditions under which an IRL policy is learnable, including in settings where UCs cannot be ruled out priori. (2) We derive new graphical condition for deciding whether an imitating policy can be computed from the available data and knowledge, which provides robust generalization of current IRL algorithms to non-Markovian settings, including GAIL (Ho & Ermon (016) and MWAL (Syed & Schapire 2008). (3) Finally, we move beyond this graphical condition and develop an effective IRI. algorithm for structural causal models (Pearl 2000) with
2

---

Published as a conference paper at ICLR 2023
arbitrary causal relationships. Due to the space constraints, all proofs are provided in (Ruan et al.) 2023 Appendix B). For a more detailed survey on imitation learning and causal inference, we refer readers to (Ruan et al. 2023] Appendix E).
1.1 PRELIMINARIES
We use capital letters to denote random variables (X) and small letters for their values (z). X represents the domain of X and Px the space of probability distributions over 2x. For set X, let X denote its dimension. The probability distribution over variables X is denoted by P(X).
Similarly, P(Y X) represents a set of conditional distributions P(Y X = x) for all realizations
a. We use abbreviations P(x) for probabilities P(X = æ); so does P(Y y X = x) P(y
2t. Finally, indicator function 1 IZZ- = }} returns if Z = z holds true; otherwise
The basic semantic framework of our analysis rests on structural causal models (SCMs) (Pearl 2000 Ch. 7). An SCM M is a tuple (UP)) with V the set of endogenous, and U exogenous variables. F is set of structural functions s.t. for fv E F. V + (v(pay, uy) with PAv S V. Uv S U. Values of U are drawn from an exogenous distribution P(U), inducing distribution P(V) over endogenous variables V. Since the leamne can observe only subset of endogenous variables, we split V into a partition O U L where variable O S are observed and L = V O remain latent the leaner. The marginal distribution P(O) is thus referred as the observational distribution. An atomic intervention on subset X S V. denoted by do(x), is an operation where values of X are set to constants æ. replacing the functions fx {fx V- } that would normally determine their values. For an SCM M. let M, be a submodel of M induced by intervention do(xr). For set Y S V, the interventional distribution P(s]do(x:)) induced by do(x)) defined as the distribution over in the submodel Ma, i.e., PM (Y (do(x)) A PM, (Y). We leave M implicit when it is obvious from the context.
Each SCM M is associated with causal diagram 9 which a directed acyclic graph where (e.g.. see Fig. D solid nodes represent observed variables O dashed nodes represent latent variables L. and arrows represent the arguments PAy of each function fv E F. Exogenous variables U are not explicitly shown; bi-directed arrow between nodes V; and V, indicates the presence of an unobserved confounder (UC) affecting both V; and Vj. We will use family abbreviations to represent graphical relationships such as parents, children, descendants, and ancestors. For example, the set of parent nodes of X in G is denoted by pa( (X)s = Uxe x x X)x; ch, de and an are similarly defined. Capitalized versions Pa, Ch, De, An include the argument as well, e.g. Pa(X) gg = (x(X), U X. For a subset X C V, the subgraph obtained from G with edges outgoing from X incoming into X removed is written as Gx/Gx respectively. 9[x] is subgraph of G containing only nodes X and edges among them. A path from a node X to node Y in G is a sequence of edges, which does not include a particular node more than once. Two sets of nodes X, Y are said to be d-separated by a third set Z in DAG G, denoted by (X 1 Y 2) g. if every edge path from nodes in X to nodes in Y is "blocked" by nodes Z. The criterion of blockage follows (Pearl 2000 Def. 1.2.3). For a more detailed survey on SCMs, we refer readers to (Pear 2000 Bareinboim et al. 2022
2 CAUSAL INVERSE REINFORCEMENT LEARNING
We investigate the sequential decision-making setting concerning a set of actions X, series of covariates Z, and a latent reward Y in an SCM M. An expert (c.g., a physician, driver), operating in SCM M, selects actions following behavior policy, which is the collection of structural functions fx = {fx X E X}. The expert's performance is evaluated as the expected reward E|Y]. On the other hand, a learning agent (i.e., the imitator) intervenes on actions X following an ordering
X, <. < X,; each action X; is associated with a set of features PA C (X;), J. A policy π over actions X a sequence of decision rules π = (m ). Each decision rule T. (;(X; Z,)
is probability distribution over action X, E X, conditioning on values of set of covariates
Z: CPA, Such policies π are also referred to as dynamic treatment regimes (Murphy et al. 2001 Chakraborty & Murphy 2014), which generalize personalized medicine time-varying treatment settings in healthcare, in which treatment is repeatedly tailored to a patient's 's dynamic state.
A policy intervention actions X following a policy π, denoted by do(rr), entails a submodel M from a SCM M where structural functions fx associated with X (i.c., the expert's 'ss behavior policy) are replaced with decision rules X; Tx(X; Zz) for every X; E X. A critical assumption
3

---

Published as a conference paper at ICLR 2023
throughout this paper is that submodel Mr does not contain any cycles. Similarly, the interventional distribution P(V do(T)) induced by policy и is defined as the joint distribution over V in Mn
Throughout this paper, detailed parametrizations of the underlying SCM M are assumed to be unknown to the agent. Instead, the agent has access to the input: (1) a causal diagram & associated with M, and (2) the expert's 's demonstrations, summarized as the observational distribution P(O). The goal of the agent is to output an imitating policy π that achieves the expert's performance.
Definition 1. For an SCM M = (U, V F, P(U)), imitating policy Te" is policy such that its expected reward is lower bounded by the expert's 's reward, i.e., E (Y do( "]] 2 EM
In words, the right-hand side is the expert's performance that the agent wants to achieve, while the left-hand side is the real reward experienced by the agent. The challenge in imitation learning arises from the fact that the reward Y is not specified and latent, i..., Y & O. This precludes approaches that identify EE [d directly from the demonstration data (e.g., through the do- or soft-do-calcllus
Pearl (2000) Correa & Bareinboim (0200).
There exist methods in the literature for finding an imitating policy in Def. Before describing their details, we first introduce some necessary concepts. For any policy и., we summarize its associated state-action domain using a sequence of pairs of variables called a policy scope S.
Definition 2 (Lee& & Bareinboim 20200). For an SCM M, a policy scope S (for short, scope) over actions X is a sequence of tuples (X,, ) where Z: C PA, for every X; E X.
We will consistently use π S to denote a policy π associated with scope S. For example, consider a policy scope S = {X1, [21]), (X2,
Zz } over actions X1. in Fig. Ic A policy π ~ S is a { Z1), (x Z2)}.
sequence of distributions
Zhang et al. (2020): : Kumor et al. (222)) provide graphical condition that is sufficient for learning
an imitating policy via behavioral cloning (BC) provided with a causal diagram G. For policy scope S - ix Z.) 1 let g())
.....,,... denote manipulated graph obtained from Ç by the (1) remove arrows coming into every action Xy; and (2) add direct arrows from nodes in Z, to X,. Formally, the sequential r--bakdoor criterion is defined as: Definition 3 (Kumor et al. (021h). Given a causal diagram G, policy scope S {X,, Z,) 1 said to satisty the sequentaal ग backdoor criterion in G (for short, b-backdoor admissible) if at each X, X, one of the following conditions hold: (1) X, is not an ancestor of Y in g(), i.c. X & An(Y) or (2) Z, blocks all backdoor path from X, to Y in g(i,, i.e. (Y 1 X, Z,) in L
following steps: for all j = i++ 1.
(ummor et al. 2021) showed that whenever a r-backdoor admissible scope S is available, one could learn an imitating policy π S by setting ni (x; 2.) P(zz | 2,) for every action X; X.
For instance, consider the causal diagram G in Fig. Ic Scope S = {(1,, { [Z1 }), (X2, [22))} is YIZz) hold G, which a super graph thus obtainable by setting (X,, 2,) P(x, Zi) and (X2 Z2) P(X, Z2). While impressive, a caveat of their results that the performance of the imitator is restricted by that of the expert, i.e., B|Y| do(>")) [ In other words, causal BC provides an efficient way to mimic the expert's performance. If the expert's ' behavior is far from optimal, the same will hold for the learning agent.
--backdoor admissible since (X, IL Y Z1) and (X2 2
containing both manipulated (1) and g(2). An imitating policy π = {Ti. n
2.1 MINIMAL SEQUENTIAL BACKDOOR CRITERION
To circumvent this issue, we take a somewhat different approach to causal imitation by incorporating the principle of inverse reinforcement learning (IRL) principle. Following the game-theoretic approach (Syed& & Schapire| 2008). we formulate the problem as learning to play two-player zero-sum game in which the agent chooses a policy, and the nature chooses an SCM instance. A key property of this algorithm is that it allows us to incorporate prior parametric knowledge about the latent reward signal. When such knowledge informative, our algorithm is about to obtain a policy that could significantly outperform the expert with respect to the unknown causal environment, while at the same time are guaranteed to be no worse. Formally, let = [VM GM = G, PM (O) = P(O)) denote the set of SCMs compatible with both the causal diagram G and the observational distribution P(O). Fix a policy scope S. Now consider the optimization problem defined as follows.
1²
min max EM [Y] - EAs [Y] do(m))].
(I)
R~ ME.K
4

---

Published as a conference paper at ICLR 2023
The inner maximization in the above equation can be viewed an causal IRL step where attempt to "guess" worst-case SCM M compatible with 9 and P(O) that prioritizes the expert's policy. That is, the gap in the performance between the expert's and the imitator's policies is maximized. Meanwhile, since the expert's reward E [Y] is not affected by the imitator's policy л, the outer minimization is equivalent to a planning step that finds a policy π optimizing the learned SCM M. Obviously, the solution π is an imitating policy if gap V' = 0. In cases where the expert is sub-optimal, i.c. Exg [ < Ex do( for some policies æ,, we may have 1 O. That is, the policy x² will dominate the expert's policy fx regardless of parametrizations of SCM M in the worst-case scenario. In other words, πT' to some extent ignores the sub-optimal expert, and instead exploits prior knowledge about the underlying model.
Despite the clear semantics in terms of causal models, the optimization problem in Eq. ① requires the learner to search over all possible SCMs compatible with the causal diagram G and observational distribution P(O). In principle, it entails a quite challenging search since one does not have access to the parametric forms of the underlying structural functions F nor the exogenous distribution P(U). It is not clear how the existing optimization procedures can be used.
In this paper, we will develop novel methods to circumvent this issue, thus leading to effective imitating policies. Our first algorithm relies on a refinement of the sequential x b-backooor, based on the concept of minimality. A subscope S' of policy scope S = {(X,, Zi) }" denoted by S' C S, is sequence {(x,, }+)=== where Z S Z; for every X, X. A proper subscope S' c S is subscope in S other than S itself. The minimal r-backdoor admissible scope is defined as follows. Definition& 4. Given causal diagram G, a π- x-backdoor admissible scope S is said to be minimal if there exists no proper subscope S' c S satisfying the sequential π n-backdoor in G.
Theorem 1. Given a causal diagram G, if there exists minimal r-backdoor admissible scope S = ((X,, 2.) in G, consider the following conditions:
1. Let effective actions X X n An os and effective covariates Z" Uxiex Z;i [VX; E X
2. For =1= 1....,, n
1, let X 21
[j 1 < i) and Z'z Ux,ex; Zj.
Then, for any policy π S, the expected reward E|Y do( is computable from P(O,Y) as:
B|y do( = Σ E|y x", x²) I (a="""""
(2)
where the occupancy measure (x", 2²) II Ixxxxx P (zz x2 2 Te (x; 21).
To illustrate, consider again the causal diagram G in Fig. lc the manipulated diagram ((2) G and g(1) is obtained from G by removing Ze ++ X2. While scope S = ((x, (21)). (X2, (2))) satisfies the sequential x rbbcckdoor, it not minimal since (X, IL Y) in (() On the other hand,
S2 = ((X1, 0), (X2, {22))} is minimal x- r-backdoor admissible since (X, I LY) 72) holds true in xx and the covariate set { {22} } is minimal due to the presence of the backdoor path X₂ + Zz + Y.
Let us focus on the minimal π bacckoo admissible scope Sz. Note that Gs, is a subgraph obtained from G by removing the bi-directed arrow Z2 ++ X2. We must have effective actions X {X,, X2) and effective covariates 2 = {Zz]. Therefore, Z = Z = 0 and Z = [Z For any policy ~ S2, Thm. [ iimplies E do( TT)
-x,,, .2 E[y 22, 22] P(z2 (r) (1 ( (z)) On
the other hand, The same result in Thm. dooes not necessarily hold for non-minimal π b-backooor ad-
missible scope. For instance, consider again the non-minimal scope S = ((X1. (Zz)), (X2, (Z2))).
The expected reward E|y | do(r)) of policy
S2 is not computable from Eq. ( and is
ultimately not identifiable from distribution P(O, Y) in g (Tian) 2008
2.2 IMITATION VIA INVERSE REINFORCEMENT LEARNING
Once minimal T- backkooo admissible scope S is found, there exist effective procedures to solve for an imitating policy in Eq. {D.. Let A be a hypothesis class containing all expected rewards E [Y
x", x²,="] compatible with candidate SCMs M E M, i.e., & = (E (Y x", 2=] VM E Al).. Ap- plying the identification formula in Thm. reduces the optimization problem in Eq. ① as follows:
min max
Σ 15"
r(0", ;0) (p(x", ",,",,,",)) x²) Pm (x",
(3)
S TES
5

---

Published as a conference paper at ICLR 2023
where the expert's is occupancy measure p(z",zz) P(z", .=²) and the agent's occupancy measure Pr (x", xx*) is given by Eq. (2). The above minimax problem is solvable using standard IRL algorithms. The identification result in Thm. J that the learned policy applies any SCM compatible with the causal diagram and the observational data, thus robust to the unobserved confounding bias in the expert's demonstrations. Henceforth,
will consistently refer to Eq. (3) as the canonical equation of causal IRL. In this paper, we solve for an imitating policy π in Eq. (3) using state-of-the
IRL algorithms, provided with common choices of parametric reward functions. These algorithms include the multiplicative-weight algorithm (MWAL) (Syed & Schapire 2008) and the generative adversarial imitation learning (GAIL) (Ho & Ermon 2016). We refer readers to Algs. 3 and 4 in (Ruan et al. 2023 Appendix C) for more discussions on the pseudo-code and implementation details.
Causal MWAL (bbeel & Ng 2004 Syed Schapire, 2008) study IRL in Markov decision expectations processes w where the reward function r(x", x²" is a linear combination of k-length
vectors <xx",, )² Particularly, let r(=" 2²) = un Φ(x=" ,²²) for coefficient vector wr contained in convex set Sh = { E R ||]]|] = and w = 0}. Let (5(+) be the i-th component of feature
vector Φ and let deterministic policies with scope S be ordered by T(1), ,r(1,...,, .....,,(() (m). The canonical equation in Eq. 3 reducible a two-person zero-sum matrix game under linearity.
Proposition 1. For a hypothesis class At {r w Φ w s. the solution v of the canonical equation in Eq. is obtainable by solving the following minimax problem:
v² min max w T GT., S wES
(4)
where G is a k x matrix given by G(i,)) = Emr... "..
(p(x", (x* xx)).
There exist effective multiplicative weights algorithms for solving the matrix game in Eq. including MW (Freund & Schapire 1999) and MWAL (Syed & Schapire 2008).
Causal GAIL (Ho & Ermon 2016) introduces the GAIL algorithm for learning an imitating policy in Markov decision processes with a general family of non-linear reward functions. In particular, r(x", =²) takes values in the real space R, i.e., r E R . where X z = {rr 2x X Oz + R). The complexity of reward function is penalized by a convex regularization function ('(r), i.e.
min max X
Σ r(x",=" (p(æ"; x²) Px (x", =²)) (r) 20
(5)
Henceforth, we will consistently refer to Eq. (5) as the penalized canonical equation of causal IRL. is often preferable to solve its conjugate form. Formally,
Proposition 2. For a hypothesis class A = fr: 2x x 2z. + IR} regularized by ib, the solution
v' of the penalized canonical equation Eq. (5) obtainable by solving the following problem:
v" min 30 (p px))
(6)
where 4²* be a conjugate function of va and is given by vh"" = maN (br).
Eq. (6) seeks a policy ा which minimizes the divergence of the occupancy measures between the imitator and the expert, as measured by the function v*. The computational framework of
generative adversarial networks (Goodfellow et al. (0014) provides an effective approach to solve
such a matching problem, e.g., the GAIL algorithm (Ho Ermon 2016).
CAUSAL IMITATION WITHOUT SEQUENTIAL BACKDOOR
In this section, we investigate causal IRL beyond the condition of minimal sequential bacckooo Observe that the key to the reduction of the canonical causal IRL equation in Eq. (3) lies in the identification of expected rewards E[Y do(rr) had the latent reward Y been observed. Next will study general conditions under which EE|Y | )] is uniquely discernible from distribution P(O, Y) in the causal diagram G, called the identifiability of causal effects (Pearl 2000 Def. 3.2.4).
Definition 5 (Identifiability). Given causal diagram G and a policy ~ S, the expected reward E|Y do(x)) ] is said to be identifiable from distribution POO,Y) computable from P(O, Y) in any SCM M compatible with G.
Y) ) in G if E|Y do(T)) is uniquely

---

Published as a conference paper at ICLR 2023
We say a policy scope S is identifiable (from P(O.Y) Y) in 9) if for all policies π ~ S, the correspond- ing expected rewards EE[ do( are identifiable from P(O,Y) in G. Our next result shows that whenever an identifiable policy scope S is found, one could always reduce the causal IRL problem to the canonical optimization equation in Eq. (3)
Theorem 2. Given a causal diagram G, a policy scope is identifiable from P(O,Y) in G if and only if for any policy π S, the expected reward EE[Y do(TT)] is computable from P(O, Y) as
E|y I do(TT) = Σ E =" (xn 2="
where subsets X" Z' X; and the imitator's occupancy measure pr 2²) is function of the observational distribution P(O) and policy π.
Thm. 2 Suggests a general procedure to learn an imitating policy via causal IRL.. Whenever identifiable scope S is found, the identification formula in Eq. 7 permits one to reduce the optimization problem in Eq. ① to the canonical equation in Eq-
(a)
S by solving Eq. (3) where the expert's occupancy measure p(z" 22² = P(x", x²)
One could thus obtain imitating policy
is given by Eq. 77. As an example, consider the frontdoor diagram described in Fig. 2aland a policy scope
and the imitator's occupancy measure p <x",, , 2²)
(b) Gs
Ex E|y doo(2)]=<zz) Figure 2: Frontdoor
{ (X, 0) ]. The expected reward E|Y do(x)]
and E do(x')) is identifiable from P(X, Y, Z) using the frontdoor adjustment formula (Pearl
2000
Thm. 3.3.4). The expected reward E[Y do TT) of any policy (X) could be written as:
E|Y | do([)]]
Σ E x. P(x()) Σ P(2)=")=(='.
(8)
Let occupancy measures p(z, 2) = P(x,,) and pT 2) = E. ((a)=")<(''). We could thus learn an imitating policy in the frontdoor diagram by solving the canonical equation given by:
Σ Er(x,2) (o(æ, 2) - Pm pm(x, (x, z)),
min max
T~8 TE
where A is a hypothesis class of the reward function r(æ, =) - E| xt =]. The solution x ='() is an imitating policy performing at least as well as the expert's behavior policy if the gap D² < 0.
Next, we will describe how to obtain the identification formula in Eq. 7 provided with an identifiable scope S. Without loss of generality, we will assume that the reward Y is the only endogenous variable that is latent in the causal diagram G, i.e., V O U (Y) We will utilize special type of clustering of nodes in the causal diagram G. called the confounded component (for short, c-component).
Definition 6 (C-component (Tian & Pearl 2002)). For a causal diagram G, a subset C C V is a c-component if any pair V, V/J E C is connected by bi-directed path in G.
For instance. the frontdoor diagram in Fig. 2a contains two e-components C - (X. Y and C2 (Z). We will utilize a sound and complete procedure IDENTIFY (Tian) 2002 2008 for identifying causal effects [Y[Y | do(r)) of an arbitrary policy ~ S. Particularly, IDENTIFY takes as input the causal diagram G. a reward Y, and a policy scope S. It returns an identification formula for E|Y do(T))] from P(O, Y) if expected rewards of all policies ~ S are identifiable. Otherwise, IDENTIFY(G, Y, = "FAIL". " Details of IDENTIFY are shown in (Zhang et al. 2020 Appendix
B). Recall that Gs is the causal diagram of submodel M2 induced by policy π ~ S. 1.Fig. 2b shows
diagram GS obtained from the frontdoor graph 9 and scope S = {XX, 0) described in Fig. 2a Let
Zy AY(Y) ) be ancestors of Y in Gs. Our next result shows that IDENTIFY (, is ensured to find an identification formula of the form in Eq. 7 when it is identifiable.
Lemma 1. Given a causal diagram G, a policy scope S is identifiable from P(O,Y) Y) in G if and only if IDENTIFY S) F "FAIL". Moreover, IDENTIFY (G, Y,S) returns an identification formula of the form in Eq. ( where X = Pa(Cy) ) n X and Z = Pa(Cy) ([Y) U X); and Cy is a c-component containing reward Y in subgraph [[An(Zr)]
"Otherwise, one could always simplify the diagram & and project other latent variahles L \ (Y) using the projection algorithm Tian 2002 Sec. 4.5), without affecting the identifiability of target query F.Y do(r)))
7

---

Published as a conference paper at ICLR 2023
For example, for the frontdoor diagram 9 in Fig. 2a the manipulated diagram Is with scope S
{(X, } is described in Fig. 2b) Since Zy = An gg = YY), Cy is thus given by Y). Lem. limplies that X
(X, {X] [X] and Z = Pa((X, {x, Y {2}.
Applying IDENTIFY Y, [(x, ()) returns the frontdoor adjustment formula in Eq.
3.1 SEARCHING FOR IDENTIFIABLE POLICY SCOPES
The remainder of this section describes an effective algorithm to find identifiable policy scopes S had the latent reward signal Y been observed. Let S denote the collection of all identifiable policy scopes S from distribution P(O, Y) in the causal diagram G. Our algorithm LISTIDSCOPE, deseribed in Alg. D enumerates elements in S. It takes as input a causal diagram G, a reward signal Y. and subsets L D and R = U PAT. More specifically, LISTIDSCOPE maintains two scopes St C S, (Step 2). It performs backtrack search to find identifiable scopes S in G such that S S S S S,. It aborts branches that either (1) all subscopes in S, are identifiable (Step 3); or (2) all subscopes containing S are non-identifiable (Step 6). The following proposition supports our aborting criterion.
Lemma 2. Given a causal diagram G. for policy scopes S' C S, S' is identifible from distribution P)O,Y) in if is identifiable from P) in G.
At Step 7, LISTIDSCOPE picks an arbitrary vari- able V that is included in input covariates R but not in L. It then recursively returns all identifi- able policy scopes S in G: the first recursive call returns scopes taking V as an input for some actions X; E X and the second call return all scopes that do not consider V when selecting
Algorithm 1: LISTIDSCOPE
1: Input: g, and subsets C
2: Output: set of identifiable policy scopes S
3: Let scopes S, = {[X,,, Rn PA 1 and
St = {(X,, n )=
4: if IDENTIFY (g, Y, S,) # "FAIL" then
5: Output S,..
values for all actions X. We say a policy π associated with a collection of policy scopes
6: end if
7: if IDENTIFY (G, Y, Si) o "FAIL" then
S. denoted by π ~ S, if there exists S E S so that π S. is possible to show that LIs- TIDSCOPE produces collection of identifiable scopes that sufficient for the imitation task.
8: Pick an arbitrary V E R R L.
9: LISTIDSCOPE G,Y, L U (V), R).
10: LISTIDSCOPP((C Y, L, R (V)).
11: end if
Theorem 3. For a causal diagram G and re- ward Y. LISTIDSCOPE(C, Y. Y.O.U PAA) enumerates a subset S* there is π S* where B|| dd((T)] = E do(TT)).
C S so that for any π ~ S,
Moreover, LISTIDSCOPE outputs identifiable policy scopes with polynomial delay. This follows from the observation that LISTIDSCOPE. searches over a tree of IU PAT| and IDENTIFY (G,Y)) terminates in polynomial steps w.r.t. the size of diagram G.
policy scopes with height at most
4 EXPERIMENTS
In this section, we demonstrate our framework on various imitation learning tasks, ranging from synthetic causal models to real-world datasets, including highway driving (Krajewski et al.) 2018) and images (LeCun) 1998). We find that our approach is able to incorporate parametric knowledge about the reward function and achieve effective imitating policies across different causal diagrams. For all experiments, we evaluate our proposed Causal- -RL based on the canonical equation formulation in Eq. (3). As baseline, we also include: (1) standard BC mimicking the expert's t's nominal behavior policy: (2) standard IRL utilizing all observed covariates preceding every X, E X while being blind to causal relationships in the underlying model; and (3) Causal-3C (Zhang et al., 2020 Kumor al. [2021) > that learn imitating policy with the sequential --backdoor criterion. We refer readers to (Ruan et al. 2023 Appendix D) for additional experiments and more discussions on the experimental setup.
Backdoor Consider an SCM instance compatible with Fig. Ic including binary observed vari- ables Z1, X1, Zz, X2, Y E (0. Causal-BC utilizes sequential π -backdoor admissible scope {(X1, (2+)), (X2, (22])}: while Causal-IRI utilizes the scope ( (X,, 0), (X2, {Zz))} satisfying the minimal sequential r-backdoor. Simulation results, shown in Fig. 3al reveal that Causal- aall-IIL consistently outperforms the expert's policy and other imitation strategies by exploiting additional parametric knowledge about the expected reward E X1, X2, Ze) ): Causa BBC is able to achieve the expert's performance. Unsurprisingy, neither BC nor IRL is able to obtain an imitating policy.
8

---

Published as a conference paper at ICLR 2023
1.0
1.0
1.0
1.0
Expert
Expert
Expert
Expert
0.8
0.8
0.8
0.8
0.6
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
IRl.
Causall
Causal AL
1RL.
Causail BC
Causal IRL
8C
IRL.
Causai BC
Causal IRL
181.
Caucal
Causal IRL
(a) Backdoor
(b) Highway Driving
(c) MNIST Digits
(d) Infinite MDPUC
Figure 3: Simulation results
a
d for our experiments, where y-axis represents the expected
reward of learned policies in the actual causal model; the grey dashed line denotes the expert's reward.
Highway Driving We consider a learning scenario where the agent learns a driving policy from the observed trajectories of human expert. Causal diagram of this example is provided in (Ruan et al.. 2023 Appendix D, Fig. 4) where X1 is the accelerations of the ego vehicle at the previous step; Zi both longitudinal and lateral historical accelerations of the ego vehicle two steps ago; X₂ is the velocity of the ego vehicle; Z2 is the velocity of the preceding vehicle; W indicates the information from surrounding vehicles. Values of X1, X2. Z,, Zg are drawn from a real-world driving dataset HighD Krajewski et al. (018). The reward Y is decided by non-linear function fr (X2, Zz, Uy Both Causal IIRL and Causa BC utilize the scope { {X,, 0). (X2, {Z Causal-IR. also exploits the additional knowledge that the expected reward B[Y |X1, X2, Z2] monotone function via reward augmentation (Li et al. 2017). Simulation results are shown Fig. 3b We found that Causal-IRL performs the best among all strategies. Causal-BC is able achieve the expert's s performance. BC and IRL perform the worst among all and fail to obtain an imitating policy.
MNIST Digits Consider again the frontdoor diagram in Fig. 2a To evaluate the performance of our proposed approach in high-dimensional domains, we now replace variable Z with sampled images drawn from MNIST digits dataset (LeCun) 1998). The reward Y is decided by linear function taking& 2 and an unobserved confounder Ux,y as input. The Causal - -IRL formulates the imitation problem as two- two-person zero-sun sum game through the frontdoor adjustment described in Eq. (2) which can be solved by the MW algorithm (Freund & Schapire 1999 Syed & Schapire 2008). j. As shown in Fig. [c] simulation results reveal that Causal IRL outperforms Causal BC and BC; while IRL performs the worst among all the algorithms.
Infinite MDPUC To demonstrate our proposed framework in the sequential decision-makinn setting with an infinite horizon, we consider a generalized Markov decision process incorporating unobserved confounders (Ruan & Di, 2022). called the MDPUC (Zhang & Bareinboim 2022). This sequential model simulates real-world driving dynamics. By exploiting the Markov property over time steps, sub-graphs, one for each time step i = 1, 2,....
able decompose the causal diagram over the infinite horizon into a collection of ... Fig. Id shows the causal diagram spanning time steps i 1,2, As comparison, BC and IRL still utilize the stationary policy {(Xi, { (Z:)). By applying
Thm. [ at each time step, we obtain a π -bbackdoor admissible policy scope {X,, [Z,.X;- X;-1. Z,-1h) for Causal- IRL and Causal- BBC, Simulation results are shown in Fig. [3] One could see by inspection that Causa IRL performs the best and achieves the expert's performance.
5 CONCLUSION
This paper investigates imitation learning via inverse reinforcement learning (IRL) in the semantical framework of structural causal models. The goal is to find effective imitating policy that performs at least as well as the expert's ''s behavior policy from combinations of demonstration data, qualitative knowledge the data-generating mechanisms represented as causal diagram, and quantitative knowl- edge about the reward function. We provide a graphical criterion (Thm. [T] based on the sequential backdoor, which allows one obtain an imitating policy by solving a canonical optimization equation of causal IRL. Such a canonical formulation addresses the challenge of the presence of unobserved confounders (UCs), and is solvable by leveraging standard IRL algorithms (Props. and 2 Finally, we move beyond the backdoor criterion and show that the canonical equation is achievable whenever expected rewards of policies are identifiable had the reward also been observed (Thms.) and 3)

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This research was supported in part by the NSF, ONR, AFOSR, DoE, Amazon, JP Morgan, and The Alfred P. Sloan Foundation.
ETHICS STATEMENT
This paper investigates the theoretical framework of causal inverse RL from the natural trajectories of an expert demonstrator, even when the reward signal unobserved. Input covariates used by the expert to determine the original values of the action are unknown, introducing unobserved confounding bias in demonstration data. Our framework may apply to various fields reality, including autonomous vehicle development, industrial automation, and chronic disease management. positive impact this work is that we discuss the potential risk of training IRL policy from demonstrations with the presence of unobserved confounding (UC). Our formulation of causal IRL is inherently robust against confounding bias. For example, solving the causal IRL problem Eq. (T) requires the imitator learn effective policy that maximizes the reward in worst-case causal model where the performance gap between the expert and imitator is the largest possible. More broadly, automated decision systems using causal inference methods prioritize safety and robustness during their decision-making processes. Such requirements are increasingly essential since black-box AI systems prevalent, and our understandings of their potential implications still limited.
REPRODUCIBILITY STATEMENT
The complete proof of all theoretical results presented in this paper, including Thms. | and 2
is provided in (Ruan et al. 2023 Appendix B). Details on the implementation of the proposed algorithms included (Ruan et al. 2023 Appendix C). Finally, (Ruan et al. 2023 Appendix D) provides a detailed description of the experimental setup. Readers could find all appendices as part of the supplementary text after "References" section. We provided references all existing datasets used in experiments, including HIGHD (Krajewski et al. 2018) and MNIST (LeCun 1998). Other experiments are synthetic and do not introduce any new assets. Source codes for all experiments and simulations are released in the complete technical report (Ruan et al. 2023).
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference Machine learning, pp. 1. ACM, 2004.
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-455 483, 2009.
Elias Bareinboim, Juan D. Correa, Duligur Ibeling, and Thomas Icard. On Pearl's Hierarchy and the Foundations of Causal Inference, pp. 507-556. Association for Computing Machinery, New York, NY, USA, 1st edition, 2022.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning, arXiv preprint arXiv: 191.v:1912.06680 2019.
Aude Billard, Sylvain Calinon, Ruediger Dillmann, and Stefan Schaal. Survey: Robot programming by demonstration. Handbook of robotics, 59, 2008
Bibhas Chakraborty and Susan Murphy. Dynamic treatment regimes. Annual review of statisties
and its application, 1:447, 2014.
J. Correa and E. Barcinboim. A calculus for stochastic interventions: Causal effect identification and surrogate experiments. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, New York, NY, 2020. AAAI Press.
Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion imitation learning. In Advances in Neural Information Processing Systems, pp. 11693-1 1174, 2019.
10

---

Published as a conference paper at ICLR 2023
Jalal Etesami and Philipp Geiger. Causal transfer for imitation learning and decision making under sensor-shift. Proceedings of the 34th AAAI Conference on Artificial Intelligence, New York, NY, 2020. AAAI Press
Yoav Freund and Robert E Schapire. Adaptive game playing using multiplicative weights. Games and Economic Behavior, 29(1-2):79-103, 1999.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforee- ment learning. arXiv preprint arXiv: 12248, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems. pp. 2672-2680, 2014.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):135, 2017
Robert Krajewski, Julian Bock, Laurent Kloeker, and Lutz Eckstein. The highd dataset: A drone dataset of naturalistic vehicle trajectories on german highways for validation of highly automated driving systems. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 2118-2125, 2018. doi: D0.1109/TSC.2218..8.8 201.85955....
Daniel Kumor, Junzhe Zhang, and Elias Bareinboim. Sequential causal imitation learning with unobserved confounders. Advances in Neural Information Processing Systems, 2021.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmist// 1998.
Sanghack Lee and Elias Bareinboim. Characterizing optimal mixed policies: Where to intervene and what to observe. Advances in Neural Information Processing Systems, 33, 2020.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. Advances in Neural Information Processing Systems, 30, 2017.
Jeffrey Mahler and Ken Goldberg. Learning deep policies for robot bin picking by simulating robust grasping sequences. In Conference on robot learning, pp. 515-524, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Joannis Antonoglon, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Urs Muller, Jan Ben, Eric Cosatto, Beat Flepp, and Yann Cun. Off-road obstacle avoidance through end-to-end learning. In Advances in neural information processing systems, pp. 739-746, 2006.
Katharina Mülling, Jens Kober, Oliver Kroemer, and Jan Peters. Learning to select and generalize striking movements in robot table tennis. The International Journal of Robotics Research, 32(3): 263-279, 2013.
S A Murphy, M J van der Laan, J M Robins, and Conduct Problems Prevention Research Group. Marginal Mean Models for Dynamic Regimes. Journal of the American Statistical Association, 96 (456):1410-142. December 2001.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pp. 278-287 1999.
Andrew Y Ng. Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, volume 1, pp. 663-670, 2000.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 7(1-2): 1-179, 2018.
11

---

Published as a conference paper at ICLR 2023
Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, New York, 2000.
Dean A Pomerleau. Alvinm: An autonomous land vehicle in a neural network. In Advances in neural information processing systems, pp. 305-313, 1989.
Jette Randløv and Preben Alstrøm. Learning to drive bicycle using reinforcement learning and shaping. In ICML, volume 98, pp. 463-471. Citeseer, 1998.
Kangrui Ruan and Xuan Di. Learning human driving behaviors with sequential causal imitation learning. Proceedings of the AAAI Conference on Artificial Intelligence, 36(4):4583-459) Jun. 2022. doi: 10.1609aaaii 2033. htthttts::/// /RRL:ttpp..//j...aaal..oo hwtp///ndxxxxce php.AAxTA
article/view/20S5
Kangrui Ruan, Junzhe Zhang, Xuan Di, and Elias Barcinboim. Causal imitation learning via inverse reinforcement learning. Technical Report R-89, Causal Al Lab, Department of Computer Science, Columbia University, 2023.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484449 2016.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 1998.
Gokul Swamy, Sanjiban Choudhury, Andrew Bagnell, and Steven Wu. Of moments and matching: A game-theoretic framework for closing the imitation gap. In International Conference on Machine Learning, pp. 10022-10033 PMLR, 2021.
Umar Syed and Robert E Schapire. game-theoretic approach to apprenticeship learning. In Advances in neural information processing systems, pp. 1449-1456, 2008
J. Tian. Studies in Causal Reasoning and Learning. PhD thesis, Computer Science Department, University of California, Los Angeles, CA, November 2002.
Jin Tian. Identifying dynamic sequential plans. In Proceedings of the Twenty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI'08, pp. 554-561, Arlington, Virginia, United States, 2008. AUAI Press. ISBN 0-9749039-4-99
Jin Tian and Judea Pearl. A general identification condition for causal effects. In Proceedings of the Eighteenth National Conference on Artificial Intelligence (AAAI 2002). pp- 567-573, Menlo Park, CA, 2002. AAAI Press/The MIT Press.
B Widrow. Pattern- recoeniin control systems. Compurter and Information Sciences, 1964.
Xingrui Yu, Yueming Lyu, and Ivor Tsang, Intrinsic reward driven imitation learning via generative model. In International Conference on Machine Learning, pp.
PMLR, 2020. J. Zhang and E. Bareinboim. Can humans be out of the loop? Technical Report R-64, Causal Artificial Intelligence Lab, Columbia University, 2020. Also, to appear: Proc. of the 1st Conference on Causal Learning and Reasoning (CLeaR), 2022.
J. Zhang, D. Kumor, and F. Bareinboim. Causal imitation learning with unobserved confounders. Advances in Neural Information Processing Systems, 33: 13:12233-12277 2020.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1430 Chicago, IL, USA. 2008
12


Reviews:
[{"review_id": "eGBm7mAiZo", "reviewer": "Reviewer_Ne7b", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new inverse reinforcement learning algorithm that's able to provably meet the performance of an expert demonstrator in the presence of uncontrolled confounders, when certain conditions are met (i.e., does the agent have the causal structure of the data generating process right?).", "summary_of_the_review": "An excellent submission to the study of causal reinforcement learning.  Please open source it :)", "strength_and_weaknesses": "Strengths: The submission does a fantastic job laying the groundwork for causal reinforcement learning, motivating its algorithms, theorems, and results within the formalisms of structured causal models.   I also greatly appreciated the extensive FAQ in the appendix.  The experiments clearly demonstrate the power of the method.\n\nWeakness: While the exposition of the paper is exceptionally high quality, there does not appear to be a code repo associated with the submission.  This method, and its uptake by the community, would greatly benefit from open sourcing of code / experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S94ZR-Le0t", "reviewer": "Reviewer_iWpt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a framework for causal imitation learning. The problem assumes that the experts has access to latent features that are not directly observable to imitators. The solution is based on identifying the minimal \\pi-backdoor admissible scope from the causal diagram. Experiments on a few datasets are provided to validate the effectiveness of the proposed approach. ", "summary_of_the_review": "The paper proposes an imitation learning framework, where the imitator can learn policies from observations with unobserved confounding. The solution is based on identifying identifiable policy scope from a causal diagram of the underlying decision process. Experiments on a few datasets are conducted to validate the effectiveness of the proposed solution. \n\nThe problem discussed in this paper is interesting, important, and technically challenging. However, paper fail to provide a clear formulation of the problem (input/output/objective, MDP formulation). The presentation, in particular the non-rigorous use of  variables and concepts make it hard to follow the details of the paper. The solution lacks the steps to complete the imitation learning loop. Experiments can be strengthened by using more relevant datasets and evaluating on additional measures (e.g., fidelity of learned polices). ", "strength_and_weaknesses": "Strengths:\n\n1. The problem described in the paper is meaningful and challenging. \n2. Theoretical proofs for theorems are provided. \n\nWeaknesses:\n\n1. The paper lacks a clear problem formulation. The input/output and objectives of the problem are not provided. \n2. The paper uses many different variables, where some of them are either undefined, or repeatedly used for different purposes. The presentation makes it hard to follow the details of the paper. \n3. The paper seems to be incomplete. The only algorithm only shows how to identify identifiable policy scope. However, how to use this algorithm with IRL or GAIL is not provided. \n4. The experiments are not convincing. Why would the MNIST dataset be used, and how to learn a policy from this dataset?\n5. How is the problem setting in this paper different from a partially observable MDP (POMDP)? More discussions are needed. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RRwnhiiDUJM", "reviewer": "Reviewer_U7KX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors considered the problem of imitation learning when the underlying causal structure of the environment is given. They provided a causal formulation of the problem of inverse RL (IRL) (equation (2)). They introduced the notion of minimal $\\pi$-backdoor admissible scope and showed that the effect of such policies on the average reward can be computed from the observational distribution over $(O,Y)$, i.e., the observable variables and the reward. Based on this, a canonical equation for IRL is given in (4) where for two settings of MWAL and GAIL, this formulation is simplified. Then, the authors used a method called IDENTIFY which is beyond the graphical condition in $\\pi$-backdoor admissible scope and it is sound and complete for checking whether a policy scope is identifiable from $P(O,Y)$. Finally, they proposed LISTIDSCOPE to enumerate all identifiable policy scopes.", "summary_of_the_review": "The submitted paper shows the advantage of considering the underlying causal diagram in the problem of inverse RL and provides a method to enumerate all identifiable policy scopes. Based on this, the canonical equation of causal IRL in (4) can be computed for such policies. The experiments showed that causal IRL can have better performance than the expert. The main ambiguity in the paper is for the case $\\nu^*>0$ and it is required to justify why the expert is not acting optimally even at least one SCM $M$.  ", "strength_and_weaknesses": "Strength:\n\n- The authors provided a nice formulation of inverse RL in the case that we have access to the causal graph. Moreover, they also gave some examples of this formulation for MWAL and GAIL.\n- They presented a method that can enumerate all identifiable policy scopes from $P(O,Y)$.\n\nWeakness:\n\n- Regarding the gap $\\nu^*$, it is not clear why the authors only considered the cases that $\\nu^*\\leq 0$. This can only happen if the expert has suboptimal performance in the environment (or in causal language, in some SCM $M$) that is acting. \n\n\nIn the following, I give my detailed comments:\n- It would be great if the authors can give some real cases that the performance of an expert is suboptimal even in the environment that is acting in it. The example in the introduction is not convincing enough as it is not clear why the expert acts based on just $X$ which results in poor performance.\n- The authors considered a ``sequential decision-making\" setting however, it seems that $Y$ is not indexed with time steps. How can the results in the paper be extended to the more general setting where we have a sequence of $Y_i$'s? In the FAQ, it is mentioned that $Y$ can be a set of variables but it might be the case that the cumulative reward is identifiable while each $Y_i$ is not identifiable from $P(O,Y)$.\n- Under what conditions, is $\\nu^*$ positive?\n- What is the exact definition of effective actions and covariates in Theorem 1? What happens to other variables $\\mathbf{X}$ and $\\mathbf{Z}$ in Theorem 1?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "78dVmc6zqb", "reviewer": "Reviewer_o8Xw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies imitation learning through the lens of structural causal models. Specifically, this paper proposes graphical conditions to enable imitator to learn an expert-level policy, when unobserved confounders (UCs) occur. Interestingly, the proposed method can extends to existing IRL algorithms when UCs are present. Two showcased algorithms are causal version of multiplicative weights algorithm (MWAL) and generative adversarial imitation learning (GAIL). The proposed methods have been evaluated using real-world and synthetic data, showing supervior performance than the original ones.", "summary_of_the_review": "Overall, this paper provides a novel causal inverse reinforcement learning method. The quality is good.", "strength_and_weaknesses": "## Strength\n\n+ The proposed method is novel. Under the structural causal model of inverse reinforcement learning, this paper extends the graphical condition for causal behavioral cloning to causal inverse reinforcement learning. This was done by formulating the problem as learning to paly a two-player zero-sum game, where the prior knowledge about the latent rewards can be incorporated. With informative knowledge, the imitator can outperform the expert in causal environment. \n\n+ The proposed method is well motivated with theoretical analysis, and naturally extended to two existing inverse reinforcement learning algorithms, i.e. MWAL and GAIL. The proposed method was well supported by the empirical results.\n\n## Weakness\n\n+ More diverse and challenging experiments are expected. \n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster