Causal Imitation for Markov Decision Processes: a Partial Identification Approach
Authors: Kangrui Ruan
Junzhe Zhang
Xuan Di
Elias Bareinboim
Venue: NeurIPS 2024 poster (2024)
Keywords: Causal Inference
Imitation Learning
Primary area: causal_inference
TL;DR: This paper presents novel causal imitation learning algorithms that adapt to confounded expert demonstrations in MDPs using partial identification techniques, demonstrating their effectiveness theoretically and empirically across various scenarios.

Abstract: Imitation learning enables an agent to learn from expert demonstrations when the performance measure is unknown and the reward signal is not specified. Standard imitation methods do not generally apply when the learner and the expert's sensory capabilities mismatch and demonstrations are contaminated with unobserved confounding bias. To address these challenges, recent advancements in causal imitation learning have been pursued. However, these methods often require access to underlying causal structures that might not always be available, posing practical challenges.
In this paper, we investigate robust imitation learning within the framework of canonical Markov Decision Processes (MDPs) using partial identification, allowing the agent to achieve expert performance even when the system dynamics are not uniquely determined from the confounded expert demonstrations. Specifically, first, we theoretically demonstrate that when unobserved confounders (UCs) exist in an MDP, the learner is generally unable to imitate expert performance. We then explore imitation learning in partially identifiable settings --- either transition distribution or reward function is non-identifiable from the available data and knowledge. Augmenting the celebrated GAIL method (Ho \& Ermon, 2016), our analysis leads to two novel causal imitation algorithms that can obtain effective policies guaranteed to achieve expert performance.

Causal Imitation for Markov Decision Processes: a Partial Identification Approach
Kangrui Ruan" Columbia University kr29100columbiee edu
Junzhe Zhang" Syracuse University j jzhan4030syr: edu
Xuan Di
Elias Bareinboim Columbia University eb0cs columbia. edu
Columbia University sharon. di@columbia, edu
Abstract
Imitation learning enables an agent to learn from expert demonstrations when the performance measure is unknown and the reward signal is not specified. Standard imitation methods do not generally apply when the learner and the expert's 's sen- sory capabilities mismatch and demonstrations confounding bias. To address these challenges, recent advancements in causal imitation learning have been pursued. However, these methods often require access to underlying causal structures that might not always be available, posing practical challenges. In this paper, investigate robust imitation learning within the frame- work of canonical Markov Decision Processes (MDPs) using partial identification, allowing the agent achieve expert performance are not uniquely determined from the confounded expert demonstrations, Specifi- cally, first, we theoretically demonstrate that when unobserved confounders (UCs) exist in an MDP, the learner is generally unable to imitate expert performance. We then explore imitation learning partially identifiable settings either transi- tion distribution or reward function is non-identifiable from the available data and knowledge. Augmenting the celebrated GAIL method (Ho Ermon, 2016), our analysis leads to two novel causal imitation algorithms that can obtain effective policies guaranteed to achieve expert performance.
contaminated with unobserved
when the system dynamics
Introduction
Children often learn how to behave in an unfamiliar environment by imitating adults. Imitation learning (IL) enables learning agent behave in unknown environment by observing expert demonstrations. It provides a viable approach for policy learning from demonstrations when the
reward function is not fully known and reward signals not specified 2 30 20
31 Imitation
learning has been widely applied across disciplines, such as autonomous driving
39 robotics 40 4T. and chronic disease management
B3
[88 natural language processing [1
52 44
has been acknowledged the literature that imitation learning could face significant challenges when unob- served confounding bias in expert demonstrations cannot be ruled out priori [17] (77 27 42). For illustration with simplicity, consider a Multi-Armen Bandit (MAB) model 28 described in Fig. 0 X {0, 1) is binary action, and
(a) P(X,Y)
(b) (X,,)
Figure I: A multi-armed bandit model.
'Equal contribution; Kangrui: work performed while Columbia University.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Identifiable Reward
Non-Identfiable Reward
Identifiable Transition
Standard IRI. (e.g., GAII [19]
CAIL-R (Alg. in Sec. 3.1b
Non-Identfiable Transition
CAIL-T (Alg. in Sec. 3.2)
Inimitable (Thm.
in Sec. 2.1
Table I: Summary of main contributions in this paper, including the analysis and proposed algorithms.
Y is the reward; U is latent covariate (to the imitator) uniformly drawn over a binary domain {0, 1}. Values of the reward are decided by reward function Y <- X @ U where @ is "xor" operator. An expert demonstrator, having access to covariate U. selecting action based on an expert policy X + -U. Evaluating the expert's performance gives [E[Y] = E[-U @ U] = 1. On the other hand, an imitator, mimicking the expert's behavior, will follow a policy π ()) = P(X)= = 0.5, selecting action uniformly at random. Evaluating the imitator's performance gives E [Y] Σ T (x))B(BUU)
0.5,
far from the expert's 's performance E[y] = 1.
Causal Inference (CI) addresses the challenges of unobserved confounding bias within the observa- tional data 32
leverages causal knowledge integral to the data generation process, typically More recently, incorporating causal inference methods into the imitation learning paradigm, causal imitation learning has evolved into a critical area of research [16] 57 27 7 49 42]. To compensate for the presence of unobserved confounding bias, these methods rely on additional structural or parametric knowledge about causal relationships among variables in the environment. By utilizing such domain knowledge, the imita- tor is able to recover the underlying system dynamics (i.e., causal effect) from confounded expert demonstrations and, in turn, obtain imitating policy that can achieve the expert's 's performance.
represented as a causal diagram or potential outcomes
32
43 5
By and large. the combination of causal knowledge and observational data does not always allow one to point-identify the causal effect, called the non-identifiable. That is, more than one parametrization of the target effect is compatible with the same observational data and model assumptions [32
Def. 3.2.2]. For instance, in the MAB environment described previously, the imitator's s performance
E,, [Y] is not identifiable from the confounded observational distribution P(X, Y) 22 Thm. 3.4.
Partial identificution methods concerned with inferring about target causal effects in non-identifiable settings, and has been a target of growing interest in the domains of causal inference 12
37
14 [8].
Among these works, two approaches are often employed: (I) bounds are derived for the target effect under minimal
econometrics [2 35 38 48, and more recently, in machine learning [25 24 23
assumptions, or (2) additional untestable assumptions are invoked under which the causal effect is identifiable, and then sensitivity analysis is conducted to assess how the target causal effect varies as the untestable assumptions are changed. Despite their effectiveness in addressing data bias, partial identification has still been rarely explored in the context of imitation learning.
This paper studies the partial identification for imitation learning in a generalized sequential decision- making environment of Markov Decision Processes (MDPs, (36)). The imitator must determine a sequence of actions, while unobserved confounders cannot be ruled out priori in expert demon- strations. We discuss the solutions case-by-case, depending on the identifiability of the underlying system dynamics from the confounded data, including the reward function R. and the transition distribution T. Specificallyy, our contributions can be summarized as follows. (1) We theoretically prove that when unobserved confounders generally exist, it is infeasible to learn robust policy that is guaranteed to achieve expert performance from the demonstration data, (2) When only the transition distribution T is identifiable, we propose novel imitation algorithm that leverages the bounds over the non-identifiable reward R; by matching the weighted occupancy measure, the imitator is able to obtain policy that can outperform the expert. (3) We propose alternative algorithm when the reward R is identifiable, but there is unobserved confounding affecting the transition T. Our proposed algorithms could be implemented by augmenting the celebrated generative adversarial imitation learning framework (GAII,, 19). Table briefly summarizes this paper's 's main contributions. Due to space constraints, all proofs are provided in Appendices A and B
1.1 Preliminaries
This section introduces the basic notations and definitions used throughout the paper. We use capital letters to denote random variables (X), small letters for their values (x), and 9x for the domain

---

of X. For an arbitrary set X, let [X| be its cardinality. Fix indices i,j E N. Let X stand for
sequence of variables {X,, Xi+ ;; : for consistency, the sequence X;j 0 jf We denote by P(X) represents probability distribution over variables X, and will consistently use P(x:) as abbreviations for probabilities P(X æ)) Finally, I { = 2) is an indicator function that
returns 1 if event Z = z holds true; otherwise, it returns O.
The basic semantical framework of our analysis rests on structural causal models (SCMs) [32 An SCM M is a tuple (V, U,F, P(U)), where V is a set of endogenous variables and U is set of exogenous variables. F is a set of functions s.t. each fv E F decides values of an endogenous variable V E V taking as argument a combination of other variables in the system. That is, V < (PAv. U.
S V.Uv S U. Exogenous variables U E U are mutually independent, values of which are drawn from the exogenous distribution P(U). Naturally, M induces a joint distribution P(V) over endogenous variables V, called the observational distribution. An intervention on a subset X V. denoted by do(ar), is an operation where values of X are set to constants a, replacing the functions {fx- VX E X) that would normally determine their values. For an SCM M, let Ma be submodel of M induced by intervention do(æ). For set Y S V. the interventional distribution Pz (Y) induced by do(x: ) is defined as the joint distribution over in the submodel My. i.c., Pæ M) & P (Y:(Y Ma). We leave M implicit when it is obvious from the context. For detailed survey on SCMs, we refer readers to 32 Ch. 7] and [4.
2 Challenges of Unobserved Confounding
We focus on the sequential pecision-making setting of an agent operating in MDP environment [36 over series of interventions =1, ..... At each time step t, the agent observes the current state S,, performs an action do(X,), receives a subsequent reward Y, and moves to the next observed state S1+11 Values of the action X, are selected by sampling from stationary policy T 8). which a function mapping from the domain of the observed state S, to the probability space over the domain of every action X₁. Let U, be an unobserved noise independently drawn from exogenous distribution (()). Values of the subsequent reward Y, and the next state S1+1 are, respectively, determined by structural functions Ye + fy Z2, u,) and 81+1 + fs (st, Zt, Ue), taking input fs
the current state St. action X,, and latent noise U2. The initial state S1 is drawn from an initial distribution P(S1). ). We will consistently use X, S, and ) denote the domain of action X,, state Si, and reward Y, Like a standard discrete MDP, domains actions X' and states S are assumed to be finite; rewards are bounded in real interval y A [0, 1] C R. Naturally, the agent operating in this environment defines an interventional distribution Pa. summarizing the consequences of its actions.
Fig. 2a shows graph describing this generative process; where nodes represent observed variables and directed arrows represent the functional relationships between them. For every time step > 1, the current state S, "blocks" all pathways from previous nodes (c.g., S,-1) to the future nodes (e.g.. S1+1) [2] Def. 1.2.3]. Applying d-separation rules leads to the following independence.
Definition (Markov Property [36]). For a joint distribution P. over sequences of states S1. Sy.....
actions X1, X2,x..... and rewards Yi, Y2, .... the Markov property holds with regard to distribution P., if for every time step 1 1,2, 1,2...., (, X Y IL Xise, S4+1:00, YE.o S.).
It follows from Def. that for any horizon T., the joint distribution Pa (X SET, YIT) generated by policy (X| 5) factorizes as follows
Ps (iLT, SI., MIT) = P(s1) II (xF sT 2r, 8c+1 R(s, Vr),
(I)
where the transition distribution T and the reward distribution R are interventionll queries given by T(s,, 24, 81-1) Pz, (se+1 se) ==)== [ (88411 fs(st, Tt, u4) P(u+)
(2)
us
R(se, Fz, vi) Pz. (ye (1)) / 1 1 fy (st, un) P(u))
(3)
"The decomposition holds since state S. blocks all backdoor paths from action X; to nodes S and
Ys, i.e., path starting with arrow X, + S,. It follows from Rule 2 of do-calculus [32 Theorem 3.4.1] that Pz (yel st, Ps. (se+1 st) and Ps (ye we) Ps. (y+ st).
3

---

(a) MDP exp
(b) MDP
Figure 2: Causal diagrams where Sy represents the state, X, represents the action (shaded blue) and Y represents the latent reward (shaded red). @a MDPea exp describes the imitator's 's interaction with the environment; (b) MDP obs shows the data-generating process for the expert demonstrations.
For analytical clarity, define reward function R(8, x)) as the expected value Ey WR(s, 2. v). Fix the discounted factor y E [0. common objective for an agent to optimize is the cumulative return R2 Y, + 7 + + + - "Yzkk. >
Imitation Learning. When a detailed parametrization of the transition distribution T and the reward function R. is available, the agent can obtain optimal policy using standard planning algorithms (36 6]. However, in many practical applications, complete knowledge these parametrizations often unavailable, necessitating a learning process. In this paper, we consider the imitation learning setting, where the agent has access to observed trajectories generated by the expert. More specifically, at each time step the expert selects an action X, + fx (st, u.) based on the current state S, = 8, and latent noise U. = Uy Fig. (2b shows the graphical representation of the data- generating process of the expert; the highlighted bi bi-directed arrows, e.g.. X, + Y,, indicate the presence of an unobserved confounder U E U, affecting both the action X, and outcome Y. We summarize the expert trajectories using the observational distribution P(X, S, Y) over sequences of variables X = {X1, X2,... ..... S {S1, S2. J. and Y -Y,, Y2,...) ). It is verifiable from Fig. 2b that Markov property holds with regard to distribution P(X, Y). For any horizon T,
T
P ( S1.T, D1T) P(s)) II P(r. (T) Ft, S1+1 ( D2,
where T and R are the expert's nominal transition distribution and reward function computed from the observational distribution follows:
7 (s) (S.-1= = s' S sx
(s,x)= = E St x, (5)
By convention imitation learning, we assume the rewards Y, are generally unobserved the learner; instead, it has access to parametric family E containing the expert's nominal reward function [Y, St, 2.]. Given the expert demonstrations sampled from X2 S₁, S2....) and the parametric reward family R, the imitator attempts to learn policy that can achieve expert -
Standard imitation methods focus on the identifiable setting where the imitator's transition distribution T and reward function R. is consistent with the expert's 's nominal transition 7 and reward R. Formally,
performance, i.e., E, |Σ --- 2
Definition 2 (Causal Consistency). For an interventional distribution P- and an observational distribution P satisfying the Markov property (Def. [ Causal Consistency said hold with respect to Px and P if the following statement is true, for every time step t 1,2.....
Pay (St+1 1 8x)
(8e41
St, mx), and PR, (ye 8t) = (y/, 81, x4)
When the invariances of Def. 2 hold, the learner could recover the parametrization of the transition distribution T from observational data P(X, S) and infer about the reward function R. from the parametric family L. An imitating policy π is obtainable by solving the following minimax program,
min max RESP
Σ
R(s, (P(x) (s)p(s) <( | (s))
where the imitator's Prr and the expert's p occupancy measures are defined as, respectively, Pa (=(s) = E Px (St s) and p(s) Ei P s). The solution is guaranteed to achieve expert

---

performance when the gap 1/' < 0. This means that the imitator following policy т performs as well as the expert, even in the worst-case environment instance compatible with the demonstration data and model assumption. Several imitation learning algorithms have been proposed to solve the optimization problem in Eq.
7 including
[ 50 19].
Graphical criteria exist 33 46, 34 to examine whether causal consistency (Def. holds from causal knowledge of the environment, including the celebrated backdoor condition [3 Def. 3.3.1 [15). In
MDPs, this means that the causal links between the latent noise U, and action X, are not effective
the graphical representation of the imitator's s (Fig. 2ap and the expert's 's (Fig. 2b data- gentratinn process coincide. However, in practice, causal consistency could be fragile and does not necessarily
hold due to the presence of unobserved confounders in the demonstration data 39). The remainder of this paper studies imitation learning when violations occur in the invariance relationships of Eq. (
2.1 Imitation with Non-Identifiable Transition and Reward
We first consider the imitation setting described in Fig. 2b where unobserved confounders generally exist in the expert demonstrations; both the transition distribution T and reward function R are not identifiable from Eq. (6. Here, we will show that expert performance is not imitable by constructing worst-case MDP instances where the expert always outperforms the imitator.
The state value function V2 (s) is defined as the expected return given the imitator's starting state S, = s following a policy x. i.c., (s)ss = E, [R, S; = s]. For any policy x, the imitator's s performance can be written as IE, [R]]] = E., to VV (s1). The value function of any state can thus be recursively defined using the celebrated Bellman Equation [6::
V=(s) = Σ | s) (((..)) + T(s,,,
(8)
where y denotes the discount factor. While the transition distribution T and the reward function R are not uniquely discernible from the observational distribution due to the unobserved confounding. it is still possible to learn about them from demonstrations using partial identification. Without loss of generality, the reward Y, is normalized in real interval [0, 1]. Through rigorous adaptation of
the bounding strategies established in [29] 55), we successfully derive the bounds for the transition
distribution T and reward function R, for every realization (s, x, s') E S x X x S,
2s) E T s') P(x s), 7 (s, 8') P(x 5)+ P s)
(9)
R (8,x)) E
R (8, P(x ( 18) R (s, P(x| 8) P(-x1 m)]
(10)
Among the above quantities, T and R. are the expert's nominal transition distribution and reward function in Eq. (5); P(x s) stands for the propensity score P (X, = S. s) and P(-x s) 1- P(x 8). We can then construct a worst-case MDP for any policy π at state s by solving the following optimization program: minimize the Bellman's is equation in Eq. (8) as the objective function, subject to the observational constraints in Eqs. (9) and (TO). Solving this program enables valid MDP construction since the transition distribution T and the reward function R are independent components induced by the underlying model and can be optimized separately.
Theorem 1. Given any positive observational distribution P(X, S, Y) > O, there exists an MDP model M compatible with the causal graph of Fig. 2b such that f(X,S,Y; = P(X, S,Y) and for any policy n. any time step t. =,,2,... any state S,
Vn s; M < E
Re S. = s; M
(11)
In other words, there always exists a candidate MDP instance M compatible with the demonstration
data such that an imitator is always unable to achieve expert performance (r.h.s. in Eq. (T]). )). regardless of the deployed policy x. It follows from Thm. that there is no policy π learnable from confounded demonstrations that is guaranteed to perform at least as the expert in all possible scenarios. This means that expert performance is not imitable when unobserved confounding generally exists. The following example demonstrates the challenges of unobserved confounding in single-stage MDP. Example 1 (Single-Stage MDP). Consider a I-stage MDP model with horizon = 1. For any policy (X1, S1), the imitator's 's expected return is Ex [Y]] = (($1,z1)T(C $1 P(1)). It follows
5

---

X3
(a) MDP obs
(b) MDP
Figure 3: (a) MDP shows a data-ge generating process for expert demonstrations where only the reward Y confounded with the action X;; (b) aDP shows a data- generating process for expert demonstrations where only the next state S is confounded with the action X₃.
from the tight lower bound in Eq. (10) that there exists an worst-case MDP model M compatible with the observational distribution P(X1, S1, Y1) such that R(11, Ø1) = E [Yi 81, x1] P( $1).
In this MDP instance M, the imitator's expected return can be further written as
Ex [1]
Σ E|y P(1))
B|G 21,21) P(s)) (12)
The last step holds since probabilities of the policy [(r 81) [0, and Σ., (( 81) 1.
Marginalizing the above equation gives E, [Yi] < [[Y]] the imitator unable achieve expert performance regardless of the deployed policy я. This analysis applies analogously to the MAB model in Fig. I, which can be thought of I-stage MDP with no initial state S We refer the readers to Appendix F for more examples about the 2-stage MDP.
3 Partial Identification for Robust Imitation
The impossibility results in Thm. Thm.Timply that robust imitation cannot be guaranteed when unobserved confounders generally exist the demonstration data. This means we must explore alternative assumptions to Iearn an imitating policy guaranteed to achieve expert performance. Meanwhile, standard imitation methods apply when causal consistency of Def. [2] holds, and no unobserved confounder affects the transition or reward function. natural question at this point arises: whether robust imitation is feasible for settings between the unconfounded (Fig. [b) and fully confounded cases (Fig. Tap. where unobserved confounding bias affects only either the transition distribution or reward function? This section aims to answer this question.
3.1 Imitation with Identifiable Transition and Non-Identifiable Reward
We first examine the setting graphically described in Fig. where the reward function is confounded, while the transition distribution is identifiable from the demonstration data. In this case, the first equation of Def. holds while the second one fails. To initiate the discussion, we write the expected return of a candidate policy in an MDP environment as follows [36).
Ex [R.]
Σ (s,2)) s)px(s)
(13)
Among quantities in the above equation, the state occupancy measure Pa (s)= x (S. =
is function of the initial state distribution P(s) = P(S s) and the transition distribution T. Specifically, Pr (s) can be recursively written as Pz (s) P (7)+ When the transition distribution is unconfounded (Fig. Bay. one could recover its parametrization T(s, s') following the first formula of Def. Therefore, what remains undetermined in Eq. (13 the non-identifiable reward function R. It follows from Eq. (10) that parametrization of R(s, can be bounded from the observational distribution. The imitator's expected return could thus
(s'', x, s) π( s' s')p= (s').
be lower bounded as E,, [R]] > Ex,r (s, P(x () (s), where R. is the nominal reward function defined in Eq. (). Similarly, the expert's expected return could be decomposed as
[R]] Σx., R(s x:) P(x ss)) where p(s) Eie P' s) the expert's 's occupancy measure. Optimizing the worst-case gap between the imitator Ey [R: and expert E [R,] leads to minimax optimization problem, the solution of which leads to possible imitating policy.

---

Algorithm 1: Causal GAIL with Confounded Reward R (CAIL-R)
1: Input: Expert demonstrations = X,)
2: for iteration & = 0, .,..... do
3: Collect expert trajectories from
4: Collect imitator trajectories based on the policy Tx [
5: Update the parameters to of discriminator D2 with gradient
E V w (og(DA((s, 2))1 E., P(x s) log(1 Dx((s
(15)
6: Update the policy Th+1 arg min, B (x: s) log(1 (11 - D)) using any forward RL algorithm
7: end for
Algorithm 2: Causal GAIL with Confounded Transition T (CAIL-T)
1: Input: Expert demonstrations = ((S),
2,... do 3: Collect expert trajectories from
2: for iteration i
4: Collect imitator trajectories based on the policy a s) from the worst-case occupancy
measure by solving the optimization problem presented in Eq. (19) and Eq. (20)
5:
Update the parameters w of discriminator D; with gradient
V (Dg(D((s, x))l + E. [V.. log(1 ()(+)));
(16)
6: Update the policy Th+1 arg min, E, [log(1 D)s, T] with any forward RL algorithm
7:
end for
Theorem 2. Given an MDP M compatible with the causal graph of Fig. 3a let R be parametric family containing the conditional reward E(, st, Consider the following optimization program,
S = min max R(s ()) (p(s) - < (8)p(()) REa
(14)
8,30
When the gap v² < 0, the solution xt is an imitating policy satisfying Ex [Ri]
E[R]].
In other words, Thm. computes an imitating policy within the environment depicted in Fig. 3a by finding a policy maximizing the worst-case reward function compatible with the demonstration data and the expert's 's nominal reward. Later in Sec. 4 we will demonstrate that such a solution exists and
robust imitation learning is feasible in Fig. 3a
The optimization program in Thm. Thm.p could be solved by augmenting some standard imitation learning such as GAIL [19]. To make the argument more precise, let the parametric family A be set of reward function R(s, 2) taking values in the real space R. We penalize the complexity of reward function R by subtracting convex regularization function (R) from Eq. (1);; the detailed definition of
(R) is given by [19, Eq. 13). Solving the optimization program of Eq. (14 is equivalent to matching
weighted occupancy measures between the imitator and the expert, shown in Appendix B
1² min 10 (P(x) 8)p(s) P(x ()) (s))
(17)
"
min max ((o,g B, [P(2 l) D(S, " 1)(0X
(18)
where 2/" maxx a T (R) is conjugate function of vi; function D S X (0. is discriminator classifier (e.g, a neural network). The above optimization problem is in the form of two neural networks competing against each other in a zero-sum game. The detailed implementation of our proposed algorithm, called CAIL-R, is provided in Alg. Compared to the standard GAIL algorithm, Alg. Tadds weight the signal generated by the discriminator for the imitator and then attempts to match the distribution between the weighted samples and expert demonstrations.
7

---

3.2 Imitation with Non-Identifiable Transition and Identifiable Reward
In this section, we examine the MDP environment as graphically depicted in Fig. 3b where the reward function is unconfounded, but UCs affect the action X, and the next state S1+1 simultaneously. In this setting, the second equation of Causal Consistency (Def. ( is satisfied, aligning the reward function R with the expert's nominal reward function. However, the first equation of Def. does not generally hold due to confounding bias, making the transition distribution T not identifiable from demonstrations. Despite these challenges, we utilize partial identification techniques to bound the transition function T, and subsequently estimate the imitator's performance.
More precisely, consider again the expected return decomposition in Eq. 113 The identifiable reward function R. must be contained in the parametric space of the expert's 'sss nominal reward % The transition distribution T can be bounded from the demonstration data using Eq. (9. One could thus obtain a lower bound over the imitator's performance by reasoning about the worst-case occupancy measure compatible with demonstrations. Formally, with the fixed reward function and the fixed policy the imitator's return E,, [Ri] is bounded by:
B [Rg] 2 min
Σ R(8, ())))))))))(()()())))))))))))))) S,
(19)
T.px
s.t.: (8) Σnr(t) T-q" and Px (8)= ((s)+ + Σ az, (<))s)) (s'))
8'.
Ts and T (s, T (s 8 P(x T (8,,2,)) (s,,,, P(x 8) + P (-xi 8)
1(x (20)
Obs. Constraints
The above optimization problem is similar to the classical linear program for planning in MDPs 36. The main difference is that the transition distribution T is no longer fixed but bounded in a convex space T specified from the observational data. Therefore, we develop an imitating policy by minimizing the performance gap between the imitator and the expert in the worst-case environment compatible with the observational data and prior knowledge.
Theorem 3. Given MDP M compatible with the causal graph of Fig. 3b let A be paramet- ric family containing the conditional reward E[Y, st. Dt]. and T be a parametric family over conditional probabilities P (st+1 s1, defined in Eq. (20). Consider the following program,
min max
max RE
R((s,a) (P(x s)p(()) - <x s)pm (s; T))
(21)
When the gap V* S 0, the solution nt is an imitating policy satisfying E., [Ri] > E[R]]]
We solve the optimization program in Thm. 3 by augmenting GAIL, standard imitation method
19.. By penalizing the complexity of a reward function R using a convex regularization function
(R) from Eq. [14. Eq. (11) is reducible to the following distribution matching problem,
min max max [oog DSS X))) + E2 [log(1 D(S, X)):
(22)
DE(O.I)SXX TET
We present the step-by-step implementation of our imitation method, CAIL-T. in Alg. 2 It is similar to the standard GAIL. [19]: however, a significant distinction arises at step 4, where the imitator collects trajectories from the worst-case occupancy measure as presented in Eq. (I9) and Eq. 20).. which obtainable by iteratively solving a series of linear programs. We refer readers to more detailed discussion, where we propose iterative algorithm designed to find the worst-case occupancy measure efficiently.
Appendix for Coo
4 Experiments
In this section, we validate the theoretical findings presented in Thm. Thm. illustrate the applications of the proposed CAIL algorithms (Alg. [ and Alg. 2 on various causal imitation learning tasks. Such tasks range from synthetic causal models to woal-world scenarios. To summarize, when both the transition and the reward are confounded, there always exists worst-case MDP instance M compatible with the expert demonstrations, but the imitator consistently fails to match expert
8

---

250
Driving (MD
2.5
Expert
200
2.0
Average Renum
150
8.5
Prbbubiity
100
1.0
50
750
1000
1250
Fjochs
Performance
CAIL
GAIL
0.0
GMLL
CAIL
(c)
Figure 4: Simulation results for our experiments. Fig- illustrates the performance gap histogram for the experiment MDP obs: where negative values indicate performance worse than expert performance. Fig. 4b shows the convergence plot for CAIL, GAIL, and BC performance. Fig. 4c shows the final performance, where y-axis represents the expected return.
performance, aligning with the proof provided in Sec. 2.1 When either the transition or the reward is confounded, we systematically evaluate our algorithms against the standard BC and GAIL methods, highlighting the importance of optimizing within the worst-case SCM. Standard BC mimics the expert's nominal behavior policy PSX via supervised learning; standard GAIL learns a policy by We provide in Appendix D more details on the experiment setup.
solving min- game 19].
MDP obs Random Instances. This experiment aims to empirically validate the theoretical findings discussed in Thm. [ Consider SCM instances compatible with Fig. [b] includingg binary observed variables St, Xz, Yr E {0, 13. 1000 random discrete MDPs are sampled, in other words, the reward functions and the transition probabilities are generally different among these models. The expert is able to observe the state S,, the unobserved variable U,. However, the imitator, lacking access to both U2 or the reward E,, YY], makes decisions solely S.. As shown in Fig. 4aj imitators consistently failed to match expert performance. Specifically, prevalent negative performance gaps indicate that most of imitators near -0.5, supporting our theoretical insights in Thm. 1 In summary, imitators fail to achieve the expert's performance when both the reward and the transition are confounded.
consistently worse than experts; only in cases did the performance gaps
MDP - Driving. To demonstrate the proposed framework, as outlined in Alg. we consider a scenario when autonomous vehicle ('ego vehicle') aims to learn optimal driving strategies from expert demonstrations. The state St contains some critical driving information, e.g.. the velocities of the ego vehicle and the leading vehicle and the spatial distance between them. The action X, represents acceleration or deceleration decisions the ego vehicle makes. The unobserved variable U, represents some information accessible the expert but inaccessible the imitator, e.g. slippery road conditions [26]. The reward Y, is designed to reflect multiple realistic driving objectives, e.g., safety, comfort, efficiency, and so on. U, has effect on the reward Y, Unlike the scenarios described in 33 (2). due to UCs between X, and Y, at each step f, it is impossible to find a r-backdoor admissible set. BC, GAIL, and CAIL utilize the same policy space ( s). The major difference between CAIL and GAIL lies in that CAIL optimizes the imitator by the weighted reward generated from the discriminator P(= 8) log(1 (ss x)). As illustrated in Fig. 4bb. where means and standard deviations are computed over 100 trajectories, CAIL consistently outperforms BC and GAIL.
MDP obs.S Medical Treatment. Consider the challenge of providing medical treatment acutely ill patients, where the primary goal is to learn policy so that the morality rate can be decreased. We utilize the real-world medical treatment dataset, i.c., Medical Information Mart for Intensive Care III (MIMIC-III) dataset [22]. MIMIC-III consists trajectories of clinical information (e.g.. heart rate, oxygen saturation, and on) recorded at various time intervals. However, due to privacy concerns, certain essential variables are masked or not properly recorded [45). e.g., socioeconomic status or the experience levels of caregivers 9 (56. Specifically, the state S, encapsulates the critical health information for the patients, e.g., prolonged elevated heart rate (peHR). The action X, represents whether to treat the medicine or not. The reward Y, is designed to represent the intent of the doctor much possible, e.g., avoiding the patient's 's mortality. The unobserved confounded

---

U2 simultaneously affects the action X, and the next state S(+1- Simulation results are illustrated in Fig. 4cc which shows that the proposed framework performs the best among all strategies. BC and IRL. fail to obtain an imitating policy that could match expert performance.
5
Conclusion
This paper investigates imitation learning in Markov Decision Processes where the unobserved confounding bias cannot be ruled out a priori. We establish theoretically that when such unobserved confounders generally exist, it is infeasible to obtain a robust imitating policy that can perform at least as well as the expert across all possible environments compatible with the demonstration data and prior knowledge. Departing from this critical realization, our research diverges into two distinct problem settings one where only the transition distribution is unconfounded, but the reward function is non non-identifiable due to unobserved confounding; and the other where the reward function is unconfounded and the transition distribution non-identifiable. We then propose novel imitation learning algorithms using partial identification techniques, which allow the imitator to obtain effective policies that can achieve expert performance for both problem settings. Through extensive experiments, empirically validate the theoretical findings and systematically evaluate our algorithms on different scenarios, ranging from simulated causal models to real-world datasets.
Acknowledgements
This research was supported in part by the NSF, ONR, AFOSR, DoE, Amazon, JP Morgan, and The Alfred P. Sloan Foundation.
References
[1] Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty- international conference on Machine learning, page 1. ACM, 2004.
[2] B. D. Argall, S. Chermova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-445 2009.
[3] A. Balke and J. Pearl. Counterfactuals and policy analysis in structural models. In Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence, pages 11-18. Morgan Kaufmann Publishers Inc., 1995.
[4] E. Bareinboim, D. Correa, D. Tbeling, and T. Ieard. On pearl's hierarchy and the foundations of causal inference. In Probabilistic and Causal Inference: The Works of Judea Pearl, page 507-556. Association for Computing Machinery, New York, NY, USA, 1st edition, 2022.
[5] F. Bareinboim and J. Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences, 1 11::7345-7352 2016
[6] R. Bellman. Dynamic programming. Science, 153(3731):34 1966.
[7] Bica, D. Jarrett, and M. van Schaar. Invariant causal imitation learning for generalizable
policies. Advances in Neural Information Processing Systems, 34:3952-39644 2021.
[8] A. Billard, S. Calinon, R. Dillmann, and Schaal. Survey: Robot programming by demonstra- tion. Handbook of robotics, 59, 2008
[9] N. B. Carnegie, Harada, and J. L. Hill. Assessing sensitivity to unmeasured confounding using simulated potential confounder. Journal of Research on Educational Effectiveness, 920, 2016.
[10] K.-W. Chang, A. Krishnamurthy, A. Agarwal, H. Daumé III, and J. Langford. Learning to search better than your teacher. In International Conference on Machine Learning, pages PMLR, 2015.
10

---

[11] A. Chen, J. Scheurer, J. A. Campos, T. Korbak, J. S. Chan, S. R. Bowman, K. Cho, and E. Perez. Learning from natural language feedback. Transactions on Machine Learning Research, 2024.
[12] D. Chickering and J. Pearl. A clinician's apprentice for analyzing non-compliance. In Proceed- ings of the Twelfth National Conference on Artificial Intelligence, volume Volume II, pages 1266.1276. MIT Press, Menlo Park, CA, 1996.
[13] S. Choi, J. Kim, and H. Yeo. Trajgail: Generating urban vehicle trajectories using genera- tive adversarial imitation learning. Transportation Research Part C: Emerging Technologies, 128: 10301,, 2021.
[14] C. Cinelli, D. Kumor, B. Chen, J. Pearl, and E. Bareinboim. Sensitivity analysis of linear structural causal models. In International Conference on Machine Learning, pages 1252-1261, 2019.
[15] J. Correa and E. Bareinboim. A calculus for stochastic interventions: Causal effect identification and surrogate experiments. In Proceedings of the 34th AAAI Conference on Artificial Intelligence. New York, NY, 2020. AAAI Press.
[16] P. de Haan, D. Jayaraman, and S. Levine. Causal confusion in imitation learning. In Advances in Neural Information Processing Systems, pages 11693-11704, 2019.
[17] J. Etesami and P. Geiger. Causal transfer for imitation learning and decision making under sensor-shift. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, New York, NY, 2020. AAAI Press.
[18] B. Fang, S. Jia, D. Guo, M. Xu, S. Wen, and F. Sun. Survey of imitation learning for robotic manipulation. International Journal of Intelligent Robotics and Applications, 3:362-369, 2019.
[19] J. Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[20] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
[21] G. W. Imbens and D. B. Rubin. Bayesian inference for causal effects in randomized experiments with noncompliance. The annals of statistics, pages 305-327, 1997.
[22] A. E. Johnson, T. J. Pollard, L. Shen, L.-w. H. Lehman, M. Feng. M. Ghassemi, B. Moody, P. Szolovits, L. Anthony Celi, and R. G. Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[23] Joshi, J. Zhang, and E. Bareinboim. Towards safe policy learning under partial identifiability: causal approach. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 13004-13012, 2024.
[24] N. Kallus, A. M. Puli, and U. Shalit. Removing hidden confounding by experimental grounding. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi. and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 10911-10920. Curran Associates, Inc., 2018
[25] N. Kallus and A. Zhou. Confounding-robus. policy improvement. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pages 9289-9299, 2018.
[26] R. Krajewski, J. Bock, L. Kloeker, and L. Eckstein. The highd dataset: A drone dataset of naturalistic vehicle trajectories on german highways for validation of highly automated driving systems. In 2018 2/st International Conference Intelligent Transportation Systems (ITSC), pages 21 18-2115, 2018.
[27] D. Kumor, J. Zhang, and E. Bareinboim. Sequential causal imitation learning with unobserved confounders. Advances in Neural Information Processing Systems, 2021.
[28] T. L. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6(1):4 - 22, 1985.

---

[29] C. Manski. Nonparametric bounds on treatment effects. American Economic Review, Papers and Proceedings. 80:319-323, 323, 1990.
[30] A. Y. Ng. D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In leml, volume 99, pages 778-287, 1999.
[31] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, J. Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 7(1-2):1-179, 2018.
[32] Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, New York, 2000.
[33] J. Pearl and J. Robins. Probabilistic evaluation of sequential plans from causal models with hidden variables. In P. Besnard and S. Hanks, editors, Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence (UAI 1995), pages 444 453. Morgan Kaufmann, San Francisco, 1995.
[34] E. Perković, J. Textor, M. Kalisch, and M. H. Maathuis. A complete generalized adjustment criterion. arXiv preprint arXiv:1507.01524, 2015.
[35] D. J. Poirier. Revising beliefs in nonidentified models. Econometric theory, 14(4):483-509, 1998.
[36] M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994.
[37] A. Richardson, M. G. Hudgens, P. B. Gilbert, and J. P. Fine. Nonparametric bounds and sensitivity analysis of treatment effects. Statistical science: review journal of the Institute of Mathematical Statistics, 29(4):596, 2014.
[38] J. P. Romano and A. M. Shaikh. Inference for identifiable parameters in partially identified econometric models. Journal of Statistical Planning and Inference, (38()))776-288))) 2008
[39] K. Ruan and X. Di. Learning human driving behaviors with sequential causal imitation learning. Proceedings of the AAAI Conference on Artificial Intelligence, (664):4583-459) 2022.
[40] K. Ruan, X. He, J. Wang, X. Zhou, H. Feng, and A. Kebarighotbi. S2e: Towards an end-to-end entity resolution solution from acoustic signal. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10441-10445, IEEE, 2024.
[41] K. Ruan, X. Wang, and X. Di. From twitter to reasoner: Understand mobility travel modes and sentiment using large language models. In 2024 IEEE 27th International Conference Intelligent Transportation Systems (ITSC), 2024.
[42] K. Ruan, J. Zhang, X. Di, and E. Bareinboim. Causal imitation learning via inverse reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
[43] D. Rubin. Direct and indirect causal effects via potential outcomes. Scandinavian Journal of Statistics, 31:161-170, 2004.
[44] S. I. H. Shah, A. Coronato, M. Naeem, and G. De Pietro. Learning and assessing optimal dynamic treatment regimes through cooperative imitation learning. IEEE Access, 10:78148 78158, 2022.
[45] Z. Shahn, N. Shapiro, P. D. Tyler, D. Talmor, and L.-w. H. Lehman. Fluid-limiting treatment strategies among sepsis patients in the icu: retrospective causal analysis. Critical Care, 24:1-9, 2020.
[46] Shpitser, T. VanderWeele, and J. Robins. On the validity of covariate adjustment for estimating causal effects. In Proceedings of the Twenty-Sive Conference Uncertainty in Artificial Intelligence, pages 527-536. AUAI, Corvallis, OR, 2010.
[47] Spirtes, C. N. Glymour, and R. Scheines. Causation, prediction, and search. MIT press, 2000.
12

---

[48] J. Stoye. More on confidence intervals for partially identified parameters. Econometrica. 17(4):199-1111,,22 2009.
[49] G. Swamy, Choudhury, D. Bagnell, and S. Wu. Causal imitation learning under temporally correlated noise. In International Conference on Machine Learning, pages 20877-20890. PMLR, 2022.
lenaeniiinssiiplliinngn.. [50] U. Syed and R. E. Schapire. A game-theoretic approach to apprenticeship learning. In Advances in neural information processing systems, pages 1449-1456, 2008.
[51] Tian. Studies in Causal Reasoning and Learning. PhD thesis, Computer Science Department, University of California, Los Angeles, CA, November 2002.
[52] L. Wang, W. Yu, X. He, W. Cheng. M. R. Ren, W. Wang, B. Zong, H. Chen, and H. Zha. Adversarial cooperative imitation learning for dynamic treatment regimes. Proceedings of The Web Conference 2020, pages 1785-1795, 2020.
[53] T.ZZ. Wang, T. Qin, and Z.-H. Zhou. Estimating possible causal effeets with latent variables via adjustment. In International Conference on Machine Learning, pages (0308-36335, PMLR, 2023.
[54] Zhang and Bareinboim. Markov decision processes with unobserved confounders: A causal approach. Technical Report R-23, Colummbia Causal Al Lab, 2016, Inttps:/ccasaala
net/mdp-causal. pdf
[55] Zhang and Bareinboim. Near-optimal reinforcement learning in dynamic treatment regimes.
In Advances in Neural Information Processing Systems, pages 3401-13411, 2019.
[56] J. Zhang and E. Bareinboim. Can humans be out of the loop? Technical Report R-64, Causal Artificial Intelligence Lab, Columbia University, 2020. Also, to appear: Proc. of the 1st Conference Causal Learning and Reasoning (CLeaR), 2022.
[57] J. Zhang, D. Kumor, and E. Bareinboim. Causal imitation learning with unobserved confounders. Advances in Neural Information Processing Systems, 33: 12:63-1222244 2020.
[58] J. Zhang, J. Tian, and E. Bareinboim. Partial counterfactual identification from observational and experimental data. In Infernational Conference Machine Learning, pages 26548-26555 PMLR, 2022.
13

---

A Proofs
In this section, we provide proofs for the theoretical claims delineated in the paper. Throughout this paper, it is important to note that detailed parametrizations of the underlying SCM are not known to the agent. Instead, the agent has access the expert's demonstrations, which are summarized as the observational distribution P Y.Y.
We begin by revisiting the distribution of state visitation. Specifically, (s( can be calculated by:
Pr (s) = P (8) Tdd,,,, (s')
(23)
where P (s) represents the initial state distribution, y represents the discount factor, T represents the transition probabilities for the imitator. Subsequently, are able to develop the occupancy measure for the policy π.
Pr (s, = Pr (s) |8)
(24)
It is important to note that, although the format of the occupancy measure P.r 2) shares formal resemblance to the one presented in GAIL. [19] P* (8, x) specifically represents an interventional distribution with policy do(m). 'he identifiability of the transition T (s, 8') directly impacts the
identifiability of Pa (st)- P. (#+) is not identifiable, P (s) and P (=(8,z) are consequently not
identifiable.
Theorem 1. Given any positive observational distribution P(X, S,Y 0, there exists an MDP model M compatible with the causal graph of Fig. 2b such that P(X, S, Y; M) P(X, S,Y and for any policy n, any time step I = 11,2,.... any state E S,
V 8; M
< E
R S. = 8; M
(11)
Proof. Without loss of generality, the reward is normalized so that has a range of [o, 1] Based on the value function defined in Eq. (8), first show how to expand it into a recursive version:
(st) = E,
k Yotk
St
(25)
k==
= =Y, [4]] + Er
A *Y+* 8t
(26)
k=1
= =Y, 8t] yBn
k- Yetk 8t
(27)
=E, [s]] + Σ Pa (S141 [
?
*Y 8141
(28)
k==0
=E 8t] +
Pa [++)V()
(29)
9t+1
where y is the discount factor, Pa (8141 st) denotes the transition probability when executing policy
π.
From the second last line to the last line is justified by the experimental markovian property, as
discussed in Sec. 2 following the graph Fig. [2a] More details could be found in (54 F_ [V,
s.] = BY2 sz, d)] denotes the expected reward received by the agent when executing policy
Similarly, the transition probability
P* 18)) Σ (s442 (T) st).
(30)
Generally speaking, when any unobserved confounder exists between S1+1 and X., the causal query Pa, (8141 st) is not identifiable
and Ps, (at+1 xt) = P(*+41 8t, do(o)(())) T81, Fe,
14

---

B2 43 5 51 Building on the previous derivations, we arrive at the recursive formulation of the value function under policy x:
(st)
Σ R (s., (( 1 + >
(
(31)
R (s) ta TTs, 24, (L)) Σ क( I se)
(32)
8441
Next, to establish the validity of the preceding claim, proceed by applying the technique of mathematical induction. Let [S] denote the number distinct states for S.
Base case / = T. For the final timestep T, for each state index where Vj, S (S], the value function V2 (*(T5)) cane be defined as follows:
(=(()))))) F Yr ST s(Ts))
Σ Ez (rr
Sr s(r))) (( s(r,)))
(33)
where s(r.) refers the scenario where the state the final timestep Sr equal the specific state j.
In order to obtain the worst-case SCM M, we need to minimize Va (ST) - V(sT) compatible with the observational distribution, by establishing its lower bound. To this end, directly employ the natural bound 299 which has been discussed in Sec. 2.I
min M
(= M) V(S(T.)); M)
Σ Em. YTr 8T s(r,)i M] I(E I s((,))) ((sergji M)
Σ E Yr s P(x. ((r.)) = (((())))) Σ E Yrr s(T,)). P( s(r,))
<0
(34)
The last step is justified because P(X, S, and 0 < T(Tr 2y 8(T.)) < 1. Intuitive examples illustrating this conclusion are provided in Sec. 2.1 and Appendix Therefore, this confirms the validity of the inequality for the base case.
Specifically, in certain degenerate cases where there only one possible action, the imitator has no choice but to follow that single option. Consequently, unobserved confounders are less likely to introduce significant effects in these scenarios. However, under such conditions, pursuing imitation learning not meaningful, there is variability in choice for the imitator to learn from. Therefore, such cases are of limited relevance to the scope this analysis.
Induction case.
Suppose f Va (8(41) < V(8+1)), need to prove V=(ss) V(s.).
(s. - EY st] + Σ Pa (1((42)))) sx)) V=((((+1,)))
(35)
<V(su)aa)
Without loss of generality, we assume that the state with the minimal value + denoted as $11+1.181) Our approach is founded on the premise that in obtaining the worst-case SCM JM, is strategic to allocate the lowest possible transition probabilities to the state with the highest value, while preferentially assigning higher probabilities to states demonstrating smaller values. Specifically, starts with the estimate P(S141 1(t+1.1). 2z St) for Pz, (S1+1 $(1+1.1) S:). Following this logic, systematically allocate probability masses for indices
|S| 1 as follows:
Ps, (St11 S(+1j) S.) + P(S++1 = s(c+1,)), 2z Ss))
15

---

In accordance with the established properties of probability distributions, it follows that:
Σ
(a. s(++1,)) St) 1.
Considering the state ((s+),|))) the corresponding probability can be assigned as:
-
Pz, (S141 8(++1)5)) (St) = Σ (S ((++1)) (t))
By substituting the assigned values, we are able to derive the following expression:
Pz, (St+1 ([()) St) - 1 1() Ss)) ((5( St) 1() St),
where the right-hand side simplifies to:
S|
(P(s St) 1 + ((5 St) ... 21) | =
Σ P (s(e+1))) 22 S.)
-P( St) ((((++1,1))) 21 St).
is established that the expression 0 < P(xe St) + (()(441))))) S.) < holds true. This
inequality is supported by the following equation:
[s|
P( S P(xa St).
To further analyze the expert policy, the associated value function V(s:) can be expanded as follows:
V(s:) YY4k St =
(36)
EE, sc] + Σ P(5(--15) 8x) ()(+41))))))))))
where y the discount factor, P((((-1,)) sc) denotes the observational transition probability. Notably, ((5(=+1.)) 82) and (s(+1)) st) generally different, because they reflect two distinct probabilities: ((s(++1.)) | st) corresponding the observational distribution and the other, Pz P=(S((+1,)) st). representing the imitator's transition dynamics.
In accordance with the established properties of probability distributions, it follows that:
[s|
Σ
P(S 8(415)) Ss)) 1.
Without loss of generality, suppose the policy is a deterministic policy. Actually, the following proof holds true regardless of the choice of v,. Subsequently, analyze the gap between V (8.) and ((t)) as follows:
V (st) (ss)
1S
se] +
Pa (8(+11.)) 1 vi) (+(C(10)))))))))
j=1
[S|
E|Y; st] +
P(s | ())(((()))))))
(37)
Is
[x st] +
Σ
Σ
Pa. (8(441))) 1 (t))((E (t)) ())
j=1 02
|S|
E|Y, st] +
P(s | (s)(((())))))))
==1
16

---

min (s1; M)
V(sc; M)
18 18
=ExIVi M6;
B|y s1; +2
Σ
( (1) (()((+1,)) I (x)) V ((((++1))
SS|-1
Σ
P (5(++15)) (
P(s (t) ((5(++1181))
j=1
Ex [Yr
16: M
B szj
7
(( 8x) P(s 8x) (V(s(+1.)) - ((50+1,15)))
<0
20
<0
(38)
where minM Bn[Y, 8;; .M 1a-MM]BJy EY,, | se; M] < 0 follows similar logic as previously introduced in the base case, and V (8(++1))) ((5((+1.18)) the state 1(++)))))) represents the minimal value.
0 is consistent with the ordering assumption, where
In some degenerated cases when E,, Y 8t] =+]=00 = and E st] O, might coincidentally follow that V (s²) O, which is equal to Vss: = 0. Another instance of degeneracy occurs when the value function V (s(e+1.5)) remains the same across all states. Such occurrences are extremely unlikely in practical scenarios, especially when P(X.S. S, >
B Derivations for Causal GAIL Algorithms
Theorem 2. Given an MDP M compatible with the causal graph of Fig. 3a let A be parametric family containing the conditional reward E|Y, 2t.. Consider the following optimization program.
min max
Σ R(s,x) 8) (p(s) - -T(x P) (s))
(14)
RES
8,20
When the gap v² < 0, the solution n' is an imitating policy satisfying Ex- [Ri] 2 E(R.].
Proof. Based on Eq. 13 we have:
Ex [R]]
Σ R (s, s)px(s) 8,30
follows from Eq. (00) that parametrization of R(s, x) can be bound from the observational distribution. The imitator's expected return could thus be lower bounded as
E [i] Σ
(s, P(x s)x(;) (s)=(s)
Note that the expert's expected return could be similarly decomposed as
E E[R]]
Σ x(s s)p(s),
where p(s) = P =
is the expert's occupancy measure.
3/ = min max
E. E M] B, [R:; M]
(39)
M
min max
Σ 2(s,2)P(a s)p(s) Σ R (s, л(x s))r(s)
(40)
R.R
min max
R(, (6s.))(e P(x 6) (p(s) <( ()))))))
(41)
R
which is the ultimate target expression.
17

---

Next, we will show the derivation details for matching weighted occupancy measures between the imitator and the expert. Suppose 1 max a aRR (R) is a conjugate function of eh.
Following similar logic in [9]. we utilize smiliar cost regularizer tGa, leading to the formulation of Alg. 1 Basically, Alg. minimizes Ansenn-Shnnoon divergence between P(a s)p(s) and
P(x 8 ()) (s)
First, we reformulate the equation into state-action occupancy measures:
* (P(x s)p(s) P( s)x(" ss)x 4 (p(s,x) P(x (s)p((,)))
(42)
Based on the definition of 2 we have:
26" (p(s, 2 - P(x ())xx()
(43)
mmxx
Σ
(p(s - P(# s) (s, R(s,
Σ
P(sx P(x s)0x(s,)))0(((ss,))
(44)
R
8,8
9,8
max 2) - P( s)p x)Φ (-o-)
(45)
R
8,8
max x-sx P( (s)) (-)
(46)
R'
8,0
max p(s, x) (=s,,)) - P(x ())
(47)
R'
where we make the change of variables R + (RR).. Suppose D E S x X + (0,
discriminator classifier (c.g, neural network). Using the logistic loss ø(x) = log (1 + +) we can
get:
1¹* (p(s,z) x) - P(x ())
(48)
Σ
max p(8,
(
+ P(x s)px (s, (3, log 11- --TE
(49)
R
8,8
max De(0
[P( +
log(( D(S,
(50)
which is the ultimate target expression.
Theorem 3. Given MDP M compatible with the causal graph of Fig. 3b let R be paramet- ric family containing the conditional reward E[Y, 8t, Re]. and T be a parametric family over
conditional probabilities P (St+1 st) defined in Eq. 20). Consider the following program,
D" min max max
R(s, (P(x s)p(s) - a<x s)pm (8; T))
(21)
RE TES
When the gap v² < 0, the solution n² is an imitating policy satisfying Ex- [R8]] > E[RR).
Proof. Based on Eq.
13
we have:
E, [R] =
Σ R
ss)x(s)
8,2
Σ R (s, x) P (s)
8,
Non-ID
The reward function R. is identifiable and must be contained in the parametric space of the expert's nominal reward M. other words,
R (s, = R (s.x).
(51)
18

---

The transition distribution T can be bounded from the demonstration data using Eq. 99. Therefore, we get:
S² = min max M
E
RR; M] - E, [R;: M]
(52)
min max T,R.R
Σ R(s, s)p(s) RR, Pr
(53)
8,8
min max T.R
Σ R (s, (P(x (s)p(s) -r(s,
(54)
min
max TED, TER REa
Σ R(s,z) (P(x ()p(s) x(x s)pn (s;
(55)
which is the ultimate desired expression.
Consider again the expected return decomposition in Eq. 13 The reward function R. is identifiable and must be contained the parametric space of the expert's 's nominal reward H. The transition distribution T can be bounded from the demonstration data using Eq. (9) One could thus obtain a lower bound over the imitator's performance by reasoning about the worst-case occupancy measure compatible with demonstrations. Formally, with the fixed reward function R. and the fixed policy the imitator's return is bounded by
Ex [Ri]
min Tip
Σ R(s ()) ())=
(56)
subject to:
(x( 2 0, and Epr(s)
1
1-7 (= = P (s) + TT('' xx, x) (( ss)))(s')
s/.z
Obs. Constraints T: T.T(s,z,s)=1 and T (s, 2. 8²) (s,z,s') (s) (s, s') P(# 8) + P(-x
(57)
The above optimization problem similar to the classic linear program for planning in MDPs [36.. The main difference that the transition distribution T is no longer fixed but bounded in a convex space 9 specified from the observational data. Similar to the previous setting, we could solve an imitating policy by minimizing the performance gap between the imitator and the expert in the worst-case environment compatible with the observational data and prior knowledge.
Next, we will provide heuristic algorithm to solve the optimization program presented in Eq. (19) and Eq. (20). Specifically, as discussed Eq. P we are able to bound the transition distribution T by:
T 28, E (8, P(xx (s), 7 (8,2, P(x 8)) P( (s)
(58)
The intuition for Alg. Bis in order to find the worst case, we need to put as less transition probability mass as possible to the state with maximal values, and allocate higher transition probabilities to states with smaller values. Without loss of generality, suppose Va, ($(41,1)))) is found have the smallest relative value. For all other states j * JS], we need allocate less transition probability mass as possible. Therefore, we take the lower bound:
Pz, (S 8((41.) 81) = P(S = 8((+1)), Wit 8t)
(59)
= P(S++1 = s((ti)) st. P(z st).
(60)
where Pz, (5 | )t) P(s441
St, do(xx))
(e,, and P(s(+1))
St,
T (81, Ne, 8141). For the state S(1+111)) ) we have:
|1-1
Pz, (S,41 s(1+1,18)) sc) == =
P(S S(++1,), st))
(61)
19

---

V² min Algorithm 3: Find Worst-Case Discounted Future Reward
Input: P(5 21 (st), the value function Vz, (st)
2: Output: Probability mass assignments for non-ID) transitions T
3: Let Va, (8(++1))))) is determined have the minimal relative value
4: Set
Pa. (S = S(++15) St) == P(S++ = =(t+1,)) st), where j # |S|
8
(S = 8(++1,151) |x) = 1
Σ
P(S++ s(++15)), Te 8e)
5: refurn
Following a similar logic in Alg. we reformulate the equation into state-action occupancy measures: 2 (P(x s)p(s) π(x (s; T)) = v5 (p(s. xs;
(62)
Based on the definition of v0, we have:
/ (p(s, - (=(ss
(63)
max Σ (p(s,z) (s () -E (s (19(1,))))))(((s522)) 7 TRR
R(s,x)
(64)
Σ
max TR
7) (-
(65)
Σ 8,0
max p T,K'
(5 (=((s)()(((())))) Po (s, T) (-
(66)
max ((8, T.R'
-x (8,
(67)
9,5
Suppose E x X' + 1) discriminator classifier (e.g, neural network). Using the logistic
loss (0xx) = log (1 + +e we can get:
3² 2)) Pr (x,
(68)
Σ max p(s, 5) log T
+ Pr
(1-
1
(69)
+ e- R2
1 1 tee R-R
max
Ellog D(S, + E, [log(1 - D(S,
(70)
T.D
Therefore, we are able to obtain the ultimate target expression:
x TES TEF.DE(0.1)Sxx. El)) E, [log(1 X))
max
(71)
C Finding the Worst-Case Transition Distribution
In this section, we provide a practical algorithm, Alg. designed to solve the optimization problem formulated in Eq. (19) and Eq. (0) The underlying rationale of Alg. is to search for the worst-case scenario by allocating the minimal transition probability mass to the state with the highest value while assigning greater transition probabilities to states with lower values. The resulting solution should still adhere to set of predefined observational constraints to ensure feasibility. This approach ensures that the most "adversarial" outcome is prioritized during the optimization process.
To further clarify the approach above, consider the following numerical example. Suppose there are only two states. The value function (5t+1) takes on two values: Va, (5(e41,1) = 0.8 and Vz, (,() the algorithm seeks the worst-case discounted future reward by allocating Pa, ($(1+1,1) st) <- P(s st) and Pa, ($(++1.2 s.) f- (($ Ft. st F] As such, we are able collect trajectories from the imitator, even though P (s se) is not identifiable.
Vz, (5(c=1,2)) = 0.2. Because Vg, (5((+11))))
IIn this case, although Pz, (s st) has a lower bound of ((s Rt | st). it cannot be set exactly equal this value. It is crucial to maintain the condition Pz, ($Q+1,1) 8t) Pa. (S(4122) st) 1.
20

---

D More Details for the Experiments
All experiments were conducted using Intel Cascade Lake processors, with vCPUs and 120 GB memory system running Ubuntu 18.04. Upon acceptance of this manuscript, we intend to make the source code available in the camera-ready version of the paper.
MDP obs Previously, 1000 random discrete causal models are sampled and all the performance gaps are less than O. other words, when both the reward and the transition are confounded, all imitators fail to match expert performance.
Specifically, let's take look at one example instance of those randomly sampled SCM instances. Its detailed parameterization is provided as follows:
((so) = 0.5,
P(si) = 0.5
P( No: s
8o) = 0.1888,
P(ro, yo. $ so) = 0.2099,
P(zo, y1 s
so) = 0.0294,
P(xo, y1, $
so) = 0.2116,
P(a, Vo: s6
so) = 0.1465,
P(rI, yos 11
80) = 0.0226,
P( y1, so
8o) 0.0645,
P(x yı, si
sn) 0.1267,
(72)
P(Zo. yo. so
$1) 0.1762,
P( yo. s
$11) 0.1775,
P( y1 s6
81) 0.0290,
P( V1, si
81)) 0.1786,
P(21, yo s/
81 0.1761,
P( yo. si
(8)) 0.0893,
P(11, y1, So
$1) 0.1472,
P(#1, 91,
(11) 0.0261,
where s' denotes the next state; P(zo, yo, $6 so) is the abbreviation format for P(X: Io, yo: S.+1 s S, so).
The expert is able observe the state S₁, the unobserved variable U., and the reward Y, However, the imitator, lacking access to both U, or the reward E, Ye), makes decisions solely on Sy. In other words, all methods utilize the same policy scope π (2: s). As shown in Fig. 4a| imitators consistently failed to match expert performance. Prevalent negative performance gaps indicate that most of imitators
significantly worse than experts; only in rare cases did the performance gaps near -0.3 0.5, supporting our theoretical insights presented in Thm. [ Furthermore, as depicted in Fig. 5a CAIL. does not achieve expert-level performance, specifically, E, [e,] E. [R.] - 1.9019.
However, although CAIL performs worse than the expert, CAIL still consistently outperforms BC and GAIL by effectively learning from the constructed worst-case MDP instances.
3,0
Expert
Expert
3
1.5
2
1.0
0.5
BC
GAIL (a)
CAIL
BC
GAIL
CAIL.
(b)
Figure 5: Simulation results for experiments that not included in the main manuscript.
MDP : Additional Experiment. Consider an SCM instance compatible with Fig. 3a including
binary observed variables S,, X,, Y, E (0, 1)- S, represents the state each time step. X, denotes the action. The unobserved variable U, represents some information accessible to the expert but
21

---

inaccessible to the imitator. Additionally, the imitator lacks access to the reward E, (X]] Its detailed parameterization is provided as follows:
P(so)= = 0.5,
P(s1) = 0.5
P( yo: 5
50) = 0.1775.
P( No. $1
80) = 0.2029,
P( y1, 80
80) = 0.0001,
P(ro, y1, s1
80) = 0.0001,
P yo. sn
sa) = 0.0993,
P(11, yo, s
su) = 0.0199,
P(ai, yl, So
5o) - 0.2001,
P(xi, V1, s1
8o) 0.3001,
(73)
P( yo: so
81) - 0.2859,
P( yo, sí
181) 0.1359,
P(Zo, V1, so
$1) 0.0001,
P( v1, s
(81) 0.0001,
P(11, No: s/
51) 0.2969,
P yo. s1
811) 0.2809,
P(Z1, (1,1,81.500 s/ (1) 0.0001,
P(x1, V1, s1 81)) 0.0001,
where s' denotes the next state; P( y/o % 80)
is the abbreviation format for P(X; = .o, Y; CAlL performs the best among all strategies. Both BC and GAIL. fail to match expert performance. Such result shows the effectiveness of Alg.
yo: S1+1 s/ S2 so). As depicted in Fig. 5b
E Broader Impacts
This paper investigates the theoretical framework of causal imitation learning from confounded demonstrations. Our framework versatile, applicable to various real-world domains such as autonomous driving, robotics, industrial automation, and medical decisions modeling. One of the positive impacts this study is the exploration of the risks associated with training IRL algorithms when demonstrations are generally contaminated by unobserved confounders. We thcoretically prove that when both the transition distribution T and reward function are not identifiable, there is no policy æ learnable from confounded demonstrations that is guaranteed to perform at least as the expert in all possible scenarios. Such theoretical findings have been validated through extensive randomly generated causal models. When either the reward function the transition distribution is confounded, we augment the GAIL framework by utilizing partial identification techniques, so that the imitator optimized within the worst-case scenarios. Specfically, the worst-case reward function in Alg. [ and the worst-case occupancy measure in Alg. 2 By mitigating the risks associated with unobserved confounders in expert demonstrations, our framework supports the development of more transparent and accountable AI systems. This transparency is crucial in high-stakes areas such as healthcare and transportation, where decision-making errors can have significant repercussions. More broadly, our framework significantly enhances the reliability and safety of autonomous systems in various fields, which prioritize safety and robustness during their decision-making processes. They are increasingly important because black-box box AI systems, whose internal workings remain opaque become more and more prevalent, and our understandings of their potential implications remain limited.
F Impossibility Result in Two-Stage MDPs
In this extension of the MAB model introduced in Sec. I we explore a two-stage framework (see Fig. 2b). Our previous discussions demonstrated that in MAB settings affected by unobserved confounders, the expert consistently outperforms the imitator: that is, i..., Ex [Y] < E[Y].
We now extend our analysis the two-stage MDPs. Specifically, the agent first observes the state St, selects an action X,, and subsequently, it receives a reward Yi. The process then progresses to the second stage, where the agent transitions to state S2. It chooses action X2, and then it receives further reward Yg. A pivotal distinction between this scenario and prior examples lies in the transition probability Ps, (S2 S1). Therefore, investigate their cumulative reward:
Em [ + Y2] and E|Y1 + Y2).
(74)
As a motivating example, we assume that all variables are binary. Our analysis begins by comparing the performance the final stage, specifically, Br.,,,(Y2]
Suppose f(22) E S2, Xx) 12S). Without loss of generality, we assume an ordering
in the functional values associated with different states: f(S, 0) > f(S² 1). To address the
22

---

non-identifiabilityy issue caused by the transition distribution Pr. (S2 S1), discussed in Eq. 99, we
formulate the worst-case SCM by allocating f(S, 0) with probability mass P(S2 0, Si).
In other words, we assign the lower bound P(S₂ = 0, 11 Z1) to the non-identifiable query
Psj (S2 = 71). ). As such, we are able to rewrite the expert's s rewards as follows:
E[ ʃ(S2 = 0) P(S2 0, X ((Z,)
(75)
+ f(S2 0) P(S2 = 0, 1 = ((2))
(76)
+ f(S2 = 1) P(S2 1, X1 (/Z))((Z)
(77)
+ f(S2 = 1) * P(S2 - 1, ((2))(())
(78)
and the imitator's 's reward can be written as
Em, Y2|
ní (X, - -
(79)
= f(S, = + - ((2)
(80)
+ = = ()):LL_(S)==0
(121))
(81)
B = f(S - P() - 122))((2)
(82)
-1_
((2)))
(83)
where is Ex [2] convex combination of the quantities A and B. Therefore, E,,, Y2 <
max [A B}. Given that f(S, = 0) > f(S, = 1), we are able to establish that A < Y2 and
B E Y22 Therefore, ,,,,,g [Y2] < [ Using similar rationale introduced in Sec.
get
Ex, Yi < E Yi] Consequently,
Ex [Y + Yg] < B + Y2].
(84)
In other words, the imitator is unable learn policy that can obtain the expert's performance the worst-case 2-stage MDP compatible with the observational distribution P(X1, X2, S1, S2, Yi, Y2).
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The major claims made in the abstract and introduction has accurately reflect the paper's contributions and scope. Specifically, we summarized our contributions in Sec. T e.g., the theoretical findings and the propose innovative algorithms. Additionally, Table provides brief summary of this paper's main contributions.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the requirements and limitations of the work in Sec. [ and Sec. 2 To address infinite horizon decision-making challenges, we utilize the Markov Property, as outlined Def. [ However, our study generalizes standard imitation methods by focusing scenarios where causal consistency does not universally hold true.
23

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The assumptions and problem settings can be found in Sec. 2 Due to space
constraints, all detailed proofs are provided in Appendices A and
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the important details reproduce the major experimental results in this paper can be found Sec. and Appendix Proposed algorithms are provided in Alg. and
Alg. 2
Guidelines:

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: If the paper is accepted, we intend to make the source code available in the camera-ready version of the paper. During the meantime, all the important details to reproduce the major experimental results in this paper can be found Sec. and Appendix D
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
Policy)oy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
guides/ CoodeSummiiii iss onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
25

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the important training and test details in this paper can be found Sec. 4 and Appendix [
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars and other appropriate information about the statistical significance the experiments could be found Sec. and Appendix D
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed information on the computer resources can found Sec. and Appendix D
26

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: ttthies/tttps://euriipp
Answer: [Yes]
Justification: This paper introduces novel causal imitation learning algorithms that adapt to confounded expert demonstrations within MDPs by using partial identification techniques. The research conducted in this conform with paper NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Societal impacts are discussed in Sec. and Appendix Our framework can be applied to various fields in reality, including autonomous driving, robotics, industrial automation, medical decisions modeling and so on. One of positive impacts of this work is that we discuss the potential risk of training IRL algorithms when demonstrations contaminated by unobserved confounders, and how to utilize partial identification techniques make the imitator robust.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
27

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual- usal-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:[Yes)
Justification: Please check Sec. and Appendix D
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com/ datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please check Sec. and Appendix D
Guidelines:
The answer NA means that the paper does not release new assets.
28

---

Researchers should communicate the details of the dataset/codeemmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "IKNqrTG3iu", "reviewer": "Reviewer_1FKA", "summary": "The paper addresses the challenges in imitation learning when expert demonstrations are contaminated with unobserved confounding bias. It proposes robust imitation learning methods within the framework of MDPs using partial identification techniques. The authors demonstrate theoretically that when unobserved confounders exist, learning a robust policy to achieve expert performance is generally infeasible. They introduce two novel causal imitation algorithms to handle settings where either the transition distribution or the reward function is non-identifiable from the available data. The proposed methods are validated through experiments, showing their effectiveness in achieving expert-like performance in the presence of unobserved confounders.", "questions": "On what tasks are these approaches experimented on, what do these tasks look like, and how does the demonstration data look like? Will the model performance change with a varying number of expert demonstrations?", "limitations": "The limitations have been discussed in the paper.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper tackles the issue of unobserved confounders in imitation learning, which is a significant challenge in practical applications.\n2. The paper provides theoretical foundations, demonstrating the infeasibility of achieving expert performance with unobserved confounders and offering rigorous proofs for the proposed algorithms.\n3. The introduction of two novel causal imitation algorithms (CAIL-$\\mathcal{R}$ and CAIL-$\\mathcal{T}$) enhances the robustness of policy learning in settings where either the transition distribution or the reward function is non-identifiable.", "weaknesses": "The experiment evaluation seems insufficient considering that there are already some causal imitation learning baselines such as [1], and some more experiment environments such as OpenAI gym for imitation learning. \n\n[1] P. de Haan, D. Jayaraman, and S. Levine. Causal confusion in imitation learning. In Advances in Neural Information Processing Systems, pages 11693–11704, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your reviews and acknowledge that some aspects of our work might have been misunderstood. Below, we provide clarifications and are open to further discussions should there be any residual concerns.\n\nOur experimental design incorporated similar baselines from Zhang et al. (2020), Kumor et al. (2021), and Ruan et al. (2023), chosen for their direct relevance to our framework. The rationale behind this selection was to showcase the potential of partial identification techniques in improving algorithms such as GAIL. These enhancements are particularly important for achieving expert-level performance despite differences in the observation spaces of the imitator and expert—an aspect often overlooked in OpenAI Gym environments. Most current setups in these environments presuppose identical observation spaces and ignore the presence of unobserved variables, both of which are addressed in our study.\n\n\n---\n> _“The experiment evaluation seems insufficient considering that there are already some causal imitation learning baselines such as [1], and some more experiment environments such as OpenAI gym for imitation learning.”_\n\n(De Haan et al., 2019) explores the underlying causal relationships in the environment, and exploits the sparsity in these relationships to improve the imitator’s performance. However, their problem setting assumes there is no unobserved confounding in the environment, which is the very motivation and the starting point of this paper. That is, our paper studies the challenges of unobserved confounding for imitation learning in MDPs, which is orthogonal to the setting in (De Haan et al., 2019). Consequently, we did not include (De Haan et al., 2019) in the experiments since their methods and ours are not comparable.\n\n\nWe did investigate some existing benchmarks including the OpenAI gym. Unfortunately, as far as we are aware of, no major RL benchmark simulates the presence of unobserved confounding without violating the Markov property. The corresponding causal diagrams of experts are shown in Fig.2 and Fig. 3. Due to these limitations, we have to build novel simulation environments to evaluate our proposed algorithms. Indeed, we intend to release our experiments and hope it could contribute to the existing RL benchmarks by highlighting the challenges of unobserved confounding. We believe that building our own benchmark should be taken as a strength, not a weakness.\n\n---\n> _”On what tasks are these approaches experimented on, what do these tasks look like, and how does the demonstration data look like? Will the model performance change with a varying number of expert demonstrations?”_\n\nOur learning task is similar to the standard imitation learning setting, where the imitator has access to offline demonstration data generated by an expert. The demonstration data is represented as a collection of sequences of trajectories in an MDP. For every sequence, it contains a finite number of state-action pairs $(s_j, x_j)$; the reward signal $y_j$ is not observed. The main difference is that we consider settings where the demonstration data **could be contaminated with confounding bias**; unobserved confounders exist affecting the action $X_j$,  subsequent state $S_{j+1}$, and reward $Y_j$ (had it been observed).\n\nSpecifically, as shown in previous research (Ruan et al. 2022), there exist lots of unobserved variables in real-world human driving datasets, e.g., road conditions. As stated in the driving experiment (L300-312), expert demonstrations include vehicle trajectories, “The state $S_t$ contains some critical driving information, e.g., the velocities of the ego vehicle and the leading vehicle and the spatial distance between them. The action $X_{t}$ represents acceleration or deceleration decisions the ego vehicle makes. The unobserved variable $U_{t}$ represents some information accessible to the expert but inaccessible to the imitator, e.g. slippery road conditions [24].” The details of our medical treatment experiments can be found in L313-325.\n\nDue to challenges caused by the confounding bias existing in expert demonstrations, our paper assumes that the imitator has access to sufficiently much demonstration data. We acknowledge that quantifying the uncertainty due to finite samples in imitation learning is an exciting problem, but it is beyond the scope of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Oc3vhxl6N", "reviewer": "Reviewer_S4Sm", "summary": "The paper addresses challenges in imitation learning when the learner and expert have mismatched sensory capabilities and demonstrations are contaminated with unobserved confounding bias. The authors propose robust imitation learning within the framework of Markov Decision Processes (MDPs) using partial identification. They demonstrate that in the presence of unobserved confounders, learning a policy that guarantees expert performance is generally infeasible. The paper introduces two novel algorithms for imitation learning in partially identifiable settings—when either the transition distribution or the reward function is non-identifiable. These algorithms, based on augmentations of the Generative Adversarial Imitation Learning (GAIL) method, are designed to achieve expert-level performance even with confounded data.", "questions": "Please see the detailed review in the Weakness part.", "limitations": "The authors have thoroughly discussed the limitations of this paper.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Interesting Problem:** The paper studies the partial identification problem in imitation learning in sequential decision making. This problem is interesting and has not been investigated before.\n2. **Systematic Theoretical Investigations**: The paper conducts systematic theoretical investigations on the partial identification problem in imitation learning, including three cases: non-identifiable transition and reward, identifiable transition and non-identifiable reward, non-identifiable transition and identifiable reward. The theoretical results demonstrate the infeasibility in the fully identifiable setting and the feasibility of the proposed approach in partially identifiable settings.", "weaknesses": "1. **Assumptions and Generalization**: The approach relies on certain assumptions about the partial identifiability of the MDP. In practice, these assumptions might not always hold. Even if these assumptions hold, one may not know the type of partial identification in advance, potentially limiting the generalizability of the methods.\n2. **Complexity and Practicality**: The proposed method CAIL-$\\mathcal{T}$ requires solving a complex constrained optimization problem in Eq. (18) and Eq. (19). It is difficult to solve this optimization problem in practice when neural networks are employed.\n3. **Missing Experimental Details:** The paper does not provide details about the implementation of the proposed algorithm and the environments. Besides, the source code is also missing. As such, it is difficult to reproduce their experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank your reviewers and recognize that certain elements of our work  might have been misunderstood, which could have influenced the evaluations. Below, we aim to clarify these aspects and remain eager to engage in further dialogue to resolve any lingering doubts. Generally, we have validated the proposed algorithms across a broad range of practical scenarios, including SCMs with various reward functions and transition dynamics, driving, and healthcare, where the proposed algorithms have demonstrated significant improvements in decision-making outcome performance.\n\n---\n> _“Assumptions and Generalization: The approach relies on certain assumptions about the partial identifiability of the MDP. In practice, these assumptions might not always hold. Even if these assumptions hold, one may not know the type of partial identification in advance, potentially limiting the generalizability of the methods.”_\n\nStandard imitation learning in MDPs assumes both the Markov property (Assumption 1) and Causal consistency (Assumption 2) in the demonstration data. In this paper, we generalize the second assumption as we allow the presence of unobserved confounding or violation of overlap. This means that our proposed methods are applicable to all standard MDP instances, and also generalize well to confounded MDPs where standard imitation learning methods do not necessarily obtain a valid solution. Given the wide adoption of MDPs and imitation learning, and the prevalence of unobserved confounding bias, we strongly believe our proposed method could be applied in many real-world scenarios, e.g., autonomous driving and healthcare.\n\nTable 1 provides a recipe for applying our methods in practice, given the violations of assumptions. First, when Assumption 2 holds and there is no unobserved confounder, the standard imitation learning algorithm obtains an effective policy. When there exist unobserved confounders affecting both the reward function and transition distribution (i.e., both equations in Assumption 2 do not hold), the expert performance is not imitable and the learner should explore additional domain knowledge. When unobserved confounders only affect either the reward function or transition distribution, the imitator could apply our proposed method and obtain a robust policy with performance guarantee.\n\n---\n> _“Complexity and Practicality: The proposed method CAIL-T requires solving a complex constrained optimization problem in Eq. (18) and Eq. (19). It is difficult to solve this optimization problem in practice when neural networks are employed.”_\n\nWe have developed specific methodologies to address these challenges. As outlined in Alg. 3 (see Appendix) and further elaborated in Thm. 3, our approach employs a structured algorithm designed to effectively navigate the optimization landscape. Specifically, as stated in L566-568, “The intuition for Alg. 3 is: in order to find the worst case, we need to put as less transition probability mass as possible to the state with maximal values, and allocate higher transition probabilities to states with smaller values.” Moreover, with some approximation techniques (Xia et al. (2023)), it is possible to solve it with NNs. \n\n\nTo substantiate the practical applicability of our methods, we have conducted experiments focusing on Driving scenarios (L300-312). These experiments, detailed in Appendix C, successfully implement the proposed framework using neural networks and demonstrate its effectiveness and efficiency in real-world settings. These results underscore our method's viability and its capability to handle the complexities associated with neural network-based implementations.\n\n\n---\n> _“Missing Experimental Details: The paper does not provide details about the implementation of the proposed algorithm and the environments. Besides, the source code is also missing. As such, it is difficult to reproduce their experiments.”_\n\nWe would like to clarify that our paper has indeed included a comprehensive description of the theoretical framework, algorithms, and step-by-step procedures required to understand and implement the proposed method. Theoretical contributions are summarized in Sec. 2 and 3, and further elaborated in Appendix A and B. Experimental and implementation details are provided in Section 4 and further elaborated in the supplementary materials (Appendix C). These environments are common in the causal imitation learning literature, ensuring that our experiments could be easily replicated. As mentioned in the checklist, we will open-source our codebase upon acceptance of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AFQ224jZgq", "reviewer": "Reviewer_P2hy", "summary": "This paper studies causal imitation learning from the perspective of partial identification. First, the authors show a hardness result that when both the transitions and the rewards are confounded, it is not possible to imitate or improve over the expert policy. Going forward, under only reward-confounding or transition-confounding, a new imitation learning objective is proposed using the partial identification bound of the corresponding non-identifiable unknowns, and it is proved that once the objective is optimized under $0$, the learned imitation policy would improve over the expert policy. Experimental result justifies the theoretical findings.", "questions": "1. The derivation of the partial identification bound, which is central to the method proposed in this work, is missing. I suggest clearly include them in the appendix to make the argument self-content and more convincing, or at least make it clear how the conclusion is made given the existing literature like [27].\n2. How to principally specify the (observational) reward function class $\\mathcal{R}$​ under the unobserved confounding? That is, since the underlying SCM is not known to the agent a priori, how to equip the learning agent a suitable function class $\\mathcal{R}$ to match the observational reward function of the unknown SCM? \n3. When the algorithm is applied to a non-confounded MDP (i.e., a standard MDP), how well would the algorithm perform when compared with prior arts designed for non-confounded MDPs? More generally, it seems unknown that how the performance of the proposed algorithms would change w.r.t. the confoundedness of the underlying MDP, e.g., how far the interventional probabilities deviate from the observational probabilities.\n4. In Theorem 1, according to the proof, it seems essential to assume that *all* the observational probabilities $P(X,S,Y)$ are positive, which seems too strong an assumption. What if one drop this assumption? Does the conclusion change drastically?\n5. Some minor typos:\n   - In the last line of the footnote in the end of page 3, the first equation should be $P_{\\pi}(s_{t+1}|s_t,x_t) = P_{x_t}(s_{t+1}|s_t)$.\n   - The imitator's policy $\\pi$ appearing in the subscript of $\\rho$ is in a bold font but sometimes not, which is not consistent.", "limitations": "Please see the weakness section and the question section above.", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Orginality and Significance:**\n\nThe idea of partial identification in causal imitation learning is somehow new. The main results are also quite theoretically sound.\n\n**Quality and Clarity:**\n\nThe paper is well written. The conclusions and the findings are clearly presented.", "weaknesses": "1. The idea seems like a relatively direct extension of the standard imitation algorithm (e.g., GAIL) with the partial identification method from the causal inference literature.\n2. Theoretically, it is still unknown when can we guarantee that the imitator can beat the expert. According to Theorem 2 (resp. Theorem 3), only when the optimization problem (14) (resp. (21)) has solutions with non-positive value would the imitator policy be guaranteed to achieve the same performance or improve over the expert policy. Is it possible to derive hardness results that in the reward-confounding-only case (or the transition-confounding-only case) there still exists instances such that it is impossible to improve over the expert?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> _“extension of the standard imitation algorithm (e.g., GAIL) with the partial identification method”_\n\nFirstly, applying partial identification to IL is nontrivial due to the complex interplay between unobserved confounders and the dynamics of MDPs. Partial identification typically deals with static settings in causal inference, but MDPs introduce a temporal dimension where past actions and states can influence future outcomes. Our work extends these methods to handle such temporal dependencies, which involves significant theoretical innovation.\n\nOur proposed algorithms might appear neat and straightforward in their final forms, but this does not mean their derivations are simple. The causal bounding results in Eqs. 9-10 are only applicable to a single time step in MDP. Our contribution significantly extends these bounds across time steps, creating a robust framework that can handle the sequential nature of decision-making in practice, where each decision can influence subsequent states and rewards. We also apply special transformations so that they are tailored to GAIL’s training process. We invite the reviewer to check the Appendix and see if our statement is factual.\n\n---\n> _“hardness results that in the reward-confounding-only case (or the transition-confounding-only case) there still exists instances such that it is impossible to improve over the expert?”_\n\nYes, it is possible. Consider the reward-confounding case as an example. One could construct an MDP model which consists of a sequence of independent contextual bandit models with side information $S_1, S_2, \\dots, S_t$. For each contextual bandit model, given any context $s_i$, the expected reward of each arm matches the worst-case lower bound $E[Y_i | s_i, x_i]P(s_i|x_i)$. It follows from Example 1 that the imitator in this worst-case model is unable to improve over the expert’s performance.\n\n---\n> _“The derivation of the partial identification bound ...”_\n\nThanks for the suggestion. The causal bounds in Eqs. 9-10 follow application of (Manski, 90). More specifically, Manski studies a canonical decision setting where $Z$ represents the context, $X$ is the treatment, and $Y$ is the primary outcome. Manski showed that for any observational distribution $P(X, Y, Z)$, the treatment distribution of intervening on $X$ is bounded by $P(y, x|z) \\leq P_x(y|z) \\leq P(y, x|z) + P(\\neg x |z)$. \n\nTo derive Eqs. 9-10, one could focus on the MDP within one timestep $t$. Applying Manski’s bound by setting $S_t$ as the context, $X_t$ as the treatment, and $S_{t+1}$ (or $Y_t$) as the primary outcome leads to the statement. We will include additional discussion in the updated manuscript. \n\n---\n> _“How to principally specify the (observational) reward function class $\\mathcal{R}$ under the unobserved confounding? That is, ..., how to equip the learning agent a suitable function class $\\mathcal{R}$ to match the observational reward function of the unknown SCM?”_\n\nWe do not require the learner to specify the reward function class $\\mathcal{R}$ over the interventional reward $E_x[Y|s]$, but only the observational quantity $E[Y|s, x]$. The observed domain of the state-action pair $(s, x)$ is well-specified and can be determined from the demonstration data. In this case, any measurable observational reward $E[Y|s, x]$ can be approximated using some non-parametric function approximators, e.g., neural networks (NNs). The learner could start with a parametric family of NNs, and further restrict it to inject domain knowledge. \n\n---\n> _“When applied to a non-confounded MDP (i.e., a standard MDP), how well would the algorithm perform when compared with prior arts designed for non-confounded MDPs? ... How far the interventional probabilities deviate from the observational probabilities.”_\n\nFor a standard MDP with no unobserved confounding, our algorithm could obtain a policy that is less effective compared to the one learned by the prior arts. The performance gap depends on how close the causal bounds are to the ground-truth interventional probabilities. However, we view this as a feature of our methods. When there is no unobserved confounding in the environment, we recommend the imitator to follow standard imitation learning procedures. When the imitator cannot exclude the unobserved confounding prior, our method obtains an imitating policy with performance guarantee. Our algorithms learn a policy by searching over the worst-case model. We believe this risk-averse approach is principled when facing uncertainty in offline imitation learning, since other models could deviate significantly from the ground-truth, leading to significantly inferior performance.\n\n---\n> _“In Theorem 1, ... $P(X, S, Y)$ are positive, ... What if one drop this assumption? Does the conclusion change drastically?”_\n\nWe require the probabilities $P(X, S, Y)$ to be positive to exclude degenerated cases that the imitator and the expert performance happen to match. As mentioned in L518-520, \"In some degenerated cases when $\\mathbb{E}\\_{\\pi}[Y_{t} \\mid s\\_{t}] = 0$ and $\\mathbb{E}[Y\\_{t} \\mid s\\_{t}] = 0$, it might coincidentally follow that $V\\_{\\pi}(s_{t}) = 0$, which is equal to $V(s\\_{t}) = 0$.”  \n\nHowever, such occurrences are highly impossible in practical scenarios, and are statistically insignificant (measure zero) when one generates MDP instances uniformly at random. Moreover, from an algorithmic perspective, focusing on degenerate cases would divert focus from more prevalent and practically significant scenarios.  As the experiment shown within Fig.4 (a), we have randomly generated over $1000$ MDP instance, and in all cases, the minimal performance gap between the imitator and the expert is negative, i.e., the imitator is unable to achieve the expert’s performance. This empirical evidence further supports the assertion that such degenerate cases, while theoretically possible, are exceedingly rare and do not affect the general applicability and robustness of our proposed methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Overall Response

We appreciate the reviewer’s feedback. We believe that a few misunderstandings of our work led to some of the evaluations being overly harsh and would sincerely ask the reviewers to reconsider our paper given the clarifications provided in the response. We will first address some main comments here and then reply to every reviewer separately.

**Unobserved Variables and Non-identifiability** Unobserved confounders generally exist in demonstrations when the sensory capabilities of the imitator and the expert differ or privacy concerns are serious, e.g., HighD (Krajewski et al., 2018) or MIMIC-III (Johnson et al. 2016).  In general, the structural assumptions (e.g., causal diagrams) required to perform causal inferences are inevitable, as shown in (Bareinboim et al., 2022, Theorem 1). Recognizing non-identifiability as a core challenge in causal inference, our approach improves previous partial identification techniques and applies them to sequential decision making settings.

**Significance and Novelty** The central focus of our paper is on exploring the effects of unobserved confounders in imitation learning (IL), an area that, while touched upon, has not been thoroughly explored in existing frameworks. Unlike previous studies such as de Haan et al. (2019), which assume the absence of unobserved confounders, our paper delves into scenarios where either the transition dynamics, the reward function, or both, are confounded (summarized in Table 1). The proposed framework is a substantial deviation from traditional causal imitation learning approaches which often ignore such complexities.

Furthermore, standard imitation learning in MDPs assumes both the Markov property (Assumption 1) and Causal consistency (Assumption 2) in the demonstration data. In this paper, we generalize the second assumption as we allow the presence of unobserved confounding or violation of overlap. This means that our proposed methods are applicable to all standard MDP instances, and also generalize well to confounded MDPs where standard imitation learning methods do not necessarily obtain a valid solution. Given the wide adoption of MDPs and imitation learning, and the prevalence of unobserved confounding bias, we strongly believe our proposed method could be applied in many real-world scenarios, e.g., autonomous driving and healthcare. 

To the best of our knowledge, we are the first to theoretically prove that “when unobserved confounders generally exist, it is _infeasible_ to learn a robust policy that is guaranteed to achieve expert performance from the demonstration data.” Additionally, our comprehensive experiments, which span various reward functions and transition dynamics, support these findings. Such findings are nontrivial, because it helps to explain why in practice lots of imitation learning cannot work very well.

Building on the theoretical foundation, to “recover” from the unobserved confounding bias, we have proposed novel imitation learning algorithms using partial identification techniques (as detailed in Alg. 1 and Alg. 2), allowing the imitator to obtain effective policies that can achieve expert performance for different problem settings. The intuition is to optimize the policy within the worst-case environment compatible with the demonstration data and model assumption. Such strategy helps to enhance the robustness and applicability of imitation learning in complex real-world environments.

**Experiment Baselines** In our experimental setup, we've chosen similar baselines from Zhang et al. (2020), Kumor et al. (2021), and Ruan et al. (2023) due to their direct relevance to our experimental framework. This selection is intended to demonstrate how partial identification techniques can enhance existing algorithms like GAIL, helping them achieve expert performance even when the observation spaces between the imitator and expert are different. This is crucial as most existing OpenAI Gym environments do not account for discrepancies between the observation spaces of the expert and the imitator, and assume unobserved variables do not generally exist, which we address in our experiments.

Overall, our paper introduces significant theoretical and practical contributions to the field of imitation learning, opening up new pathways for research and application in environments affected by unobserved confounders.

Meta-review: The paper studies imitation learning for MDPs. The focus is on using causal methods to battle confounding. 

The contributions are:
- an impossibility result about unobserved confounders (Theorem 1)
- a result about how to construct a policy when the transition dynamics are identifiable but the reward is not (Theorem 2)
- a result about how to construct a policy when the reward is identifiable but the transition is not (Theorem 3)

While reviewers have voiced concerns, the meta-reviewer disagrees with their assessment. Specifically:
- the lack of extensive experiments shouldn't be a main reason to reject a theory paper.
- saying it's hard to validate assumptions in practice is a vacuous statement, which applies to (almost) all theory papers, especially to causality papers
- saying the paper isn't practical because it involves constrained optimisation is unfair - constrained optimisation is required by many NeurIPS papers.
- criticisms of reproducibility are exaggerated given the paper is theoretical and authors promised to release code upon acceptance
- some said the paper needs more baselines, but didn't provide what baselines to use (and an earlier suggestion turned out irrelevant)
- similarity in algorithms doesn't mean the theory is trivial. The algorithms are different from GAIL in ways which are crucial for battling confounding.

This paper is a solid contribution and pushes on what is possible in causal imitation learning. For this reason, I recommend acceptance.
Decision: Accept (poster)