Causal Imitation Learning under Expert-Observable and Expert-Unobservable Confounding
Authors: Daqian Shao
Thomas Kleine Buening
Marta Kwiatkowska
Venue: ICLR 2026 Poster (2026)
Keywords: Imitation Learning
Hidden Confounders
Causal Inference
Reinforcement Learning
covariate shift
latent variable
Primary area: reinforcement learning

Abstract: We propose a general framework for causal Imitation Learning (IL) with hidden confounders, which subsumes several existing settings. Our framework accounts for two types of hidden confounders: (a) variables observed by the expert but not by the imitator, and (b) confounding noise hidden from both. By leveraging trajectory histories as instruments, we reformulate causal IL in our framework into a Conditional Moment Restriction (CMR) problem. We propose DML-IL, an algorithm that solves this CMR problem via instrumental variable regression, and upper bound its imitation gap. Empirical evaluation on continuous state-action environments, including Mujoco tasks, demonstrates that DML-IL outperforms existing causal IL baselines.

Published as a conference paper at ICLR 2026
CAUSAL IMITATION LEARNING UNDER EXPERT-OBSERVABLE AND EXPERT- UNOBSERVABLE
CONFOUNDING
Dagian Shao"
Thomas Kleine Buening
Marta Kwiatkowska
University of Oxford
ETH Zurich Zurich, Switzerland
University of Oxford Oxford, UK
Oxford, UK dagian shwolcs acw.kkcckkkkk
ABSTRACT
We propose a general framework for causal Imitation Learning (IL) with hidden confounders, which subsumes several existing settings. Our framework accounts for two types of hidden confounders: (a) variables observed by the expert but not by the imitator, and (b) confounding noise hidden from both. By leveraging trajectory histories as instruments, we reformulate causal IL in our framework into Conditional Moment Restriction (CMR) problem. We propose DML-IL, an algorithm that solves this CMR problem via instrumental variable regression, and upper bound its imitation gap. Empirical evaluation on continuous state-action environments, including Mujoco tasks, demonstrates that DML-II. outperforms existing causal II. baselines.
INTRODUCTION
Imitation Learning (IL) has emerged a prominent paradigm in machine learning. where the objec- tive is to learn a policy that mimics the behaviour of an expert by learning from their demonstrations. While classical IL theory suggests that, with infinite data, the IL error should vanish (Ross al., 2011), practical implementations often yield suboptimal and unsafe behaviours (Lecun et al., 2005; Kuefler et al., 2017; Bansal et al., 20.8). Prior work attributes these failures to various factors, in- cluding spurious correlations (de Haan et al., 2019; Codevilla et al., 2019; Pfrommer et al., 2023), temporal noise (Swamy al., 2022b), expert-exclusive knowledge (Choudhury et al., 2017; Chen et al., 2019; Swamy et al., 2022a; Vuorio et al., 2022) and causal delusions (Ortega & Braun, 2008; Ortega et al., 2021), which act as confounding variables unobserved by the imitator. Previous work typically addresses these factors in isolation. In practice, however, these challenges coexist, making partial solutions insufficient. This calls for a holistic approach that accounts for multiple confounding factors simultaneously.
We propose a general framework for causal imitation learning that models hidden confounders, i.e., variables present in the environment but not recorded in demonstrations. Importantly, distinguish between expert-observable confounders, which influence expert decisions but are not accessible the imitator, and expert-inobservable confounders, which introduce spurious correlations and remain hidden from both the imitator and the expert. As a result, our framework generalises prior settings and enables broader, more realistic problem formulation. In previous work, it has been shown that the application of an interactive IL algorithm such as DAgger (Ross et al., 2011), 1)1, which allows us to directly query the expert, can be effective dealing with hidden confounders. However, an interactive expert is not realistic assumption in many domains and applications. Therefore, we aim to develop approaches that solely rely on a fixed set of demonstrations.
Specifically, we propose IL method that leverages trajectory histories Insirumental Variables (IVs) to mitigate spurious correlations caused by expert-unobservable confounders. Additionally, by learning history-dependers policy, can infer information about expert-observable confounders, which enables us to better imitate the expert despite lacking access to said variables. We show that
A lternative address: shaodagian @amaill.mmmmmmmmmmmmmmmmmm

---

Published as a conference paper at ICLR 2026
IL in our framework can be reformulated Conditional Moment Restriction (CMR) problem a well-studied problem in econometrics and causal inference, which allows us to design practical algorithms with theoretical guarantees on the imitation gap.
In summary, our main contributions are as follows:
We introduce a framework for causal IL (Section 3) that incorporates both expert-observable and expert-nnobeerable confounding variables to unify and generalise many of the settings in previous work (c.g., Swamy et al. (2022b;a); Ortega et al. (2021); Vuorio et al. (2022)).
We reformulate the problem of confounded II. in our framework as solving CMR problem, where we aim to learn a history-dependent policy by leveraging trajectory histories as instruments to break the confounding (Section 4).
We propose DML-IL, a novel IL algorithm in our framework, for which we prove an upper bound on the imitation gap that recovers prior works' results as special cases (Theorem 4.5).
We empirically validate our algorithm in both custom and MuJoCo environments with both expert-observable and expert-unobservable confounders and demonstrate that DML-IIL outper- forms existing causal IL baselines (Section 5). This highlights the need to explicitly account for both types of hidden confounders.
1.1 RELATED WORKS
Causal Imitation Learning. Imitation learning considers the problem of learning from demon- strations (Pomerleau, 1988; Lecun et al., 2005). Standard IL methods include Behaviour Cloning (Pomerleau, 1988), inverse RL (Russell, 1998), and adversarial methods (Ho Ermon, 2016). Interactive IL (Ross al., 2011) extends standard IL by allowing the imitator to query an interactive expert, facilitating recovery from mistakes. However, in this paper, we do not assume query access to an interactive expert. Recently, it has been shown that IL from offline trajectories can suffer from the existence of latent variables (Ortega et al., 2021; Bica et al., 2021), which cause causal delusion. This can be resolved by learning interventional policy. Following this discovery, various methods (Vuorio et al., 2022: Swamy et al., 2022a) consider IL when the expert has access to the full hidden context that is fixed throughout each episode, whereas the imitator does not observe the hidden context, They aim to learn an interventional policy through on-policy IL algorithms that require an interactive demonstrator and/or an interactive simulator (e.g.. DAgger (Ross al., 201 1)).
Orthogonal to these works, Swamy et al. (2022b) consider latent variables unobserved by the expert, which act as confounding noise that affects the recorded expert demonstrations, but not the transition dynamics. To address this challenge, the problem is then cast into an IV regression problem. Our work combines and generalises the above works (Vuorio et al., 2022: Swamy et al., 2022a;b) to allow the latent variables to be (a) only partly known to the expert, (b) evolving through time in each episode, and (c) directly affecting both the expert policy and the transition dynamics. Solving this generalisation implies solving the above problems simultaneously.
Causal confusion (de Haan et al., 2019; Pfrommer et al., 2023) considers the situation where the expert's ' actions are spuriously correlated with non-causal features of the previous observable states. While it is implicitly assumed that there are no latent variables present in the environment, we can still model this spurious correlation as the existence of hidden confounders that affect both previous states and current expert actions. Slight variations of this setting have been studied in Wen et al. (2020); Spencer et al. (2021); Codevilla al. (2019). In Appendix A, we explain and discuss how these works can be reduced to special cases of our general framework. From the perspective of causal inference (Kumor al., 2021; Zhang al., 2020), previous work has studied the theoretical conditions on the causal graph under which the imitator can exactly match the expert performance through backdoor adjustments (imitability). Hereto related, Ruan et al. (2023) extended such condi- tions and backdoor adjustments to inverse RI. We instead consider setting where exact imitation impossible and aim to minimise the imitation gap. Beyond backdoor adjustments, imitability has also been studied theoretically using context-specifi independence relations (Jamshidi et al., 2023). Finally, Ruan et al. (2024) analyse IL under unobserved confounding and show that exact imitation is impossible without additional assumptions. They develop robust IL algorithms tailored to such partially identifiable regimes. In contrast, we adopt structural assumptions (finite- (finite-horizon and addi- tive confounding noise) which induce valid instrumental-variahte relation in the trajectory history.
2

---

Published as a conference paper at ICLR 2026
These stronger assumptions avoid their impossibility result and yield point identification of the history-dependent policy, although the expert's 's latent variables themselves remain unidentifiable.
IV Regression and Conditional Moment Restrictions (CMRs). In this paper, transform the causal IL problem into solving a CMR problem through IVs, to which end we provide a brief overview over IV regression and approaches for solving CMRs. The classic IV regression algo- rithms mainly consider linear functions (Angrist et al., 1996) and non-linear basis functions (Newey & Powell, 2003; Chen & Christensen, 2018; Singh et al., 2019). More recently, deep neural net- works have been used for function approximation and methods such as DeepIV (Hartford et al., 2017), DeepGMM (Bennett et al., 20)9b,, AGMM (Dikkala et al., 2020), DFIV (Xu et al., 2020) and DML,IIV (Shao et al., 2024) have been proposed. More generally, IV regression algorithms can be generalised to solve CMRs (Liao al., 2020; Dikkala et al., 2020; Shao et al., 2024), specifically linear CMRs, where the restrictions are linear functionals of the function of interest. In paper, we derive linear CMRs for causal IL so that the above methods can be adopted.
2 PRELIMINARIES: IVS AND CMRS
We first introduce the concept of Instrumental Variables (IVs) and its connection to Conditional Moment Restrictions (CMRs). Consider a structural model for outcome Y and treatment X:
Y f(X) + =(U) with E[E(U)) 0,
where U is a hidden confounder that affects both X and Y so that E(=(U) X] 0 0. Due to the presence of this hidden confounder, standard regressions (e.g., ordinary least squares) generally fail to produce consistent estimates of the causal relationship between X on Y. i.e., f(X). If we only have observational data, classic technique for learning is IV regression (Newey & Powell, 2003). An IV Z is an observable variable that satisfies the following conditions:
Unconfounded Instrument: Z Ill U:
Relevance: P( is not constant in Z;
Exclusion: does not directly affect Y: JL Y (X,U).
Using IVs, we are able to formulate the problem of learning f into a CMR problem (Dikkala et al., 2020), where we aim to solve for ʃ satisfying E s(X) Z = 0. In our work, we show that we can use trajectory histories as instruments to learn the causal relationship between states and expert actions by transforming the problem of causal IL into a CMR problem (Section 4).
A GENERAL CAUSAL IMITATION LEARNING FRAMEWORK
MDPs with Hidden Confounders. We now introduce a general framework for causal IL in the presence of hidden confounders. We begin by introducing a Markov Decision Process (MDP) for- mulation with hidden confounders, (S,A, U, (o, T), where S is the state space, A is the action space and 2/ is the confounder space. Importantly, parts of the hidden confounder at time may be available to the expert but not to the imitator due to imperfect data logging or expert knowledge. We model this by segmenting the hidden confounder at time into two parts u, = (uf, uz), where uf is observable to the expert and uf is not. Intuitively, uf corresponds to the additional information that only the expert observes and uf acts confounding noise in the environment that affects both the state and action.' As a result, the transition function P(- 8t, at (uā, ut] at time depends on both hidden confounders, but the reward function r( de; uz) only depends on the state, action, and the observable confounder uf since the confounding noise only directly affects the state and actions. Finally, on is the initial state distribution and I' the time horizon. A causal graph illustrating these relationships is provided in Figure 1. This nuanced distinction between up and uf is crucial for determining the appropriate method for IL, and we begin with an example to motivate our setting and illustrate the importance of considering u, (uf, uf).
Example 3.1. Consider an airline ticket pricing scenario (Wright, 1928), where the goal is to learn pricing policy by imitating actual airline pricing based on expert-set profit margins. Suppose
In our framework, we allow the actual actions taken in the environment to be affected by the noise. Noise that only perturbs data records be considered as special case of our framework.
3

---

Published as a conference paper at ICLR 2026
T--1 TEB
T1g Qc-1
RC
S--1
Figure I: A causal graph of MDPs with hidden confounders = (u,, wi). The black dashed lines represent the causal effect of the expert-observable confounder uy, which directly affects the expert action or. It also directly affects st+1 and rt. The red dashed lines represent the causal effect of the expert-unobservable uf, which acts as confounding noise and directly affects the states and actions. uf does not directly affect (following Swamy et al. (2022b)) because the expert policy does not take uf into account, and letting uf, directly affect ra would only add noise to the expected return.
that seasonal patterns and external events are known only experts, but missing from the dataset. Hence, these latent variables serve as expert-observable confounders uf. Meanwhile, actual airline prices are confounded (additively) by fluctuating operating costs, which are unknown to the experts when they set the profit margin and are not contained in the dataset. Consequently, such fluctuating operating costs act as confounding noise uf. We conduct experiments on a toy environment inspired by this example in Section 5, and show that IL algorithms that do not distinguish between ug and uf fail to correctly imitate the expert.
Causal Imitation Learning. We assume that an expert is demonstrating a task following some expert policy Th (which will specify in more detail later) and we observe of N 2 expert demonstrations (d1, ...,..ddd dv); ). Each demonstration is a state-action trajectory (81, 41,-., 8r, ar) ) where, at each time step t, we observe the state s, and the action at taken in the environment. The next state is sampled from the transition function P( St, az (ug, uz)).
Let hr US x A) x S is the set of all possible trajectory historics. Importantly, we observe nei- ther the reward nor the confounders (ug, uf) at time t. Given the observed trajectories, to learn a history-dependent policy Th : H - A(A), where ((AA) denotes the set of prob- ability measures over A and the policy class Th E II is convex and compact. The Q-function of policy TA is (n az, uz) EYNEA Ei ar, ut)] and the value of policy is
H denote the trajectory history at time where H. C
goal
J( = Er [Ei=1 r(su, Op', u..)], )]. where 7 is the trajectory following The
In order learn policy Th that matches the performance of nE. need to break the spurious cor- relation between states and expert actions by inferring what the expert would do if we intervened and placed them in state s; when observing uf. Unfortunately, the causal inference literature (Shpitser & Pearl, 2008) tells us that, without further assumptions, it is generally impossible to identify ag- To determine the minimal assumptions that allow Te to be identifiable, we first observe that of can be correlated for all time steps i, making it impossible to distinguish between the intended actions of the expert and the confounding noise. However, in practice, the confounding noise at far-apart time steps is often independent. For example, the effect of the confounding noise ui at time / on future states and actions often diminishes over time, which is typically the case for random environment noise such as wind. In addition, when the confounding noise ui at time becomes observable at future time t', e.g., previous operating costs are observed eventually as in Example 3.1, the unob- servable confounding noise times f. and t! becomes independent. We formalise this intuition as the notion of confounding noise horizon k.
Assumption 3.2 (Confounding Noise Horizon). For every t, the confounding noise up has a horizon of l where 1 < k < T. More formally, of II. ui VI. > k.
4

---

Published as a conference paper at ICLR 2026
This assumption is essential for decoupling the spurious correlation between the state and action pairs. We also assume that the confounding noise is additive to the action, which is standard in causal inference (Pearl, 2000; Shao et al., 2024). Without this assumption, the causal effect becomes unidentifiable (see, e.g., Balke & Pearl (1994)) and the best we can do is to upper/lower bound it. Assumption 3.3 (Additive Noise). The structural equation that generates the actions in the observed trajectories
a. = ( ug) + ui,
where w.l.o.g. Flug] 0 as any non-zero expectation of uf can be included as constant in Tg-
Next, we show that, with the above two assumptions, becomes possible to identify the true causal relationship between states and expert actions, and to imitate Tg.
4 CAUSAL IL AS A CMR PROBLEM
In this section, we demonstrate that performing causal in our framework is possible using trajec- tory histories as instruments. We show that the problem can be reformulated as a CMR problem and propose an efficient algorithm to solve it.
The typical target for IL would be the expert policy Te itself. However, since the expert has access to privileged information, namely uf, which the imitator does not, the best thing an imitator can do is to learn a history-dependent policy Ta that is the closest to the expert. A natural choice for learning objective is the conditional expectation of ( up) on the history he:
TA (ha) = Ep(ag A.) ( x]]] = [s ue) he],
because the conditional expectation minimises the least squares criterion (Hastie et al., 2001) and Ty is the best predictor of mg given /4- In The the distribution P(u ht)) captures the information about up that can be inferred from trajectory histories.
Remark 4.1. Learning Th is not trivial. Policies learnt naively using behaviour cloning, i.e., Ela, hel, fail match ge. To see this, note that, in view of Equation (2), we have
Elaz h4] -E (t,,
(st, ue) h4] + Elu h4]
(h,) Elu, he),
(3)
where Eluj ht] o 0 due the spurious correlation between ui and the trajectory history h As result, E[az hi] becomes biased, which can lead to arbitrarily worse performance compared to TE.
Derivation of the CMR Problem. Leveraging the confounding horizon from Assumption 3.2, are able to break the spurious correlation using the independence of uf and ur_k. We propose to use the k-step history he-k (81, a1 (.., St-k) as an instrument for the current state s,." Taking the expectation conditional on h & on both sides of Equation (3) yields
[[ h
E [[ae h.] h EETT ) he-k] + E[E[u [t]] he-k]
E ( (b) ho-k] + Eluf h4-
E[TT he=x] + Efu = ([(Tp(he) he-
where we use the fact that hy- a ( -eeasurable because hę= S he, of IL wi_x and [[ by Assumption 3.2. As result, the problem of learning Tr reduces to solving for nn that satisfies the following identity
Ela, (h+) h==*]= =0.
(4)
which is CMR problem as defined in Section 2. In this case, both o and hy observed in the confounded expert demonstrations, and 1.- acts as the instrument.
To ensure that the instrument hy is valid, we formally verify that the three IV conditions from Section 2: uf 11. h4-k. P(h; h4-1)) is not constant in h and h4- doesn't 't directly affect at are satisfied by hr- in Appendix B. However, the strength of the instrument hy- representing its correlation with hr. influences how well TA. (hee can be identified by solving the CMR problem in Equation (4). As the confounding horizon i increases, this correlation weakens, making hi-k a less effective instrument. We formally analyse this relationship in Proposition 4.3 and further validate it experimentally in Section 5.
Note that this requires prior knowledge (or an upper bound) of the confounding horizon k. We discuss this assumption and practical ways choose e.g., conditional independence tests, in Appendix F.
5

---

Published as a conference paper at ICLR 2026
Algorithm Double Machine Learning for Causal Imitation Learning (DML-IL)
1: input Dataset D of expert demonstrations, confounding noise horizon A
2: Initialize the roll-out model M as Gaussian mixture model
3: repeat
4: Sample (h,, az) from data Dø
5: Fit the roll-out model (/4, a1) M (h+-k) to maximize the log likelihood
6: until convergence
7: Initialize the expert model nt as a neural network
8: repeat
9: Sample h4-k from DE
10:
Generate h4 and âg using the roll-out model M
11:
Update n to minimise the loss := ||xe R (h+ 112
until convergence
13:
return A history-dependent imitator policy n
4.1 PRACTICAL ALGORTHTM FOR CAUSAL IL
There are various techniques (Bennett et al., 2019a; Xu et al., 2020; Shao et al., 2024) for solving the CMR problem [ha ET (h,) [e in (4). Here, the CMR error that aim minimise
is given by
E Bla tn (Ite)
[ |[ (h: [r-])22
In Algorithm 1, we introduce DML-IL, an algorithm adapted from the regression algorithm DML-IV (Shao et al., 2024), which solves our CMR problem by minimising the above CMR error. The first part of the algorithm (lines 3-7) learns roll-out model M that generates a trajectory steps ahead given h4-k- Then, fy takes the generated trajectory he from M (hę-k) input and minimises the mean square error to the next action (lines 8-13).
Using generated trajectories is crucial for breaking the spurious correlation caused by uj, and the trajectory history before h4- allows the imitator to infer information about up. In particular, the expert's 's future trajectory after h is confounded with the current state and action through the unobserved noise nf. so it does not represent draws from the conditional distribution of future histories given -4 Rolling out from h with M removes this dependence and yields the correct conditional distribution needed for the CMR moment. We refer to Appendix F for discussion of the theoretical convergence rate guarantees of DML-IL and the choice of the confounding noise horizon i as input.
Moreover, once we set the learning objective as the conditional (h,)) = E[Tg(St ut) [he] we can learn TA (h+) for both continuous and discrete action spaces as the derivation of the CMR problem in (4) remains valid for both. However, in the algorithm and the subsequent theoretical analysis of the imitation gap, we implicitly assume that a, is continuous such that क (h+)) is valid action by the imitator. In practice, if the action space is discrete, we require mapping that maps ny (he) to the action space, e.g., treating na (hz) the logits output to the action space.
4.2 THEORETICAL ANALYSIS
In this section, we derive theoretical guarantees for our algorithm, focusing on the imitation gap and its relationship to existing work. All proofs in this section are deferred to Appendix B.
On a high level. in order bound the imitation gap the learnt policy मy, i.e. J(TØ) J (Ax), we need to control:
(i) the amount of information about the hidden confounders that can be inferred from trajectory histories hạ:
PDL. stands for double machine learning (Chernozuukoy et al., 2018), which is statistical technique to ensure fast convergence rate for two-step regression, as is the in Algorithm I.

---

Published as a conference paper at ICLR 2026
(it) the ill- ill-posedness (or identifiability) of our CMR problem, which intuitively measures the strength of the instrument hi-k:
(iit) the disturbance of the confounding noise to the states and actions at test time.
These factors are all determined by the environment and the expert expert To control (i), measure how much information about uf is captured by the trajectory history h, by analysing the Total Variation (TV) distance between the distribution of uy and Elug [by] along the trajectories of TT. To control (fi) and (ivi), we need to introduce the following two key concepts.
Definition 4.2 (Il-Posednesss of CMRs (Dikkala et al., 2020)). Given the derived CMR problem in Equation (4), the ill possddsssssss (III, k) of the policy space II with confounding noise horizon k is
|T π 2/2 Fla (he
v (II, k) sup TEll
The ill-posedness vIII, k) measures the strength of the instrument, where a higher (III, k) indicates a weaker instrument. bounds ratio between the L2 error of the imitator to the expert policy, and the learning emrrr of the imitator following CMR objective.
As discussed previously, intuitively, the strength of the instrument would decrease as the confound- ing horizon & increases. This is confirmed by the following proposition.
Proposition 4.3. v (II,, k) is monotonically increasing as the confounded horizon l increases.
Next, we introduce the notion of c-TV stability.
Definition 4.4 (c-Total Variation Stability (Bassily et al., 2021; Swamy al., 2022b)). Let P(X) be the distribution of a random variable X f - X. P(X) is c-TV stable if for all a1, E X and A
|a a2ll s A
òrv(a) +X, a2 + X) cA,
where ||-|| is some norm defined on X' and ôrv is the TV distance.
wide range of distributions are c-TV stable. For example, standard normal distributions are 1-TV stable. We apply this notion to the distribution over uf to bound the disturbance it induces in the trajectory and the expected return.
analyse and upper bound the imita- tion gap (TTE) - ( by controlling the three previously discussed components (i) (iii). Theorem 4.5 (Imitation Gap Bound), Let Rn be the learni policy with CMR error e and let v(II, k) be the ill-posedness of the problem. Assume that δrv (ug Erg [u] [ha)) S ó for ö E IRt, P(uf) is c-TV stable and Tg deterministic. Then, the imitation gap is upper bounded by
With the notion of ill- polsddessssss and c-TV stability, we
J(TE) ((rr) T (cev(111, 26) (6+2
This upper bound scales the rate of T2, which aligns with the expected behaviour of imitation learning without an interactive expert (Ross & Bagnell, 2010). Next, we show that the upper bounds on the imitation gap from prior work (Swamy et al., 2022b;a) are special cases of Theorem 4.5. The proof' are deferred to Appendix B.4.
Corollary 4.6. In the special case that uf = 0, i.e. there are no expert-observable confounders, or ul E. [uq] he], i.e., uf is (he))-measuii -measurable (all information about ut is contained in the history), the imitation gap is upper bounded by
J(Tg) - J(RN) (EE((II))) = OT?ss)
which coincides with Theorem 5.1 of Swamy al. (2022b).
In the other extreme case, when there are no hidden confounders, i.e., of = 0, our framework is reduced to that Swamy et al. (2022a). However, Swamy et al. (2022a) provided an abstract bound that directly uses the supremum of key components in the imitation gap over all possible Q-functions to bound the imitation gap. We further extend and concretise the bound using the learning error and the TV distance bound 5 instead of relying the supremum.

---

Published as a conference paper at ICLR 2026
BC-SEQ
ResidulL
DML-IL (ours)
Noised Expert
Squared (log scale)
Ticket Pricing
Ticket Pricing
1.0
Average Reward
0.8
20
0.6
0.4
2-1
0.2
0.0
11
15
37
19
20
13
15
19 20
harizon
horizon
(a) MSE in log scale, lower is better.
(h) Average reward, higher is better.
Figure 2: Plane Ticket Environment (Example 3.1): On the left, the MSE in log scale between the learnt policy and the expert. On the right, the average reward of our approach and baselines.
Corollary 4.7. In the special case that uf = 0, if the learnt policy has optimisation error e, the imitation gap is upper bounded by
2
((TE) - ((th) S TT
ddim(A) E 28
where (A))(A) denotes the dimension A. This is concrete bound that extends the abstract bound in Theorem 5.4 of Swamy el al. (2022a).
Remark 4.8. If both uf, and zero, then recover the classic setting IL without founders (Ross & Bagnell, 2010), and the imitation gap bound is T where is the optimisation error of the algorithm.
EXPERIMENTS
We empirically evaluate the performance of Algorithm 1 (DML-IL) on the toy environment mod- elling the ticket pricing scenario with continuous state and action spaces introduced from Exam- ple 3.1 and the Mujoco environments (Todorov et al., 2012): Ant, Half Cheetah and Hopper. We compare with the following existing methods: Behavioural Cloning (BC), which naively minimises El-log [ar] BC-SEQ (Swamy et al., 2022a), which learns a history-dependen policy han- dle expert-observable hidden confounders; ResidulL (Swamy et al., 2022b), which we here adapt to setting by providing h4- instruments to learn distory-independen policy; and the noised expert, which is the performance the expert in the confounded environment, and corresponds to the maximally achievable performance. In Appendix C.I, include additional evaluations when using other IV regression algorithms, including DFIV (Xu et al., 2020) and DeepGMM (Bennett al., 2019b), the core CMR solver, but found inconsistent and subpar performance. In Ap- pendix C.2, we also provide further discussion and empirical evaluations of DML-IL under mis- specification of the confounding noise horizon
We train imitators with 20000 samples (40 trajectories of 500 steps each) of the expert trajectory using each algorithm and report the average reward when tested online in their respective environ- ments. The reward is scaled such that 1 is the performance of the un-noised expert, and 0 is that of random policy. We also report the Mean Squared Error (MSE) between the imitator's and expert's actions. The purpose of evaluating the MSE is to assess how well the imitator learnt from the expert, and importantly whether the confounding noise problem is mitigated. When the confounding noise uf is explicitly handled, we should expect to observe much higher MSE. All results plotted with one standard deviation as a shaded area. In addition, we vary the confounding noise horizon from to 20 in order increase the difficulty of the problem with weaker instruments h4-k-
5.1 PLANE TICKET PRICING ENVIRONMENT
Experimental Setup. We first consider the plane ticket pricing environment described Exam- ple 3. Here, the expert-unobservable confounding noise uf corresponds operating costs and the
8

---

Published as a conference paper at ICLR 2026
expert-observable confounder uf models seasonal demand patterns and events. We set uf to tinuously vary with a rate of change of approximately every 30 steps. A detailed description of this environment provided in Appendix D.1.
The results are presented in Figure 2. DML-IL performed best with the lowest MSE and the highest average reward that is closest to the expert, especially when the uf horizon is 1. This implies that DML-IL is successful in handling both uf and ug. ResiduIL is able to reduce the confounding effect of uf, evident by the lower MSE compared to the two other methods that do not deal with uf- However, since it does not explicitly consider ug, the imitator has information on oi and the best it can do is to assume some average value (or expectation) of uf. Therefore, while ResidulL still achieves some reward, its considerable performance gap to DML-IL can be explained by its ignorance of ug. Both BC and BC-SEQ fail entirely in the presence of confounding noise uf, with orders of magnitude higher MSE and average reward close to a random policy. From the similar performance of BC-SEQ and BC, we see that using trajectory histories to infer uf is not helpful when the confounding noise is not handled explicitly. This demonstrates that only partially accounting for the effect of uz ug is insufficient to learn good imitator.
Results.
Moreover, as the confounding noise horizon & increases (x-axis), the performance of DML-II de- creases. This supports our intuition and theoretical results that the instrument becomes weaker, and less information about ug can be inferred from h as k increases. When k = 20, we find that the performance of DML-IL is close to that of ResidulL, which does not consider the effect of ut, because very limited information about the current expert-observabb confounder uf can be inferred based on the history from 20 steps ago. 5.2 MUJOCO ENVIRONMENTS
Experimental Setup. In Figure 3, we consider the Mujoco tasks. While the original environment implementations (Todorov et al., 2012) do not have hidden confounding variables, modify the environment to introduce uf and u.- Specifically, instead of travelling as fast as possible, the goal to control the agent to travel speed is ug, which is observed by the expert but not recorded in the dataset. In addition, we add confounding noise uf to 8; and 0 to mimic confounding noise such wind. Additional details about the modification made to the environments are provided in Appendix D.2.
target speed that varying throughout an episode. This target
Results. DML-IL outperforms other methods in all three Mujoco environments as shown in Fig- ure 3. Similarly to the plane ticket environment, ResidulL is effective in removing the confounding noise but fails match the average reward of DML-IL confounders ug. BC and BC-SEQ have much higher MSE and fail to learn meaningful policies. As the confounding horizon of uf increases, the performance of DML-IL drops, which is expected as the instruments weaken and less information about up can be inferred from the histories. This is most visible in the Ant and Half Cheetah environments.
does not account for expert-observabll
6 DISCUSSION
We proposed framework for causal imitation learning with hidden confounders that unifies sev- eral previous causal IL settings. Specifically, we considered IL from a fixed set of confounded expert demonstrations without further interactions with the confounded MDP, where the hidden confounders are partially observable to the expert. We demonstrated that causal II. under this frame- work can be reduced to CMR problem when using the histories as instruments. We proposed novel algorithm, DML-IL, to solve the CMR problem and imitate the expert, and provided upper bounds the imitation gap of DML-IL that subsume previous results. Finally, empirically evaluated DML-IL on multiple tasks, including Mujoco environments, and demonstrated improved imitation performance against other causal IL algorithms in the presence of expert-observable and expert-unobservable confounding.
Limitations. One limitation is the explicit assumptions made in Section 3, which are essential for the expert policy to be identifiable. Therefore, it is important for practitioners to validate that their specific environment and task satisfy these assumptions. We provided in the paper some examples where these assumptions are known to hold (c.g.. drone and ticket sales), while we acknowledge that method is not applicable all scenarios, especially in the healthcare domain where non-linear

---

Published as a conference paper at ICLR 2026
BC-SEQ
ResidulL
DML-IL (ours)
Noised Expert
Mean Squared Error tlog scale)
Ant
1.0
Average Reward
0.8
2-1
0.6
2-a
0.4
2-
0.2
2.4
0.0
5
15
19
13
15
17
19 20
horizon
harizon
(a) MSE in log scale in Ant.
(h) Average reward in Ant.
MMan Souared Error (log scale)
Half Cheetah
Half Cheetah
1.0
Average Reward
0.8
2
2-1
2-2
2-3
2-4
5
11
15
19
13
15
17
19
20
honzon
horizon
MSE in Half Chectah.
(d) Average reward in Half Cheetah.
Mean Squared Error (leg scale)
lopper
lopper
Average Reward
22
0.2
0.0
2-1
6 horzon
J honzon
(c) MSE in Hopper.
(f) Average reward Hopper.
Figure 3: MuJoCo: On the left, the MSE in log scale between the learnt policy and the expert (lower MSE is better). On the right, the average reward in the MuJoCo environments Ant, Half Cheetah and Hopper (higher values better). The confounding horizon increases along the x-axis.
confounding is typical. However, causal identification comes at cost it requires non-trivial assumptions that don't it hold in all real-world applications.
In addition, we assume knowledge of the confounding noise horizon k upper bound on it for Algorithm 1. Unfortunately, the value of generally cannot be verified empirically. However, there exist tests that can indirectly check whether candidate IV is valid, such conditional inde- pendence tests (Gretton et al., 2005), which we discuss in Appendix F.
Future Works. There are many active research fronts that consider causal identification with non- additive noise, partially observable covariates and invalid instruments. They are beyond the scope of this paper and are orthogonal our work. It would be an interesting research direction consider our confounded MDP framework in these problem settings.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the EPSRC Prosperity Partnership FAIR (grant number EP/V056883/1) DS acknowledges funding from the Turing Institute and Accenture collaboration. Part of this work was done while TKB was at The Alan Turing Institute. TKB is supported by an ETH Al Center Postdoctoral Fellowship. MK receives funding from the ERC under the European Union's Hori- zon 2020 research and innovation programme (FUN2MODEL, grant agreement No. 934115) and participates in Erlangen Al Hub.
REFERENCES
Joshua D. Angrist. Guido W. Imbens, and Donald B. Rubin. Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91:444 455, 6 1996. ISSN 1537274X. doi: 10. 10.1080001214459.100 1996. 10476902.
Alexander Balke and Judea Pearl. Counterfactual probabilities: Computational methods, bounds and applications. Uncertainty Proceedings 1994, pp. 46-54, 1994. doi: 10.1016/
B978- 1778-1-55800-33-333 332-5.50011-0.
Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. Chauffeurnet: the best and synthesizing the worst. Robotics: Science and Systems, 5:5986, 12 2018. ISSN 2330765X. doi: NL.16077RSS.2019.XX.2 03079v1.
Learning to drive by imitating
URL
Raef Bassily, Thomas Steinke, Kobbi Nissim, Uri Stemmer, Adam Smith, and Jonathan Ullman. Algorithmic stability for adaptive data analysis. SIAM Journal on Computing, pp. 1046-1059, 11 2021. ISSN 07378017. doi: 10. (0.1145/897518.288977 1511.0251300
URL hwtpstt//axxii..tttt//
Andrew Bennett, Nathan Kallus, and Tobias Schnabel. Deep generalized method of moments for instrumental variable analysis. Advances in Neural Information Processing Systems, 32, 2019a. ISSN 10495258.
Andrew Bennett, Nathan Kallus, and Tobias Schnabel. Deep generalized method of moments for instrumental variable analysis. Advances in Neural Information Processing Systems, 32, 2019b. ISSN 10495258.
Ioana Bica, Daniel Jarrett, and Mihaela van der Schaar. Invariant causal imitation learning for generalizable policies. Advances in Neural Information Processing Systems, 01:3952-3964, 2021.
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. Pro-
ceedings of Machine Learning Research, 100:66-75, 2019. ISSN 26403498. doi: 10.1126/
acir URL https: aawiiviiivvv.//vv org. aabs//////// / 912 12294v1.
Xiaohong Chen and Timothy M. Christensen. Optimal sup-norm rates and uniform inference nonlinear functionals of nonparametric iv regression. Quantitative Economics, 9:39-84, 2018.
ISSN 17597331. doi: 10.3982/ge722..
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whit- ney Newey, and James Robins. Double/debiased machine learning for treatment and struc- tural parameters. The Econometrics Journal. 21(1):C1-66) 2018. ISSN 1368-4221. doi:
(0.111//CCT..1299.,
Sanjiban Choudhury. Mohak Bhardwaj, Sankalp Arora, Ashish Kapoor, Gireeja Ranade, Sebas- tian Scherer, and Debadeepta Dey. Data-driven planning via imitation learning. Interna- lional Journal of Robotics Research, 37:1632-1672, 11 2017. ISSN 17413176. doi: 10.1177/
027836491878100 URL /
11111.111111..111..1 06391v1.
Felipe Codevilla, Eder Santana, Antonio Lopez, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. Proceedings of the IEEE International Conference Computer Vision, 2ctoeerctoe 9937,9933.444 4 2019. ISSN 15505499. doi: 10.1009//
00942. URL https: //ta::////xiiii wtw.///////aaaaoooooooo/.. 0004..09880 08980v1.

---

Published as a conference paper at ICLR 2026
Timothy G. Conley, Christian B. Hansen, and Peter E. Rossi. Plausibly exogenous. The Review of Economics and Statistics, 94:260- 272, 2012. ISSN 0034-6535, doi: 10.1162/RES.A.000339.
URL https:://///mm / dx doi. www.////oooo /ttpg///.............../ RESSE 00139.
Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. Advances in Neural Information Processing Systems, 32, 2019. ISSN 10495258. URL
https:/ / arxii.... .rgg alos/ 905. 119792..
Nishanth Dikkala, Greg Lewis, Lester Mackey, and Vasilis Syrgkanis. Minimax estimation of con- ditional moment models. Advances in Neural Information Processing Systems, 2020- Deeembee,m,er, 2020. ISSN 10495258. URL httpt:t/p//oo. rxi v. wwg.or///aao.... abs 20000 07201v1.
Kenji Fukumizu, Arthur Gretton, Xiaohai Sun, and Bernhard Schölkopf. Kernel measures of condi- tional dependence. Advances in Neural Information Processing Systems, 20, 2008
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schlkopf. Measuring statistical depen- dence with hilbert-schmide norms. Algorithmic Learning Theory, 3734 LNAI:63-77, 2005. ISSN 161 1611-3349. doi: 10.1007/1 1564089 7. URL https: //t::///nniie springer com chapter/ 10. 000000/1111499999
Hahn and Hausman. Estimation with valid and invalid instruments. Annales d' É'conomie et de Statistique, pp. 25, 2005. ISSN 0769489X. doi: 10.2307/2077569..
Lars Peter Hansen. Large sample properties of generalized method of moments estimators. Econo- metrica, 50:1029, 7 1982. ISSN 00129682. doi: 10.2307/1912775.
Jason Hartford, Greg Lewis, Kevin LeytonnB Brown, and Matt Taddy. Deep IV: A flexible approach for counterfactual prediction. Proceedings of the 34th International Conference on Machine Learn- ing, 2017. doi: 10.5555/33053813330553
Jason Hartford, Victor Veitch, Dhanya Sridhar, and Kevin Leyton-Brown. Valid causal inference with (some) invalid instruments. Proceedings of Machine Learning Research, 139:4096-4106, 2020. ISSN 26403498. URL https: //tpi/aaaa..///////aa/eee/aaweva org/ wttp//t/g///ooooooooooo pdf/2006. 11386.
Trevor Hastie, Robert Tibshirani, and Jerome Fricdman. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in Neural Information Processing Systems, pp. 4572-45880 6 2016. ISSN 10495258. URL https://
arxiv. wttp://gggbbb///ooooobs /606. 03476v1
Guido W. Imbens and Whitney K. Newey. Identification and estimation of triangular simul- taneous equations models without additivity. Econometrica, 77:1481-1512, 9 2009. ISSN URL
1468-0262. doi: 10.3982/ECTA7108 ///o/inneiibraaaaaaeeeeeeeeeaeae wiley com doi httpp///aa/....../////1111111111 //onlinelibraryy wiley com wttp////...////111111/3
Fateme Jamshidi, Sina Akbari, and Negar Kiyavash. Causal imitability under context-specific inde- pendence relations. 2023. URL htts::/s:///axt org/ abs/2306. 00585v2.
Kakade and J. Langford. Approximately optimal approximate reinforcement learning, Interna- tional Conference on Machine Learning, 2002.
Zhaobin Kuang, Frederic Sala, Nimit Sohoni, Sen Wu, Aldo Córdova-Palomera, Jared Dunnmon, James Priest, and Christopher Ré. Ivy: Instrumental variable synthesis for causal inference. Proceedings of Machine Learning Research, 108:398 410, 4 2020. ISSN 26403498. URL arxiv wot.or////..... / 2004.000 05316.
https: //rpsi///xxvvv
Alex Kuefler, Jeremy Morton, Tim Wheeler, and Mykel Kochenderfer. Imitating driver behavior with generative adversarial networks. IEEE Intelligent Vehicles Symposium, Proceedings, 5:204- 211. 2017. doi: 10.1109/1VS2011.7995711. URL https: axxiv. org/ abs/ /1701. 06699v1.
12

---

Published as a conference paper at ICLR 2026
Daniel Kumor, Junzhe Zhang, and Elias Bareinboim. Sequential causal imitation learning with unobserved confounders. Proceedings of the 35th Conference on Neural Information Processing Systems, 8 2021. URL https:: arxiv. org/ abs 22008 06276v1
Yann Lecun, Urs Muller, Jan Ben, Eric Cosatto, and Beat Flepp. Off-road obstacle avoidance through end-to-end learning, Advances in Neural Information Processing Systems, 18, 2005. URL www.thttp//... /aann. lecun. com.
Luofeng Liao, You Lin Chen, Zhuoran Yang, Bo Dai, Zhaoran Wang, and Mladen Kolar. Prov- ably efficient neural estimation of structural equation model: An adversarial approach. Advances in Neural Information Processing Systems, 2020-December, 2020. ISSN 10495258. URL
https: /atpii///aaivv///aa/eeeeeeaaaava org/ abs 2007 01290v3.
Whitney K. Newey and James L. Powell. Instrumental variable estimation of nonparametric models. Econometrica, 11:1565-1577 9 2003. ISSN 44680226. doi: 10.1111/1468 0t00:/14466-0222200555599
Pedro A. Ortega and Daniel A. Braun. A minimum relative entropy principle for learning and acting. Journal of Artificial Intelligence Research, 38:475-511, 10 2008. ISSN 10769757. doi:
10. 3/.ai33. URL https: : /aaxivivv.///////////a//////aava www.xi..rgggaaaa.. org/abs 000000810.3000.0 3605v3.....
Pedro A. Ortega, Markus Kunesch, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Joel Ve- ness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Perolat, Tom Everitt, Corentin Tallec, Emilio Parisotto, Tom Erez, Yutian Chen, Scott Reed, Marcus Hutter, Nando de Freitas, and Shane Legg. Shaking the foundations: delusions sequence models for interaction and control. 10 2021. URL https:/ arxiv. org/ als 2110 10819v1.
Judea Pearl. Causality: Models, reasoning, and inference. Econometric Theory, 2000.
Samuel Pfrommer, Yatong Bai, Hyunin Lee, and Somayeh Sojoudi. Initial state interventions for de- confounded imitation learning. 7 2023. URL https: /arxivv...... org/ abs 2307. 15980v3.
Dean A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in Neural Information Processing Systems, 1, 1988.
Antonin Rafin,, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL 20 1364 html.
Stéphane Ross and J Andrew Bagnell. Efficient reductions for imitation learning. Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, 2010.
Stéphane Ross, Geoffrey Gordon, and Andrew Bagnell. A reduction imitation learning and structured prediction to no-regret online learning. Journal of Machine Learning Research, 15: 627-635, 11 2011. ISSN 15324435.
Kangrui Ruan, Junzhe Zhang, Xuan Di, and Elias Bareinboim. Causal imitation learning via inverse reinforcement learning. Proceedings the International Conference on Learning Representa- fions, 2023.
Kangrui Ruan, Junzhe Zhang, Xuan Di, and Elias Bareinboim. Causal imitation for markov decision processes: A partial identification approach. Advances in neural information processing systems, 37:87592-87620, 2024.
Stuart Russell. Learning agents for uncertain environments (extended abstract). In The Eleventh Annual Conference on Computational Learning Theory, 1998.
J. D. Sargan. The estimation of economic relationships using instrumental variables. Econometrica, 26:393, 7 1958. ISSN 00129682, doi: 10.2307/1907619.
Daqian Shao, Ashkan Soleymani, Francesco Quinzan, and Marta Kwiatkowska. Learning deci- sion policies with instrumental variables through double machine learning. Proceedings of the International Conference Machine Learning, 2024.
13

---

Published as a conference paper at ICLR 2026
Daqian Shao, Ashkan Soleymani, Francesco Quinzan, and Marta Kwiatkowska. Double machine learning for conditional moment restrictions: regression, proximal causal learning and beyond. 149500 6 2025. URL
org/ wwtpo/tggppp//////e/ee/t/ pdfpppdfs//s//eee 14950.
Iyy Shpitser and Judea Pearl. Complete identification methods for the causal hierarchy. Journal of Machine Learning Research, 9:1941-1979, 2008. ISSN 1533-7928. URL http:// org/ axg/paperse httpp/pppsss////// shpitser08a. html.
Rahul Singh, Maneesh Sahani, and Arthur Gretton. Kernel instrumental variable regression. Ad- vances in Neural Information Processing Systems, 32, 6 2019. ISSN 10495258.
Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian Ziebart, and Andrew Bagnell. Feedback imitation learning: The three regimes of covariate shift. 2021. URL https:
/ //arxiv.org//////...../.oo..o..a abs 220.. 02872v2.
Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell, and Zhiwei Steven Wu. Sequence model imitation learning with unobserved contexts. Advances in Neural Information Processing Systems, 35, 2022a. ISSN 10495258.
Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell, and Zhiwei Steven Wu. Causal imita- tion learning under temporally correlated noise. Proceedings of Machine Learning Research, 162:20877-20890, 2 2022b. ISSN 26403498. URL http:://axxiv.org/// wtt.o//gg/abb//////ea 2202. 01312v1.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi: 10.1009/R0S.22222.36611
Risto Vuorio, Johann Brehmer, Hanno Ackermann, Daniel Dijkman, Taco Cohen, and Pim Haan. Deconfounded imitation learning. 11 2022. URL https::///// http:s///axxvvvv/a org/ abs 2211.
02667v1.
Chuan Wen, Jierui Lin, Trevor Darrell, Dinesh Jayaraman, and Yang Gao. Fighting copycat agents in behavioral cloning from observation histories. Advances in Neural Information Processing Sys- tems, 2020-December, 10 2020. ISSN 10495258. URL http://///axiiooo/a////////0// 14876v1.
Philip G. Wright. The tariff on animal and vegetable oils. http::/4bb.orr//.0055544444 38: 619-620, 10 1928. ISSN 0022-3808. doi: 10.1086/254144,
Liyun Xu, Yutian Chen, Siddarth Srinivasan, Nando de Freitas, Arnaud Doucet, and Arthur Gret- ton. Learning deep features in instrumental variable regression. ICLR 2021 9th International
Conference Learning Representations, 10 2020. URL htLhttss://aaae /atp:i////xivv.///a wwwpxxivvvrrg.oooooooooo atsg/abbb2010 07154v4.
Junkun Yuan, Anpeng Wu, Kun Kuang, Bo Li, Runze Wu, Fei Wu, and Lanfen Lin. Auto iv: Counterfactual prediction via automatic instrumental variable decomposition. ACM Transactions
on Knowledge Discovery from Data 16:1-20, 1 2022. doi: 10. 10.1145/3494568. URL http:
//arxiv.erg/aaaaaaii....a abss 2210. 0588 4htt::http///m /x. doi. org 1tte///11.11//44455551
Junzhe Zhang, Daniel Kumor, and Elias Bareinboim. Causal imitation learning with unobserved confounders. Proceedings of the 34th Conference on Neural Information Processing Systems, 8
2020. URL https::// //axiiv
/2000 06267v2
14

---

Published as a conference paper at ICLR 2026
A REDUCING OUR UNIFYING FRAMEWORK TO RELATED LITERATURE
In this section, we discuss how the various previous works can be obtained as special cases of our unifying framework.
A.1 TEMPORALLY CORRELATED NOISE (SWAMY AI.,, 2022B)
The Temporally Correlated Noise (TCN) proposed in Swamy et al. (2022b) is a special case of our setting where 2² 0 and only the confounding noise uf present. Following Equation 14-17 of Swamy et al. (2022b), their setting can be summarised as
$t T(s-11,
= (5 + u4-1 + u4-2)
TT (st) u4 U1-1,
where is the transition function and u, are the TCN. can be seen that TCN is the confounding noise of since the expert policy doesn't 't take it into account, and it affects (or confounds) both the state and action.
It can be seen that this is special case of our framework when uf O, where (s))s) + +e(u)) from Equation (2), and more specifically when the confounding noise horizon Theorem 3.2 is 2. In addition, the theoretical results in Swamy et al. (2022b) can be deduced from our main results as shown in Corollary 4.7.
A.2 UNOBSERVED CONTEXTS (SWAMY ET AL., 2022A)
The setting considered Swamy et al. (2022a) is special of our setting when uf 0 and only u² are present. Following Section of Swamy et al. (2022a), their setting can be summarised
A D(S)
V T:Sx x:SxAxC= (-1,
a = C))
where e E C' is the context, which is assumed to be fixed throughout an episode, There are no hidden confounders in this setting and the context is included in u² under our framework. Note that in our setting we also allow u" to vary throughout an episode. In addition, the theoretical results in Swamy al. (2022a) can be deduced from our main results, as shown in Corollary 4.6.
A.3 IMITATION LEARNING WITH LATENT CONFOUNDERS (VUORIO ET AL., 2022)
The setting considered by Vuorio et al. (2022) is also a special case of our setting when af = 0 and only u are present, which is very similar to Swamy et al. (2022a). In Section 2.2 of Vuorio et al. (2022), they introduced a latent variable 0 θ that is fixed throughout ( 0). There are no hidden confounders this setting and the latent variable is included in u in framework. No theoretical imitation gap bounds are provided in Vuorio al. (2022). However, Corollary 4.6 can be directly applied to their setting and bound the imitation gap.
episode and a,
A.4 CAUSAL DELUSION AND CONFUSION (ORTEGA ET AL.,, 2021; DR HAAN ET AI., 2019; PFROMMER ET AL.,, 2023; SPENCER AL., 2021; WEN ET AI., 2020)
The concept of causal delusion (Ortega et al., 2021) and confusion is widely studied in the liter- ature (de Haan al., 2019; Pfrommer al., 2023; Spencer et al., 2021; Wen et al., 2020) from different perspectives. A classic example of causal confusion is learning to brake in an autonomous driving scenario. The states are images with a full view of the dashboard and the road conditions. The brake indicator in this scenario is the confounding variable that correlates with the action of braking in subsequent steps, which causes the imitator to learn to brake the brake indicator light is already Therefore, another name for this problem is the latching problem, where the imitator latches to spurious correlations between current action and the trajectory history. In the setting of Or- tega et al. (2021), this is explicitly modelled latent variables that affect both the action and state,
15

---

Published as a conference paper at ICLR 2026
causing spurious correlation between them and confusing the imitator. In other settings (de Haan et al., 2019; Pfrommer et al., 2023; Spencer et al., 2021; Wen et al., 2020), there are no explicit un- observed confounders, but the nuisance correlation between the previous states and actions can be modelled as the existence of hidden confounders uf in our framework. Specifically, in de Haan et al. (2019), 2t-1 and 0t-1 are considered confounders that affect the state variable 24, which causes spurious correlation between previous state action pairs and Oz. The spurious correlation between variables is typically modelled as the existence of hidden confounder uf that affects both variables in causal modelling. For example, the actual hazard or event that causes the expert to brake will be the hidden confounder uf that affects both the brake and the brake indicator.
However, despite the fact that this setting can be considered a special case of our general framework, we stress that the concrete and practical problems considered in de Haan al. (2019); Pfrommer et al. (2023); Spencer et al. (2021); Wen et al. (2020) are different from ours, where they assumed implicitly that the hidden confounders of are embedded the observations or outright observed.
B PROOFS OF MAIN RESULTS
In this section, we provide the proof' for the main results and corollaries in this paper.
B.1 IV CONDITIONS FOR ht-k
In this section, we verify that h is a valid instrument. Firstly, we derive uf IL h This follows from standard d-separation rules for causal graphs (Pearl, 2000). To establish this, we must verify that all paths from h4-* ($1, 01, S(=x) to uf are blocked in the graph, meaning that h is d-separated from uf, which implies h JL u². From our causal graph in Figure 1, we see that any paths from h, to uf must pass through a collider structure, specially through either 8, - a, + uf or at - 81+1 <- uf. Furthermore, potential paths through hidden confounders are ruled out because there are no direct causal paths between - and uf, as required by Assumption 3.2. Thus, all paths from h4= to uf, are blocked by d-separation, and can conclude that h2- IL uf. Secondly, Phe he=k) is not constant in fy- because we can assume that the environment is non trivial and the past state have an impact
future states. Finally, h doesn't directly affect de, specifically hi-k IL ag (st, uy, uf), by the Markov property - the next action at and the trajectory history conditionally independent given the current state 8t.
B.2 PROOF OF PROPOSITIONS
Proposition 4.3: The ill-posedness vTTI, k) is monotonically increasing as the confounded horizon increases.
Proof. From definition, we have that
I T
(III, k) sup TEll
E (a
(1t) 1 11
We would like to show for each E II, would imply that v(TI, k) is increasing. For each π E IT, we see that the numerator constant want the horizon k. Therefore, it is enough to check that for each II, the denominator |[aa, T (he decreases as k increases. For any two integer horizon k1 >> K2.
li |Tg- 7lle
is increasing k increases, which
Elae- T
= B[E[a, = =(hx)) =/ 1b,
(5)
< E|E(a, -=(-==)())))))----)) =(hh)) hic
(6)
Flaa (h4
by the tower property of conditional expectation as conditional expectations, and the fact that Eja, - x(hr)]/4 is he-ki measurable, respectively for each line. Therefore, we have that Efa, [h, is decreasing, which implies |E[a, T (h+))/a (h, is decreasing and vIII k) is increasing as & increases, which completes the proof.
C al14 Jensen's inequality for
16

---

Published as a conference paper at ICLR 2026
B.3 MAIN RESULTS FOR GUARANTEES ON THE IMITATION GAP
Theorem 4.5: Let it. be the learnt policy with CMR error and let vIII, k) be the ill-posedness of the problem. Assume that δrv (ug. Ex [m]]4.]] deterministic. Then, the imitation gap is upper bounded by
for 8 RR,, P(u5) c-TV stable and n is
J (TT) - J(+)) I (IE k) + 26) (6++))
Proof of Theorem 4.5. Recall that ((r) is the expected reward following म, and we would like to bound the performance gap ((T)) (Tr. between the expert policy TE and the learned history- dependent policy thh Let Qr (St, ae. uf) be the Q-function of т.. Using the Performance Differ- ence Lemma (Kakade & Langford, 2002), we have that for any Q-function (h,, ax) that takes in the trajectory history h2 and action dt.
T
J(TE) - J(Fh)) = EYTOE
Σ t=1
Qri (s, ac, ug) - Eawxi 1Qth (8t, a, 4,)]]
BTwTE (2ri (s,, ug) Q(is, + Q(1,, - Faawi, 1Q -Q+ +]]
E 12
Eanxi [²]
E E 10th. -Q -Eaws, 1Q+,
Q]] (8)
We first bound the second part of Equation (8). Denote by drv the total variation distance. For two distributions P. Q, recall the property of total variation distance for bounding the difference in expectations:
[p[f(=)] - [o] (r
In order to bound the second part of Equation (8), for any Q function, consider inferred using the conditional expectation of u2 the history h,
(h,, a+) == Qst,, 0e, F [x
where note that 8 E he We have that, when the transition trajectory (se, ue. uf, rz) ~ aE follows the expert policy, for any action à ~ π following some policy (in our case, it can be TE Rr),
[E [Q(s,, à, u.) Q(, a)] = ETnar (Q à, 4t) - Q(s à EToop (1,11
EugNER E (0 à, ui 14 - Euilhorn E [Q(s.. â, ux]l
(9)
E 1Q(se, à ui) u/ll & Egg (w]l)()) T drv (u) E (x]
(10)
(11)
T8
(12)
where Equation (9) uses the tower property of expectations, Equation (10) uses the total variation distance bound for bounded functions, Equation (11) uses the fact that the Q function is bounded by T' and Equation (12) uses the condition that (ug Eng [u]]/e]) < 5 in the theorem statement.
Since Equation (8) holds for any choice of Q we choose Qu, (h,, ax) = Q (st, ag Erong hh])) such that we can apply Equation (12) twice to bound the second part of Equation (8):
ErwER Er -rr, - EEnsi, 1Q -xn.ll S Braza 12r, - Q + Ennwi 1ri, 2rh
ES Q2i, Q.r,] + BRa. 1Q
Q
< ETTaE 12th - Qmall +88
(13)
$ 2T8
where Equation (13) holds by applying Equation (12) because the expectation of the trajectories (and their transitions) are over Ty, and the actions which are used only as arguments into the Q function are sampled from oी.-
17

---

Published as a conference paper at ICLR 2026
Next, bound the first part of Equation (8). Recall that the ill ill-posedness of the problem for policy class
TE
(I1, k) - sup
Ela - π (11 h
where ||T π 2 is the RMSE and | π
is the CMR error from our algorithm. Since the learned policy Th has CMR error of have that
|| 1lll S v (II, k) [[ th (h.))l S (III,
Next, recall that c-total variation stability of distribution P(u") where of E A for some space A implies for two elements ú1, 02 E A,
||a 02 la SA
ôrv(a1 + u ,2 + uf) S cc.
Since P(uf) c-TV stable w.r.t the action space A, we have that for all history trajectories h, H (note that 8, E lt)
ÔTV + us, (he) + uf) s cl|TE(S)) - ti (h:)ll..
Then, we have that by Jensen's 's inequality,
Bl (8T ui, 1f (h+) + ut]]² S (TV uT, (h, (h,) + x5) - ( + ni, +ं (h+) + +1)l s v ( u.. मM (1+) + x
S / Ene tं (h+)
= - fnl S cIv(
Therefore, by applying the total variation distance bound for expectations of Qxi over different distributions of action az, we have that
En 1mm - Eansi, 1Qma]] Er~ag 12 (he, ax) - ElQr, tf, (n.,))(())))
(14)
EN rx [1 (le, (8)) + ut)] - E[Qx, (he, t (h+)
uf]l]
(15)
< 1 al FAi (drv (F(TT(())) (F + wτ), ((r + +())]
(16)
s TCEv(II, k)
(17)
Combining all of above, we see that from Equation (8). by selecting Qr,, (he,
Qxn (St, a2, Erang [u] hel), the imitation gap can be bounded by
T
J(TE) - J(Th)) =
E Erwwo 12m. -Eanti 12+ Σ 1=1
Σ
Eraea 1Qri. - Q+i -Eanii Eantil1r - Q
1=1
(18)
T
Tesv(II, + Σ 2T8
(19)
t=1
< T (Tesv(L1, k) 2T')
(20)
= (eEv((I), k) + 25) = O
(21)
which concludes the proof.
B.4 PROOFS OF COROLLARIES
Corollary 4.6:
the special case that uf = 0, meaning that there is no confounder observable to the expert, ut = Eng [u]] [h),, meaning that uf is ohh) measurable (all information regarding uf is represented in the history), the imitation gap bound (IIV k)), which coincides with
Theorem 5.1 of Swamy et al. (2022b).
18

---

Published as a conference paper at ICLR 2026
Proof. If up O, then we have in Exp [he] since uz is constant. If uz Erro [ug] have that
êTV (ug, Eig [u] (he]) 8TV (ug. u() 50
By plugging å = O into Theorem 4.5, we have that ((TT) J( I T (IE k:), which is the same as the imitation gap derived in Swamy et (2022b) and completes the proof.
Corollary 4.7: In the special case that ui 0, if the learned policy via supervised BC has error E, then the imitation gap bound is ( (d) + 26), which is a concrete bound that extends the abstract bound in Theorem 5,4 of Swamy et al. (2022a).
Proof. Theorem 5.4 of Swamy et al. (2022a), for the offline case, which the setting we are considering (as opposed to the online settings), they defined the following quantities for bounding the imitation gap in a very general fashion,
Eoft sup E, EYsTE 122 Eawx' [Q]]
6ott sup EranE 19th - Q Eawnt 12 QQ]. QxQ
The imitation gap by Theorem 5.4 in Swamy al. (2022a) under the assumption that uf = 0 is (EaM + SofF), which can also deduced from Equation (8) by naively applying the above supre- mum. obtain a concrete bound, we can provide tighter bound for Eyang 12th Ea 12,,.l, which is the first part of Equation (8), given that uf 0.
For two elements 01, 02 E A, we have that by Cauchy -chhaa,
δTV (a: + 0, 02 + 0) = 2 ||al - a2)1 S /iim A)
|al - a2|12.
2
Then, we have that
2
||.1 02/12 < A orv(ai, 5
A
Vdim(A) A)
so that by Theorem 4.5,
Enwog 1 -Eaoti, [Q+ ET 14r,, (he, uz) B(Qx,, (lu,, +in (h+))]]
(22)
ENNNTE [E] Qx,, (he (g()))) El (he oi (hx))]
(23)
2
12 1l
| 1
(24)
V ddim(A)
2
ST
(25)
V (A)
since when of = 0 the learning error via supervised learning is e : || T/12. Therefore, the final imitation bound following Theorem 4.5 is
T
J(#g) - J(rh)) =
E E 12m - -Eanti 12 11
E 1Qri. - Q+ - -Eanii 1Qri - Q
1=1
(26)
T
2
T
Σ
2T8
(27)
dim A)
1=1
t=1
2 /im (A)
+ 26).
(28)
This bound a concrete bound, obtained through detailed analysis of the problem at hand, that coincides with the abstract bound 2 (Eott + don)) provided in Theorem 5.4 of Swamy al. (2022b). Note that this bound independent of the ill-posedness v (III, k) and the c-TV stability of uf, which present in the bound of Theorem 4.5, because of the lack of hidden confounders uf- n
19

---

Published as a conference paper at ICLR 2026
Piane Ticket Mrcng
Plane Ticket Pricing
1.0
Reward
0.5
methed
Squared Error
6C
EC-DO
2-4
Noleed Expert
DPV
Deap
3
11 horizon
13
15
17
29:20
3
13
13
17
29:20
L honizon
Figure 4: Additional results for the MSE between learnt policy and expert, and the average reward, in the plane ticket environment (Example 3.1), with DFIV and DeepGMM as the CMR solver.
1.0
2-4
0.5
0.5
2-2
Resiall
Squaren
0.4
DMLIL (ou5))
2.4
Noleed Expert
DFVV
2.4
13
17
39
13
17
horzon
honzon
Figure 5: Additional results for the MSE between learnt policy and expert, and the average reward, Ant Mujoco environment, with DFIV and DeepGMM as the CMR solver.
C ADDITIONAL EXPERIMENTS
C.I ADOPTING OTHER IV REGRESSION ALGORITIMMS
In this paper, we have transformed causal IL with hidden confounders into CMR problem as de- fined in Equation (4). Therefore, in principle, many IV regression algorithms can be adopted to solve our CMR problem. We also experimented with other IV regression algorithms that have been previously shown be practical (Shao et al., 2024) for different tasks and high-dimensional input. Specifically, we experimented with DFIV (Xu et al., 2020), which is an iterative algorithm that inte- grates the training of two models that depend on each other, and DeepGMM (Bennett et al., 2019b), which solves minimax game by optimising two models adversarially. Note that DeepIV (Hartford et al., 2017) can be considered a special case of DML-IV (Shao et al., 2024), so we did not evaluate it.
The additional results for using DFIV and DeepGMM as the CMR solver are provided in Figure and Figure 5. It be seen from Figure 4 that only DFIV achieves good performance in the airline ticket pricing environment, surpassing the performance of ResidulL. For the Ant Mujoco task Figure 5, both DFIV and DeepGMM fail to learn good policies, with only slightly lower MSE than BC and BC-SEQ. We think this is mainly due to the high-dimensional state and action spaces and the inherent instability in the DFIV and DeepGMM algorithms. For DFIV, the interleaving of training of two models causes highly non-stationary training targets for both models, and, for DeepGMM, the adversarial training procedure of two models is similar to that of generative adversarial networks (GANs), which are known to be unstable and difficult to train. In addition, when the CMR problem weakly identifiable, as the case of a weak instrument, the algorithms may converge to local minima that are far away from the true solution in the face of instabilities in the algorithm.
We conclude that solving our CMR problem can sensitive to the choice solver as well as to the choice of hyperparameters. In addition, some IV regression algorithms do not work well with high-dimension inputs. Our IV algorithm of choice, DML-IV, provides a robust base for the DML-IL algorithm that demonstrated good performance across all tasks and environments. This demonstrates the benefit of using double machine learning, which can debias two-stage estimators and provide good empirical and theoretical convergence.
20

---

Published as a conference paper at ICLR 2026
C.2 PERFORMANCE UNDER MISSPECIRICATION OF i
When past unobservable confounders ui_ are weakly correlated with the current uf, the uncon- founded instrument condition for valid IV is mildly violated. Empirically. when the violation is mild, it typically induces small bias. This is especially true if the correlation between the IV and hidden confounder weak relative IV strength (Hahn & Hausman, 2005), i.c., the correlation be- tween h4- and the current state 8t- It is also often observed that there is threshold effect (Kuang al., 2020), where once the violation rises above a certain threshold, IV regression begins to induce large bias.
However, to the best our knowledge, there no theoretical framework that can analyse IV regres- sion bias with respect to IV violation with guarantees. In fact, in theoretical worst-case, a weak correlation between the IV and the hidden confounder could potentially cause the causal effect to be unidentifiable, rendering causal inference tools ineffective.
That being said, there also exist methods that can combine weak or mildly invalid IVs to synthesise valid IVs (Kuang et al., 2020; Hartford et al., 2020; Yuan et al., 2022) and it would be possible to combine the trajectory history h4- which may contain invalid IVs, into a valid IV.
To empirically evaluate this, we conduct additional experiments where the true confounding horizon 10. but DML-IL is given the misspecified = to With & = 10 as the baseline without misspecification, performance (avg reward) in Half Cheetah stays within 5% of the baseline down to & 6, and remains acceptable down to = 8 in the plane ticket task, after which DMI-III starts induce larger bias. We report the average reward together with its standard deviation (in parentheses).
Misspecified k k= 10 (no misspecification)
Half Cheetah
Plane Ticket
0.7183 (0. (0.1789)
0.6181 (0.0356)
k=9
0.7108 (0.1193)
0.5973 (0.0242)
k=8
0.7209 (0.1717)
0.5546 (0.0325)
k=7
0.6675 (0.1595)
0.4801 (0.0614)
k=6
0.6903 (0.1393)
0.3944 (0.0682)
k=5
0.3471 (0.1989)
0.3241 (0.0773)
k=4
0.3243 (0.2329)
0.1561 (0.0961)
k=3
0,2749 (0.1643)
0.1076 (0. 1310)
k=2
0.1082 (0.2155)
0.0801 (4.14
k=1
0.0896 (0.3080)
0.0656 (0. 1227)
Table Performance across misspecified values for Half Cheetah and Plane Ticket.
D ENVIRONMENTS AND TASKS
D.1 DYNAMIC AEROPLANE TICKET PRICING
Here, we provide details regarding the dynamic aeroplane ticket pricing environment introduced in Example 3.1. The environment and the expert policy are defined follows:
S = R
(29)
A = [-1,1]
(30)
8t sign(s 4n u
(31)
TE = clip( s/u,, -
(32)
0: mE 10 uf
(33)
" mean(pt ~
(34)
uf mean(g: Normal(O, 0.1 .... 4::1. 1.... (t-k+1)
(35)
where M is the influence horizon of the expert-observable u², which set to 30. The states 8; the profits at each time step, and the actions a, are the final ticket price. uf represent the seasonal patterns, where the expert mh will try to adjust the price accordingly. of represent the operating
21

---

Published as a conference paper at ICLR 2026
costs, which are additive both to the profit and price. Both ug and uf are the mean ii.d samples, qt and Pt. and vary across the time steps by updating the elements the set at each time step. This construction allows uf and uī_x k to be independent since all set elements q will be re-sampled from time step t to t. We multiply the standard deviation of qe by vk to make sure uf, which is the average over k i.i.d. variables, has the same standard deviation for all choices of k.
set of
D.2 MUJOCO ENVIRONMENTS
We evaluate DML-IL three Mujoco environments: Ant, Half Cheetah, and Hopper. The original tasks do not contain hidden variables, so we modify the environment to introduce u and wº. We the default transition, state, and action space defined in the Mujoco environment. However, changed the task objectíves by altering the reward function and added confounding noise to both the state and action. Specifically, instead of controlling the ant, half cheetah, and hopper, respectively, to travel as fast as possible, the goal is to control the agent to travel at a target speed that is varying throughout an episode. This target speed is u", which is observed by the expert but not recorded in the dataset. In addition, we add confounding noise uf to and to mimic the environmental noise such as wind noise. In all cases, the target speed uf, confounding noise uf, and the action a, are generated as follows:
+ 20 of
(36)
ut mean
2: P......MM
(37)
" mean(1: (0,mal 0.01 vk), (1), lt-k+1)
(38)
where M = 30, the state transitions follow the default Mujoco environment and the expert policy TE learned online the environment, uf and uf follow the aeroplane ticket pricing environment to be the average over a queue of i..dd. random variables. The reward is defined to be the bealthy (current velocity - x2)² 2 control loss, where 1 healthy gives reward 1 as long as the agent is in healthy state as defined the Mujoco documentation. The second penalty term penalises deviation between the current agent's 's velocity and the target velocity ug. The control loss term is also as defined in default Mujoco, which is 0.1 (a²) at each step to regularize the size of actions.
D.2.1 ANT
In the Ant environment, we follow the gym implementation with an 8-dimensional action space and 28-dimensional observable state space, where the agent's position is also included in the state space. Since the speed uf is not recorded in the trajectory dataset, we scale the current position of the agent with respect to the target speed, post = post-1 post ppcs-ppos-1 and use the new agent position pos, in the observed states. This allows the imitator to infer information regarding ug from trajectory history, namely from the rate of change in the past positions.
D.2.2 HALF CHEETAH
In the Half Cheetah environment, we follow the gym implementation 5 with a 6-dimensional action space and an 18-dimensional observable state space, where the agent's position also included in the state space. Similarly to the Ant environment, we scale the current position of the agent to pos;. = pos,-1 + pos. -oos-1 such that the imitator can infer information regarding ug from trajectory history.
D.2.3 HOPPER
In the Hopper environment, we follow the gym implementation with a 3-dimensional action space and 12-dimensional observable state space, where the agent's 's position is also included in the state space. Similarly to the Ant environment, we scale the current position of the agent to pos
Half Cheetah environment: http://ww..ooo...eeeeeeeeeeeeeee cheetah/
Ant environment: https:
Hopper environment: https:/ http///wwwwggg www.gyml abbarrrrrrr. devryddve/evviii www.errononnnnnnnneee
22

---

Published as a conference paper at ICLR 2026
Table Network architecture for DML-IL. For mixture of Gaussians output, we report the number of components. No dropout is used.
(a) Roll-out model M
Layer Type
Configuration
Input
state dim x
FC + ReLU
Out: 256
FC + ReLU
Out: 256
MixtureGaussian
components; Out: state dim x k
(b) Policy model fr.
Layer Type
Configuration
Input
state dim x (k+3)
FC + ReLU
Out: 256
FC ReLU
Out: 256
FC
Out: action dim
p +
pos: 1 "?
such that the imitator can infer information regarding ug from trajectory
history.
E IMPLEMENTATION DETAILS
Experiments are carried out Linux server (Ubuntu 18.04.2) with two Intel Xeon Gold 6252 CPUs, and each experiment run uses a single NVIDIA GeForce RTX 2080 Ti GPU for neural net- work training.
E. EXPERT TRAINING
The expert in the aeroplane ticket pricing environment is explicitly hand-crafted. For the Mujoco environments, we used the (SAC) and the default hyperparameters for each task outlined by SabbleBaseeline.. The expert policy an MLP with two hidden layers of size 256 and ReLU activations, and we train the expert for 10" steps.
(Raffin et al., 2021) implementation of soft actor-critic
E.2 IMITATOR TRAINING
With the expert policy TE, we generate 40 expert trajectories, each of 500 steps, following our previ- ously defined environments. Specifically, the confounding noise is added the state and actions and crucially uf is not recorded in the trajectories. The naive BC directly learns E[a, 8t] via supervised learning. Residull. mainly follows the implementation of Swamy al. (2022b), where we adapt to allow a longer confounding horizon & >> 1. For DML-IL and BC-SEQ. history -estory-de policy used, where we fixed the look-back length to be & + 3, where
the confounding horizon. BC- SEQ then just learns Efoa h,] via supervised learning, and DML-IL is implemented with K- following Algorithm 2. The policy network architecture for BC, BC-SEQ, and ResidulL are layer MLPs with 256 hidden size. The policy network it and the mixture of Gaussians roll-out model M for DML-IL have similar architecture, with details provided in Table 2. We the AdamW optimizer with a weight decay of 10 10-4 and learning rate of 10-4 The batch size is 64 and each model is trained for 150 epochs, which sufficient for their convergence.
E.3 IMITATOR EVALUATION
The trained imitator is then evaluated for 50 episodes, each 500 steps in the respective confounded environments. The average reward and the mean squared error between the imitator's action and the expert's action are recorded.
23

---

Published as a conference paper at ICLR 2026
F PRACTICAL CONSIDERATIONS FOR DML-IL
DML-IL can also be implemented with K'fold cross-fitting, where the dataset is partitioned into K folds, with each fold alternately used to train Rn and the remaining folds to train M. This ensures unbiased estimation and improves the stability of training. The base IV algorithm DML-IV with K-fold pross-fitting is theoretically shown to converge at the rate of O(N (Shao et al., 2024), where N is the sample size, under regularity conditions. DML-IL with K-fold cross-fitting (see Appendix G for details) will thus inherit this convergence rate guarantee.
Discussion on the Confounding Noise Horizon. Note that Algorithm 1 requires the confounding noise horizon & as input. Although the exact value of k can be difficult to obtain practice, any upper bound & of k is sufficient to guarantee the correctness of Algorithm 1, since f is also valid instrument. Ideally, we would like a data-driven approach to determine k. Unfortunately, the confounding horizon k equivalently the validity of h4- as IV, generally cannot be definitively verified using empirical data, especially the unconfounded instrument condition (i.c., h4- -l uf).
Therefore, rely the user provide sensible choice of i based the environment that does not substantially overestimate k, informed by domain knowledge about the task. However, there exist tests that can indirectly check whether candidate IV is valid, such the overidentification tests (Hansen, 1982; Sargan, 1958), conditional independence tests between the instrument and the residual (Gretton al., 2005; Fukumizu al., 2008), and sensitivity analysis (Conley al., 2012). It would be interesting future work to incorporate these methods to help identify k. In Appendix C.2, we additionally evaluate the performance and sensitivity of DML-IL under misspecification of k.
Discussion on the Additive Noise Assumption. The additive noise assumption in Theorem 3.3 is key identification assumption and is standard in IV regression (Pearl, 2000). If the additive noise is misspecified,, e.g., multiplicative or complex non- non-linearity, then the derivation of the CMR in Equa- tion (4) breaks down. However, this limitation of DML-IL arises from the fact that, with non-additive confounding noise and without further assumptions, the causal effect is generally unidentifiable (Im- bens & Newey, 2009). Therefore, while the additive noise assumption may be simplistic in complex settings such as healthcare, it is the best we can do without further assumptions.
The validity of additive noise can often be justified through domain knowledge. For example, in physical systems such as drones or aircraft, directional environmental noises such as wind and vi- brations affect the position of a drone or plane additively. In econometrics applications, confounding noises, when quantified in monetary terms, naturally aggregate additively into total cost or revenue. Finally, it is worth noting that this assumption only requires the expert action (i.e., the outcome) to have additive noise, whereas the relationship between the confounding noise and the state (i.e., the treatment) is unrestricted.
G BACKGROUND ON DML AND DML-IL WITH K-FOLD CROSS-FITTING
Double Machine Learning (DML) (Chernozhukov et al., 2018) is a statistical technique that debiases two-stage regressions. In the DML framework, a function of interest f is estimated in two stages. In the first stage, some parameters (which can be ciinite-dimensional functionals) that are necessary for the second stage estimation are estimated. In the second stage, first stage estimators are plugged in to estimate the function of interest f. Shao et al. (2024) utilised the DML. framework to propose DML-IV, which is two-stage IV regression algorithm. DML-IV is also a general CMR solver (see DML-CMR, a generalisation of DML-IV proposed by Shao et al. (2025)) that can be used to solve general CMR problems. In Shao al. (2025), score (criterion) function that describes general CMR problems was proposed; the score function guarantees Neyman orthogonality for estimating solutions to CMR problems. Our CMR objective Ela, (h+) h = 0 fits directly into the
CMR framework of DML-CMR. In our adaptation in Algorithm 1, the rollout model M serves as the nuisance component, and the second stage estimates Th using this orthogonal score.
In Shao et al. (2025), the authors show that DML-CMR can achieve 0(2-1(2) convergence rate, where N the sample size, if implemented with K K-fold cross-fitting under some standard DML conditions. Next, we introduce Algorithm 2, which is version of DMI.-II with K-fold cross-
24

---

Published as a conference paper at ICLR 2026
Algorithm 2 DML-IL with K-fold cross-fitting
Input: Dataset D of expert demonstrations, Confounding noise horizon ki, number of folds K for cross-fitting
Output: A history-dependent imitator policy #t Get a partition (Ix) of dataset indices [M] of trajectories for A to K do
E = []] \\ 1
Initialize the roll-out model M, mixture of Gaussians model repeat
Sample (h, an) from data {(DE,)) i 1
Fit the roll-out model (h4 (1) M,(h+x)) maximize log likelihood until convergence
end for
Initialize the expert model a as neural network repeat
for k == to K do Sample hĄ from {(DE,) : 1x} Generate h2 and â using the roll-out model M, Update n.h to minimise the loss l : l|ā, nn (h:) l|2 end for until convergence
fitting, and discuss the specific conditions required for DML-IL to achieve 0(2-1/2) convergence rate,
G.1 DML-IL WITH K-FOLD CROSS-FITTING
Here, outline DML-IL with K-fold cross-fitte cross-fitting. The algorithm is shown in Algorithm 2. The dataset is partitioned into K folds based on the trajectory index. For each fold, we use the leave-out data, that is, indices g = [N] TR. to train separate roll-out models M, for E [1..K. Then, to train single expert model n.. we sample the trajectory history hę- from each fold and use the roll-out model trained with the leave-out data to complete the trajectory and train The This technique is very important in Double Machine Learning (DML) literature (Shao al., 2025; Chernozhukov al., 2018) for it provides both empirical stability and O(N-1/2) convergence rate guarantees.
The conditions required for this root-N consistency are standard DML-CMR conditions ((Shao al., 2025), Condition 4), which includes identifiability conditions, orthogonality and nuisance convergence rate of -1N44 The identifiability conditions are satisfied if we have valid instru- ment, and the orthogonality is guaranteed by the score function in Shao et al. (2025). The nuisance rate requires that our nuisance parameter converges at ||M M1 = o(N-1/4), which is usually achieved by density estimation models such as mixture Gaussian (see discussion before Theorem in Shao et al. (2025)). Therefore, DML-IL with K-fold cross-fitting will thus inherit this conver- gence rate guarantee if all the above conditions are satisfied.
25


Reviews:
[{"review_id": "8qyJox1cE2", "reviewer": "Reviewer_iKXV", "summary": "This paper proposes a framework for causal imitation learning with hidden confounders that distinguishes two types of latent confounders: (i) expert-observable confounders (visible to the expert but not to the imitator) and (ii) expert-unobservable confounders (hidden to both). Using trajectory histories as instrumental variables, the approach reduces imitation learning to a conditional moment restriction problem. They prove an imitation-gap upper bound that recovers prior bounds as special cases, and provide empirical validation on a toy pricing environment and modified MuJoCo tasks where their algorithm outperforms some baselines.", "questions": "1. Could you please clarify why algorithm 1 is a double machine learning algorithm? What are the nuisance parameters? How is the Neyman orthogonality guaranteed? It is simply mentioned that under \"regularity assumptions\" the estimator converges at root-n rate. I do not think achieving root-n convergence (especially without cross-fitting) is as simple as that, perhaps Donsker-type conditions are meant as \"regularity conditions?\" I believe the authors should at least name the assumptions for convergence, and with the Gaussian mixture roll-out, a more careful treatment is required.\n\n---\n2. When proving Eq. (4), under the derivation of CMR problem on page 5, the authors use $u^\\epsilon_t \\perp\\\\!\\\\!\\\\!\\perp h_{t-k}$, whereas the assumption they have made is $u^\\epsilon_t \\perp\\\\!\\\\!\\\\!\\perp u^\\epsilon_{t-k}$. Could you elaborate on how this assumption implies $u^\\epsilon_t \\perp\\\\!\\\\!\\\\!\\perp h_{t-k}$? It does not sound trivial to me.\n\n---\n3. Example 3.1 is meant to make the paper clearer, but it sort of adds to the confusion. What is the action set here? Does the agent set the price or does it set the profit margin? If it sets the price, then the example is confusing and does not fit into the paper. If it sets the profit margin, then I find the example unrealistic as this would mean that the airline can set the profit margin but not the price?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Casting imitation under confounding as a CMR with trajectories as instruments is conceptually simple and lets the authors leverage existing IV/CMR machinery.\n\n- The paper shows corollaries recovering Swamy et al. results as special cases.", "weaknesses": "A confusing point of this paper is the arrow from $u_\\epsilon$ to $a$, which means that even the expert does not have full control over its actions--the expert might choose to take one action, but another action take place in the end because of $u_\\epsilon$. I did not get how this is justified.\n\n---\n>\"We assume the confounding noise is additive to the action, which is standard in causal inference. Without this assumption, the causal effect becomes unidentifiable.\"\n\nI do not think that this is really 'standard'. The claim of unidenitifiability in case it is not assumed also requires more elaboration, as the setting of this paper is not exactly that of Balke and Pearl 1994. Most importantly, this assumption, as well as the first equation on page 5 at the beginning of Section 4 only make sense if the action $a_t$ is continuous, which is a point that needs be clarified.\n\n---\nIt is not clear how Assumption 3.2 implies what is being used in the derivations afterwards. There is a missing step in the proofs which doesn't sound straightforward to me. Unless I'm missing something, there is an implicit assumption made that is not mentioned in the paper. See my question 2 below.\n\n---\nAlgorithm 1 is adapted from Shao et al, 2024, but with the current form of adaptation, it is not clear whether or not convergence guarantees transfer from that paper. See my question 1.\n\n---\nThere are a few symbols and notions that are used on the go without being defined at all, which makes following the paper difficult. For example, policy $\\pi_h$ is defined as a function from $\\mathcal{H}$ to $\\Delta(A)$, but $\\Delta()$ is never defined. So it is not even clear what we are looking for.\nSame goes for dim(A) in Corollary 4.7 which is never defined, although it is guessable unlike $\\Delta(A)$.\n\n---\nThe plots of Figures 2 and 3 are too small, and the font size is so small that the text is almost not readable. \n\n---\nMinor:\n\nFigure 1: dotted -> dashed, last paragraph of page 9: one limitation are -> one limitation is", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xNfr1boj3o", "reviewer": "Reviewer_YSLm", "summary": "This paper proposes a unifying framework for causal imitation learning (IL) with hidden confounders, distinguishing between expert-observable confounders (variables observed by the expert but not the imitator) and expert-unobservable confounders (confounding noise hidden from both). By leveraging trajectory histories as instrumental variables, the authors reformulate causal IL as a Conditional Moment Restriction (CMR) problem and propose DML-IL, an algorithm that solves this CMR via instrumental variable regression. They provide an upper bound on the imitation gap and demonstrate empirically on continuous state-action environments that DML-IL outperforms existing causal IL baselines.", "questions": "- How do the experimental results quantitatively compare to the theoretical bounds from Theorem 4.5? Can the authors provide concrete values for the variables in the theorem in the experimental settings to assess bound tightness?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality: The distinction between expert-observable and expert-unobservable confounders provides a novel and more realistic problem formulation that generalizes several prior works. The neat reframing of causal IL into instrumental variables and CMR problems is elegant and theoretically well-motivated.\n- Quality: The theoretical analysis is solid, with the neat theory on the imitation gap bound (Theorem 4.5) that recovers prior results as special cases (Corollaries 4.6 and 4.7).\n- Clarity: The paper is generally well-written with clear motivation through Example 3.1 and good use of figures (especially Figure 1). The connection between hidden confounders and the CMR formulation is explained systematically.\n- Significance: The unifying framework addresses a realistic scenario where multiple confounding factors coexist, which is practically important and has been overlooked in prior work that typically addresses confounders in isolation.", "weaknesses": "- Limited scope of unification: The framework doesn't truly subsume Vuorio et al. (2022), which solves the problem through environment interaction. The experimental results don't include comparisons to interactive scenarios, limiting the claim of being a \"unifying\" framework. The paper focuses exclusively on offline IL from fixed demonstrations.\n- Gap between theory and experiments: No analysis is provided comparing experimental results to the theoretical bounds from Theorem 4.5, making it unclear how tight these bounds are in practice.\n- Algorithm derivation unclear: The connection between the CMR error minimization and Algorithm 1's two-stage procedure needs better motivation. The paragraph below Algorithm 1 explaining why training on expert trajectories does not work, is difficult to follow.\n- Restrictive assumptions: The framework requires knowledge of (or an upper bound on) the confounding noise horizon k, which cannot be verified empirically. The additive noise assumption (Assumption 3.3) is quite strong and limits applicability. The authors don't fully discuss the effect of these limitations.\n\nMinor:\n- In Definition 4.2, the text describes \"the ratio between the learning error... and its L2 error\" but the equation shows the reciprocal of this ratio (L2 error in numerator, learning error in denominator).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5WPy1VKrSI", "reviewer": "Reviewer_mgqr", "summary": "This paper proposes a unifying framework for causal imitation learning in the presence of hidden confounders, distinguishing between two types: variables observed by the expert but not the imitator (expert-observable), and noise hidden from both parties (expert-unobservable). By leveraging trajectory histories as instrumental variables and assuming a bounded confounding noise horizon along with additive noise structure, the authors reformulate the imitation problem as a Conditional Moment Restriction (CMR). They introduce DML-IL, a novel algorithm based on double machine learning to solve this CMR, and provide a theoretical upper bound on the imitation gap that generalizes prior results. Empirical evaluations on both a custom airline pricing task and modified MuJoCo environments demonstrate that DML-IL outperforms existing causal imitation baselines when both types of confounders are present.", "questions": "- How robust is the proposed method against model misspecification, particularly with respect to the noise horizon at Assumption 3.2?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is clearly structured and presents a compelling case for a more holistic approach to addressing hidden confounding in imitation learning. The proposed framework elegantly subsumes several prior settings, and the experimental results convincingly validate the method’s effectiveness across diverse environments. The theoretical analysis is rigorous and meaningfully connects to established results in causal inference and econometrics.", "weaknesses": "- The framework relies on two key assumptions that may limit its applicability in certain domains. First, the additive noise assumption (i.e., that unobservable confounders affect actions additively) is essential for identifiability but may be overly restrictive in complex systems where interactions are nonlinear or multiplicative (e.g., biological or financial systems). While this assumption is standard in instrumental variable literature, its validity must be carefully assessed per application.\n\n- The method requires knowledge (or a valid upper bound) of the confounding noise horizon k —the temporal range over which unobservable noises remain correlated. Although the paper discusses practical heuristics and conditional independence tests to estimate k, this parameter is not identifiable from data alone, and misspecification (especially severe underestimation) can degrade performance, as shown in the ablation studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lwtcD4hvQ0", "reviewer": "Reviewer_2XU2", "summary": "This paper proposes a causal imitation learning (IL) framework that explicitly models two forms of hidden confounding. It leverages trajectory histories as instruments to convert confounded IL into a conditional moment restriction (CMR) problem and introduces DML-IL, an instrumental-variables–based algorithm to solve it.  The authors establish an upper bound on the imitation gap that recovers earlier results as special cases, and demonstrate in continuous-control benchmarks (including MuJoCo) that DML-IL consistently outperforms existing causal IL baselines.", "questions": "## Identifiability under missing expert covariates\n\n- Your formulation assumes that the expert’s additional covariates $u_{t}^{o}$ are never directly observed in demonstrations (only indirectly via histories). Can you clarify whether your identification or bounds require any minimal observability or recoverability conditions on $u_{t}^{o}$?\n- [1] studies the scenario where exact imitation is impossible. How does your analysis relate to partial-identification results showing impossibility from demonstrations when state/action/reward are jointly confounded [1]? You note that without assumptions the expert policy is unidentifiable; please delineate precisely which of your assumptions (e.g., Additive Noise, Confounding Noise Horizon) are sufficient to escape [1]’s impossibility and where residual partial identification remains. It might be better if the authors could add such discussions comparing with [1] in their paper.\n\n## Confounding noise horizon\n\n- Algorithm 1 requires k as input. However, in practice, how do we know k beforehand? Do we need to run causal discovery algorithms to obtain k? Additionally, can you provide a data-driven procedure used in your experiments, report sensitivity to $k$ or $\\bar{k}$, and include overidentification or CI-based diagnostics per environment?\n- In the worst case k=T−1, the instrument is the entire past. With continuous, high-dimensional states, both statistical and computational complexity can scale poorly. In that case, the scenario is similar to POMDP. Please characterize how sample complexity and runtime of DML-IL scale with k, T and state/action dimensionality.\n- Since the validity $h_{t-k} \\perp u_{t}^{\\varepsilon}$ is very hard to be verified, how do the proof steps change if the instrument is slightly invalid (e.g., $\\delta_{\\mathrm{IV}}$ violation)? Robustness version of Theorem 4.5?\n- Is $\\delta$ treated as a structural constant, or can it be bounded/estimated from data (e.g., via proxy tests) ? Can you provide an example showing how large $\\delta$ can be in typical MuJoCo settings?\n\n## Additive-noise assumption\n\n> Actions are generated via $a_{t}=\\pi_{E}\\left(s_{t}, u_{t}^{o}\\right)+u_{t}^{\\varepsilon}$.\n\n-  What if in general like $a_{t}=\\pi_{E}\\left(s_{t}, u_{t}^{o}, u_{t}^{\\varepsilon}\\right)$ non-additive cases [1][2], how could you deal with them?\n- Could you bound the bias of DML-IL under structured departures from additivity (e.g., small multiplicative terms) or provide a sensitivity analysis (e.g., \"plausibly exogenous\") approaches?\n\n## Experiments\n\n> we generate 40 expert trajectories, each of 500 steps, following our previously defined environments. \n\n- Why 40×500 specifically? Please report sensitivity to the number/length of demos.\n\n- K-fold details. You mention K-fold cross-fitting and give Algorithm 2. What K did you use in each experiment? Were folds stratified by environment/noise setting? How sensitive are results to K (1, 2, 5, 10)? Please include means ± CIs across seeds.\n\n- Code and seeds. Please release code, random seeds, and environment configs (including the exact noise processes) so others can reproduce the roll-out model training and cross-fitting setup described in Appendix\n\n## Scope\n\nFinally, I strongly recommend the authors to change their claim about \"**Unifying Framework**\" for Causal Imitation Learning. There are lots of cases that cannot be covered by your framework (i.e., the current scope excludes several practically common regimes), e.g., violations of the additive-noise assumption (which is quite common), stochastic experts, sub-optimal experts, invalid history instruments, unknown or unbounded confounding horizons, partial observability, different causal diagrams and so on. These are important and active research fronts; acknowledging them explicitly—ideally with a short taxonomy of covered vs. uncovered regimes and a softened headline claim —would make the paper’s contributions more precise and credible while pointing concretely to valuable directions for future work.\n\nI would be _inclined to raise my score_ if the revision clarifies scope as suggested and properly answers my questions above.\n\n-----\n\n\n[1] \"Causal imitation for markov decision processes: A partial identification approach.\" Advances in neural information processing systems 37 (2024): 87592-87620.\n\n[2] \"Causal imitation learning via inverse reinforcement learning.\" The Eleventh International Conference on Learning Representations. 2023.\n\n[3] \"Causal imitation learning under temporally correlated noise.\" International Conference on Machine Learning. PMLR, 2022.\n\n[4] \"Invariant causal imitation learning for generalizable policies.\" Advances in Neural Information Processing Systems 34 (2021): 3952-3964.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- some important assumptions are clearly stated\n- some motivating examples (e.g., Example 3.1) are provided\n- an expansion of [3], with a longer confounding horizon $k > 1$", "weaknesses": "- Some important relevant papers are not included (e.g., [1]).\n- Claims outpace evidence. The \"unifying\" scope is asserted more strongly than it is demonstrated.\n- Some assumptions are hard to be verified in practice\n\nSpecific questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)