What Would the Expert $do(\cdot)$?: Causal Imitation Learning
Authors: Gokul Swamy
Sanjiban Choudhury
Drew Bagnell
Steven Wu
Venue: ICLR 2022 Submitted (2022)
Keywords: imitation learning
causal inference
reinforcement learning

Abstract: We develop algorithms for imitation learning from data that was corrupted by unobserved confounders. Sources of such confounding include (a) persistent perturbations to actions or (b) the expert responding to a part of the state that the learner does not have access to. When a confounder affects multiple timesteps of recorded data, it can manifest as spurious correlations between states and actions that a learner might latch onto, leading to poor policy performance. By utilizing the effect of past states on current states, we are able to break up these spurious correlations, an application of the econometric technique of instrumental variable regression. This insight leads to two novel algorithms, one of a generative-modeling flavor ($\texttt{DoubIL}$) that can utilize access to a simulator and one of a game-theoretic flavor ($\texttt{ResiduIL}$) that can be run offline. Both approaches are able to find policies that match the result of a query to an unconfounded expert. We find both algorithms compare favorably to non-causal approaches on simulated control problems.

Under review as a conference paper at ICLR 2022
WHAT WOULD THE EXPERT do(.))? CAUSAL IMITATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We develop algorithms for imitation learning from policy data that was corrupted by unobserved confounders. Sources of such confounding include (a) persistent perturbations to actions or (b) the expert responding to part of the state that the learner does not have access to. When a confounder affects multiple timesteps of recorded data, it can manifest as spurious correlations between states and actions that learner might latch on to, leading to poor policy performance. To break up these spurious correlations, we apply modern variants of the classical instrumental variable regression (IVR) technique, enabling underlying policy without requiring access to an interactive expert. In particular, we present techniques, one of generative-modeling flavor (DoubTI.) that can utilize access to simulator and one of a game- game-theoretic flavor (Res i duIL) that can be run entirely offline. We discuss, from the perspective of performance, the types of confounding under which it is better to use an IVR-based technique instead of behavioral cloning and vice versa. We find both of our algorithms compare favorably to behavioral cloning on simulated rocket landing task.
recover the causally correct
INTRODUCTION
Much of the theory of imitation learning (IL) indicates that with enough demonstrations, we should be able to accurately recover the expert's ' policy. When we apply II. algorithms in practice however, sometimes see them produce manifestly incorrect estimates of the expert's policy (Muller al.) 2006 Codevilla et al. 2019; de Haan et al. 2019 Bansal et al., 2018: Kuefler et al.) 2017). One possible reason for this phenomenon is that empirically, we only have access noisy recordings of what the expert did. This critical detail has been thus far neglected by most prior theoretical work in imitation learning. We focus in this paper on how best to learn from two kinds of noisy data:
Exogenous noise: When observe expert actions corrupted by persistent noise (e.g. faulty joystick that persistently perturbs actions before they are executed in the game).
Endogenous noise: When we do not observe the full state an expert used to pick an action (e.g. the learner not knowing there's 's an enemy behind a door).
The net effect of either kind of persistent noise (more formally, an unobserved confounder) is to in- troduce temporal correlations in the recorded actions that do not have their true cause in the recorded state. For example, consider recordings of an expert driver slowing down stop sign. If all we present the leamnr with as state input is whether the expert was slowing down at the last timestep, they will likely learn to simply repeat the expert's past action. Thus, once the car begins to slow down, it continues to slow down, regardless of whether there is stop sign present. At a more ab- stract level, these sorts of inertia problems can result from temporal correlations between pairs of actions (c.g. the effect of the stop sign) being reflected in the state (e.g. the past action variable), leading to spurious correlations between state and action that the learner might unfortunately latch onto (e.g. repeating the past action).
What should we hope to learn then in these confounded settings? Given we do not have access to the unobserved confounder, reasonable choice is to ensure that match the behavior of an expert that has access to the same information we do. That is, if we could query the expert for an action with only the information have available, we should strive to produce action that matches

---

Under review as a conference paper at ICLR 2022
a
(a)
(b)
(c)
We focus in imitation learning in the presence of temporally correlated perturbations (exogenous noise, (a)) not having access to the full state (endogenous noise, (b)). We formalize both in a graphical model (c) that allows us to leverage a technique known as instrumental variable regression find a policy that isn't corrupted by spurious correlations introduced by the confounder.
Figure 1:
this queried action. While applying an interactive imitation learning algorithm (Ross et al., 2011) would allow to collect a dataset uncorrupted by confounding, queryable expert is not realistic assumption for many domains. We therefore focus on approaches for the off-policy setting. We base algorithms data known an instrumental variable regression (IVR) (Angrist et al. 1996). The high-level idea of IVR is to leverage an instrument, a source of random variation independent of the confounder, to deconfound inputs to learning procedure via conditioning on the instrument. In dynamical systems, history act as this source of variation, 2015). Our key insight is that we can leverage past states as instruments to break the spurious correlation between states and actions caused by an unobserved confounder.
technique from econometrics for dealing with confounding in recorded
it is unaffected by future confounding (Hefny et al.
Our work provides the following contributions:
1. We formalize confounding in imitation learning. We provide structural causal model that captures inertia effects that result from temporally correlated exogenous or endogenous noise. We also derive test to detect whether this sort of confounding is present dataset.
2. We present a unified derivation of modern instrumental variable regression techniques. We show how two recent extensions of the classical IVR technique share common structure. We also extend the theoretical analysis of previous work by deriving accuracy bounds.
3. We provide two novel algorithms deal with confounding in imitation learning. We derive two novel IVR-based algorithms:
Doub IL generative modeling approach that and can utilize access to a simulator for reduced sample complexity.
Resí du IL a simulator-free, game- thee-the approach.
We derive performance bounds for policies produced by these algorithms under exogenous noise. We empirically investigate the effect of the persistence of the confounder on this bound. We also compare the performance of these approaches to behavioral cloning under endogenous noise.
2 RELATED WORK
Imitation Learning. Broadly speaking, imitation learning approaches can be grouped into three classes: offline, online, and interactive. Our work is most similar to offline imitation learning al- gorithms (c.g. Behavioral Cloning (Pomerlcau) 1989), ValueDice (Kostrikov et al.) 2019). AdVIL (Swamy al. 2021)) that operate purely on collected data. Unlike previous work however, we consider the effect of unobserved confounding. Our work shares the goal of interactive imitation learning algorithms (e.g. DAgger (Ross et al., 2011I, AggreVaTe (Ross & Bagnell| (0144), in that seek to match the output of query to the expert. However, a query on recorded data, rather than on learner rollouts, as is standard for interactive approaches. This is because the confounder decouples the actions recorded in the data and queried actions. Zhang ct al, (2020) consider imitation learning through the lens of causal inference but focus on the one-step setting, while consider multiple timesteps. Kumor al. (2021) contemporaneouslly consider the
focus matching the output of
2

---

Under review as a conference paper at ICLR 2022
PROBLEM
CORRELATED STATE
ACTION
CONFOUNDER
Gridlock
Distance to intersection
Crossing
Cars on other side of intersection (endog.)
Stationary
Previously Braking
Braking
Traffic light (endog.)
Faulty Brakes
Speed
Braking
Brakes not responding presses (exog.)
Table I:
Concrete examples of confounding from the driving domain. The stationary problem was observed empirically by Codevilla et al. (2019).
multi-step setting and come to similar conclusions as us about the challenges of endogenous noise. They derive necessary and sufficient structural condition for successful imitation learning, while we focus on practical algorithms with performance guarantees for a particular graphical model un- der exogenous noise. In contrast to imitation learning methods that seeks to match moments of the expert's 's behavior (Swamy et al.) (02I). we focus only on matching average expert actions. We leave matching arbitrary moments to future work.
Inertia Effects in Imitation Learning. Several authors have empirically observed a latching effect in policies trained via imitation learning: (Muller et al. [2006] Codevilla et al.) 2019 de Haan al., 2019 [aasal et al.) 22118] Kuefler et al. (0017), where learned policies tend to inappropriately repeat the same action. We seek to provide a plausible explanation and correction for the phenomenon reported in these works, and list several examples in Table ! We note that when attempting to explain inertia effects, de Haan et al. (2019) propose causal confounding as the root cause of the error. However, as previously pointed out by Spencer et al. (2021), there is no actual confound in the theoretical or empirical examples in the work de Haan et al. al. (2019). This because the learner observes all of the variables the expert was using to make decisions.
Instrumental Variable Regression. The classical approach to instrumental variable regression (Wright 1928) is two-stage least squares procedure (e.g. in [Angrist et al. (1996)'s textbook). We focus on the more general nonlinear setting and instead base our approaches on the more recent DEEPIV (Hartford et al. 2017) and AGMM (Dikkala et al., (020). We present extensions to the work in these papers, including a unified derivation of both methods and error analysis for DEEPIV.
A BRIEF REVIEW OF INSTRUMENTS IN CAUSAL MODELING
We begin by discussing the concept of an instrument before de- riving algorithmic approaches in a simplified. non-sequential setting. Let X, Y, and Z be random variables on (potentially infi- nite) sample spaces X, V, and Z. Assume that X, Y, and Z have the causal, rather than statistical, dependency structure in Fig. 2 Given a dataset of (I. y. 2) tuples, we are interested in determin- ing the causal relationship between X and Y. E [do(z)]. where do(-) is the interventional operator Pearl et al. (2016). Intuitively, []Y(d)(=)) is the expected value of Y when we intervene and set X x,, rather than observe such an X. In the SCM to the right, ((x) E[Y [[o())( Because of the presence of an unobserved confounder, U, that affects both X and Y, standard regression (e.g. OLS) generically produces inconsistent estimates. Coarsely, this occurs because OLS will over-estimate the influence of the parts of X that are affected by the confounder. If we only have observational data and are unable to perform randomized control trials, canonical technique recover h is IVR (Winship) & Morgan 1999). Formally, an instrument Z must satisfy three structural conditions:
Z
X
Y
g
h
The structural causal model (SCM) we con- sider. We are interested in finding h, the causal rela- tionship from X to Y, even though there is an unobserved confounder, U. To do so, we leverage the effect Z, which provides independent randomness from U.
Figure 2:
Ordinary Least Squares
i.e. independent randomization from confounder. 2. Exclusion: Z IL Y i.e. no extraneous paths.
1. Unconfounded Instrument: Z IL U i.e.
3. Relevance; Z V. X -.e. conditioning has an effect.
3

---

Under review as a conference paper at ICLR 2022
Z satisfies these three conditions in the SCM of Fig. 2 Without loss of generality, assume that B[U] = 0. This allows us to concisely derive a set of conditional moment restrictions (CMR):
0 = E = E
Y h(X)[=]
= 42 2, y = [
(2)
In words, these constraints are saying that necessary condition for recovery of h(x)) is that for all values of the instrument, the actual and predicted expected values of Z are equal. We further assume that noise U enters additively to Y y and write out the following equations:
X (Z. Y h(X) + U.
(3)
We now derive an appropriate loss function for finding an h that approximately satisfies the CMR. If we only have finite samples and can therefore only estimate conditional expectations up to some tolerance, it natural relax the CMR
min REH. s.L.
[0
(4)
E
[x))==II 62, 6. 2.00 0,
V2 E 2,
where the 6₂ are slack variables. Then, the Lagrangian (with the natural P(=)- ((eihtted inner prod- uct that captures how often each we expect each z to occur) is
Σ PE)) - T(x)==] 6.) P(3) -2
L(h,
(5)
=EZ
where X is the vector of Lagrange multipliers. By the stationarity component of the KKT conditions,
vs, L(h, 8, 2) P(=)A; + P(z)6- = 0,
(6)
implying that ó_ = >a- Plugging this back into the Lagrangian, we can simplify our function to
T(, = P(:)).EEYY-RA
1 -
SEZ
We refer to 7 as the Regularized Lagrangian or ReLa for short. Now, solving for the optimal Lagrange multipliers via stationarity, we arrive at
Vh, A) P())E[Y - 7(x))2] P(=)A₂ = 0,
(8)
which implies the optimal A₂ is equal to E[y ](())=z) Plugging this back into recovers the loss function,
(hh) = Σ P(:)ELY - =x())=11 = PRMSER (h).
(9)
ZEZ
This expression is the square of the Projected Root Mean Squared Error (PRMSE) of Chen & Pouzo 2012). To recap, by minimizing Eq. ( we are attempting to find an h that approximately satisfies the CMR. Minimizing PRMSE is necessary condition for recovering E do()), For it to be sufficient condition, one needs the natural identifiability assumptions - we refer interested readers to Chen & Pouzo 2012) for more thorough discussion.
3.1 GENERATIVE MODELING APPROACH
How should we minimize the PRMSE then? One option is learning the distribution P(X|=) = ==().
passing samples from it to candidate h, and trying to match E [ This is generalization of the standard Two-Stage Least Squares (2SLS) (Angrist al. 1996) procedure to nonlinear functions. The nonlinearity of the second stage means that one cannot simply compute the first moment of the P[X[z) distribution, which is recovered by linearly regressing from X Z in the 2SLS procedure. This sort of approach was first proposed for the IVR setting by Hartford et al. (2017) and amounts to first learning a g(2) (e.g. via maximum likelihood estimation) and then solving
min Ez (E|Y kch
E Ex~g(e) ((())))]
(10)
The work of Hartford al. (2017) did not have theoretical analysis regarding the effect of errors in
g(z) upon attempts to learn h(z)). We prove the following in Appendix A
TTee inclusion of V makes our model generalization the standard IVR model, so we confirm the validity of the instrument in Appendix[A]
Without this assumption, can only upper/lower hound h(x:) Balke Pearl 2013
4

---

Under review as a conference paper at ICLR 2022
Theorem 1. Assume we learn g(2) s.t. max Bz (2((tgg()))((())) - [(x)s [mx=))² s
Then, optimizing 11 to value e corresponds to recovering h(x)) s.t. (RMSE(h) S vō+ Vc.
3.2 GAMB-THEORETIC APPROACH
One can also proceed by instead solving the two-player zero-sum game with the ReLa 7 as the payoff. Denoting by f E F = {2 - R] the function that maps 2's to their Lagrange multipliers, we can write this game as
min max E - -((x))((2) bEN fE
-
(11)
This game is the core objective of the AGMM method of Dikkala al. (2020). Importantly, one does not need to learn a generative model of P(X) 22) for these sorts of tame-theoretic approaches. We prove the following theorem in Appendix A
Theorem 2. Assume that H. and F are bounded, closed under negation, convex, compact, and that H and wh H. f(z) = B| h[X)]=] E F. Then, an c-approximate Nash equilibrium of corresponds recovering a h(x)) s.t. PRMSE(h) S VE.
One can find such an equilibrium via a standard reduction to no-regret online learning (Freund & Schapire 1997).
In summary, one can frame nonlinear IVR as a generative modeling or game-theoretic problem, leading to different error characteristics. We now turn our attention to applying these methods to imitation learning with unobserved confounders.
4 CAUSAL CONFOUNDING IN IMITATION LEARNING
We begin with brief, intuitive sketch illustrate the challenges of confounding for IL: consider an expert trying to fly a quadcopter straight but their actions being perturbed by wind (i.e. form of ex- ogenous noise). Because it attempts to reproduce expert actions, be- havioral cloning would reproduce these deviations, producing tra- jectories that deviate even further from straight path in a windy environment. In contrast, by filtering out the effects of the con- founder, policy trained via IVR would only be affectedd by the wind present at test time and therefore produce trajectories similar to those of the expert.
TE
BC
RIV
We now formalize this sort of confounding and how one can use IVR to mitigate its effects. We use A(S) to mean the set of distri- butions over S and focus on a Markov Decision Process (MDP) parameterized by (S,A, r, T where S is the state space, A the action space, T S x A + A(S) is the transition operator, r S A 1-1,1] is the reward function, and T is the horizon of the problem. Let J(T) = E,-- EE r(s,, at)]. II C {S + A(A) be the policy class we optimize over and d, the visitation distribution of policy æ. In the presence of unobserved confounding, the trajectories generated by the ex- pert can be captured by the structural causal model (SCM) in Fig.
Figure 3: Behavioral cloning amplifies the effect of exoge- nous noise, unlike IV.
Fig. captures both the exogenous and endogenous noise settings. In the exogenous noise setting, the confounder W4-1 could be a persistent noise that affects pairs of actions, while in the endogenous noise set- ting, u can be thought of a response a part of the state that the learner does not have access to (see Table [ for several example) In either setting, the confounding travels through the dynamics influence the next state, leading spurious correlations between the recorded states and actions. We can also see this correlative effect by writing out the structural equations:
X st T(s 11-1 ((s TE(3-1) + u4-1 + 2222
(12)
Y = a, (g( + + ut-1.
(13)
'One technically needs to add another input to nh (i.e. (E,s, R), where H is random input) to allow for non-deterministic expert. In this work, we focus techniques for minimizing the PRMSE, for which matching average expert actions is sufficient. Thus, we suppress the dependence on the other input.
5

---

Under review as a conference paper at ICLR 2022
Fig. also tells us that Z = St-1 satisfies the three con-
ditions to make it a valid instrument for countering the effects of U 1--1- Intuitively, this is because the past state is independent of the current confounder, allowing it to function
Z
8t-1
X
RE
independent source of randomness. One can imagine longer time-scale correlations induced between actions than just one step confounding our ap- proaches naturally extend to this setting by using state further back in the past as the instrument.
0t-1
Y
Unlike standard imitation learning approaches like behav- ior cloning which attempt to recover [[a[s]. an approach based on IVR enables us to instead recover the interven- tional effect of the policy [s) = [[]] Con- ceptually, [a [o{s is asking what the expert would do on average if they were placed in state 8, i.c. the kind of answer would get from queryable expert in DAg- ger (Ross et al.] 2011) or DAeQuIL (Swamy et al.) (021). only interested in the result of queries on states from expert demonstrations, we are able to get a similar effect via IVR to an interactive approach without requiring access to a queryable expert. We now build upon this intuition to derive two algorithms.
Ut-2
De-1
MK
Figure The SCM for imitation learn- ing with unobserved confounders. The confounding is mediated via the dynam- ics into the state, introducing spurious correlations between states (X 8t) and actions (Y az). To break the founding, we can utilize the past state as instrument (Z 8t-1).
However,
ALGGRRIHMMS FOR CAUSAL. IMITATION LEARNING
We now present tWO approaches for causal imitation learning that be applications of the generative modeling and game-theoretic approaches of Sec. 3
At their core, both algorithms are
attempting to minimize a PRMSE objective.
min (s,s'' ,a') [EEaa' (ss)sss)){1,
(14)
mell
instead of the usual II. objective,
min muinErs.oo (ss.)))daa [(a -(s))²
(15)
In both the exogenous and endogenous settings, minimizing equation I4 corresponds to recovering [s]]do(s)]. What differs is the performance implications of doing so, now discuss further.
5.1 EXOGENOUS NOISE
Exogenous noise is present both in the demonstrations well test time. Our goal in this setting is to eliminate the effect of the confounder so at test time we do not needlessly reproduce its effects (e.g. the increased swerving in our quadcopter example). Notice that under exogenous noise, minimizing (15) to would not recover the expert's policy while minimizing Equation (14) would. First, let distribution P(U) be e- Total Variation stable (Bassily et al. 2021 if:
||a
b|12 S 5 - drv (a + U. U) S c8.
(16)
This property satisfied by a wide variety of distributions. For example, for standard normal random variables, = 1/2. Next, in our setting, the measure of ill-posedness (Dikkala et al. 2020 Chen &
Pouzo 2012)
Eander (TE(S) {()))
sup RMSE(T)
<III sup TEll
(17)
Ba,s' dd [Bla' laaa--a [+')|=1]]
sell PRMSE(#))
We prove the following bound on policy performance in Appendix B
Theorem 3. Assume P(U) is c-VV Stable exog. noise, TE is deterministic, and let k(II) be the measure of the ill- ill-posedness of the problem. Then, PRMSE(T) J(TE) J(r) S (T)

---

Under review as a conference paper at ICLR 2022
Intuitively, x(II) measures the strength of the strength of the instrument. Consider the extreme case where s' = s. Then, x(II)= = 1. As the past state becomes a weaker instrument, x(II) > 1. Thus, if the confounding affects multiple timesteps, further back in time to find a valid instrument, leading to a looser performance bound. We investigate the effect of the length of confounding the ill ill-posedness of the problem empirically in Sec. 6
would expect k(II) to grow needs to reach
5.2 ENDOGENOUS NOISE
Endogenous noise is present only in the expert demonstrations and not applied during learner roll- outs. Thus, in contrast to the exogenous setting, we do not need to eliminate the effect of the confounder perform well as the expert. Instead, hope to effectively reduce our uncertainty over the confounder (e.g. the state of the traffic light). We begin by defining the following policies:
Ys E supp(dræ); (Bc(a = d (s, a)/dr (al(s = p(aldo(s)),
(18)
where supp denotes support. We prove the following results under endog. noise in Appendix A
Lemma 1. There exist MDPs for which TE, TBC, and πιV have different trajectory distributions.
Lemma 2. If reward is function of state and action only, (TT)) = (TBC) always, while there exist MDPs for which J (Tæ)
) > (TIV).
Lemma 3. If reward r E {S x A x - R] (i.e. the reward additionally depends on the con- founder), then there exist MDPs for which ((TE) > J(m) Vn (S A(A)).
One take-away from these lemmas is the fundamental difficulty of producing value equivalent policy to the expert's under endogenous noise, result that was concurrently derived via a graphical condition by Kumor et al.) 2021) (Defn. 2.3). Consider, for example, intersection when a traffic light is red. the learner does not see this light, there is no way for them ensure they match the behavior of such expert. However, in the special case where the reward function does not directly depend on the confounder (i.e. eliminating the natural reward function that penalizes the learner for not obeying the traffic light), TBC value equivalent to the expert, while, perhaps counter-intuitively, causally consistent TTV is not. The value difference is because TV marginalizes out u by sampling from P(u+-1) while TBC samples from P( (see proof of Lemma iin Appendix A). TBC is therefore better able to estimate the value of the confounder by utilizing the information in the current state, which is advantageous in the endogenous setting.
expert driver that stops an
5.3 WITH A SIMULATOR: DOUBIL
Algorithm DoubIL
Input:
Dataset DE of expert trajectories, Policy class II, Simulator T
Output: Trained policy 22
Ti = arg min Eg I log π [a[s)]
(Train preliminary policy via cooment-matching
Drv ((((s, mi (8)), (V)\V a²) E Dz)
(Pass 's actions through simulator. [Train final policy on new dataset and output.)
n2 = arg min Ex,a- (aa T [8
Algorithm [ be seen a variation of generative modeling approach of Sec. 3 and Hartford et al. (2017) where one lever- ages knowledge just learns the other factor. Via the Markov assumption, we
one factor of the P(X distribution and
can factorize P(X 2) = P(S'(s) (s) Eaes ((als))(s.a))
9-11
S1
Assuming access simulator 7 that closely
BC
DoubIL
approximates the true transition dynamics, we can instead focus learning the P(a]s) component: the standard im- itation learning task. Notably, this first-stage policy is bi- ased as it includes the effect of the confounder: P(als) P(U + T (s)s)ss) However, when we use it to simulate tran- sitions, the next states that are produced no longer have the
Ox
Figure 5: DOubIL deconfounds inputs to the second stage by re- simulating state transitions.

---

Under review as a conference paper at ICLR 2022
particular instantiation of the confounder present in the recorded dataset's next actions. Using tilde to denote a fresh draw from a distribution, simulated states are drawn from
8t ( #1 (8t-1))
(19)
while the observed next actions are drawn from
04 FE (s-- + we-1 + ux-2)) + U4-1 + Ut.
(20)
Notice that there are no shared noise terms. This allows us to apply standard imitation learning to this new dataset of (5t, az) to learn causally consistent policy. The two applications of imitation learning lead us to term this algorithm DoubIL. We can translate the guarantee of Theorem to our factored context:
Lemma 4. Assume we learn I (s) s.t.
max max Ee..1
( ()) [T(+)]] - -., / 8
TEll
Then, optimizing the second-stage MSE corresponds to recovering a m2 S.I.
PRMSE(T2)
> Egndd BiTz(s') - S v Vc.
We prove this lemma in Appendix A Combining this lemma with Theorem allows one derive
performance bound of (TTE) [(T) < cx(IT)(vō Ve)T2 under exogenous noise. We note that one could simply learn the mapping P(s'(s) but this can be far less sample efficient than merely learning policy when [A] s (S), as is often true in practice.
5.4 WITHOUT STATE RE- SA-MALLING: RESIDUIL
Algorithm Resi duTL
Input: Dataset Dg of expert trajectories, Policy class II, Discriminator class F. Learning rate Output: Trained policy æ Set II, F, 9T = 0, Øs 0
while x not satisfactory do
L(T, f) = ((s.s',a') (2 x(s))f((s) f(s)3] [
{Payoff of zero-sum game.)
V L, Of
(Perform Optimistic Mirror Descent.)
π +- π - (12g gm)
f + f (2gs 9f)
Jn f Ix 95 t 9f
end while
Algorithm is the direct application of the game-theoretic approach of Sec. 3 and Dikkala et al. (2020) imitation learning. We term ResiduIL because the adversary attempts predict the residual between the learner and the expert's actions while the learner attempts to minimize this residual. Notably, this algorithm can be run completely offline (i.e. without access to simulator). We use the Optimistic Mirror Descent approach of Syrgkanis et al. (2015) to find approximate Nash equílibria in our experiments. Once again, we can extend our past results to II. setting:
Lemma 5. An e-approximate equilibrium for the policy player corresponds recovering policy т S.I PRMSE(TT) I Ve.
This lemma dovetails with Theorem 3 to prove that J(TR) (TT) S (EI) under exogenous noise (Appendix A
EXPERIMENTS
We test DoubIL and ResiduIL on a slightly modified version of the OpenAl Gym (Brock- man et al. 2016) LunarL environment against behavioral cloning baseline. We generate
8

---

Under review as a conference paper at ICLR 2022
Confounded
Demoised MDP
[[x]s]]] on Expert Trajs.
MSE (e))
40
15
MSE from El
30
Dl
80
Beab
1.0
Residell.
doll
20
40
0.5
10
20
20
30
10
30
50
-0.75
-0.50
-0.25
0.00
0.25
0.50
Num. Export Traj.
Num. Expert Traj.
Figure 6: We train behavioral cloning, DoubIL, and Res du on trajectories from modified Lu- narl Lander environment, computing standard errors across 5 runs. The left plot shows that Doub L and Res duII are better able to match the desired E[a]do(s on states from expert rollouts, while the middle plot shows they are able to generalize better to the state dist. of an expert w/o noise. The right plot shows how we can compare the results of behavioral cloning and causal IL procedures to identify areas of the state space where the effect of confounding is strong (the red dots).
demonstrations by simulating rollouts of expert policy trained via PPO (Schulman et al. 2017 adding fresh Gaussian noise to the expert's 's action as well as cached noise from the last timestep. The latter noise the confounder. See Appendix for full parameters. We judge policy quality by computing the MSE between the output of a deconfounded expert query and a learner's ss proposed actions on states from expert rollouts. We see that both of our methods are able to more closely match E[a [o) than behavioral cloning, especially in the low-data regime (Fig. 6 left). We also measure the MSE on states from deconfounded expert rollouts while there are no clear guarantees on this state distribution, we see that our methods generalize better than BC empirically (Fig. 6 middle). One might wonder how, given dataset of expert demonstrations, one detects whether there is unobserved confounding in the data. We can answer this question by comparing the results of behavioral cloning and either of our above algorithms. We prove the following in Appendix A
Lemma 6. Assume अ (s) TR(a (s] and ( [[a]do())]. Then, Elu s] TBC()) TVV (8).
The implication of this lemma is that comparing the outputs of IVR-based procedures to behavioral cloning can help us detect causal confounding - if they greatly differ with a sufficiently sized dataset, there is likely an unobserved confounding effect in our data. Moreover, the states where they differ represent the parts of the state space where the influence of the confounder is highest. Fig. 6 right an empirical example of how the test of Lemma [6 can be used to identify areas of the state space where the effect of the confounder is especially strong (e.g. the center).
For linear problems, we can bound xx (II) (the measure of ill- posedness) via eigenvalue ratio (Dikkala et al. 2020). Extending our previous model to include the effect of the last H confounders
(a: = =T(s)) + Ej= Uj.), we arrive at the bound
Aman [E [sss
I(II: H) S
(21)
Amin (E [[[[[s St E[se (
Hf fnum. steps confounded)
We compute this quantity empirically for a linear-quadratic problem with Gaussian confounding and plot results in Fig. [ As expected, that increasing the length of confounding leads to weaker instruments as one has to use states further back in time. Theorem 3 tells us that under exogenous noise, we should expect weaker instrument to lead to a larger performance gap between the learner and expert. See Appendix for full experimental setup details.
Figure 7: We compute s(II) for an LQG problem where we vary the number of steps confounder sticks around for.
CONCLUSION
We present model that captures confounding in imitation learning and derive two algorithms, DoubIL and ResiduIL, that are able to utilize history as an instrument to mitigate the effects of unobserved confounders. We prove performance bounds and validate their empirical efficacy under exogenous noise. We also discuss the challenges of learning with endogenous noise.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Joshua Angrist, Guido W Imbens, and Donald Rubin. Identification of causal effects using instrumental variables. Journal of the American statistical Association, 91(434):444-455 1996.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995, pp. 90-37. Elsevier, 1995.
Alexander Balke and Judea Pearl. Counterfactual probabilities: Computational methods, bounds and applications, 2013.
Mayank Bansal, Alex Krizhevsky, and Abhijit S. Ogale. Chauffeurnet: Learning drive by imitating the best and synthesizing the worst. CoRR, abs/1812.03079, 2018. URL http:
//arxiv. wwwxii..rr//bb//11111. 03079
Raef Bassily, Kobbi Nissim, Adam Smith, Thomas Steinke, Uri Stemmer, and Jonathan Ullman. Algorithmic stability for adaptive data analysis. SIAM Journal Computing, (0):STOC16-377, 2021.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Xiaohong Chen and Demian Pouzo. Estimation of nonparametric conditional moment models with possibly nonsmooth generalized residuals. Econometrica, 80(1): 30(1):2777321, 2012.
Felipe Codevilla, Eder Santana, Antonio M. López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. CoRR, abs/1904.08980, 2019. URL http://
arxiv. wrx.v.org/abs//Soee
08980
Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning, Ad- vances Neural Information Processing Systems, 32:1 (1698-11709 2019.
Nishanth Dikkala, Greg Lewis, Lester Mackey, and Vasilis Syrgkanis. Minimax estimation of con- ditional moment models, 2020.
Yoav Freund and Robert E Schapire. A decision-theoreti generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 55(1):119-139, 1997.
Jason Hartford, Greg Lewis, Kevin Lcyton-Brown, and Matt Taddy. Deep iv: A flexible approach for counterfactual prediction. In International Conference Machine Learning, pp. 1414 1423. PMLR, 2017.
Ahmed Hefny, Carlton Downey, and Geoffrey J Gordon. Supervised learning for dynamical system learning. Advances in neural information processing systems, 28: 1963-19771 2015.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In IN PROC. 19TH INTERNATIONAL CONFERENCE ON MACHINE LEARNING, pp. 267-274, 2002.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint 11X.v:1412.6980, 2014.
Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off off-policy distribution matching, 2019.
Alex Kuefler, Jeremy Morton, Tim Wheeler, and Mykel Kochenderfer. Imitating driver behavior with generative adversarial networks. In 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 204-211. IEEE, 2017.
Daniel Kumor, Junzhe Zhang, and Elias Bareinboim. Sequential causal imitation learning with unobserved confounders. 2021.
Urs Muller, Jan Ben, Eric Cosatto, Beat Flepp, and Yann L Cun. Off-road obstacle avoidance through end-to-end learning. In Advances in neural information processing systems, pp. 739- 746. Citeseer, 2006.
10

---

Under review as a conference paper at ICLR 2022
Judea Pearl, Madelyn Glymour, and Nicholas P Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016.
Dean A Pomerleau. Alvinm: An autonomous land vehicle in neural network. 1989.
Stephane Ross and J. Andrew Bagnell. Reinforcement and imitation learning via interactive no- regret learning. 2014.
Stephane Ross, Geoffrey Gordon, and Andrew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning, 2011.
Stuart Russell and Peter Norvig. Artificial intelligence: modern approach. 2002.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian Ziebart, and J. Andrew Bagnell. Feedback in imitation learning: The three regimes of covariate shift, 2021.
Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell, and Zhiwei Steven Wu. moments and matching: A came-theoretic framework for closing the imitation gap. 2021.
Vasilis Syrgkanis, Alekh Agarwal, Haipeng Luo, and Robert Schapire. Fast convergence of regularized learning in games. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar-
nett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Asso- www.:pp.....uu.p..c.........
ciates, Inc., 2015, URL 7fea637fd6d02brr Oad (at.///200000aoooeooaaeeeee33332
pddf
Christopher Winship and Stephen Morgan. The estimation of causal effects from observational data. Annual review of sociology, 25(1):659-706, 1999.
Philip G Wright. Tariff animal and vegetable oils. Macmillan Company, New York, 1928.
Junzhe Zhang, Daniel Kumor, and Elias Bareinboim. Causal imitation learning with unobserved confounders. Advances in Neural Information Processing Systems, 33, 2020.

---

Under review as a conference paper at ICLR 2022
A PROOFS
A.1 PROOF OF VALIDITY OF INSTRUMENT
Proof. We check the instrument conditions in order:
1. Unconfounded Instrument: Z TT U: The Z - x triples are blocked by standard d-separation rules (Pearl et al. 2016). All paths from Z to U must pass through one these triples so Z 1l U.
U, - + U, and X + + U
2. Exclusion: Z TL Y|X, U: The Z + + Y, X + +Y, and V + X + Y triples are blocked by standard d-separation rules. All paths from Z to Y must pass through one of these triples so Z IL Y|X, U.
+
3. Relevance: 1/4 X: There is a Z + X edge, which is assumed to be non-degenerate.
Thus, Z is valid instrument for determining the causal relationship between X and Y.
A.2 PROOF OF THEOREM I
Proof. We simplify notation for clarity in our proof. Consider two vectors of the same dimension,
a and b. Assume that Σ af S e and Σ; bi s 6. This implies that ||al|2 S ve and ||bll2 I V8. Then, by the triangle inequality, ||a bll2 S ||al| + ||bll2 I vc + v8. Setting = (E Eing(=) (x())) and b, / P(2) (B [(+)]] E(]())|=]) proves that
max Ez REN
[((r)) - E2- P(x (x)))) 5
(22)
Eg (E[Y| - Ea (()))) <
(23)
(RMSE()) E. [E[ -x==))))))) VE+ v8
(24)
A.3 PROOF OF THEOREM 2
Proof. The population version of is
min max R[2(Y - - kEH JEF
(25)
An c-approximate equilibrium is an (h. f) pair such that:
- max E[2(Y - -(()) 2 FEF
(26)
$ E/2(Y ((x))((2)-P12
(27)
< min - -(X))
(28)
NEW
Taking the derivative w.r.t f()) of the payoff and setting equal to 0, we arrive
2P(=)E[Y - [= (=) = 0 - f(z) = Ely T(())22).
(29)
Plugging this back into 49 gives us the inequality
Ez EE
7 =
e min E[2(Y M())) (2 + 2
(30)
2
Assuming we are in the realizable setting (e.g. M(x) = E[Y[do(z)] E H), mingeR EE2((
M())((()) (Z)) < 0. Thus, we can write that:
E [(X))=]3] < C
PRMSE(1) S Ve.
(31)
2
We note that Theorem Saaremm@[ee follows somewhat directly from the main theorems of Dikkala et al. (2020)
but that it was not stated in this precise form in their work.
12

---

Under review as a conference paper at ICLR 2022
A.4 PROOF OF LEMMA
Proof. We focus on the first two timesteps of the problem. By construction, p(so) is the same for all three policies. The following statements follow from standard conditional independence rules (Russell & Norvig 2002):
(BC(aa)so) = p(ao
Σ P(uo) uo).
(32)
(oolo))) = p(ao)d)(so)) = Σ P(uo)
uo) Σ P(uo) so, uo).
(33)
The last equality follows from the fact so has no parents. Thus, when combined with the fact that the transition dynamics P(s1 So. ao) are the same for all three policies, we arrive at the following equality:
Png (8o: 0o, #1) = Prac (8o, 40: 81) Paiv (80, ao 81)
(34)
The first difference between the trajectory distributions starts at a By definition, the expert chooses actions via
Te(ai 181, uo, u1) p(an (81; uo, uo, u1).
(35)
while the learners instead follow
BBC (a1(s)) = ((ai(s1) = >> pluo 1 (p())(u))ppp (pp(u)pp(aaa) uo: un);
(36)
ng
*rv(ai|si) p((a)(o(ss))
ΣΣ p(uo)p((i)p(a) (d(s))(ss))))))))))))))))))))))) uo. uo. u1)
(37)
p(uo)p(u1))) (opp(())p(ass))) u0, u1).
(38)
The first line follows from the fact u1 IL uo. 81. The last equality follows from Rule 2 of do-caleulus (Pearl et al., 2016). Thus, given the action distributions at the second step are not equal, so long as a 11 Ug and $1 I ug all three policies have different trajectory distributions.
A.5 PROOF OF LEMMA 2
Proof. We prove J (gg) J J(T via application of the Performance Difference Lemma (PDL)
(Kkkade & Langford 2002). We note that this is equivalent to the graphical argument used by Kumor et al. (2021) (Defn. 2.3 in their paper).
First, we note that p(so, an) is equal for both policies. Assuming that p(s Qt-1) 1) is equal, p(s)) must be equal by the fact the transition dynamies are the same. Then, via the fact that л&C matches the conditional distribution of or 0r/ p(s,, ax) must also be equal. Thus, by induction, drge drg-
This means that TBC is defined properly everywhere within it's own state visitation distribution. We
then apply the PDL as follows:
(TTE)
J (TBC) TE TEs,,, u.d I TQT (s, a) pla' 1QTH (s, )]]
(39)
TE, d, 1QTH (s, a)
E, (s 1 (s, ']]] TE 1QTH TH (s, Euje / [
(40)
(41)
TEg 1QTH (s, a) (a'np(a()s) Q (s, a')l]
(42)
0.
(43)
The third equality follows from the fact the learner's s actions are independent of the confounder, so under dazc. P(uls) P(u). From the proof of the previous lemma, p(a, [s:) o p(a, (do(s))) )) in general, so πy and n can have different state-action visitation distributions after the first timestep.
Therefore, for two-step problem where a, K uo and 8 1 ug.
( - (TIV) = T(E [r(s, a)] EEgandrry (r(s, > 0
(44)
for reward function
(1, (1, > dayy (8,0)
r(s, a)
(45)
0, 0.w.
13

---

Under review as a conference paper at ICLR 2022
A.6 PROOF OF LEMMA 3
Proof. Let Ur be a Rademacher random variable (1 w.p. and -1 otherwise) 74 E [T]. Let ar|8t, Ue, Ut-1 Ue and there be single fixed state that action can leave. Set r(s, dt Up) 1aa = un]. Then, ((T = T. Notice that p(+1|s) = p(-1(s) = p(+ [do(s)) = (d-)))d(ss)
0.5. Thus, J(TE) > (TTIV) = J(TCC) = 2. T. the expected number of correct answers of randomly guessing the outcome of fair coin T' times. Furthermore, notice that 1 is the best any II can do. One can see this by considering the problem with T 1. If there one would be able to use such a policy to predict the outcome fair coin better than the Bayes- optimal classifier for coin-betting (p (heedss) p(tails) 2), violating the fact the Bayes-optimal classifier minimizes Bayes error.
policy with J(x) > 1.
A.7 PROOF OF LEMMA 4
Proof. Notice that
max B., (EB (B, 7 [T()]] - E Es ()((() < 6
(46)
can be re-written as
max Ez (Bewg(=) [T))] [[--(()))((((]]]] 8
(47)
#ell
Thus, the proof of Theorem holds as written.
A.8 PROOF OF LEMMA 5
An e-approximate equilibrium for the policy player is such that - < min E[2(a, M42
max - (s)) /EF
(48)
#ell
With change of notation, we can re-write this as:
max B| (()
min ( -((2) (49) Hell
JEF
Thus, the proof of Theorem holds as written.
A.9 PROOF OF THEOREM
3
Proof. By definition,
PRMSE(T)
BEand. g Bla' [88))
(50)
Recall that the measure of ill-posedness of the problem Diikalla et al. 2020 can be defined as
Chen & Pouzo
2012)
Ex-d [(TE(S) - (s)))
RMSE( TT)
(III sup nENl
sup rell PRMSH(T)
(51)
Eg.e',a'dee a² a' [Eaa' [s5))ss]12
Directly,
RMSE(T) S <R((TT
(52)
We repeat the definition of total variation stability of distribution P(U):
||a bl| s 8 (a U.b U) S c8.
(53)
We proceed by noting that TV-stability implies that Vs S.
Un +U) < c (s)) -g((s] drv( + U1
(54)
drv(m(ss + + U & (m(mm =
(55)
14

---

Under review as a conference paper at ICLR 2022
BEwdde ( + U, ( +U) Sc Egdd [|| TE(())}* MSE(T))
(56)
By Jensen's inequality,
Esdd, [dr (T (s) TE( +U)] S Redd,
@MSBBT)) (57)
Taking the square root both sides, we arrive at
Egndxy g ddv (T()) +U, U, U)] RMSE(T)) - cx([I)e.
(58)
Lastly, apply the Performance Difference Lemma of Kakade & Langford (2002)
follows:
J(TB) - J(x) =
[Q" (s, a) E. (s==
(59)
= TE and 1Q" TT()) + u ūi) EEQ" (s, ((8) + ū2)]]
(60)
E drv (T()) U,) U
(61)
S cx(11)
(62)
We use the fact that the same would be added to both the learner and the expert's actions and that rewards are in the range [-1, 1] in the third step.
A.10 PROOF OF LEMMA
Proof.
Bla;)do(s:)] ([ + 4 do(s:)] π ( + E[u]] + E[uc-1] (g(s))
(63)
E[a./s:] ((T + u4 - [sc] = (st)
+ E [4 - E E(u--1/s = TT(s)) + E(44-1s11
(64)
(B)(s) - TE (s) = Bla, [s,] E [a (s.)) = (0(u-11ss1 = [B[uss] (65)
B EXPERIMENT DETAILS
B.1 LUNARLANDER EXPERIMENTS
For case of simulation, we remove the legs from the LunarLander vehicle (the joints connecting them to the main body have state that recorded in the observed state), remove the dispersion noise, and generate trajectories with fixed ground layout.
For all learned functions, we use two-layer ReLu MLPs with 64 hidden units. We use the Adam optimizer (Kingma & Ba 2014) for behavioral cloning and Doub IL and use the optimistic variant for Resicu We apply a weight decay le-3 all. We train all methods for 50k steps.
PARAMETER
VALUE
LEARNING RATE
3E-4
BATCH SIZE
128
Table Parameters for behavioral cloning.
For computational ease, we only learn the mean of P(a)s) s) for DoubIL and add fresh standard normal noise on-top of it to simulate drawing actions. For more complex noise models, one would need to use moment matching algorithm (Sanny et al. 2021 in the first stage.
Importantly, DoubIL suffers from a couble-sample" issue (Baird 1995) where multiple indepen- dent samples of g(z) are required to compute gradients h. To see this, note that the gradient with respect to of [10 is
2 (())))
Ez
( 2]
(66)
15

---

Under review as a conference paper at ICLR 2022
PARAMETER
VALUE
LEARNING RATE
3E-4
BATCH SIZE
128
NUM. SAMPLES FOR E
Table 3: Parameters for DoubTL.
Notice that 2'r appears under two separate expectations that are then multiplied together. To get an unbiased estimate of this product, two samples of f are required, one for each expectation. There- fore, it is most correct to use multiple samples from g(=) 3) for each update.
Thus, for implementing the "double samples" for the gradient, we compute E, - (s')) and Eg xss'){s) |s] using independent samples. Then, we apply stop-gradient operator to the former expectation before taking product between the expectations and averaging over
E,
[E, ( ss) - =(s')|ss]]
(67)
This loss function has the correct gradient as it uses independent samples for computing the two expectations.
PARAMETER
VALUE LEARNING RATE
5B-5
BATCH SIZE
128
BC REGULARIZER WEIGHT
5E-2
f NORM PENALTY
1B-3
ADAM Is
0, 1E-2
Table 4: Parameters for Resi ILIL.
B.2 LQG EXPERIMENTS
We compute the optimal policy for the following canonical linear system via solving Discrete-Time Al Algebraic Ricatti Equation via the standard iterative method:
Ft Ax,-1 + Bu--1
(68)
J(K)
Σ
of 2y Qu: + (Kr.) RKKr
(69)
[
A7
(0.(()) AT
1 0
A
B
Q
R= [0.1] AT = 0.1
0 1
This is the dynamics of "sliding brick on frozen lake." We then simulate rollouts of 200 timesteps with d, being drawn i.i.d. from the standard normal distribution. We confound actions with the sum of confounders going H steps back:
a = K*
Σ 4j.
(70)
We simulate 1000 such rollouts compute 21 empirically. We calculate [[X] -E(s
(A + BK*)* ) s.-H analytically instead of via samples due to the small value of the quantity in comparison the variance of the noise.
16


Reviews:
[{"review_id": "taETptvbRUA", "reviewer": "Reviewer_Jzke", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think the authors consider an exciting problem, since imitation learning has been widely applied in reinforcement learning and the presence of unobserved confounders is always prevalent in practical applications. However, it does not seem that the problem formulation of this paper properly addresses the challenges of causal imitation learning. More importantly, cloning the causal effect of the state S on action A does not necessarily allow the imitator to achieve the expert's performance. For instance, consider a causal model where S and A are both determined by an unobserved variable U in {0, 1}. That is, S <- U and A <-U, P(U = 1) = 0.5. The reward Y <- A \\xor \\neg S. In this case, the interventional distribution P(A = 1 | do(S = 1)) = P(A = 1 | do(S = 0)) = 0.5, i.e., do(S = s) has no causal effect on the expert's action A. If the imitator naively learns a policy copying P(A | do(S)), it is verifiable that it will achieve the expected reward equal to 0.5, which is far from the optimal expert's reward E[Y].\n\nIndeed, the optimal imitation strategy here is to copy the observational distribution P(A|S), which leads to a policy f: A <- S. This allows the imitator to match the expert's reward. Such an imitation strategy is effective since S is backdoor-admissible w.r.t. A and Y. The completeness of the backdoor criterion in the single-stage causal imitation has been discussed in (Zhang et al., 2020) [27]. I understand the authors' goal to extend (Zhang et al., 2020) to the general sequential decision-making setting. However, it would be recommended if the authors could build on the existing framework of causal imitation learning. Finally, the backdoor condition has been extended to perform imitation learning in the sequential decision-making settings and is shown to be sufficient and necessary (Kumor, Zhang and Bareinboim, NeurIPS'21).", "summary_of_the_paper": "This paper attempts to study imitation learning from a causal inference perspective. More specifically, let S be an observed state, A be action and Z be an instrumental variable. The authors propose that one should perform causal imitation learning by learning a policy function that could induce the same interventional distribution P(A|do(Z)).", "summary_of_the_review": "This paper studies an interesting problem in imitation learning with the presence of unobserved confounders. The causal identification method with instrumental variables in Section 3 seems interesting. However, as demonstrated in the example in \"Main Review\", I am not sure if estimating the causal effect of observed state S on action A is sufficient for a successful imitation. In other words, I am not convinced that the proposed approach in this paper is technically sound.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qGMA1k7fDOe", "reviewer": "Reviewer_aRw4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe authors have adapted the instrumental variable regression of Angrist etal. to estimate a better state transition for a linear dynamical system that has a Gaussian noise.\n\nWeaknesses:\n\nThe paper describes a linear dynamical system expressed in causal language.  Fig 4 depicts a dynamical system that only has a similar form to the SCM in Angrist metal, but is not a SCM.  The DAG is not depicting the mechanism of data formation.  If the model does not account explicitly for all the causal factors [wind force, traveling medium (air density, air humidity, air temperature), copter state (position, velocity, acceleration, battery power), navigator actions, etc.], it is unclear that the Gaussian noise assumption is reasonable.  How was the noise distribution estimated?   \n\nBy the authors' own admission they are not estimating the effects of interventions.  \n\n\"We note that minimizing PRMSE does not guarantee recovery of E[Y|do(X)]. However, given that E[Y |do(X)] always minimizes PRMSE, a low PRMSE is indicative of being close to E[Y |do(X)].\"  \n\nWhat is the definition of \"close\"?  \"Close\" by comparison to what? Given that the authors are not recovering E[Y|do(X)], the title of the paper \"What would the Expert  do(.)?\"  is misleading.  ", "summary_of_the_paper": "The authors derive two algorithms based on instrumental variable regression and game theoretic approach for imitation learning.", "summary_of_the_review": "The authors have adapted the instrumental variable work of Angrist etal. to estimate a better state transition for a linear dynamical system when the noise is Gaussian.  However, the authors do not seem to be modeling the mechanism of data formation, and they are not estimating the effects of interventions.  The Gaussian noise assumption has not been justified. \n\nThe authors are performing regression and the causal language is misleading.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jOAnH-4_hwH", "reviewer": "Reviewer_X9N9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Formulating latent confounders in imitation learning in the framework of structural causal models\n- Proposing two algorithms to address the problem of imitation learning for the latent confounders considered in this work: In the first algorithm, if we have access to a simulator, the algorithm focuses on learning $P(a|s)$ by first drawing the next state from the simulator, and then try to learn a causally consistent policy. In the second algorithm, the authors utilized the approach in Dikkala et al. (2020).\n\nWeaknesses:\n- Some parts of the paper are not written rigorously and it is hard to follow these parts. For instance: 1- In Section 3.1, the authors argued that it is \"most correct\" to use multiple samples from $g(z)$. What do you mean by \"most correct\"? 2- At the beginning of Section 4, the authors mentioned some events that result in poor performance in behavior cloning. Again, this part cannot be easily justified. 3- On page 6, the exact modelling of exogenous and endogenous noises is not clear. What are the main differences between these two kinds of noises? 4- For the right plot in Figure 6, it seems that there is no explanation in the text. There are only some descriptions in the caption but it does not help to understand this plot. \n- Issues regarding the models of latent confounders: Could the authors give a real-world example of latent confounders that can be modelled by Figure 4? It seems to me that additive noises are too restrictive in this model.\n- About the first algorithm in Section 6, I did not understand how we can connect to instrumental variables and how the results in the previous section can be useful in this section.\n- It would be great if the authors could explain how they removed the absolute function in eq. (4) when they wrote the Lagrangian in eq. (5).\n- In the experiment section, it is better to evaluate the algorithms in more practical settings instead of adding some Gaussian noise.\n\n\n\n", "summary_of_the_paper": "The authors proposed a SCM to model latent confounder in the problem of imitation learning. In particular, two cases of exogenous and endogenous noises are considered. For the instrumental variable, they studied the effect of error in estimation $P(X|z)=g(z)$ on Projected Root Mean Squared Error (PRMSE) where $Z$ is the instrumental variable. They also proposed a game-theoric approach to estimate $h(x)= \\mathbb{E}[Y|do(X)=x]$. Moreover, they proposed two algorithms to deal with latent confounders in imitation learning. Experimental results showed the proposed algorithms can perform better with respect to behavior cloning.", "summary_of_the_review": "In overall, some parts of the paper are not well-written. Moreover, the model of latent confounder (additive noise with special form in Figure 4) is not justified and the application of the proposed algorithms might be too restrictive. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4IYMB6KgtH", "reviewer": "Reviewer_A5Fo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n•\tOne of the first papers to consider causal imitation learning\n\n•\tError bounds are provided for two known IVR approaches\n\n•\tExperiments show the benefit of explicitly accounting for hidden confounders\n\nWeaknesses:\n\n•\tCan the author provide some examples and justifications of the following two assumptions made by the proposed method (Fig 4)? \n\nStates and confounders are marginally independent. Take the authors’ quadcopter as an example, the positions (which I assume these are the states) are most likely not independent of the wind unless the expert can perfectly fly it straight? \n\nConfounders are independent of each other. Using the same example, the wind at this time step is probably not independent of the wind of the last time step? If the confounders have temporal dependence, it seems to introduce additional dependence between states and confounders via action (e.g., considering the path u[t], u[t-1], u[t-2], a[t-1], s[t])\n\n•\tExperiments. This is related to the first weakness. Right now, the experiment is set up so that the assumptions are perfectly satisfied by design. What if the assumptions are violated to different degrees? How robust is the proposed method?\n", "summary_of_the_paper": "This paper proposes two causal imitation learning algorithms to address the confounding issue that causes spurious correlations. The method relies on instrumental variable regression to correct for confounding effects. The main idea is to use the state variable from the last time point as instrumental variable for the state-act variables at the current time point in a Markov decision process. ", "summary_of_the_review": "Overall, I find the paper to be novel and interesting.  It could be improved, in my opinion, in terms of the suitability/justification of the IVR assumptions in the considered application domain as I mentioned in the weaknesses above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies imitation learning from a causal inference perspective. The authors propose a method to remove the effects of confounders on expert action a using instrumental variable regression, which presumably leads to better estimation of P(a|s), and hence better imitation.  The reviews were negative overall at the start. After the discussions, one reviewer stated that he would change his recommendation to accept, although his score is not changed on the review form. However, another reviewer is still not convinced that the causal formalism introduced in the paper improves over the existing RL literature.
Decision: Reject