What Would the Expert $do(\cdot)$?: Causal Imitation Learning

1 INTRODUCTION . Much of the theory of imitation learning ( IL ) indicates that with enough demonstrations , we should be able to accurately recover the expert ’ s policy . When we apply IL algorithms in practice however , we sometimes see them produce manifestly incorrect estimates of the expert ’ s policy ( Muller et al. , 2006 ; Codevilla et al. , 2019 ; de Haan et al. , 2019 ; Bansal et al. , 2018 ; Kuefler et al. , 2017 ) . One possible reason for this phenomenon is that empirically , we only have access to noisy recordings of what the expert did . This critical detail has been thus far neglected by most prior theoretical work in imitation learning . We focus in this paper on how best to learn from two kinds of noisy data : • Exogenous noise : When we observe expert actions corrupted by a persistent noise ( e.g . a faulty joystick that persistently perturbs actions before they are executed in the game ) . • Endogenous noise : When we do not observe the full state an expert used to pick an action ( e.g . the learner not knowing there ’ s an enemy behind a door ) . The net effect of either kind of persistent noise ( more formally , an unobserved confounder ) is to introduce temporal correlations in the recorded actions that do not have their true cause in the recorded state . For example , consider recordings of an expert driver slowing down at a stop sign . If all we present the learner with as state input is whether the expert was slowing down at the last timestep , they will likely learn to simply repeat the expert ’ s past action . Thus , once the car begins to slow down , it continues to slow down , regardless of whether there is a stop sign present . At a more abstract level , these sorts of inertia problems can result from temporal correlations between pairs of actions ( e.g . the effect of the stop sign ) being reflected in the state ( e.g . the past action variable ) , leading to spurious correlations between state and action that the learner might unfortunately latch onto ( e.g . repeating the past action ) . What should we hope to learn then in these confounded settings ? Given we do not have access to the unobserved confounder , a reasonable choice is to ensure that we match the behavior of an expert that has access to the same information we do . That is , if we could query the expert for an action with only the information we have available , we should strive to produce an action that matches this queried action . While applying an interactive imitation learning algorithm ( Ross et al. , 2011 ) would allow us to collect a dataset uncorrupted by confounding , a queryable expert is not a realistic assumption for many domains . We therefore focus on approaches for the off-policy setting . We base our algorithms on a technique from econometrics for dealing with confounding in recorded data known an instrumental variable regression ( IVR ) ( Angrist et al. , 1996 ) . The high-level idea of IVR is to leverage an instrument , a source of random variation independent of the confounder , to deconfound inputs to a learning procedure via conditioning on the instrument . In dynamical systems , history can act as this source of variation , as it is unaffected by future confounding ( Hefny et al. , 2015 ) . Our key insight is that we can leverage past states as instruments to break the spurious correlation between states and actions caused by an unobserved confounder . Our work provides the following contributions : 1 . We formalize confounding in imitation learning . We provide a structural causal model that captures inertia effects that result from temporally correlated exogenous or endogenous noise . We also derive a test to detect whether this sort of confounding is present in a dataset . 2 . We present a unified derivation of modern instrumental variable regression techniques . We show how two recent extensions of the classical IVR technique share a common structure . We also extend the theoretical analysis of previous work by deriving accuracy bounds . 3 . We provide two novel algorithms to deal with confounding in imitation learning . We derive two novel IVR-based algorithms : • DoubIL is a generative modeling approach that and can utilize access to a simulator for reduced sample complexity . • ResiduIL is a simulator-free , game-theoretic approach . We derive performance bounds for policies produced by these algorithms under exogenous noise . We empirically investigate the effect of the persistence of the confounder on this bound . We also compare the performance of these approaches to behavioral cloning under endogenous noise . 2 RELATED WORK . Imitation Learning . Broadly speaking , imitation learning approaches can be grouped into three classes : offline , online , and interactive . Our work is most similar to offline imitation learning algorithms ( e.g . Behavioral Cloning ( Pomerleau , 1989 ) , ValueDice ( Kostrikov et al. , 2019 ) , AdVIL ( Swamy et al. , 2021 ) ) that operate purely on collected data . Unlike previous work however , we consider the effect of unobserved confounding . Our work shares the goal of interactive imitation learning algorithms ( e.g . DAgger ( Ross et al. , 2011 ) , AggreVaTe ( Ross & Bagnell , 2014 ) ) , in that we seek to match the output of a query to the expert . However , we focus on matching the output of a query on recorded data , rather than on learner rollouts , as is standard for interactive approaches . This is because the confounder decouples the actions recorded in the data and queried actions . Zhang et al . ( 2020 ) consider imitation learning through the lens of causal inference but focus on the one-step setting , while we consider multiple timesteps . Kumor et al . ( 2021 ) contemporaneously consider the multi-step setting and come to similar conclusions as us about the challenges of endogenous noise . They derive a necessary and sufficient structural condition for successful imitation learning , while we focus on practical algorithms with performance guarantees for a particular graphical model under exogenous noise . In contrast to imitation learning methods that seeks to match moments of the expert ’ s behavior ( Swamy et al. , 2021 ) , we focus only on matching average expert actions . We leave matching arbitrary moments to future work . Inertia Effects in Imitation Learning . Several authors have empirically observed a latching effect in policies trained via imitation learning : ( Muller et al. , 2006 ; Codevilla et al. , 2019 ; de Haan et al. , 2019 ; Bansal et al. , 2018 ; Kuefler et al. , 2017 ) , where learned policies tend to inappropriately repeat the same action . We seek to provide a plausible explanation and correction for the phenomenon reported in these works , and list several examples in Table 1 . We note that when attempting to explain inertia effects , de Haan et al . ( 2019 ) propose causal confounding as the root cause of the error . However , as previously pointed out by Spencer et al . ( 2021 ) , there is no actual confound in the theoretical or empirical examples in the work of de Haan et al . ( 2019 ) . This is because the learner observes all of the variables the expert was using to make decisions . Instrumental Variable Regression . The classical approach to instrumental variable regression ( Wright , 1928 ) is a two-stage least squares procedure ( e.g . in Angrist et al . ( 1996 ) ’ s textbook ) . We focus on the more general nonlinear setting and instead base our approaches on the more recent DEEPIV ( Hartford et al. , 2017 ) and AGMM ( Dikkala et al. , 2020 ) . We present extensions to the work in these papers , including a unified derivation of both methods and error analysis for DEEPIV . 3 A BRIEF REVIEW OF INSTRUMENTS IN CAUSAL MODELING . We begin by discussing the concept of an instrument before deriving our algorithmic approaches in a simplified , non-sequential setting . Let X , Y , and Z be random variables on ( potentially infinite ) sample spaces X , Y , and Z . Assume that X , Y , and Z have the causal , rather than statistical , dependency structure in Fig . 2 . Given a dataset of ( x , y , z ) tuples , we are interested in determining the causal relationship between X and Y , E [ Y |do ( x ) ] , where do ( · ) is the interventional operator of Pearl et al . ( 2016 ) . Intuitively , E [ Y |do ( x ) ] is the expected value of Y when we intervene and set X = x , rather than observe such an X . In the SCM to the right , h ( x ) = E [ Y |do ( x ) ] . Because of the presence of an unobserved confounder , U , that affects both X and Y , standard regression ( e.g . Ordinary Least Squares or OLS ) generically produces inconsistent estimates . Coarsely , this occurs because OLS will over-estimate the influence of the parts ofX that are affected by the confounder . If we only have observational data and are unable to perform randomized control trials , a canonical technique to recover h is IVR ( Winship & Morgan , 1999 ) . Formally , an instrument Z must satisfy three structural conditions : 1 . Unconfounded Instrument : Z ⊥⊥ U – i.e . independent randomization from confounder . 2 . Exclusion : Z ⊥⊥ Y |X , U – i.e . no extraneous paths . 3 . Relevance : Z 6⊥⊥ X – i.e . conditioning has an effect . Z satisfies these three conditions in the SCM of Fig . 2 . 1 Without loss of generality , we assume that E [ U ] = 0 . This allows us to concisely derive a set of conditional moment restrictions ( CMR ) : 0 = E [ U ] = E [ U |z ] = E [ Y − h ( X ) |z ] ( 1 ) ⇒ ∀z ∈ Z , E [ Y |z ] = E [ h ( X ) |z ] . ( 2 ) In words , these constraints are saying that a necessary condition for recovery of h ( x ) is that for all values of the instrument , the actual and predicted expected values of Y |Z are equal . We further assume that noise U enters additively to Y ,2 and write out the following equations : X = g ( Z , U , V ) , Y = h ( X ) + U . ( 3 ) We now derive an appropriate loss function for finding an ĥ that approximately satisfies the CMR . If we only have finite samples and can therefore only estimate conditional expectations up to some tolerance , it is natural to relax the CMR to minĥ∈H , δ 1 2Ez [ δ 2 z ] s.t . |E [ Y − ĥ ( X ) |z ] | ≤ δz , δz ≥ 0 , ∀z ∈ Z , ( 4 ) where the δz are slack variables . Then , the Lagrangian ( with the natural P ( z ) -weighted inner product that captures how often each we expect each z to occur ) is L ( ĥ , δ , λ ) = ∑ z∈Z P ( z ) λz ( E [ Y − ĥ ( X ) |z ] − δz ) + P ( z ) 1 2 δ2z , ( 5 ) where λ is the vector of Lagrange multipliers . By the stationarity component of the KKT conditions , ∇δzL ( ĥ , δ , λ ) = −P ( z ) λz + P ( z ) δz = 0 , ( 6 ) implying that δz = λz . Plugging this back into the Lagrangian , we can simplify our function to L ( ĥ , λ ) = ∑ z∈Z P ( z ) λzE [ Y − ĥ ( X ) |z ] − P ( z ) 1 2 λ2z . ( 7 ) We refer to ( 7 ) as the Regularized Lagrangian or ReLa for short . Now , solving for the optimal Lagrange multipliers via stationarity , we arrive at ∇λzL ( ĥ , λ ) = P ( z ) E [ Y − ĥ ( X ) |z ] − P ( z ) λz = 0 , ( 8 ) which implies the optimal λz is equal to E [ Y − ĥ ( X ) |z ] . Plugging this back into ( 7 ) recovers the loss function , L ( ĥ ) = ∑ z∈Z P ( z ) E [ Y − ĥ ( X ) |z ] 2 = PRMSE2 ( ĥ ) . ( 9 ) This expression is the square of the Projected Root Mean Squared Error ( PRMSE ) of Chen & Pouzo ( 2012 ) . To recap , by minimizing Eq . 9 , we are attempting to find an ĥ that approximately satisfies the CMR . Minimizing PRMSE is a necessary condition for recovering E [ Y |do ( X ) ] . For it to be a sufficient condition , one needs the natural identifiability assumptions – we refer interested readers to Chen & Pouzo ( 2012 ) for a more thorough discussion .

Summary: This paper proposes two causal imitation learning algorithms to address the confounding issue that causes spurious correlations. The method relies on instrumental variable regression to correct for confounding effects. The main idea is to use the state variable from the last time point as instrumental variable for the state-act variables at the current time point in a Markov decision process.