What Would the Expert $do(\cdot)$?: Causal Imitation Learning

1 INTRODUCTION . Much of the theory of imitation learning ( IL ) indicates that with enough demonstrations , we should be able to accurately recover the expert ’ s policy . When we apply IL algorithms in practice however , we sometimes see them produce manifestly incorrect estimates of the expert ’ s policy ( Muller et al. , 2006 ; Codevilla et al. , 2019 ; de Haan et al. , 2019 ; Bansal et al. , 2018 ; Kuefler et al. , 2017 ) . One possible reason for this phenomenon is that empirically , we only have access to noisy recordings of what the expert did . This critical detail has been thus far neglected by most prior theoretical work in imitation learning . We focus in this paper on how best to learn from two kinds of noisy data : • Exogenous noise : When we observe expert actions corrupted by a persistent noise ( e.g . a faulty joystick that persistently perturbs actions before they are executed in the game ) . • Endogenous noise : When we do not observe the full state an expert used to pick an action ( e.g . the learner not knowing there ’ s an enemy behind a door ) . The net effect of either kind of persistent noise ( more formally , an unobserved confounder ) is to introduce temporal correlations in the recorded actions that do not have their true cause in the recorded state . For example , consider recordings of an expert driver slowing down at a stop sign . If all we present the learner with as state input is whether the expert was slowing down at the last timestep , they will likely learn to simply repeat the expert ’ s past action . Thus , once the car begins to slow down , it continues to slow down , regardless of whether there is a stop sign present . At a more abstract level , these sorts of inertia problems can result from temporal correlations between pairs of actions ( e.g . the effect of the stop sign ) being reflected in the state ( e.g . the past action variable ) , leading to spurious correlations between state and action that the learner might unfortunately latch onto ( e.g . repeating the past action ) . What should we hope to learn then in these confounded settings ? Given we do not have access to the unobserved confounder , a reasonable choice is to ensure that we match the behavior of an expert that has access to the same information we do . That is , if we could query the expert for an action with only the information we have available , we should strive to produce an action that matches this queried action . While applying an interactive imitation learning algorithm ( Ross et al. , 2011 ) would allow us to collect a dataset uncorrupted by confounding , a queryable expert is not a realistic assumption for many domains . We therefore focus on approaches for the off-policy setting . We base our algorithms on a technique from econometrics for dealing with confounding in recorded data known an instrumental variable regression ( IVR ) ( Angrist et al. , 1996 ) . The high-level idea of IVR is to leverage an instrument , a source of random variation independent of the confounder , to deconfound inputs to a learning procedure via conditioning on the instrument . In dynamical systems , history can act as this source of variation , as it is unaffected by future confounding ( Hefny et al. , 2015 ) . Our key insight is that we can leverage past states as instruments to break the spurious correlation between states and actions caused by an unobserved confounder . Our work provides the following contributions : 1 . We formalize confounding in imitation learning . We provide a structural causal model that captures inertia effects that result from temporally correlated exogenous or endogenous noise . We also derive a test to detect whether this sort of confounding is present in a dataset . 2 . We present a unified derivation of modern instrumental variable regression techniques . We show how two recent extensions of the classical IVR technique share a common structure . We also extend the theoretical analysis of previous work by deriving accuracy bounds . 3 . We provide two novel algorithms to deal with confounding in imitation learning . We derive two novel IVR-based algorithms : • DoubIL is a generative modeling approach that and can utilize access to a simulator for reduced sample complexity . • ResiduIL is a simulator-free , game-theoretic approach . We derive performance bounds for policies produced by these algorithms under exogenous noise . We empirically investigate the effect of the persistence of the confounder on this bound . We also compare the performance of these approaches to behavioral cloning under endogenous noise . 2 RELATED WORK . Imitation Learning . Broadly speaking , imitation learning approaches can be grouped into three classes : offline , online , and interactive . Our work is most similar to offline imitation learning algorithms ( e.g . Behavioral Cloning ( Pomerleau , 1989 ) , ValueDice ( Kostrikov et al. , 2019 ) , AdVIL ( Swamy et al. , 2021 ) ) that operate purely on collected data . Unlike previous work however , we consider the effect of unobserved confounding . Our work shares the goal of interactive imitation learning algorithms ( e.g . DAgger ( Ross et al. , 2011 ) , AggreVaTe ( Ross & Bagnell , 2014 ) ) , in that we seek to match the output of a query to the expert . However , we focus on matching the output of a query on recorded data , rather than on learner rollouts , as is standard for interactive approaches . This is because the confounder decouples the actions recorded in the data and queried actions . Zhang et al . ( 2020 ) consider imitation learning through the lens of causal inference but focus on the one-step setting , while we consider multiple timesteps . Kumor et al . ( 2021 ) contemporaneously consider the multi-step setting and come to similar conclusions as us about the challenges of endogenous noise . They derive a necessary and sufficient structural condition for successful imitation learning , while we focus on practical algorithms with performance guarantees for a particular graphical model under exogenous noise . In contrast to imitation learning methods that seeks to match moments of the expert ’ s behavior ( Swamy et al. , 2021 ) , we focus only on matching average expert actions . We leave matching arbitrary moments to future work . Inertia Effects in Imitation Learning . Several authors have empirically observed a latching effect in policies trained via imitation learning : ( Muller et al. , 2006 ; Codevilla et al. , 2019 ; de Haan et al. , 2019 ; Bansal et al. , 2018 ; Kuefler et al. , 2017 ) , where learned policies tend to inappropriately repeat the same action . We seek to provide a plausible explanation and correction for the phenomenon reported in these works , and list several examples in Table 1 . We note that when attempting to explain inertia effects , de Haan et al . ( 2019 ) propose causal confounding as the root cause of the error . However , as previously pointed out by Spencer et al . ( 2021 ) , there is no actual confound in the theoretical or empirical examples in the work of de Haan et al . ( 2019 ) . This is because the learner observes all of the variables the expert was using to make decisions . Instrumental Variable Regression . The classical approach to instrumental variable regression ( Wright , 1928 ) is a two-stage least squares procedure ( e.g . in Angrist et al . ( 1996 ) ’ s textbook ) . We focus on the more general nonlinear setting and instead base our approaches on the more recent DEEPIV ( Hartford et al. , 2017 ) and AGMM ( Dikkala et al. , 2020 ) . We present extensions to the work in these papers , including a unified derivation of both methods and error analysis for DEEPIV . 3 A BRIEF REVIEW OF INSTRUMENTS IN CAUSAL MODELING . We begin by discussing the concept of an instrument before deriving our algorithmic approaches in a simplified , non-sequential setting . Let X , Y , and Z be random variables on ( potentially infinite ) sample spaces X , Y , and Z . Assume that X , Y , and Z have the causal , rather than statistical , dependency structure in Fig . 2 . Given a dataset of ( x , y , z ) tuples , we are interested in determining the causal relationship between X and Y , E [ Y |do ( x ) ] , where do ( · ) is the interventional operator of Pearl et al . ( 2016 ) . Intuitively , E [ Y |do ( x ) ] is the expected value of Y when we intervene and set X = x , rather than observe such an X . In the SCM to the right , h ( x ) = E [ Y |do ( x ) ] . Because of the presence of an unobserved confounder , U , that affects both X and Y , standard regression ( e.g . Ordinary Least Squares or OLS ) generically produces inconsistent estimates . Coarsely , this occurs because OLS will over-estimate the influence of the parts ofX that are affected by the confounder . If we only have observational data and are unable to perform randomized control trials , a canonical technique to recover h is IVR ( Winship & Morgan , 1999 ) . Formally , an instrument Z must satisfy three structural conditions : 1 . Unconfounded Instrument : Z ⊥⊥ U – i.e . independent randomization from confounder . 2 . Exclusion : Z ⊥⊥ Y |X , U – i.e . no extraneous paths . 3 . Relevance : Z 6⊥⊥ X – i.e . conditioning has an effect . Z satisfies these three conditions in the SCM of Fig . 2 . 1 Without loss of generality , we assume that E [ U ] = 0 . This allows us to concisely derive a set of conditional moment restrictions ( CMR ) : 0 = E [ U ] = E [ U |z ] = E [ Y − h ( X ) |z ] ( 1 ) ⇒ ∀z ∈ Z , E [ Y |z ] = E [ h ( X ) |z ] . ( 2 ) In words , these constraints are saying that a necessary condition for recovery of h ( x ) is that for all values of the instrument , the actual and predicted expected values of Y |Z are equal . We further assume that noise U enters additively to Y ,2 and write out the following equations : X = g ( Z , U , V ) , Y = h ( X ) + U . ( 3 ) We now derive an appropriate loss function for finding an ĥ that approximately satisfies the CMR . If we only have finite samples and can therefore only estimate conditional expectations up to some tolerance , it is natural to relax the CMR to minĥ∈H , δ 1 2Ez [ δ 2 z ] s.t . |E [ Y − ĥ ( X ) |z ] | ≤ δz , δz ≥ 0 , ∀z ∈ Z , ( 4 ) where the δz are slack variables . Then , the Lagrangian ( with the natural P ( z ) -weighted inner product that captures how often each we expect each z to occur ) is L ( ĥ , δ , λ ) = ∑ z∈Z P ( z ) λz ( E [ Y − ĥ ( X ) |z ] − δz ) + P ( z ) 1 2 δ2z , ( 5 ) where λ is the vector of Lagrange multipliers . By the stationarity component of the KKT conditions , ∇δzL ( ĥ , δ , λ ) = −P ( z ) λz + P ( z ) δz = 0 , ( 6 ) implying that δz = λz . Plugging this back into the Lagrangian , we can simplify our function to L ( ĥ , λ ) = ∑ z∈Z P ( z ) λzE [ Y − ĥ ( X ) |z ] − P ( z ) 1 2 λ2z . ( 7 ) We refer to ( 7 ) as the Regularized Lagrangian or ReLa for short . Now , solving for the optimal Lagrange multipliers via stationarity , we arrive at ∇λzL ( ĥ , λ ) = P ( z ) E [ Y − ĥ ( X ) |z ] − P ( z ) λz = 0 , ( 8 ) which implies the optimal λz is equal to E [ Y − ĥ ( X ) |z ] . Plugging this back into ( 7 ) recovers the loss function , L ( ĥ ) = ∑ z∈Z P ( z ) E [ Y − ĥ ( X ) |z ] 2 = PRMSE2 ( ĥ ) . ( 9 ) This expression is the square of the Projected Root Mean Squared Error ( PRMSE ) of Chen & Pouzo ( 2012 ) . To recap , by minimizing Eq . 9 , we are attempting to find an ĥ that approximately satisfies the CMR . Minimizing PRMSE is a necessary condition for recovering E [ Y |do ( X ) ] . For it to be a sufficient condition , one needs the natural identifiability assumptions – we refer interested readers to Chen & Pouzo ( 2012 ) for a more thorough discussion .

Summary: This paper attempts to study imitation learning from a causal inference perspective. More specifically, let S be an observed state, A be action and Z be an instrumental variable. The authors propose that one should perform causal imitation learning by learning a policy function that could induce the same interventional distribution P(A|do(Z)).