Deep Proxy Causal Learning and its Application to Confounded Bandit Policy Evaluation

1 Introduction . In causal learning , we aim to estimate the effect of our actions on the world . For example , we may be interested in measuring the impact of flight ticket prices on sales [ 2 , 34 ] , or the effect of grade retention on cognitive development [ 4 ] . We refer to our action as a treatment , which results in a particular outcome . It is often impossible to determine the effect of treatment on outcome from observational data alone , since the observed joint distribution of treatment and outcome can depend on a common confounder which influences both , and which might not be observed . In our example on sales of plane tickets given a price , the two might even be positively correlated in some circumstances , such as the simultaneous increase in sales and prices during the holiday season . This does not mean that raising the price causes sales to increase . In this context , people ’ s desire to fly is a confounder , since it affects both the number of ticket sales and the prices people are willing to accept . Thus , we need to correct the bias caused by the confounder . One common assumption to cope with confounding bias is to assume no unobserved confounders exist [ 8 ] , or more generally , the ignorable treatment assignment assumption [ 28 ] , which states that the treatment assignment is independent of the potential outcomes caused by the treatment , given the background data available . Although a number of methods are proposed based on this assumption [ 7 , 9 , 36 ] , it can be too restrictive , since it is often difficult to determine how the confounder affects treatment assignments and outcomes . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A less restrictive assumption is that we have access to proxy variables , which contain relevant side information on the confounder . In the flight tickets example , we can use the number of views of the ticket reservation page as a proxy variable , which reflects peoples ’ desire for flights . Note that if we can completely recover the confounder from proxy variables , the ignorable treatment assignment assumption can be satisfied . Motivated by this , Lee et al . [ 14 ] and Louizos et al . [ 17 ] aim to recover the distribution of confounders from proxy variables using modern machine learning techniques such as generative adversarial networks [ 5 ] or variational auto-encoders ( VAE ) [ 11 ] . Although these methods exhibit powerful empirical performance , there is little theory that guarantees the correct recovery of the causal effects . Kuroki and Pearl [ 13 ] first considered the necessary conditions on proxy variables to provably recover the underlying causal effect via direct recovery of the hidden confounder . This work was in turn generalized by Miao et al . [ 20 ] . In their work , it is shown that two types of proxy variables are sufficient to recover the true causal effects without explicitly recovering the confounder . One is an outcome-inducing proxy , which correlates with confounders and causes the outcome , and the other is a treatment-inducing proxy which is caused by confounders and correlates with the treatment . In the flight ticket example , we can use the number of views of the ticket reservation page as the outcome-inducing proxy , and the cost of fuel as the treatment-inducing proxy . Given these proxy variables , Miao et al . [ 20 ] show that the true causal effect can be recovered by solving a Fredholm integral equation , which is referred to as the proxy causal learning ( PCL ) problem . The PCL problem is known to have an interesting relation to the causal inference with multiple treatments , which uses a subset of treatments as proxy variables in PCL [ 22 , 33 ] . Although the PCL problem has a solid theoretical grounding , the question of how to estimate the causal effect remains a practical challenge , in particular when we consider nonlinear causal relationships or high-dimensional treatments . In Miao et al . [ 20 ] , the treatment and outcome are assumed to be categorical variables . In a follow-up study , Miao et al . [ 21 ] show that we can learn a linear causal effect by a method of moments . Deaner [ 3 ] models the causal effect as a linear combination of nonlinear basis functions , which is learned by solving two stage regression . These two methods are extended by Mastouri and Zhu et al . [ 18 ] , who learn the causal effect in a predefined reproducing kernel Hilbert space ( RKHS ) . We provide an overview of the PCL problem and the two-stage regression in Section 2 . Although these methods enjoy desirable theoretical properties , the flexibility of the model is limited , since all existing work uses pre-specified features . In this paper , we propose a novel Deep Feature Proxy Variable ( DFPV ) method , which is the first work to apply neural networks to the PCL problem . The technique we employ builds on earlier work in instrumental variable ( IV ) regression , which is a related causal inference setting to PCL . A range of deep learning methods has recently been introduced for IV regression [ 1 , 6 , 35 ] . We propose to adopt the Deep Feature Instrumental Variable method [ 35 ] , which learns deep adaptive features within a two-stage regression framework . Details of DFPV are given in Section 3 . In Section 4 , we empirically show that DFPV outperforms other PCL methods in several examples . We further apply PCL methods to the off-policy evaluation problem in a confounded bandit setting , which aims to estimate the average reward of a new policy given data with confounding bias . We discuss the setting in Section 3 , and show the superiority of DFPV in experiments in Section 4 . 2 Preliminaries . In this section , we introduce the proxy causal learning ( PCL ) problem and describe the existing two-stage regression methods to solve PCL . Notation . Throughout the paper , a capital letter ( e.g . A ) denotes a random variable , and we denote the set where a random variable takes values by the corresponding calligraphic letter ( e.g . A ) . The symbol P ( · ) denotes the probability distribution of a random variable ( e.g . P ( A ) ) . We use a lowercase letter to denote a realization of a random variable ( e.g . a ) . We denote the expectation over a random variable as E [ · ] and ‖f‖P ( · ) as the L2-norm of a function f with respect to P ( · ) ; i.e . ‖f‖P ( A ) = √ EA [ f2 ( A ) ] . 2.1 Problem Setting for Proxy Causal Learning We begin with a description of the PCL setting . We observe a treatment A ∈ A , whereA ⊂ RdA , and the corresponding outcome Y ∈ Y , where Y ⊂ R. We assume that there exists an unobserved confounder U ∈ U that affects both A and Y . The goal of PCL is to estimate the structural function fstruct ( X ) defined as fstruct ( a ) = EU [ EY [ Y |A = a , U ] ] , which we assume to be continuous . This function is also known as the Average Treatment Effect ( ATE ) . The challenge of estimating fstruct is that the confounder U is not observable — we can not estimate the structural function from observations A and Y alone . To deal with this , we assume access to a treatment-inducing proxy variable Z , and an outcome-inducing proxy variable W , which satisfy the following structural assumption and completeness assumption . Assumption 1 ( Structural Assumption [ 3 , 18 ] ) . We assume Y ⊥⊥Z|A , U , and W ⊥⊥ ( A , Z ) |U . Assumption 2 ( Completeness Assumption on Confounder [ 3 , 18 ] ) . Let l : U → R be any square integrable function ‖l‖P ( U ) < ∞ . The following conditions hold : E [ l ( U ) | A = a , W = w ] = 0 ∀ ( a , w ) ∈ A×W ⇔ l ( u ) = 0 P ( U ) -a.e . E [ l ( U ) | A = a , Z = z ] = 0 ∀ ( a , z ) ∈ A×Z ⇔ l ( u ) = 0 P ( U ) -a.e . Figure 1 shows the causal graph describing these relationships . In our setting , we assume that there is no observable confounder , though this may be easily included [ 18 , 31 ] as presented in Appendix D. Here , the bidirectional arrows mean that we allow both directions or even a common ancestor variable . Given these assumptions , it is shown that the structural function can be expressed using a solution of an integral equation . Proposition 1 ( Miao et al . [ 20 ] ) . Let Assumptions 1 , 2 and Assumptions 4 , 5 , 6 in Appendix B hold . Then there exists at least one solution to the functional equation E [ Y |A = a , Z = z ] = ∫ h ( a , W ) ρW ( w|A = a , Z = z ) dw , ( 1 ) which holds for any ( a , z ) ∈ A×Z . Here , we denote ρW ( w|A = a , Z = z ) as the density function of the conditional probability P ( W |A = a , Z = z ) . Let h∗ be a solution of ( 1 ) . The structural function fstruct is given as fstruct ( a ) = EW [ h∗ ( a , W ) ] . ( 2 ) For completeness , we present a proof in Appendix B ( Lemma 2 and Corollary 1 ) , which is due to Miao et al . [ 21 ] and Deaner [ 3 ] . Following Miao et al . [ 21 ] , we call h∗ the bridge function . From Proposition 1 , we can see that the estimation of the structural function reduces to the estimation of the bridge function , since once we obtain the bridge function , the structural function directly follows from ( 2 ) . 2.2 Two-stage Regression Scheme for Proximal Causal Learning with Fixed Features . To obtain the bridge function , we need to solve the functional equation ( 1 ) . However , directly solving ( 1 ) in a rich function space can be ill-posed ( see discussion in Nashed and Wahba [ 25 ] ) . To address this , recent works [ 3 , 18 ] minimize the following regularized loss LPV to obtain an estimate of the bridge function ĥ : ĥ = arg min h∈Hh LPV ( h ) , LPV ( h ) = EY , A , Z [ ( Y −EW |A , Z [ h ( A , W ) ] ) 2 ] + Ω ( h ) , ( 3 ) where Hh is an arbitrary space of continuous functions and Ω ( h ) is a regularizer on h. Note that this loss can be interpreted as the deviation of the r.h.s and the l.h.s in ( 1 ) measured in L2-norm with respect to the distribution P ( Y , A , Z ) . Deaner [ 3 ] and Mastouri and Zhu et al . [ 18 ] solve the minimization problem ( 3 ) by successively solving two-stage regression problems . They model the bridge function as h ( a , w ) = u > ( ψA ( a ) ⊗ψW ( w ) ) ( 4 ) where u is a learnable weight vector , ψA ( a ) , ψW ( w ) are vectors of fixed basis functions , and ⊗ is a Kronecker product , defined as a⊗ b = vec ( ab > ) for any finite dimensional vectors a , b.1 An estimate û is obtained by solving the successive regression problems . In Stage 1 , we estimate the conditional expectation EW |A=a , Z=z [ ψW ( W ) ] as a function of a , z . In Stage 2 , we substitute the model ( 4 ) into the inner conditional expectation in LPV , EW |A=a , Z=z [ h ( a , W ) ] = u > ( ψA ( a ) ⊗EW |A=a , Z=z [ ψW ( W ) ] ) , and then minimize LPV with respect to u using the estimate of EW |A=a , Z=z [ ψW ( W ) ) ] from Stage 1 . The above idea can be implemented as follows . We model the conditional expectation as EW |A=a , Z=z [ ψ ( W ) ] = V ( φA ( a ) ⊗φZ ( z ) ) , where φA ( a ) , φZ ( z ) are another set of basis functions , and V is a matrix to be learned . Note that we can use different basis functions for φA ( a ) and ψA ( a ) . In Stage 1 , the matrix V is learned by minimizing the following loss , L1 ( V ) = EW , A , Z [ ‖ψW ( W ) −V ( φA ( A ) ⊗φZ ( Z ) ) ‖2 ] + λ1‖V ‖2 , ( 5 ) where λ1 > 0 is a regularization parameter . This is a linear ridge regression problem with multiple targets , which can be solved analytically . In Stage 2 , given the minimizer V̂ = arg minV L1 ( V ) , we can obtain û by minimizing the loss L2 ( u ) = EY , A , Z [ ‖Y −u > ( ψA ( A ) ⊗ ( V̂ ( φA ( A ) ⊗φZ ( Z ) ) ) ) ‖2 ] + λ2‖u‖2 , ( 6 ) where λ2 > 0 is another regularization parameter . Stage 2 corresponds to another linear ridge regression from input ψA ( A ) ⊗ ( V̂ ( φA ( A ) ⊗φZ ( Z ) ) ) to target Y , and also enjoys a closed-form solution . Given the learned weights û = arg minuL2 ( u ) , the estimated structural function is f̂struct ( a ) = û > ( ψA ( a ) ⊗EW [ ψW ( W ) ] ) . When fixed feature dictionaries are used , this two-stage regression benefits from strong theoretical guarantees [ 3 , 18 ] . The use of pre-specified feature maps , limits the scope and flexibility of the method , however , especially if the treatment and proxies are high dimensional ( e.g . images or text ) , and the causal relations are nonlinear . To overcome this , we propose to use adaptive features , expressed by neural nets , as described in the next section .

Summary: This paper presents a novel method: deep feature proxy variable. It applies a neural network to PCL, allowing non-linear relationships between variables. The authors prove the consistency of the method and demonstrate how this method can be applied to off-policy evaluation. The authors test the methods using two simulation studies, showing that DFPV outperforms existing methods.