Deep Proxy Causal Learning and its Application to Confounded Bandit Policy Evaluation

1 Introduction . In causal learning , we aim to estimate the effect of our actions on the world . For example , we may be interested in measuring the impact of flight ticket prices on sales [ 2 , 34 ] , or the effect of grade retention on cognitive development [ 4 ] . We refer to our action as a treatment , which results in a particular outcome . It is often impossible to determine the effect of treatment on outcome from observational data alone , since the observed joint distribution of treatment and outcome can depend on a common confounder which influences both , and which might not be observed . In our example on sales of plane tickets given a price , the two might even be positively correlated in some circumstances , such as the simultaneous increase in sales and prices during the holiday season . This does not mean that raising the price causes sales to increase . In this context , people ’ s desire to fly is a confounder , since it affects both the number of ticket sales and the prices people are willing to accept . Thus , we need to correct the bias caused by the confounder . One common assumption to cope with confounding bias is to assume no unobserved confounders exist [ 8 ] , or more generally , the ignorable treatment assignment assumption [ 28 ] , which states that the treatment assignment is independent of the potential outcomes caused by the treatment , given the background data available . Although a number of methods are proposed based on this assumption [ 7 , 9 , 36 ] , it can be too restrictive , since it is often difficult to determine how the confounder affects treatment assignments and outcomes . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A less restrictive assumption is that we have access to proxy variables , which contain relevant side information on the confounder . In the flight tickets example , we can use the number of views of the ticket reservation page as a proxy variable , which reflects peoples ’ desire for flights . Note that if we can completely recover the confounder from proxy variables , the ignorable treatment assignment assumption can be satisfied . Motivated by this , Lee et al . [ 14 ] and Louizos et al . [ 17 ] aim to recover the distribution of confounders from proxy variables using modern machine learning techniques such as generative adversarial networks [ 5 ] or variational auto-encoders ( VAE ) [ 11 ] . Although these methods exhibit powerful empirical performance , there is little theory that guarantees the correct recovery of the causal effects . Kuroki and Pearl [ 13 ] first considered the necessary conditions on proxy variables to provably recover the underlying causal effect via direct recovery of the hidden confounder . This work was in turn generalized by Miao et al . [ 20 ] . In their work , it is shown that two types of proxy variables are sufficient to recover the true causal effects without explicitly recovering the confounder . One is an outcome-inducing proxy , which correlates with confounders and causes the outcome , and the other is a treatment-inducing proxy which is caused by confounders and correlates with the treatment . In the flight ticket example , we can use the number of views of the ticket reservation page as the outcome-inducing proxy , and the cost of fuel as the treatment-inducing proxy . Given these proxy variables , Miao et al . [ 20 ] show that the true causal effect can be recovered by solving a Fredholm integral equation , which is referred to as the proxy causal learning ( PCL ) problem . The PCL problem is known to have an interesting relation to the causal inference with multiple treatments , which uses a subset of treatments as proxy variables in PCL [ 22 , 33 ] . Although the PCL problem has a solid theoretical grounding , the question of how to estimate the causal effect remains a practical challenge , in particular when we consider nonlinear causal relationships or high-dimensional treatments . In Miao et al . [ 20 ] , the treatment and outcome are assumed to be categorical variables . In a follow-up study , Miao et al . [ 21 ] show that we can learn a linear causal effect by a method of moments . Deaner [ 3 ] models the causal effect as a linear combination of nonlinear basis functions , which is learned by solving two stage regression . These two methods are extended by Mastouri and Zhu et al . [ 18 ] , who learn the causal effect in a predefined reproducing kernel Hilbert space ( RKHS ) . We provide an overview of the PCL problem and the two-stage regression in Section 2 . Although these methods enjoy desirable theoretical properties , the flexibility of the model is limited , since all existing work uses pre-specified features . In this paper , we propose a novel Deep Feature Proxy Variable ( DFPV ) method , which is the first work to apply neural networks to the PCL problem . The technique we employ builds on earlier work in instrumental variable ( IV ) regression , which is a related causal inference setting to PCL . A range of deep learning methods has recently been introduced for IV regression [ 1 , 6 , 35 ] . We propose to adopt the Deep Feature Instrumental Variable method [ 35 ] , which learns deep adaptive features within a two-stage regression framework . Details of DFPV are given in Section 3 . In Section 4 , we empirically show that DFPV outperforms other PCL methods in several examples . We further apply PCL methods to the off-policy evaluation problem in a confounded bandit setting , which aims to estimate the average reward of a new policy given data with confounding bias . We discuss the setting in Section 3 , and show the superiority of DFPV in experiments in Section 4 . 2 Preliminaries . In this section , we introduce the proxy causal learning ( PCL ) problem and describe the existing two-stage regression methods to solve PCL . Notation . Throughout the paper , a capital letter ( e.g . A ) denotes a random variable , and we denote the set where a random variable takes values by the corresponding calligraphic letter ( e.g . A ) . The symbol P ( · ) denotes the probability distribution of a random variable ( e.g . P ( A ) ) . We use a lowercase letter to denote a realization of a random variable ( e.g . a ) . We denote the expectation over a random variable as E [ · ] and ‖f‖P ( · ) as the L2-norm of a function f with respect to P ( · ) ; i.e . ‖f‖P ( A ) = √ EA [ f2 ( A ) ] . 2.1 Problem Setting for Proxy Causal Learning We begin with a description of the PCL setting . We observe a treatment A ∈ A , whereA ⊂ RdA , and the corresponding outcome Y ∈ Y , where Y ⊂ R. We assume that there exists an unobserved confounder U ∈ U that affects both A and Y . The goal of PCL is to estimate the structural function fstruct ( X ) defined as fstruct ( a ) = EU [ EY [ Y |A = a , U ] ] , which we assume to be continuous . This function is also known as the Average Treatment Effect ( ATE ) . The challenge of estimating fstruct is that the confounder U is not observable — we can not estimate the structural function from observations A and Y alone . To deal with this , we assume access to a treatment-inducing proxy variable Z , and an outcome-inducing proxy variable W , which satisfy the following structural assumption and completeness assumption . Assumption 1 ( Structural Assumption [ 3 , 18 ] ) . We assume Y ⊥⊥Z|A , U , and W ⊥⊥ ( A , Z ) |U . Assumption 2 ( Completeness Assumption on Confounder [ 3 , 18 ] ) . Let l : U → R be any square integrable function ‖l‖P ( U ) < ∞ . The following conditions hold : E [ l ( U ) | A = a , W = w ] = 0 ∀ ( a , w ) ∈ A×W ⇔ l ( u ) = 0 P ( U ) -a.e . E [ l ( U ) | A = a , Z = z ] = 0 ∀ ( a , z ) ∈ A×Z ⇔ l ( u ) = 0 P ( U ) -a.e . Figure 1 shows the causal graph describing these relationships . In our setting , we assume that there is no observable confounder , though this may be easily included [ 18 , 31 ] as presented in Appendix D. Here , the bidirectional arrows mean that we allow both directions or even a common ancestor variable . Given these assumptions , it is shown that the structural function can be expressed using a solution of an integral equation . Proposition 1 ( Miao et al . [ 20 ] ) . Let Assumptions 1 , 2 and Assumptions 4 , 5 , 6 in Appendix B hold . Then there exists at least one solution to the functional equation E [ Y |A = a , Z = z ] = ∫ h ( a , W ) ρW ( w|A = a , Z = z ) dw , ( 1 ) which holds for any ( a , z ) ∈ A×Z . Here , we denote ρW ( w|A = a , Z = z ) as the density function of the conditional probability P ( W |A = a , Z = z ) . Let h∗ be a solution of ( 1 ) . The structural function fstruct is given as fstruct ( a ) = EW [ h∗ ( a , W ) ] . ( 2 ) For completeness , we present a proof in Appendix B ( Lemma 2 and Corollary 1 ) , which is due to Miao et al . [ 21 ] and Deaner [ 3 ] . Following Miao et al . [ 21 ] , we call h∗ the bridge function . From Proposition 1 , we can see that the estimation of the structural function reduces to the estimation of the bridge function , since once we obtain the bridge function , the structural function directly follows from ( 2 ) . 2.2 Two-stage Regression Scheme for Proximal Causal Learning with Fixed Features . To obtain the bridge function , we need to solve the functional equation ( 1 ) . However , directly solving ( 1 ) in a rich function space can be ill-posed ( see discussion in Nashed and Wahba [ 25 ] ) . To address this , recent works [ 3 , 18 ] minimize the following regularized loss LPV to obtain an estimate of the bridge function ĥ : ĥ = arg min h∈Hh LPV ( h ) , LPV ( h ) = EY , A , Z [ ( Y −EW |A , Z [ h ( A , W ) ] ) 2 ] + Ω ( h ) , ( 3 ) where Hh is an arbitrary space of continuous functions and Ω ( h ) is a regularizer on h. Note that this loss can be interpreted as the deviation of the r.h.s and the l.h.s in ( 1 ) measured in L2-norm with respect to the distribution P ( Y , A , Z ) . Deaner [ 3 ] and Mastouri and Zhu et al . [ 18 ] solve the minimization problem ( 3 ) by successively solving two-stage regression problems . They model the bridge function as h ( a , w ) = u > ( ψA ( a ) ⊗ψW ( w ) ) ( 4 ) where u is a learnable weight vector , ψA ( a ) , ψW ( w ) are vectors of fixed basis functions , and ⊗ is a Kronecker product , defined as a⊗ b = vec ( ab > ) for any finite dimensional vectors a , b.1 An estimate û is obtained by solving the successive regression problems . In Stage 1 , we estimate the conditional expectation EW |A=a , Z=z [ ψW ( W ) ] as a function of a , z . In Stage 2 , we substitute the model ( 4 ) into the inner conditional expectation in LPV , EW |A=a , Z=z [ h ( a , W ) ] = u > ( ψA ( a ) ⊗EW |A=a , Z=z [ ψW ( W ) ] ) , and then minimize LPV with respect to u using the estimate of EW |A=a , Z=z [ ψW ( W ) ) ] from Stage 1 . The above idea can be implemented as follows . We model the conditional expectation as EW |A=a , Z=z [ ψ ( W ) ] = V ( φA ( a ) ⊗φZ ( z ) ) , where φA ( a ) , φZ ( z ) are another set of basis functions , and V is a matrix to be learned . Note that we can use different basis functions for φA ( a ) and ψA ( a ) . In Stage 1 , the matrix V is learned by minimizing the following loss , L1 ( V ) = EW , A , Z [ ‖ψW ( W ) −V ( φA ( A ) ⊗φZ ( Z ) ) ‖2 ] + λ1‖V ‖2 , ( 5 ) where λ1 > 0 is a regularization parameter . This is a linear ridge regression problem with multiple targets , which can be solved analytically . In Stage 2 , given the minimizer V̂ = arg minV L1 ( V ) , we can obtain û by minimizing the loss L2 ( u ) = EY , A , Z [ ‖Y −u > ( ψA ( A ) ⊗ ( V̂ ( φA ( A ) ⊗φZ ( Z ) ) ) ) ‖2 ] + λ2‖u‖2 , ( 6 ) where λ2 > 0 is another regularization parameter . Stage 2 corresponds to another linear ridge regression from input ψA ( A ) ⊗ ( V̂ ( φA ( A ) ⊗φZ ( Z ) ) ) to target Y , and also enjoys a closed-form solution . Given the learned weights û = arg minuL2 ( u ) , the estimated structural function is f̂struct ( a ) = û > ( ψA ( a ) ⊗EW [ ψW ( W ) ] ) . When fixed feature dictionaries are used , this two-stage regression benefits from strong theoretical guarantees [ 3 , 18 ] . The use of pre-specified feature maps , limits the scope and flexibility of the method , however , especially if the treatment and proxies are high dimensional ( e.g . images or text ) , and the causal relations are nonlinear . To overcome this , we propose to use adaptive features , expressed by neural nets , as described in the next section .

Summary: This work extends known Proxy Causal Learning methods that aim at learning the causal effect of treatment in cases where unobserved confounders are assumed yet there is observable information in form of proxy variables to these confounders. In particular, authors extend the theory to allow deep neural nets to be used as building blocks of the two stage regression proposed by Mastouri et al instead of RKHS. The paper also explains how to use the method to perform off-policy evaluation in a bandit setting where arms are confounded. Synthetic experiments are also reported to compare the performance with a few baselines of the same family.