Deep Proxy Causal Learning and its Application to Confounded Bandit Policy Evaluation
Authors: Liyuan Xu
Heishiro Kanagawa
Arthur Gretton
Venue: NeurIPS 2021 Poster (2021)
Keywords: Causal Inference
Deep Learning
Off-policy evaluation

Abstract: Proxy causal learning (PCL) is a method for estimating the causal effect of treatments on outcomes in the presence of unobserved confounding, using proxies (structured side information) for the confounder. This is achieved via two-stage regression: in the first stage, we model relations among the treatment and proxies; in the second stage, we use this model to learn the effect of treatment on the outcome, given the context provided by the proxies. PCL  guarantees recovery of the true causal effect, subject to identifiability conditions. We propose a novel method for PCL, the deep feature proxy variable method (DFPV), to address the case where the proxies, treatments, and outcomes are high-dimensional and have nonlinear complex relationships, as represented by deep neural network features. We show that DFPV outperforms recent state-of-the-art PCL methods on challenging synthetic benchmarks, including settings involving high dimensional image data. Furthermore, we show that PCL can be applied to off-policy evaluation for the confounded bandit problem, in which DFPV also exhibits competitive performance.


Deep Proxy Causal Learning and its Application to Confounded Bandit Policy Evaluation
Liyuan Xu Gatsby Unit
Heishiro Kanagawa
Gatsby Unit
liyuan. jtr.an....19uc..eee 19wucl..c..c. ac.
heishiroo.amagaae@mmil. com
Arthur Gretton
Gatsby Unit
arthur wtw.bur.gretttn@maal..cco
Abstract
Proxy causal learning (PCL) is method for estimating the causal effect of treat- ments on outcomes in the presence of unobserved confounding, using proxies (structured side information) for the confounder. This is achieved via two-stage regression: in the first stage, we model relations among the treatment and proxies; in the second stage, we use this model to learn the effect of treatment on the outcome, given the context provided by the proxies. PCL guarantees recovery of the true causal effect, subject to identifiability conditions. We propose a novel method for PCL, the deep feature proxy variable method (DFPV), to address the case where the proxies, treatments, and outcomes are high-dimensional and have nonlinear complex relationships, represented by deep neural network features. We show that DFPV outperforms recent state-of-the-art PCL methods on challeng- ing synthetic benchmarks, including settings involving high dimensional image data. Furthermore, we show that PCL can be applied to off-policy evaluation for the confounded bandit problem, in which DFPV also exhibits competitive performance.
Introduction
In causal learning, we aim to estimate the effect of our actions on the world. For example, we may be interested in measuring the impact of flight ticket prices sales [2, 34], or the effect of grade retention on cognitive development [4]. We refer to our action as a treatment, which results in particular outcome. It is often impossible to determine the effect of treatment on outcome from observational data alone, since the observed joint distribution treatment and outcome can depend common confounder which influences both, and which might not be observed. In our example sales of plane tickets given price, the two might even be positively correlated in some circumstances, such the simultaneous increase in sales and prices during the holiday season. This does not mean that raising the price causes sales to increase. In this context, people's desire to fly is a confounder, since it affects both the number of ticket sales and the prices people willing accept. Thus, need to correct the bias caused by the confounder.
One common assumption to cope with confounding bias is assume unobserved confounders exist [8]. or more generally, the ignorable treatment assignment assumption [28], which states that the treatment assignment is independent of the potential outcomes caused by the treatment, given the background data available. Although a number of methods are proposed based on this assumption
[7, 9, 36], it can be restrictive, since it is often difficult determine how the confounder affects treatment assignments and outcomes.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

A less restrictive assumption is that we have access to proxy variables, which contain relevant side information on the confounder. In the flight tickets example, we can use the number of views of the ticket reservation page proxy variable, which reflects peoples' desire for flights. Note that if we can completely recover the confounder from proxy variables, the ignorable treatment assignment assumption can be satisfied. Motivated by this, Lee et al. [14] and Louizos et al. [17] aim to recover the distribution of confounders from proxy variables using modern machine learning techniques such generative adversarial networks [5] or variational auto-encoders (VAE) [11]. Although these methods exhibit powerful empirical performance, there is little theory that guarantees the correct
recovery of the causal effects.
Kuroki and Pearl [13] first considered the necessary conditions on proxy variables to provably recover the underlying causal effect via direct recovery of the hidden confounder. This work was in turn generalized by Miao et al. [20]. In their work, it is shown that two types of proxy variables are sufficient to recover the true causal effects without explicitly recovering the confounder. One is an outcome- -irruiinn proxy, which correlates with confounders and causes the outcome, and the other is treatmentinnductin proxy which is caused by confounders and correlates with the treatment. In the flight ticket example, we can use the number of views of the ticket reservation page as the outcome-inducing proxy, and the cost of fuel as the treatment-inducing proxy. Given these proxy variables, Miao et al. [20] show that the true causal effect can be recovered by solving a Fredholm integral equation, which is referred to as the proxy causal learning (PCL) problem. The PCL problem is known to have an interesting relation to the causal inference with multiple treatments, which uses a subset of treatments as proxy variables in PCL [22, 33].
Although the PCL problem has a solid theoretical grounding, the question of how to estimate the causal effect remains practical challenge, in particular when we consider nonlinear causal relationships or high-dimennsionl treatments. In Miao et al. [20], the treatment and outcome are assumed to be categorical variables. In a follow-up study, Miao et al. [21] show that we can learn a linear causal effect by method of moments. Deaner [3] models the causal effect as a linear combination of nonlinear basis functions, which is learned by solving two stage regression. These two methods are extended by Mastouri and Zhu et al. [18], who learn the causal effect in a predefined reproducing kernel Hilbert space (RKHS). We provide two-stage regression in Section 2. Although these methods enjoy desirable theoretical properties, the flexibility of the model is limited, since all existing work uses pre- ppeesppeified features.
overview of the PCL problem and the
In this paper, we propose a novel Deep Feature Proxy Variable (DFPV) method, which the first work to apply neural networks to the PCL problem. The technique we employ builds on earlier work in instrumental variable (/V) regression, which is related causal inference setting to PCL. A range of deep learning methods has recently been introduced for IV regression [1, 6, 35J. We propose to adopt the Deep Feature Instrumental Variable method [35], which learns deep adaptive features within a two-stage regression framework. Details of DFPV are given in Section 3. In Section 4, empirically show that DFPV outperforms other PCL methods in several examples. We further apply PCL methods to the off-policy evaluation problem in a confounded bandit setting, which aims to estimate the average reward of a new policy given data with confounding bias. We discuss the setting in Section 3, and show the superiority of DFPV in experiments in Section 4.
2 Preliminaries
In this section, we introduce the proxy causal learning (PCL) problem and describe the existing two-stage regression methods to solve PCL.
Notation. Throughout the paper, a capital letter (e.g. A) denotes random variable, and we denote the set where random variable takes values by the corresponding calligraphic letter (e.g. A). The symbol P (-) denotes the probability distribution of a random variable (e.g. IP (A)). We use a lowercase letter to denote a realization of a random variable (e.g. a). We denote the expectation over a random variable as B(.) and [l//) as the L--norm of function ʃ with respect to P (-); i.e. |l = v (A))]
2.1 Problem Setting for Proxy Causal Learning
2

---

We begin with description of the PCL setting. We observe a treatment A E A, where C R and the corresponding outcome V, where )' C R. We assume that there exists an unobserved confounder U E l that affects both A and Y. The goal of PCL. is to estimate the structural function fstruct( (X) defined as
W
Sstruct (a) = Ev Ey [ = a. U]]
which we assume be continuous. This function is also known as the Average Treatment Effect (ATE). The challenge of estimating fstruct Figure 1 Causal Graph. that the confounder observations A and Y alone. To deal with this, we assume access to a treatment-inducinn proxy variable Z, and an outcome-inducing proxy variable W, which satisfy the following structural assumption and completeness assumption.
is not observable
we cannot estimate the structural function from
Assumption 1 (Structural Assumption [3, 18)). We assume Y LLLZIA,U, and W [L(A
Assumption (Completeness Assumption on Confounder [3, 18]). Let : + R be any square integrable function [|I (ee()) oc. The following conditions hold:
E [(U)) a, W w| 0 V(a E WAW
+> I(u) (UU)
E [((U) A = a, Z = z] = - v(a,z) Ax
+ ((u) =0 P (UU) -a.e.
Figure shows the causal graph describing these relationships. In our setting, we assume that there no observable confounder, though this may be easily included [ [18, 31] presented in Appendix D. Here, the bidirectional arrows mean that we allow both directions or even a common ancestor variable. Given these assumptions, it is shown that the structural function can be expressed using solution of an integral equation.
Proposition (Miao al. [20]. Let Assumptions 1, 2 and Assumptions 5, in Appendix hold. Then there exists at least one solution to the functional equation
B B = a, Z = ==1= f hiaa, pw (w) a, 2 = =z)dw,
which holds for any (a, 2) E A Z. Here, denote Pw (W]A a, Z 2) as the density function of the conditional probability P (WIIA= = a 2). Let he be solution of (1). The structural function Sstruct is given as
fatruct (a) = Ew [h" W)].
(2)
For completeness, we present a proof in Appendix (Lemma and Corollary 1), which is due to Miao et al. [21] and Deaner [3]. Following Miao et al. [21], call ht the bridge function. From Proposition 1, we can see that the estimation of the structural function reduces to the estimation of the bridge function, since from (2).
obtain the bridge function, the structural function directly follows
2.2 Two-stage Regression Scheme for Proximal Causal Learning with Fixed Features
To obtain the bridge function, we need to solve the functional equation (1), However, directly solving (1) in a rich function space can be ill-posed (see discussion in Nashed and Wahba [25]). To address this, recent works [3, 18] minimize the following regularized loss Lpv to obtain an estimate of the bridge function h:
h arg min (h), (ry(h) Ev.A, [(Y EEw A.2 [4(A, W)D) 2(h),,
(3)
kEM,,
where Hr is an arbitrary space of continuous functions and 22h)) is regularizer on h. Note that this loss can be interpreted as the deviation of the r.h.s and the l.h.s in (1) measured L norm with respect to the distribution I (Y, Z)
Deaner [3] and Mastouri and Zhu al. [18]solve the minimization problem (3) by successively solving two-stage regression problems. They model the bridge function as
h(a, w) u (a) @ ww (w))
(4)
3

---

where is learnable weight vector, (a),),, 2ow are vectors of fixed basis functions, and & a Kronecker product, defined as a @ = vec(ab" T for any finite dimensional vectors a, b. An estimate is obtained by solving the successive regression problems. In Stage I, we estimate the conditional expectation Ew 2= (w//::/:wwwww/// [W]] as a function of a, a. In Stage 2, we substitute the model (4) into the inner conditional expectation in LpV.
B [h(a, W)] u (a) @ EWIA=o,Z=: ( (W))),
and then minimize Lpv with respect to u using the estimate of Ew|A=o, w (W)) from Stage 1.
The above idea can be implemented as follows. We model the conditional expectation as
E 1== (bb (a) (z(z)),
where (a), (z) are another set of basis functions, and V is matrix be learned. Note that we can use different basis functions for (a and (a). In Stage 1, the matrix V is learned by minimizing the following loss,
L (VV = Ew,A, [| (W) (A) @ +
(5)
where A > is a regularization parameter. This a linear ridge regression problem with multiple targets, which can be solved analytically. In Stage 2, given the minimizer V can obtain by minimizing the loss
arg miny Li (V),, we
(z( Zy, ||X - u (AA(A) (V( oz (z A211u11?
(6)
where X2 > 0 is another regularization parameter. Stage corresponds to another linear ridge re- gression from input (A) @ (V( @ (z())) to target Y, and also enjoys closed-form solution. Given the learned weights arg min, (2))). the estimated structural function is fstreer(a) = á (a) 8 Ew (W))).
When fixed feature dictionaries are used, this two-stage regression benefits from strong theoretical guarantees [3, 18]. The use of pre-specified feature maps, limits the scope and flexibility of the method, however, especially if the treatment and proxies are high dimensional (e.g. images or text), and the causal relations are nonlinear. To overcome this, we propose to use adaptive features, expressed by neural nets, as described in the next section.
Deep Feature Proxy Causal Learning
In this section, we develop the DFPV algorithm, which learns adaptive features modeled by neural nets using a technique similar to Xu et al. [35]. As in Mastouri and Zhu al. [18], assume that we do not necessarily have access to observations from the joint distribution of (A, Z, W). Instead, we are given m observations of (A, Z. W') for Stage and n observations of (A, Y) for Stage 2. We denote the stage 1 observations by (a,, 21, w)) and the stage 2 observations by (ā,, 2, 3:). If observations of (A,Y; Y, W) are given for both stages, we can evaluate the out-of-sample loss of Stage 1 using Stage 2 data and vice versa, and these losses can be used for hyper-paramed tuning of \1, A2 (Appendix A). We first introduce two-stage regression with adaptive feature maps and then describe the detailed learning procedure of DFPV.
3.1 Two-stage regression with adaptive features
In DFPV, consider the following models of the bridge function h(a, w) and conditional feature mean EWIA. (zoo (W)):
where and V parameters, and Φ Φo a tha neural nets parametrized by h(a, w) u." (abo (a) & show (w)), EwjA=a,Z== l (W)
(a) @ Φo (=)),
A(()), Oz, A(2), Ow. respectively. Again, we may use different neural nets in the treatment fea- tures Φ and 4
Throughout this paper. we assume the number of basis functions to be finite. Mastouri and Zhu et al. [18] consider an infinite number of basis function in a reproducing kernel Hilbert space, and use the definitions of the inner and Kronecker products for that space.

---

As in the existing work [3, 18), we learn Ewja,> (vew (w)] in Stage I and h(a, w) in Stage 2, but in addition to the weights u and V, we also learn the parameters of the feature maps. Specifically, in Stage I, we learn V and parameters 0 ACL)) Oz by minimizing the following empirical loss:
(V (A(I) -2)=
Σ || Now (wi) - V (Φe (a:) & Φo (2i) + (2 IVI?
Note that Ê1 an empirical estimate of L1 in (5) with adaptive feature maps. Although &1 depends on Ow. at this stage, we do not optimize Ow with L1 as spow (w) is the "target variable" in Stage 1. Given the minimizers (V, AACI), 0z) arg min L1, we learn weights u and parameters Ow, A(()
by minimizing the empirical stage 2 loss,
E2(u, Ow, AA22)
1 = n2 Σ
J u (Noscn (a:) V (0BACO) (ā.) @ dēz (z))))
2 + Ao A2/|ua|?
Again, the loss Lg an empirical estimate of C2 in (6) with adaptive feature maps. Although the expression of L2 does not explicitly contain Ow, implicitly depends on Ow as (A)) 0z) the solution of a minimization problem involving 6w. This implicit dependency makes it challenging to update Ow. as we cannot directly obtain its gradient. One possible solution is to use the implicit gradient method [16], but this approach might be too computationally expensive. Instead, we use the method proposed Xu et al. [35], in which we ignore the dependency of Ow on parameters (A(1): (z. and compute the gradient via the closed-form solution of V.
3.2 Deep Feature Proxy Variable Method
We now describe the learning procedure for DFPV. First, we fix parameters in the adaptive feature maps (9 (()) 02 8 A(2). ((2),0w) Given these parameters, optimal weights V, â can be learned by mini- mizing the empirical stage loss Ê1 and empirical stage 2 loss L2, respectively. These minimizations can be solved analytically, where the solutions are
( 1 o (0) 11 + +m
a(0) = (#2 t2 + nd22))
D2 y2.
where we denote 0 (0)))) 02.0 (1)) 0W) and define matrices as follows:
V w (w1) (u1),..... ... sow (wm))
Φ [1n (01, 21) )),,... (am 2m
y2 = [uI..... ......... -unl
$ = (āz, &n) )....., U2 (ān 2n, 2T
vn (a, =) = A (a) 3 Φ0oz (z);
02 (a. =) = a (a) s (V()) (oo 3 d0z (3))
Given these weights i (0), we can update feature parameters by gradient descent method with respect to the residuals of the loss of each stage, while regrading V and à as functions of parameters. Specifically, we take the gradient of the losses
EDFPV (0)
m Σ>
1
Now (w:) V(0) (Do (ai) doz (1 (1)11(
ADFPV (0) L2
jr- i(0)T (Deace) (6:)) & (0) (De (ā,) & 20 (2)))) +a)l(ū())13
where V(0), (()) are given in (7). Given these losses, (0 0z) are minimized with respect to C CDPPV (0), and (PA(2), Ow) are minimized with respect to L2 (DFPV (0). Finally, we take the empirical mean of Lgggs based on additional output-proxy data Sw (wfxtta Je=1 which is used for estimating the structural function. Here, we assume access to Sw for proving consistency results, but empirically, stage data compute this mean.
The complete procedure is presented Algorithm 1. Note that we may use any sophisticated bradient-based learning method to optimize, such as Adam [10]. As reported in Xu et al. [35], we observe that the learning procedure is stabilized by running several gradient descent steps on the stage 1 parameters AC1))), Oz) before updating the stage 2 features (Q 0w). Furthermore,
5

---

may use mini- batch updates, which sample subsets of the data at the beginning of each iteration and only use these subsamples update the parameters.
Algorithm 1 Deep Feature Proxy Causal Learning
Input: Stage I data S = {a;, Zi, w;), Stage 2 data S2 (ā Di), Additional outcome- proxy data Sw = {wfxtra), ). Regularization parameters (A1. Initial values Q (0(0) A(1): /
A(2)) /// ( 2 Learning rate a.
Output: Estimated structural function fetruct (a)
1: f
2: repeat
3: Compute V(O("), ((()) in (7)
4: Update parameters in features g(++1) t (f(t+))
.00 glt+1) ((t+t)
A(2) @
((++1) as follows
A(1)
(((+1) + ((1) aV
A(1)
A(1) (
21 CPFPV(00)e=00r (0))e=001)
1(++1
0
Voz 02 1
DFPV (0)le=00))
((++1)
g(1) A(2) Von
f2
CDFPV (OPPP(()))o=2)) (0))
((++1) 8
V Vow L2
DFPV (0) 0=
A(2)
5: Increment counter t + 1+1;
6: until convergence
7:
Compute ((())) from (7)
8: Compute mean feature for W using stage 1 dataset: pow < 1 Σ &ogz (wfxtra)
9: return Satruct (a) (a)+)
Sgges A(2)
(a)
How
3.3 Consistency of DFPV
In this section, show that the DFPV method yields consistent estimate of the bridge function. Our approach differs from the consistency result of Deaner [3] and Mastouri and Zhu et al. [18], which depend the assumption that the true bridge function lies in a specific functional space (such as an RKHS [18J) or satisfies certain smoothness properties [3]. Since our features are adaptive, however, we instead build our result on Rademacher complexity argument [24], which holds for wide variety of hypothesis spaces. Furthermore. we do not assume that our hypothesis space contains the true bridge function, which makes this theory of independent interest.
To derive the excess risk bound, we need another completeness assumption.
Assumption (Completeness Assumption on Outcome-Induee Proxy [3, 18). Let : W - E be any square integrable function [/4]l(w>> < oo, We assume the following condition:
E [[WW) A = Z = 2] = v(a,z) E Ax
I(w) 0 IP (W)-a.c.
This assumption is not necessary for identification (Proposition 1), but we need for connecting the two-stage loss and the deviation of the bridge function. Given this assumption, we have the following consistency result.
Let Assumption 3 and Assumptions 7 and 8 in Appendix C hold. Given stage I data 5 = [wi, as, 24)= stage data S₂ {Ti, āi, ) and additional output-proxy data Sw [w.xtra] then for the minimizer of L2 denoted as (á, (a,0 0z), ) we have
Proposition 2.
||Sstruct SetrucellP(A) SOpp (Vi= Rs. (HL)
1
+22+ Vm
Rs, (H2)
Vn
Vnw
where Hi, H2 are the functional classes that are considered in each stage, comprising both the predictions and their associated losses, which are defined in (12) and (15), respectively; and Rs (H1), (HH(H) are their respective empirical Rademacher complexities; the quantities K1, K2 measure the misspecification in each regression, and are defined in (13) and (16), respectively.
We formally restate Proposition in Appendix (Theorem 3), in which the definitions of hypothesis spaces and constants can be found. From this, we can see that if correctly identify the hypoth-

---

esis space KK 0) and have vanishing Rademacher complexities (Rs, (H1), (H2)) - 0 in probability), then the estimated bridge function converges in probability to the true one. Note that the Rademacher complexity of certain neural networks is known; e.g.. ReLU two-layer networks are known to have the complexity of order Ol (/v1/n) [26]. However, it might be not straightforward to derive the Rademacher complexities of the specific classes of neural networks we use, because we are employing the outer product of two neural networks in (5) and (6), whose Rademacher complexity remains a challenging open problem. The analysis in Proposition is similar to Xu et al. [35], with two important distinctions: first, the target fstruct is estimated using the bridge function h marginalized over one of its arguments W, but PCL learns the bridge function itself. Second, our result is more general and includes the misspecified case, which was not treated in the earlier work.
One limitation of our analysis is that we leave aside questions of optimization. As we previously discussed, &2 does not explicitly include Ow. thus is difficult to develop a consistency result that includes the optimization. We emphasize that Algorithm does not guarantee to converge to the global minimizer, since we ignore the implicit dependency of Ow on 0 A1)) 0z when calculating the gradient. As will see in the Section 4, however, the proposed method outperforms competing methods with fixed feature dictionaries, which do not have this issue.
3.4 DFPV for Policy Evaluation
PCL methods can estimate not only the structural function, but also other causal quantities, such as Conditional Average Treatment Effect (CATE) [29] and Average Treatment Effect on Treated (ATT) [3, 29]. Beyond the causal context, [32] uses a PCL method to conduct the off-policy evaluation in a partially observable Markov decision process with discrete action-state spaces. Here, we present the application of PCL methods to bandit off off-policy evaluation with continuous actions, and propose a method to leverage adaptive feature maps in this context.
In bandit off-policy evaluation, we regard the outcome Y as the reward and aim to evaluate the value function of given policy. Denote policy π C - AA and define the value function v(T) as
u(T) Eu,c E Y = (C),U1]
where C' C is a variable on which the policy depends. In the flight ticket sales prediction example, a company might be interested in predicting the effect of discount offers. This requires us to solve the policy evaluation with C A since a new price is determined based on the current price. Alternatively, if the company is planning to introduce new price policy that depends on the fuel cost, we need to consider the policy evaluation with C' Z, since the fuel cost can be regarded as the treatment-inducing proxy variable. Here, we assume the policy π to be deterministic, but this can be easily generalized to stochastic policy. Although the value function contains the expectation over the latent confounder U, it can be estimated through bridge function.
Proposition 3. Assume that the true bridge function (a, w) : A W + is jointly measurable. Suppose C IT WIU. Under Assumptions 1, 2 and Assumptions 4, 5, 6 in Appendix B, have
u(T) = E..w (=(C), W)).
The proof can be found in Appendix B (Corollary 2). From Proposition 3. we obtain the value function by taking the empirical average over the bridge function (T((C), , W). In DFPV, we estimate the bridge function as h(a, w) (ū) (4b (a) @ Vi (w)) where ú, i A(2) Aw minimize
Eg. Hence, given n' observation of (C, W) denoted {č,, th Yr' can estimate the value function v(n as
(
Σ â nC i=1
1
(Heac (m(()) & bo (wi)))
We derive the following consistency result for policy value evaluation in a bandit with confounding bias.
Proposition 4. Let Assumption 3 and Assumptions 7, 8, 9 in Appendix C hold. Denote the
minimizers of Ce as (0, (0,0 0z). Given stage I data S₁ [ww,, ag, 24) Jm stage 2 data
7

---

S2 = (()). 2,) and data for policy evaluation S3 = {(,,, wi))
we have
(u(T) ((T)) ( (Hz)+
1
1
+ xiz Rsg (H2) Vn
Vm
Vnl
where H2 are the function classes that are considered in each stage, comprising both the predictions and their associated losses, which are defined in (12) and (15), respectively; and their empirical Rademacher complexities are given by Rs, (H1), (H2): the quantities K1,K2 measure the misspecification in each regression, which are defined in (13) and (16), respectively.
The formal statement of Proposition 4 can be found in Appendix (Theorem 3). As in the structural function case, we can that estimated value function converges in probability to the true one, provided that we have vanishing Rademacher complexities and there is no misspecification.
4 Experiments
In this section, we report the empirical performance of the DFPV method. First, we present the results of estimating structural functions; we design two experimental settings for low-dimensional treatments and high-dimensional treatments, respectively. Then, we show the result of applying PCL methods to the bandit off- off-polic evaluation problem with confounding. We include the results for problems considered in prior work in Appendix E. The experiments implemented using Py Torch [27]. The code is included in the supplemental material. All experiments can be run in a few minutes on Intel(R) Xeo CPU B5-2698 v4 2.20GHz.
Experiments for Structural Function We present two structural function estimation experiments. One is a demand design experiment based on a synthetic dataset introduced by Hartford et al. [6], which is a standard benchmark for the instrumental variable regression. Here, we modify the data generating process to provide a benchmark for PCL methods. We consider the problem of predicting sales Y from ticket price P, where these are confounded by a potential demand D [0, 10). To correct this confounding bias, we use the fuel price (C1, C2) as the reatment-inducinn proxy, which has an impact price P, and the number of views of the ticket reservation page V as the outcome-ine outcome-inducing proxy. Details of the data generation process can be found in Appendix F.1.
Our second structural function estimation experiment considers high-dimensional treatment variables. We test this using the dSprite dataset [19]. which is an image dataset described by five latent parame- ters (shape, scale, rotation, posX and posY). The images are 64 64 -09e-nimmnii Based on this, Xu et al. [35] introduced the causal experiment, where the treatment each figure, and the confounder is posY. Inspired by this, we consider the PCL setting that learns the same structural functions with nonlinear confounding, which is not possible to handle in the instrumental variable setting. Specifically, the structural function fstruct and outcome Y defined as
fstruct(a) (a) || Ba - 5000
1 1
Y
12 (posy-O.5) 0.5rue (A) +E, E~ (00,05)..)
1000
where each element of the matrix B R 10x was generated from Unif 1.0) and fixed throughout the experiment. We fixed the shape parameter to heart and used other parameters as the ceatment-inducinn proxy Z. We sampled another image that shared the same posY as treatment A, which is used as output-inducing proxy W. Details of data generation process can be found in Appendix F.2.
We compare the DFPV method to three competing methods, namely KPV [18], PMMR [18], and an autoencoder approach derived from the CEVAE method [17]. In KPV, the bridge function is estimated through the two- stage regression as described in Section 2. where feature functions are fixed via their kernel functions. PMMR also models the bridge function using kernel functions, but parameters are learned by moment matching. CEVAE is not a PCL method, however, it represents in correcting for hidden confounders using observed proxies. The causal graph for CEVAE is shown in Figure 2, and CEVAE uses VAE [11] to recover the distribution of confounder U from the "proxy" Q. We make two modifications to CEVAE to apply it in our setting. First, we include
ofate-of-the-arre approach
Figure Graph in CEVAE
Causal
8

---

Demand
dSprite
Cost Policy
Price Policy
3000
10
Out-of-Sample MSE
1000
20
Out-of-Sample MSE
KPV
Algorithm
300
PMMR
RPV
PMMR
10
CEVAE
100
DFPV
DFPV
0.1
30
5
1000
5000
1000
5000
1500
7500
1500
7500
Data Size
Data Size
Figure 3: Result of structural function experiment in demand design setting (Left) and dSprite set- ting (Right).
Figure 4: Result of OPE experiment when the policy depends on the costs (Left) and on the current price (Right).
both the treatment- induuiingiiiiiggg proxy and output-inducing proxy W as Q in CEVAE (we emphasize that this does not follow the causal graph in Figure 2, since there exist arrows from Q to A, Y). Second, CEVAE originally used in the setting where Q is conditioned on particular value, whereas we marginalize Q. See Appendix F.4 for the choice of the network structure and syper-parameters, We tuned the regularizers λ1 X2 as discussed in Appendix A, with the data evenly split for Stage 1 and Stage 2. We varied the dataset size and ran 20 simulations for each setting. Results are summarized in Figure 3.
In both experiments, DFPV consistently outperforms existing methods. This suggests that DFPV is capable learning complex structural functions by taking the advantage of the flexibility of neural networks. KPV and PMMR perform similarly in all settings, but KPV tends to perform slightly better when the data size is large. This might be because KPV expresses its solution using a larger number of parameters than PMMR. Although CEVAE also learns flexible model with neural network, it's highly unstable the demand design experiment and underperforms kernel methods in the dSprite experiment. This is because CEVAE does not take advantage of the relations between proxy variables and the structural function.
Experiments for Offline Policy Evaluation We now describe the offline policy evaluation experi- ment based on the demand design data. We set up synthetic experiments to evaluate two types of policy. In the first case, a policy depends on costs C' (C1, C2). address the question How much would we gain if we decide the price based on the fuel costs. In the second case, the policy depends on current action C' P. which addresses How much would we gain if we cut the price by 30 percent. The detailed policies can be found in Appendix F.3.
We only compare PCL methods here, it is not straightforward to apply CEVAE to the off-policy evaluation problem. We evenly split the data for Stage 1, Stage 2, and policy evaluation (i.e we set m We 20 simulations for each setting. Results are summarized in Figure 4, in which DFPV performs better than existing PCL methods. This is not surprising since, as demonstrated in the structural function experiments, DFPV estimate complex bridge functions, which results in more accurate estimation of the value function.
5 Conclusion
We have proposed novel approach for proxy causal learning, the Deep Feature Proxy Variable (DFPV) method, which performs two-stage least squares regression on flexible and expressive features. Motivated by the literature on the instrumental variable problem, we showed how to learn these feature maps adaptively with deep neural networks. also showed that PCL learning can be used for off-policy evaluation in the bandit setting with confounding, and that DFTV performs competitively in this domain. This work thus brings together research from the worlds of deep offline RL and causality from observational data.

---

In future work, it would be interesting to explore different approaches to learning deep models in the PCL problem. One direction would be adapt the method of moment matching, which has been studied extensively in the instrumental variable literature [1]. Moreover, Liao et al. [15] recently developed a novel adversarial method to solve a class of functional equations, which would be also promising approach in the PCL problem. In the RI. context, Tennenholtz et al. [32] shows an interesting connection between PCL learning and off-policy evaluation partially observable Markov decision process, and we believe that adapting DFPV to this setting will be of value.
References
[1] A. Bennett, N. Kallus, and T. Schnabel. Deep generalized method of moments for instrumental variable analysis. In Advances in Neural Information Processing Systems, volume 32, 2019.
[2] R. Blundell, J. Horowitz, and M. Parey. Measuring the price responsiveness of gasoline demand: Economic shape restrictions and nonparametric demand estimation. Quantitative Economics, 3: 29-51, 2012.
[3] B. Deaner. Proxy controls and panel data, 2018.
[4] J. Fruehwirth, S. Navarro, and Y. Takahashi. How the timing of grade retention affects outcomes: Identification and estimation of time-varying treatment effects. Journal of Labor Economics, 34 (4):979 1021, 2016.
[5] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, 2014.
[6] Hartford, G. Lewis, K. Leyton-Brown, and M. Taddy. Deep IV: flexible approach for counterfactual prediction. In Proceedings of The 34th International Conference Machine Learning, 2017.
[7] J. Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20:217-240, 2011.
[8] G. W. Imbens. Nonparametric estimation of average treatment effects under exogeneity: review. Working Paper 294, National Bureau of Economic Research, October 2003.
[9] F. Johansson. U. Shalit, and D. Sontag. Learning representations for counterfactual inference. In Proceedings of The 33rd International Conference on Machine Learning, 2016.
[10] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[11] D. Kingma and M. Welling. Auto-encoding variational Bayes. International Conference on Learning Representations, 2014.
[12] R. Kress. Linear Integral Equations. Applied Mathematical Sciences. Springer New York, 1999.
[13] M. Kuroki and J. Pearl. Measurement bias and effect restoration in causal inference. Biometrika, 101(2):423 437, 2014.
[14] C. Lee, N. Mastronarde, and M. van der Schaar. Estimation of individual treatment effect in latent confounder models via adversarial learning. In Advances in Neural Information Processing Systems, volume 32, 2018.
[15] L. Liao, Y.-.. Chen, Z. Yang, B. Dai, M. Kolar, and Z. Wang. Provably efficient neural estimation of structural equation models: An adversarial approach. In Advances in Neural Information Processing Systems, volume 33, 2020.
[16] J. Lorraine, P. Vicol, and D. Duvenaud, Optimizing millions of hyperparameters by implicit differentiation. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, 2020.
10

---

[17] C. Louizos, U. Shalit, J. M. Mooij, D. Sontag, R. S. Zemel, and M. Welling. Causal effect inference with deep latinl-variable models. In Advances in Neural Information Processing Systems, volume 31, 2017.
[18] A. Mastouri, Y. Zhu, L. Gultchin, A. Korba, R. Silva, M. Kusner, A. Gretton, and K. Muandet. Proximal causal learning with kernels: Two-stage estimation and moment restriction. In Proceedings of The 38th International Conference on Machine Learning. 2021.
[19] L. Matthey, L. Higgins D. Hassabis, and A. Lerchner. dSprites: Disentanglement testing sprites dataset, 2017. URL https: //t:::////tttt com/ http//deeemnnnndeeeeeeee seteiiinddddpriiteee seepppann//pppiiessddtaeaee//
[20] W. Miao, Geng, and E. Tchetgen Tchetgen. Identifying causal effects with proxy variables of unmeasured confounder. Biometrika, 105(4))887-993, 2018.
[21] W. Miao, X. Shi, and E. T. Tchetgen. confounding bridge approach for double negative control inference on causal effects, 2018.
[22] W. Miao, W. Hu, E. L. Ogburn, and X. Zhou. Identifying effects of multiple treatments in the presence of unmeasured confounding, 2020.
[23] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
[24] M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of Machine Learning. MIT Press, 2012.
[25] M. 7. Nashed and G. Wahha. Generalized inverses reproducing kernel spaces: An approach regularization of linear operator equations. SIAM Journal Mathematical Analysis, 5(6): 974-987, 1974.
[26] B. Neyshabur, Z. Li. S. Bhojanapalli, Y. LeCun, and N. Srebro. The role of over-paramettrization in generalization of neural networks. International Conference on Learning Representations, 2019. URL https: //tp:rev///peeevieee net hodd/oorumiiddBygggha
[27] A. Paszke, S. Gross, F. Massa, Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. Vetito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32, 2019.
[28] P. R. Rosenbaum and D. B. Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55 1983.
[29] R. Singh. Kernel methods for unobserved confounding: Negative controls, proxies, and instruments, 2020.
[30] R. Singh, M. Sahani, and A. Gretton. Kernel instrumental variable regression. Advances in Neural Information Processing Systems, volume 32, 2019.
[31] E. J. T. Tchetgen, A. Ying, Y. Cui, X. Shi, and W. Miao. An introduction proximal causal learning, 2020.
[32] G. Tennenholtz, U. Shalit, and S. Mannor. Off-policy evaluation in partially observable environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10276-10283, 2020.
[33] Y. Wang and D. Blei. A proxy variable view of shared confounding. M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10697-10707. PMIR, 18-24 Jul 2021. URL https: /pppppeeeingg.../////// mlr. press/ vvtss//vv/v////
v139/wang21111 wang2ic html.
[34] Wright. The Tariff on Animal and Vegetable Oils. Investigations in International Commercial Policies. Macmillan Company, 1928.

---

[35] L. Xu, Y. Chen, S. Srinivasan, N. de Freitas, A. Doucet, and A. Gretton. Learning deep features in instrumental variable regression. In International Conference on Learning Representations. 2021.
[36] L. Yao, S. Li, Y. Li, M. Huai, J. Gao, and A. Zhang. Representation learning for treatment effect estimation from observational data, Advances in Neural Information Processing Systems, volume 31, 2018.
12


Reviews:
[{"review_id": "cRdSlaY0dCu", "reviewer": "Reviewer_q48d", "summary": "The paper proposes an algorithm that operationalize the proxy variable\nstrategy for causal identification. It builds on existing approaches\n(Deaner [3] and Mastouri and Zhu et al. [18]) and additionally\nconsider adaptive features as in Deep Feature Instruemtal Variable\nmethod [32]. The key idea is to consider features parameterized by\nneural nets, and alternate between learning $V$ by minimizing the\nempirical loss and learning the parameters of the feature maps. The\npaper shows that the additional flexibility of the features helps the\nproxy variable method achieve better out of sample performance and\nsmaller errors in empirical studies.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes an algorithm that operationalize the proxy variable\nstrategy for causal identification. It builds on existing approaches\n(Deaner [3] and Mastouri and Zhu et al. [18]) and additionally\nconsider adaptive features as in Deep Feature Instruemtal Variable\nmethod [32]. The key idea is to consider features parameterized by\nneural nets, and alternate between learning $V$ by minimizing the\nempirical loss and learning the parameters of the feature maps. The\npaper shows that the additional flexibility of the features helps the\nproxy variable method achieve better out of sample performance and\nsmaller errors in empirical studies.\n\nBoth the theoretical and empirical results of the paper make sense.\nThe methods and the results of the paper build closely on the two\nstage formulation of Deaner [3] and Mastouri and Zhu et al. [18] and\nthe adaptive feature idea in Deep Feature Instruemtal Variable method\n[32]. I have a few major questions.\n\nThe paper claim that Assumption 1 and 2 (the completeness assumption\non the unobserved confounder) are sufficient for causal\nidentification. In particular, the paper says that: Assumption 3,\nwhich was needed in the original proxy variable paper (Miao et al\n2018), is not necessary. Can the authors explain the intuition here?\nWhy is it not necesary here? The proof of Miao et al 2018 does rely on\nthis assumption. Moreover, do these completeness conditions in\nAssumptions 1 and 2 have observable implications? Do they imply the\nconditional of the proxy given treatment is complete in any ways? Or\ndo they have provably *no* observable implications? These observables\nimplications are important for causal estimation in principle.\n\nFurther, the completeness assumptions  plays a key role for\nidentification in the proxy variable strategy. They are not regularity\nconditions. Rather, the completeness conditions are quite strong; they\nare used to guarantee the solution to the integral equation is unique.\n(Solutions to integral equations are not unique in general.) Despite\nthe importance of these identification assumptions, nowhere in the\nalgorithm are these completeness conditions enforced. As above, even\nthough the completeness assumption of Assumptions 1 and 2 are about\nunobserved confounders, they may have observable implications and\nshall be enforced in the proposed algorithm. (Or we should prove that\nthere exist *no* observable implications.) Without enforcing these\nconditions, the causal estimation can in general by biased.\n\nFinally, the paper could benefit from a discussion on the connections\nto other recent work on proxy variables, which also discussed\npractical estimation approaches to proxy variables that may be\ncompatible with the adaptive feature idea:\n\nMiao, W., Hu, W., Ogburn, E. L., & Zhou, X. (2020). Identifying effects of multiple treatments in the presence of unmeasured confounding. arXiv preprint arXiv:2011.04504.\n\n\nWang, Y., & Blei, D. (2021, July). A Proxy Variable View of Shared Confounding. In International Conference on Machine Learning (pp. 10697-10707). PMLR.\n\nMiao, W., & Tchetgen Tchetgen, E. (2018). A confounding bridge approach for double negative control inference on causal effects (supplement and sample codes are included). arXiv preprint arXiv:1808.04945.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper could benefit from a more elaborate discussion on its limitations. Is the use of adaptive features free lunch?", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "LcoWuPkElhz", "reviewer": "Reviewer_Rp98", "summary": "This work extends known Proxy Causal Learning methods that aim at learning the causal effect of treatment in cases where unobserved confounders are assumed yet there is observable information in form of proxy variables to these confounders.\nIn particular, authors extend the theory to allow deep neural nets to be used as building blocks of the two stage regression proposed by Mastouri et al instead of RKHS. The paper also explains how to use the method to perform off-policy evaluation in a bandit setting where arms are confounded. Synthetic experiments are also reported to compare the performance with a few baselines of the same family.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n- the two main original contributions seem to be the theory that enables deepnets in place of RKHS and the explanation of how to do OPE with the method \n\nQuality & Clarity:\n- I appreciate the summary of related works which makes it easy to refresh one's memory before diving in the main development\n- The notation and mathematical developments are relatively easy to follow thanks to a nice introduction of details and assumptions when they are needed\n- I appreciate also that the corresponding proofs in appendix are detailed enough to follow and that required a dedicated effort from the authors\n- I'm not completely convinced by some of the arguments, especially on why we need deepnets (see below)\n- I'm not convinced that the experimental setup is providing evidence for that either (see below)\n- there are a few typos (l242 Propositionn) but it doesn't affect reading\n\nSignificance:\n- I understand in principle that the work enables application of PCL to more complex settings, even though I think the authors don't make an especially good argument on why it is required (the running argument in my head being that it might be the next natural thing to do in the field if deepnets were not tried before)\n- the OPE reduction is usable for all PCL methods and thus might be interesting for this community which means for a large number of potential application domains where off-policy is interesting or the only option possible\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would be keen to read a developed argument about some of the claims in the discussion phase. I believe it could firm up my mind on the significance of the proposed method.\n\n- l32: authors indicate that ref 7/9/33 (BART, CFRNet, ...) have too restrictive assumptions; in what sense exactly ? if they are too restrictive can you give a small example explaining the limitations ?\n- l42: authors indicate that ref 14/17 provide \"little theory\" to guarantee recovery of causal effect - could you elaborate and point out what is missing in your opinion and how your work differs ?\n- l60: authors seem to imply that deepnets can learn more complex functions than RKHS - is that right ? if so in what sense ? are the cases where it makes a difference significant from an application perspective ?\n- l101: the proposed setup assumes the existence of both outcome- and treatment-inducing proxies - isn't that a bit restrictive ? esp. compared to latent variables methods like the Deconfounder or ref 14/17 ?\n- l207: did you try to measure such Rademacher complexities to compute this bound in practice (and verify its empirical tightness) ?\n- in Sec 4, it seems to me the proposed setups are maybe more complex than it should. Esp. the Demand experiment equations seem a bit random and not very realistic (just try to imagine the shape of the curve from the equations in your head ?). At this point I can't make up my mind if the setup was designed to be so arbitrary that deepnets would be the only viable modeling option or if it resembles anything realistic.\n- in the same idea I'd like to see what the method does when the setup is simpler. I believe it is part of the authors duty to study the limits of their method and the regimes where it is recommended to use it or not.\n- the choice of CEVAE as the only baseline not from the CPL family is not very well motivated and I wonder of other methods could have been tried as well ?\n- Ideally, I would like to have pointers on the performance of these baselines in the same setup in different papers to make sure the baseline methods perform as expected (i.e. why another benchmark ?)", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "JNp7S9SFFLS", "reviewer": "Reviewer_fc4M", "summary": "This paper presents a novel method: deep feature proxy variable. It applies a neural network to PCL, allowing non-linear relationships between variables. The authors prove the consistency of the method and demonstrate how this method can be applied to off-policy evaluation. The authors test the methods using two simulation studies, showing that DFPV outperforms existing methods. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I enjoyed reading this paper. I believe some analysis on the usability of the method could make the paper more impactful. I would a consider increasing my score if the authors address the concerns.\n\npros\n- The paper is clear and well written. It’s clear in its notation and setup. It also clearly stated the limitations of the theoretical contribution.\n- The paper is well researched and motivated. It provides a crisp overview of proximal causal learning. It articulates the challenges in PCL and provides a reasonable solution.\n- The experiments are reasonably designed. The authors provide detailed records of the experiment and source code. \n\nArea of improvements\n- The captions in figure 3 and 4 are not clear. Why is DFPV so high variance?\n-  I would be interested in knowing the performance of DFPV on a relatively simple data generating process. What sort of behaviour would DFPV exhibit in that case? \n- I would like an analysis about when we should use DFPV in contrast to the parametric methods.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed the limitation of the theoretical performance, showing that the method works well in practice. I would like to see an non-trivial failure case of the method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "DaQ-890uyzP", "reviewer": "Reviewer_bAic", "summary": "The work proposes a causal effect estimation method in presence of unobserved confounding, building on proximal causal learning. It starts from an existing two-stage regression approach for tractably solving the proximal causal learning problem and extends it to handle high-dimensional and non-linearly related data using neural networks to learn adaptive features which are used as input to the regression approach.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work lifts the limitation of using linear or fixed kernel-based function classes in proximal causal learning, showing how to use neural networks which provide a more expressible and data-adaptive function class. The past approach based on two-stage regression is described in great detail, and the proposed modifications are clear. While the changing the method from static kernel-based features to adaptive neural network-based features is conceptually close to the approach of [32,18], the work demonstrates the significant technical effort required to make the change. The estimation method is shown to be consistent given assumptions on estimation of the regression functions in two stages and Rademacher complexity of the function classes used. The experiments show significant improvement from using neural networks.\n\nMy first concern is the artificial nature of the two experiments. The demand design task, while being realistic, simulates both the covariates and outcomes. Experiment on dSprite uses more realistic covariate distribution but the task itself (which is estimating the causal effect of an image on a synthetic outcome function) does not seem realistic or modelled after some real world application. I recognise that these are good benchmarks for testing the proposed methods that account for high-dimensional features. I would have appreciated demonstration of the method on a real world dataset or a synthetic one modelled after a task of practical interest and designed using realistic data distribution (in contrast to the demand design task).\n\nMy second concern is the limited motivation provided for the confounded off-policy evaluation problem which seems to be of practical value. But the description and possible applications of the problem is not very detailed. This can be improved by adding more example applications which are suitable to be modelled as confounded off-policy evaluation problem.\n\nQuestions:\n\nExperiment setup:\nHow did the authors control the degree of complexity of confounder-treatment/outcome or outcome-treatment functions and dimensions of the three? Is it the case that Demand task has a  complex outcome function while dSprite task has complex confounder- treatment/outcome functions.\n\nThe method assumes continuous treatments, line 88, whereas in many practical settings we are concerned with quantifying treatment effect between two levels of the treatment. Does it extend to binary treatments? If not, then what is the difficulty on handling binary treatments?\n\nCan you please provide example applications where the complexity of the datasets necessitate adaptive features?\n\nTheoretical results:\nThe consistency results are left in terms of the Rademacher complexity of the function classes used. However, to justify the use of neural networks through these results, could you please comment on the Rademacher complexity of possibly a restricted class of neural networks? For example, a 2 layer ReLU network https://arxiv.org/abs/1805.12076. \nAdditionally, please discuss whether the Rademacher complexities can be shown to vanish and at what rate, even in some restricted class of neural networks. \nWithout these, the theoretical results are disconnected from the proposed method.\n\nMethod:\nCan deep kernel learning (e.g. https://arxiv.org/abs/1511.02222) be a practical way of achieving adaptivity to data in the kernel-based methods [3,18]? Are there difficulties in learning such kernels for causal effect estimation tasks which in turn justify the proposed approach?\n\n---\nAuthor response addressed my concerns on experiments on realistic dataset and ideal application settings.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Consider adding limitation on only handling continuous actions.\n\nSuggestions:\n\nAverage treatment *effect* is defined typically as the contrast between potential outcomes at two treatment levels and not for a single treatment level as in line 88. Consider rewording to average potential outcome or a similar name.\n\nThe experiment setup for demand design experiment states it as a prediction task line 277. Instead, please consider posing it as a causal effect estimation task to justify the need for removing confounding bias.\n\nAssumptions 4,5,6 directly referenced in the Proposition 1 without introducing the nature of these assumptions. Please add a few words e.g. saying these are regularity conditions on conditional expectation functions or that these are not structural but functional assumptions.\nSimilarly, for Assumptions 7,8.\n\nThe approximation for computing gradient w.r.t. theta_W as mentioned in line 171 can be clarified in the update equations in Algorithm 1.\n\nLine 97, may be easily included -> can be, since Appendix D extends the proposed method, as I understand.\n\nLine 284, clarify what figure in this phrase mean “treatment is each figure”.\n\nMinor\nLine 27 confouder, line 195 frm, line 336 DFIV", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {"ethical_concerns": "I do not foresee any ethical implications of the presented applications. As a positive implication, the work enables making more robust causal inferences when unobserved confounders exist. Thus, it may be applicable to problems in social sciences where all possible confounders cannot be measured such as socio economic indicators of individuals in treatment efficacy studies."}}]

Author rebuttal:

Meta-review: The expert reviewers all appreciated the paper and agree it provides a useful new method and that the paper should be accepted. The reviewers appreciated the authors' response which better highlighted the significance of the contribution, including the relevant applications and what is lacking in previous work. The authors are expected to address the points raised by reviewers in a final version, including incorporating the additional clarifying discussion in their responses.
Decision: Accept (Poster)