Learning Deep Features in Instrumental Variable Regression
Authors: Liyuan Xu
Yutian Chen
Siddarth Srinivasan
Nando de Freitas
Arnaud Doucet
Arthur Gretton
Venue: ICLR 2021 Poster (2021)
Keywords: Causal Inference
Instrumental Variable Regression
Deep Learning
Reinforcement Learning

Abstract: Instrumental variable (IV) regression is a standard strategy for learning causal relationships between confounded treatment and outcome variables from observational data by using an instrumental variable, which affects the outcome only through the treatment. In classical IV regression, learning proceeds in two stages: stage 1 performs linear regression from the instrument to the treatment; and stage 2 performs linear regression from the treatment to the outcome, conditioned on the instrument. We propose a novel method, deep feature instrumental variable regression (DFIV), to address the case where relations between instruments, treatments, and outcomes may be nonlinear. In this case, deep neural nets are trained to define informative nonlinear features on the instruments and treatments. We propose an alternating training regime for these features to ensure good end-to-end performance when composing stages 1 and 2, thus obtaining highly flexible feature maps in a computationally efficient manner.
DFIV outperforms recent state-of-the-art methods on challenging IV benchmarks, including settings involving high dimensional image data. DFIV also exhibits competitive performance in off-policy policy evaluation for reinforcement learning, which can be understood as an IV regression task.

Published as a conference paper at ICLR 2021
LEARNING DEEP FEATURES IN INSTRUMENTAL VARIABLE REGRESSION
Liyuan Xu Gatsby Unit liyuan. 190ucl ac. uk
Yutian Chen DeepMind
Siddarth Srinivasan
University of Washington
yut.ancdgoogle..om
sidsrini&cs. washington. edu
Nando de Freitas DeepMind nardolefrriita@@gogle..oo
Arnaud Doucet
DeepMind arnauddoocett@googee.ccm
Arthur Gretton
Gatsby Unit arthur. teaurggeeetonggma..com
ABSTRACT
Instrumental variable (IV) regression is standard strategy for learning cansal relationships between confounded treatment and outcome variables from observational data by using an instrumental variable, which affects the outcome only through the treatment. In classical IV regression, learn- ing proceeds in two stages: slage 1 performs linear regression from the instrument to the treatment; and stage 2 performs linear regression from the treatment to the outcome, conditioned on the instrument. We propose a novel method, deep feature instrumental variable regression (DFIV), to address the case where relations between instruments, treatments, and outcomes may be nonlinear. In this case, deep neural nets are trained to define informative nonlinear features on the instruments and treatments. We propose alternating training regime for these features to ensure good end-to-end performance when composing stages and 2, thus obtaining highly flexible feature maps in computationally efficient manner. DFIV outperforms recent. state-of-the-art methods on challenging IV benchmarks, including settings involving high dimensional image data. DFTV also exhibits competitive performance in off- off-policy policy evaluation for reinforcement learning, which can be understood as an IV regression task.
INTRODUCTION
The aim of supervised learning is to obtain model based on samples observed from some data generating process, and to then make predictions about new samples generated from the same distribution. If our goal is predict the effect of our actions the world, however, our aim becomes to assess the influence of interventions on this data generating process. To answer such causal questions, supervised learning approach is inappropriate, since our interventions, called frealments, may allect the underlying distribution of the variable of interest, which is called the outcome.
To answer these counterfactual questions, we need to learn how treatment variables causally affect the distribution process of outcomes, which is expressed in a structural function. Learning structural function from observational data (that is, data where we observe, but not intervene) is known to be challenging if there exists an unmeasured confounder, which inffuences both treatment and outcome. To illustrate: suppose predicting sales of airplane tickets given price. During the holiday season, we would observe the simultaneous increase in sales and prices. This does not. mean that raising the price
interested in

---

Published as a conference paper at ICLR 2021
causes the sales to increase. In this context, the time of the year is confounder, since it affects both the sales and the prices, and we need to correct the bias caused by it.
One way of correcting such bias is via instrumental variable (IV) regression (Stock and Trebbi, 2003). Here, the structural function is learned using instrumental variables, which only affect the treatment directly but not the outcome. In the sales prediction scenario, we can use supply cost shifters as the instrumental variable since they only affect the price (Wright, 1928; Blundell et al., 2012). Instrumental variables can be found in many contexts, and IV regression is extensively used by economists and epidemiologists. For example, IV regression is used for measuring the effect of drug in the scenario of imperfect compliance (Angrist et al., 1996), or the influence of military service on lifetime earnings (Angrist, 1990). In this work, we propose a novel IV regression method, which can discover non-linear causal relationships using deep neural networks
Classically, IV regression is solved by the two-stage least squares (2SLS) algorithm; we learn mapping from the instrument to the treatment in the first stage, and learn the structural function in the second stage as the mapping from the conditional expectation of the treatment given the instrument (obtained from stage 1) to the oulcome. Originally, 2SLS assumes linear relationships in both stages, but this has been recently extended to non-linoe -inlinner settings.
One approach has been to non non-linear feature maps. Sieve IV performs regression using a dictionary of nonlinear basis functions, which increases in size as the number of samples increases (Newey and Powell, 2003; Blundell et al., 2007; Chen and Pouzo, 2012; Chen and Christenson, 2018). Kernel IV (KIV) (Singh et al., 2019) and Dual IV regression (Muandet et al., 2020) use different (and potentially infinite) dictionaries of basis functions from reproducing kernel Hibert spaces (RKHS). Although these methods enjoy desirable theoretical properties, the flexibility of the model is limited, since all existing work uses pre-specified features.
Another approach is to perform the stage 1 regression through conditional density estimation (Carrasco et al., 2007: Darolles et al., 2011; Hartford et al., 2017). One advantage of this approach is that it allows for flexible models, including doep neural nets, as proposed in the DecpIV algorithm of (Hartford et al., 2017). It is known, however, that conditional deusity estimation costly and often suffers from high variance when the treatment is high-dinnensional.
More recently, Bennett et al. (2019) have proposed DeepGMM, a method inspired by the optimally weighted Generalized Method of Moments (GMM) (Hansen, 1982) to find structural function ensuring that the regression residual and the instrument are independent. Although this approach can handle high-dimensional treatment variables and deep NNs as feature extractors, the learning procodure might not be as stable as the 2SLS approach, since it involves solving smooth zero-sum game, as when training Generative Adversarial Networks (Wiatrak el al., 2019).
In this paper, we propose Deep Feature Instrumental Variable Regression (DFIV), which aims to combine the advantages of all previous approaches, while avoiding their limitations. In DFIV, we use deep neural nets to adaptively learn feature maps in the 2SLS approach, which allows us to fit highly nonlinear structural functions, as in DeepGMM and DeepIV. Unlike DeepIV, DFIV does not rely on conditional density estimation. Like sieve IV and KIV, DFIV learns the conditional expectation of the feature maps in stage 1 and uses the predicted features in stage 2, but with the additional advantage of learned features. We empirically show that DFIV performs better than other methods several IV benchmarks, and apply DFTV successfully to off- off-policy policy evaluation, which is a fundamental problem in Reinforcement Learning (RL).
The paper is structured as follows. In Section 2, we formulate the IV regression problem and introduce two-stage least-squares regression. In Section 3, we give detailed description of our DFIV method. We demonstrate the empirical performance DFIV in Section 4, covering three settings: classical demand prediction example from econometrics, challenging IV set ing where the treatment consists of high-dimensional image data, and the problem of off off-policy policy evaluation in reinforcement learning.
2

---

Published as a conference paper at ICLR 2021
2 PRELIMINARIES
2.1 PROBLEM SETTING OF INSTRUMENTAL VARIABLE REGRESSION
We begin with a description of the IV setting. We observe treatment X E X. where &' C Rdx, and an outcome Y Vr where y C R. We also have an unobserved confounder that affects both X and Y. This causal relationship can be represented with the following structural causal model:
Y fstruc E, [ 0, TE [c] #0,
(1)
where fstruct is called the structural function, which we assume to be continuous, and ε is an additive noise term. This specific confounding assumption is necessary for the IV problem. In Bareinboim and Pearl (2012), is shown that we cannot learn fstruct if we allow any type of confounders. The challenge is that E [=[ # O, which reflects the existence a confounder. Hence, we cannot use ordinary supervised learning techniques since fstruct (z) + E [Y|X =z. Here, we assumne there is no observable confounder but we may easily include this, as discussed in Appendix C.
Figure 1: Causal Graph.
To deal with the hidden confounder e, assume to have access lo an instrumental variable Z E Z which satisfies the following assumption.
A Assumption 1. The conditional distribution P(X nol consland Z and E [e| =
Intuitively, Assumption means that the instrument Z induces variation in the treatment X but is uncorrelated with the hidden confounder E. Again, for simplicity, we assume Z c Rd The causal graph describing these relationships is shown in Figure 1.", Note that the instrument Z cannot have an incoming edge from the latent confounder that is also parent of the outcome.
Given Assumption 1, we can see that the function fatruct satisfies the operator equation E [ = E (Xstreet(X) [2] by laking expectation conditional on of both sides of (1). Newey and Powell (2003) provide necessary and sufficient conditions, known as completeness assumptions, to ensure identifiability of fstruct (X). Solving this equation, however, is known to be ill- ill-posed (Nashed and Wahba, 1974). To address this, recent works (Carrasco et al., 2007; Darolles et al., 2011; Muandet et al., 2020; Singh et al., 2019) minimize the following regularized loss L to obtain the estimate fatruct:
S arg min L(J), L()) Eyz |Y Ex1z [S(X))) + R(S),
(2)
fEF
where F is an arbitrary space of continuous functions and {(f) is a regularizer on f.
2.2 Two STAGE LEAST SQUARES REGRESSION
number of works (Newey and Powell, 2003; Singh et al., 2019) tackle the minimization problem (2) using two-stage least squares (2SLS) regression, in which the structural function modeled as fstruc(()) = u (x),, where u is a learnable weight vector and (b(x) a vector of fixed basis functions. For example, linear 2SLS used the identity map vf(x) x, while sieve IV (Newey and Powell, 2003) uses Hermite polynomials.
In the 2SLS approach, estimate á is obtained by solving two regression problems succes- sively. In stage 1, we estimate the conditional expectation Ex [xb(X) as a function of z. Then in stage 2, as Ex [/(X)] u 1x [xb(X)), we minimize L with Exis [S(X) being replaced by the estimate obtained in stage
Specifically, we model the conditional expectation as Exie [x)) = VΦ(2), where Φ(2) is another vector of basis functions and V matriz to be learned. Again, there exist many
We show the simplest causal graph in Figure 1 It entails Z IL E, but we only require Z and e to be uncorrelated in Assumption 1. Of course, this graph also says that Z is not independent of when conditioned observations X.
3

---

Published as a conference paper at ICLR 2021
choices for Φ(2), which can be infiite-dimmnsiional but assume the dimensions of (b(x) and ((2) to be d1, d2 < oo respectively.
In stage 1, the matrix V is learned by minimizing the following loss,
V arg mrg Li(V), (V( Ex,2 [[lxb(X) V + VER4x422
(3)
where A > 0 is a regularization parameter. This is a linear ridge regression problem with multiple targets, which can be solved analytically. stage 2, given V, we can obtain u by minimizing the loss
arg min C2(2), C2(u) By,z |I u (Z()) 12|lu]
(4)
wER
where X2 > 0 is another regularization parameter. Stage 2 corresponds to a ridge linear regression from Vz( to Y, and also enjoys closed-form solution. Given the learned weights ú, the estimated structural function is fstruet(x) = ú (zx)..
3 DFIV ALGORITHM
In this section, we develop the DFIV algorithm. Similarly to Singh et al. (2019), we assuine that we do not necessarily have access to observations from the joint distribution of (X, Y, Z). Instead, we are given m observations of (X, Z) for stage 1 and n observations of (Y, Z) for stage? We denote the stage observations If observations of (X, Y, 2) are given for both stages, we can evaluate the out-of-sample losses, and these losses can be used for hyper-parameter tuning of A1 (Appendix A).
(Ir4, and the stage observations (Ur, z.).
DFIV uses the following models
fstruct (I) = T "baxx (x) and Ex|= [bbox (X) = V Vo20(2)
(5)
where u E Rdi and V E Bd are the parameters, and tba (x) E Rdh and Φo (z) E Rd are the neural nets parameterised by 0x E @x and 0g E θ2, respectively. As in the original 2SLS algorithm, we learn Ex|= [xbe (X)] in stage 1 and fatruct (z) in stage 2. In addition to the weights u and V, however, we also learn the parameters of the feature maps, 0x and 0z. Hence, we need to alternate between stages 1 and 2, since the conditional expectation Exix [tbex (X)) changes during training.
Stage 1 Regression The goal of stage 1 estimate the conditional expectation Exi= [4bx (X)] ~ V (z) by learning the matrix V and parameter 0z, with 0x = 0x given and fixed. Given the stage data (x), this can be done by minimizing the empirical estimate of Li.
1 m
(m) Oz
arg min clm)( (V.82). clTh VERdi Xd2 8260z
m Σ
Vosz (2.)11 + V 1 (x:)
Note that the feature map bb (X) is fixed during stage 1, since this is the "target variable." we fix 0z, the minimization problem (6) reduces linear ridge regression problem with multiple targets, whose solution as a function of 6x and Oz is given analytically by
(m) (0x,0z) 0z) V t { Φ + m
where P1, v1 are feature matrices defined as 1 = [ebe (I1)..., &oy (Im))T E mm (21)...., Φ2 (zm)) madd We can then learn the parameters Oz of the and Φ, = (60 = adaptive features 4b,z by minimizing the loss c(m) at V (r(m) 0z) using gradiont descent. For simplicity, we introduce small abuse of notation by denoting as Oz the result of a user-chosen number of gradient descent steps the loss (6) with (mm) (0x, 0z) from (7), even though Ūz need not attain the minimum of the non-convex loss (6). We then write
4

---

Published as a conference paper at ICLR 2021
((m) (0x, While this trick of using an analytical estimate of the linear output weights of a deep neural network might not lead to significant gains in standard supervised learning, it turns out to be very important in the development of our 2SLS algorithm. As shown in the following section, the analytical estimate (( (fxx) 0z) (now considered function of 0x) will be used to backpropagate to 0x in stage 2.
Stage 2 Regression In stage 2, we learn the structural function by computing the weight vector and parameter 0x while fixing Oz 0z, and thus the corresponding feature map Φ8 (z). Given the data (ūi, 21); we can minimize the empirical version of L2, defined as
ú()),0 mig XER"1xXE@X
(u,
c
y u (m) 0b (2))) + A2 22l. (8)
72
Again, for a given 0x, we can solve the mininization problem (8) for as function of (m(m) ((x) fz) by linear ridge regression
a (0x 0z) = (v(m) 22 (m)) + +nd21) 4m y2. where Φ2 = 1 (a) ( ( 88 (Zx)]
(9)
E Ruxda and y22 [ūr..... ..]] E R
The loss -)" explicitly depends on the parameters 0x and we can backpropagate it to 0x via (( (0x. 0z), even though the samples of the treatment variable X do not appear in stage 2 regression. We again introduce a small abuse of notation for simplicity, and denote by Ox the estimate obtained after a few gradient steps on (8) with (0x, 0z) from (9), even though 0x need not minimize the non-convex loss (8). We then have (!n) a(m) (0x, 0z). After updating 0x, we need to update 6z accordingly. We do not attempt to backpropagate through the estimate Oz to do this, however, as this would be too computationally expensive; instead, we alternate stages and 2. We also considered updating 0x and 0z jointly to optimize the loss c(m),, but this fails, as discussed in Appendix F.
Computational Complexity and Convergence The computational complexity of the algorithm is ((d, + d)) for stage 1, while stage 2 requires additional ( dg + d computations. This is small compared to KIV (Singh et al., 2019), which takes O(m²) and O(n²), respectively. We can further speed up the learning by using mini-batch training as shown in Algorithm 1. Here, (mm) and (mm) are the functions given by (7) and (9) calculated using mini-batches of data. Similarly, c(m)) and c(mb) are the stage and 2 losses for the mini-batches. We recommend setting the batch size large enough so that (mms), (mm) do not diverge from (mm),, á (()) computed on the entire dataset. Furthermore, we observe that setting Ti > T2, i.c. updating 0g more frequently than 0x, stabilizes the learning process.
In Appendix B, we provide regularity conditions under which the function learned by DFIV converges to the true structural function in probability. The derivation based on Rademacher complexity bounds (Mohri et al., 2012).
4 EXPERIMENTS
In this section, we report the empirical performance of the DFIV method. The evaluation considers both low and high-dimensional treatment variables. We used the demand design dataset of Hartford and we propose new setting for the high-dimensional case based on the dSprites dataset (Matthey et al., 2017). the deep RL context, we also apply DFIV to perform off-policy policy evaluation (OPE). The network architecture and hyper-parameters are provided in Appendix G. The algorithms in the first two experiments are implemented using PyTrrc (Paszke et al., 2019) and the OPE experiments implemented using TensorFlow (Abadi et al., 2015) and the Acme RL framework (Hoffman et al., 2020). The code included in the supplemental material.
al. (2017) for benchmarking in the low and high-dimensional cases,
5

---

Published as a conference paper at ICLR 2021
Algorithm 1 Deep Feature Instrumental Variable Regression
Input: Stage data (a,, 24), Stage 2 data (0,2,,, Regularization parameters (A1, A2). Initial values 0x, Oz. Mini-batch size (ma, no). Number of updates in each stage (Ti, T2).
Output: Estimated structural function fstruct (4)
1: repeat
2: Sample my stage data (xt (2( and ny stage data (7!") =
3: for t= 1 to Tj do
4: Return function (0 Oz) in (7) using (z; (b) )
5: Update 02 +- 8z - a 2o (m)b) (ma) (omm)(B. (0x, , 82), 82 82) dz=
Stage 1 learning
6: end for
7:
for t= 1 to T/ do
8: Return function ((ms)) (0x, 0z) in (9) using (7!") ()) and function ((m) 0z)
9: Update 0x + 0x a Vox C2 (a (m) (0xx 02), ) 0x 1xx Ōx
V Stage 2 learning
10: end for
until convergence
Compute a(m), := à ((m) (0x, Oz) from (9) using entire dataset.
12:
13:
return (x)) = (a T "0xxxx (x)
4.1 DEMAND DESIGN EXPERIMENTS
The demand design dataset is synthetic dataset introduced by Hartford et al. (2017) that now a standard benchmarking dataset for testing nonlinear IV methods. In this dataset, we aim to predict the demands on airplane tickets Y given the price of the tickets P. The dataset contains two observable confounders, which are the time of year T E [o, 10] and customer groups S {L...., 7} that are categorized by the levels of price sensitivity. Further, the noise in Y and P is correlated, which indicates the existence of an unobserved confounder. The strength of the correlation is represented by P E [0, 1]. To correct the bias caused by this hidden confounder, the fuel price C' is introduced as an instrumental variable. Details of the data generation process can be found Appendix E.1. In DFIV notation, the treatment is X = P. the instrument is Z = C, and (T, are the observable confounders.
We compare the DFIV method three leading modern competitors, namely KTV (Singh et al., 2019), DeepIV (Hartford et al., 2017), and DeepGMM (Bennett et al., 2019). We used the DFIV method with observable confounders, as introduced in Appendix C. Note that DeePGMM does not have explicit mechanism for incorporating observable confounders. The solution we use, proposed by Bennett et al. (2019, p. 2). is to incorporate these observables in both instrument ond treatment; hence we apply DeepGMM with treatment X = (P.T, S) and instrumental variable Z = (C,T, S). Although this approach is theorelically sound, this makes the problem unnecessary difficult since it ignores the fact that we only need uo consider the conditional expectation of P given Z.
We used a network with similar number of parameters DeepIV as the feature maps in DFIV and models in DeepGMM. We tuned the regularizers λι, A2 as discussed in Appendix A, with the data evenly split for stage 1 and stage 2. We varied the correlation parameter p and dataset size, and ran 20 simulations for each setting. Results are summarized in Figure 2. We also evaluated the performance via the estimation of average treatment effect and conditional average mreatment. effect, which is presented in Appendix F.2 Next, we consider a case, introduced by Hartford et al. (2017), where the customer type S E (1...., 7} is replaced with an image of the corresponding handwritten digit from the MNIST dataset (LeCun and Cortes, 2010). This reflects the fact that we cannot know the exact customer type, and thus we need to estimate it from noisy high-dinensional data. Note that although the confomner is high-dimensional, the treatment variable is still real- valued, i.c the price P of the tickets. Figure presents the results for this high-dimensional confounding case. Again, we train the networks with a similar number of learnable parameters to DeepIV in DFIV and DeepGMM, and pyrer-parameters set. in the way discussed in

---

Published as a conference paper at ICLR 2021
Out-of-Sameee Sampie MSE
10
Algorithm
DeepCiM
10"
KIv
DRV
10a
19000
1000
Training Sample Size
Figure MSE for demand design dataset with low dimensional confounders.
30
Out Sample MSE
Algortthm
10-
Algorithm
Out-af- Sample MSE
DeegCMM
DeepSMM
DeeplV
5000
10000
0.1
0.25
0.5
0.75
0.9
Training Sample Size
Confounding Level
Figure MSE for dSprite datasct. DeepIV did not yield meaningful predictions for this experiment.
Figure 3: MSE for demand design dataset with high dimensional observed confounders.
Appendix A. We ran 20 simulations with data size m 5000 and report the mean and standard error.
Our first observation from Figure 2 and 3 is that the level p of correlation has significant impact on the error under any of the IV methods, indicating that approaches correctly account for the effect of the hidden confounder. This is consistent with earlier results this dataset using DeepIV and KIV (Hartford et al., 2017; Singh et al., 2019). We note that. DeepGMM does not perform well in this demand design problem. This may be due to the current DeepGMM approach to handling observable confounders, which might not be optimal. KIV performed reasonably well for small sample sizes and low-dimensional data, but it did less well in the high-dimensional MNIST case due to its less expressive features. In high dimensions, DeepIV performed well, since the treatment variable is unidimensional. However, DFIV performed consistently better than all other methods in both low and high dimensions, which suggests can learn flexible structural function in stable manner.
4.2 DSPRITES EXPERIMENTS
To test the performance of DFTV methods for high dimensional treatment variable, we utilized the dSprites dataset (Matthey et al., 2017). This is an image dataset described by five latent parameters (shape, scale, rotation, posX and posY). The images 64 x 64 4096-dimensional. In this experiment, fixed the shape parameter to heart, i.c. we only used heart- shaped images. An example is shown in Figure 5.
10
From this dataset, we generated data for IV regression in which each figure treatment variable X. Hence, the treatment variable is 4096-dimensional in this experiment. To make the task more challenging, used posY as the hidden confounder, which is not revealed to the model. We used the other three latent variables as the instrument variables Z. The structural function fatruct and outcome are defined as
60
25
50
Figure 5: dSprite image
|AX - 5000
Sstruet
Y = Sstrnct (X) +32(posY - 0.5) + (00,00))
1000
7

---

Published as a conference paper at ICLR 2021
where each element of the matrix A E R 10x40066 is generated from (n0, 1.0) and fixed throughout the experiment. See Appendix E.3 for the detailed data generation process.
We tested the performance of DFIV with KIV and DeepGMM, where the lyper-parameters are determined as in the demand design problem. The results are displayed in Figure 4. DFIV consistently yiclds the best performance of all the methods. DeepIV is not included in the figure because it fails to give mcaningful predictions due to the difficulty of performing conditional density estimation for the high-dimensional treatment variable. The performance of KIV suffers since it lacks the feature richness to express a high-dimensional complex structural function. Although DeepGMM performs comparatively DFIV, we observe some instability during the training, see Appendix E.4.
4.3 OFF-POLICY POLICY EVALUATION EXPERIMENTS
We apply our IV methods to the off off-policy policy evaluation (OPE) problem (Sutton and Barto, 2018), which is one of the fundamental problems of deep RL. In particular, it has been realized by Bradtke and Barlo (1996) that 2SLS could be used to estimate a linearly parameterized value function, and we use this reasoning as the basis of our approach. Let us consider the RI, environment (S, A, P. R, (oo 7), where S is the state space, is the action space, P: P S x A 8 [0, 1] is the transition function, R: S x A S x R + R is the reward distribution, Po S + [0, 1] is the initial state distribution, and discount factor y E (0, 1]. Let æ be a policy, and denote (a)s) as the probability of selecting action in stage E S. Given policy म, the Q-function is defined as
(T(s,a) E
Are 80
s, 0u
with at ~ T[- st), 8141 ~ P(- at), rt ~ R(-1s;, a, s(+1). The goal of OPE is to evaluate the expectation of the Q-function with respect to the initial state distribution for a given target policy x, Rewpo IQ (s, learned from a fixed dataset of transitions (s, a, 8"), where and are sampled from some potentially unknown distribution μ and behavioral policy (-(+(s) respectively. Using the Bellman equation satisfied by Q", we obtain a structural causal model of the form (1),
structural function (s,, r= + (Q"(s'')))) Essr((lss) (Q" as))) +r- Ep- [r],
- (s"
(10)
confounder e
where X = (s. s.aas'.0. a'), (s
We have that E E 0, [c|x] * 0, and Assumption is verified. Minimizing the loss (2) for the structural causal model (10) corresponds to minimizing the following loss LoPE
LopE = Eear (r ((yy
(11)
and we can apply any IV regression method to achieve this. In Appendix D, we show that minimizing CoPE corresponds to minimizing the mean squared Bellman error (MSBE) (Sutton and Barto, 2018, p. 268) and we detail the DFIV algorithm for OPE. Note that MBSE is also the loss minimized by the residual gradient (RG) method proposed in (Baird, 1995) estimate Q-functions. However, this method suffers from the "double-sample" issue, ie. it requires two independent samples of s' starting from the same (s,a) a) due to the inner conditional expectation (Baird, 1995), whereas IV regression methods do not suffer from this issue.
We evaluate DFIV on three BSuite (Osband et al., 2019) tasks: catch, mountain car, and cartpole. See Section E. for description of those tasks. The original system dynamics are deterministic. To create a stochastic environment, we randomly replace the agent action by uniformly sampled action with probability p E [0, 0.5). The noise level p controls the level of confounding effect. The target policy is trained using DQN (Mnih et al., 2015), and we subsequently generate offline dataset. for OPE by executing the policy in the
8

---

Published as a conference paper at ICLR 2021
cartpole
catch
mountain car
100
Algorithm
Absolute Error
10
10-
DeepGMM
10
DeepiV
FOE
10-1
10-1
DFIV
10
10-3
0.0
0.1
0.2
0.4
0.5
0.0
0.1
0.4 0.5
0.0
0.1
0.2
0.3
0.4
0.5
Noise Level
Figure 6: Error of offline policy evaluation.
same environment with a random action probability of 0.2 (on top of the environment's random action probability p). We compare DFIV with KIV, DeepIV, and DeepGMM; as well as Fitted Q Evaluation (FQE) (Le ct al., 2019; Voloshin et al., 2019), a specializod approach designed for the OPE setting, which serves our "gold standard" baseline (Paine et al., 2020) (see Section E.6.2 for details). All methods use the same network for value estimation. Figure 6 shows the absolute error of the estimated policy value by each method with standard deviation from runs. In catch and mountain car, DFIV comes closest in performance to FQE, and even matches it for some noise settings, whereas DeepGMM is somewhat worse in catch, and significantly worse in mountain car. In the case of cartpole, DeepGMM performs somewhat better than DFIV, although both are slightly worse than FQE. DeepIV and KIV both do poorly across all RL benchmarks.
CONCLUSION
We have proposed a novel method for instrumental variable regression, Deep Feature TV (DFTV), which performs two-stage least squares regression on flexible and expressive features of the instrument and treatment. As a contribution to the IV literature, we showed how to adaptively learn these feature maps with deep neural networks. We also showed that the off-policy policy evaluation (OPE) problem in deep RL can be interpreted as a nonlinear IV regression, and that DFIV performs competitively in this domain. This work thus brings the research worlds of deep offline RL and causality from observational data closer.
In terms of future work, it would be interesting adapt the ideas from (Angrist and Krueger, 1995; Angrist et al., 1999; Hansen and Kozbur, 2014) to select the regularization hyperparameters of DFIV as well as investigate generalizations of DFIV beyond the additive model (1) considered in (Carrasco et al., 2007, Section 5.5). In RL, problems with additional confounders are common, see e.g. (Namkoong et al., 2020; Shang et al., 2019), and believe that adapting DFIV to this setting will be of great value.
REFERENCES
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, 1. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Vičgas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke,
Y. Yu, and X. Zheng. TensorFlow: Large-scale machine learning on helerogeneous systems, 2015. URL http://tensorfloo.ct
J. D. Angrist. Lifetime carnings and the Vietnam era draft lottery: Evidence from social security administrative records. The American Economic Review, 80(3):313 336, 1990.
J. D. Angrist and A. B. Krueger. Split-sample instrumental variables estimates of the return to schooling. Journal of Business & Economic Statistics, 13(2):225 22552,555, 1995.

---

Published as a conference paper at ICLR 2021
J. D. Angrist, G. W. Imbens, and D. B. Rubin. Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434)::44-455 1996.
Angrist, G. W. Imbens, and A. B. Krueger. Jackknile instrumental variables estimation. Journal of Applied Econometrics, 14(1):57 67, 1999.
L. Baird. Residual algorithms: Reinforcement learning with function approximation. In Proceedings of the 12th International Conference on Machine Learning, 1995.
E. Barcinboim and J. Pearl. Causal inference by surrogate experiments: Z-identifiability. In Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence, page 113-120, 2012.
A. G. Barto, R. S. Sutton, and C. W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions Systems, Man, and Cybernetics, (5):834-844 846,,866,11 1983.
A. Bennett, N. Kallus, and T. Schnabel. Deep generalized method of moments for instrumental variable analysis. In Advances Neural Information Processing Systems 32, pages 3564- 3574. 2019.
R. Blundell, X. Chen, and D. Kristensen. Semi-nonparametric IV estimation of shape- invariant engel curves. Econometrica, 75(6):1613 1669, 2007.
R. Blundell, J. Horowitz, and M. Parey, Measuring the price responsiveness of gasoline demand: Economic shape restrictions and nonparamotric demand estimation. Quantitative Economics, 3:29-51, 2012.
S. J. Bradtke and G. Barto. Lincar least-squares algorithms for temporal difference learning. Machine Learning, 22(1-3):33 57, 1996.
M. Carrasco, J.-P. Florens, and E. Renault. Linear inverse problems in structural econo- metrics estimation based on spectral decomposition and regularization. In Handbook of Econometrics, volume 6B, chapter 77. 2007.
X. Chen and T. M. Christensen. Optimal sup-norm rates and uniform inference on nonlinear functionals of nonparametric IV regression: Nonlinear functionals of nonparametric IV. Quontitative Economics, 9:39 84, 2018.
X. Chen and D. Pouzo. Estimation of nonparametric conditional moment models with possibly nonsmooth generalized residuals. Econometrica, 80(1):277 321, 2012.
S. Darolles, Y. Fan, J. P. Florens, and E. Renault. Nonparametric instrumental regression. Econometrica, 79(5):1541-1566) 2011.
D. Frnst, P. Geurts, and L. Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503 1563--55,,20 2005.
C. Hansen and D. Kozbur. Instrumental variables estimation with many weak instruments using regularized jive. Journal of Econometrics, 182(2):290-22 -0090000------ 2014.
L. P. Hansen. Large sample properties of generalized method of moments estimators. Econometrica, 50(44:1029-105:) 1982.
J. Hartford, G. Lewis, K. Leylon-Brown, and M. Taddy. Deep IV: A flexible approach for counterfactual prediction. In International Conference on Machine Learning, 2017.
M. Hoffman, B. Shahriari, J. Aslanides, G. Barth-Maron, F. Behbahani, T. Norman, A. Ab-
dolmaleki, A. Cassirer, F. Yang, K. Baumli, S. Henderson, A. Novikov, S. G. Colmenarejo, S. Cabi, C. Gulcchre, T. L. Paine, A. Cowie, Z. Wang, B. Piot, and N. de Freitas. Aeme: A research framework for distributed reinforcement learning, arXiv preprint arXix:2006. 00979, 2020.
10

---

Published as a conference paper at ICLR 2021
H. Le, C. Voloshin, and Y. Yue. Batch policy learning under constraints. In International Conference on Machine Learning, 2019.
Y. LeCun and C. Cortes. MNIST handwritten digit databasc. 2010. URL http://yann. lecun. cwm/co/eeeaaae..
wttp:/ow//ooccecttttttttttt..cpt exdb/mn htdb/mnist//
L. Matthey, Higgins, D. Hassabis, and A. Lerchner. dSprites: Disentanglement testing sprites datasct, 2017. URL https:/ thub. com/ http///eeemnnddpppptttttttaaaaee dataset/.
T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540))529 533, 2015.
M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of Machine Learning. MIT Press, 2012.
A. W. Moore. Efficient Memory-Baae Based Learning for Robot Control. PhD thesis, Cambridge University, 1990.
K. Muandet, A. Mehrjou, K. Lee, and A. Raj. Dual IV: A single stage instrumental variable regression. In Advances in Newral Information Processing Systems 34, 2020.
H. Namkoong, R. Keramati, S. Yadlowsky, and E. Brunskill. Off-policy policy evaluation for sequential decisions under unobserved confounding. In Advonces in Neural Information Processing Systems 34, 2020.
M. Z. Nashed and G. Wahba. Generalized inverses in reproducing kernel spaces: An approach to regularization of linear operator equations. SIAM Journal on Mathematical Analysis, 5 (6) (0):97449880 1974.
W. K. Newey and J. L. Powell. Instrumental variable estimation nonparametric models. Econometrica, (1(::15565 1578, 2003.
I. Osband, Y. Doron, M. Hessel, J. Aslanides, E. Sezener, A. Saraiva, K. McKinney, T. Lat- timore, C. Szepesvari, S. Singh, et al. Behaviour suite for reinforcement lcarning. In International Conference on Learning Represeniations, 2019.
T. L. Paine, C. Paduraru, A. Michi, C. Gulcehre, K. Zolna, A. Novikov, Z. Wang, and N. de Freitas. Hyperparameter selection for offline reinforcement learning, arXiv preprint ar orXix:2007.00 000055, 2020.
A. Paszke, S. Gross, Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and Chintala. Pytorch: An imperative style, pigh-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 5024-8035. 2019.
A. Rahimi and B. Rocht. Random features for large-scale kernel machines. In Advances in Newral Information Processing Systems 20, pages 1177 1184. 2008.
W. Shang, Yu, Q. Li, Z. Qin, Y. Meng, and Ye. Environment reconstruction with hidden confounders for reinforcement learning based recommendation. In Proceedings of the 25th A CM SIGKDD International Conference on Knowledge Discovery Data Mining, pages 566-576, 2019.
R. Singh, M. Sahani, and A. Gretton. Kernel instrumental variable regression. In Advances in Neural Information Processing Systems 32, pages 3593-4605. 2019.
J. H. Stock and F. Trebbi. Retrospectives: Who invented instrumental variable regression? Journal of Economic. Perspectives, 1773 2003.
11

---

Published as a conference paper at ICLR 2021
R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2018.
C. Voloshin, H. M. Le, N. Jiang, and Y. Yue. Empirical study of oft-policy policy evaluation for reinforcement learning. arXie preprint arXi::1911.0685, 2019.
M. Wiatrak, S. V. Albrecht, and A. Nystrom. Stabilizing generative adversarial networks: A survey. Xiv preprint 191: 00922, 2019.
P. Wright. The Tariff on Animal and Vegetable Oils. Investigations in International Commercial Policies. Macmillan Company, 1928.
12

---

Published as a conference paper at ICLR 2021
A HYPER-PARAMETED TUNING
If observations from the joint distribution of (X, Y; Z) are available in both stages, we can tune the regularization parameters λ1 A2 using the approach proposed in Singh et al. (2019). Let the complete data of stage 1 and stage 2 be denoted as (I,, 2,) and (æ,, De. &,). Then, we use the data not used in each stage to evaluate the out-of-sample performance of the other stage. Specifically, the regularizing parameters are given by
x = arg min L
(n)
1 (T.) (mm) 08 (2/)
1-nos
Ø=1
X2 arg min (m)
(m)
1
(v) (a(")) T ( Pūzz (2i))²
2-005
2-006
where (m)), (m), Box: doz are the parameters learned in (6) and (8).
B CONSISTENCY OF DFIV
In this appendix, we prove consistency of the DFIV approach. Our contribution is to establish consistency of the end-to-end procedure incorporating Stages and 2, which we achieve by first showing a Stage 1 consistency result (Lemma 1), and then establishing the consistency of Stage 2 when the empirical Stage solution is used as input (Lemma 2). The desired result then follows in Theorem 1.
Consistency results will be expressed in terms of the complexity of the function classes used in Stages and 2, as encoded in the Rademacher complexity of functionals of these functions (sce Proposition below). Consistency for particular function classes can then be shown by establishing that the respective Rademacher complexities vanish. We leave for future work the task of demonstrating this property for function classes interest.
B.1 OPERATOR VIEW OF DFIV
The goal of DFIV is to learn structural function fstruct, which satisfies Eriz [Y] = Exiz (X)]
(12)
We model Sstruct as Setruct (X) = 2 T 4be (X) and denote the hypothesis spaces for vbax and fatruct follows:
Hs = { X + Rd | 0x E @x],
F {u T "bx X + y bo E Ho, u E Rd)..
To learn the parameters, we minimize the following stage 2 loss:
(m)) 0x = arg min ch wE341,8x
cls) (u, 0x),
clm) = n
1 (ū T L Exiz [bbox (Z))
We denote the resulting estimated structural function as farmee (X) (1) 1/xxx (X). For simplicity, we set all regularization lerms to zero. Here, Exiz [wbex (X) is the empirical conditional expectation operator, which maps an element of Hy to some function g : Z Idd that
Êx [ (X)] arg min ( (V,0z), clm) gEG
1 m
I 9(2)))1
In DFIV, we define g as
V VE 2d2d, A2 E θz). This formulation is equivalent to the one introduced in Section 3. With slight abuse of notation, for f(X) = u <box (X) F, we define É Ëx1z [X]] to be
( Φ0 :Z d
Exiz [f(X)] u TÉx1 Lbbex (X)
since this is the empirical estimate of Exiz (xx)
13

---

Published as a conference paper at ICLR 2021
B.2
GENERALIZATION ERRORS FOR REGRESSION
Here, we bound the generalization errors of both stages using Rademacher complexity bounds (Mohri et al., 2012).
Proposition (Theorem 3.3 Mohri al., 2012, with slight modifieation) Lel. S be measurable space and H be family of functions mopping from S to [0, M Given fixed dataset ($1, Sp) E S", empirical Rademacker complexity is given by
Rs(H) = Eo
sup WEH
ogh((s)
where o = (o1.....o.) with a, independent uniform random variables taking values in {-1,+1}. Then, for any 5 >> O, with probability at least - over the draw of on i.i. sample S of size n, each of followingg holds for all h E H:
log 2/8
E [ - Σ ((s)) + 29 + 3M
2n
log 2/8
h(s)) S E [h(s)] 2 + 3M
2n
1=1
We list the assumptions below.
Assumption 2. The following hold:
1. Bounded oulcome variable |Y] M.
2. Bounded stage hypothesis space: V2 E Z, [[g(2))] < 1.
3. Bounded stage feature map Vx E X, || (x)]l < 1.
4. Bounded stage weight: ||u|| S M.
5. Identifiable stage 1 hypothesis space: Vabox E Hwp Fg E 9, Exiz [4bxx]
6. Identifiable stage 2 hypothesis space: fetruct E F.
Note that we leave aside questions of optimization. Thus, assume that the optimization procedure over (0z, V) is sufficient to recover Exiz, and that the optimization procedure over (u, 0xx is sufficient to recover Sstruct (which requires, in turn, the correct Exiz, for this d. ). We emphasize that Algorithm does not guarantee these properties.
Based on these assumptions, we derive the generalization error in terms of L2-norm || (le(2) defined as
[ ( [1M22)PaP(2))
The following lemma proves the generalization error for stage regression.
Lemma 1. Under Assumption 2, and given stage data S = {(x,, 2,) ) for any 8 > 0, with at least probability 1 28, we have
S MM 19s, (Hi) + 24 |Bx12 [S(X) - ExIZ [(X) P(Z)
log 2/5
2m
for any f u Tex E F, where hypothesis space H, is defined as
Hi ((x,2) X 2 - |lubox (x) 2 R. g E bbx Hp).
(13)
14

---

Published as a conference paper at ICLR 2021
Proof. From Cauchy-Schwarz Schwarz inequality, we have
- Ex - [f(x))
P(Z) = |(uT (Ex1z 1xxxx (X)] - Ex12 [xrox (x31) p(Z)
$ SM
M Exiz (oxx (X) Exi2 (4 P(Z)
By applying Proposition to hypothesis space H1, we have
Exz |ox (x) -Ex12 [x (x]]](]
vex (x;) - Exiz lxb (X) (2)) 29s, (H4) + 12
log 2/5 2m
with probability 1 8. Indeed all functions E Hi satisfy ||||| < since || (X)ll S 1, [lg(Z)]1 1 from Assumption 2.
Also, since Vapax E Hyp 3g E 9, Exiz [bbax] = 9, again from Proposition 1, we have
xox (I.) Exiz fxbox (X)] (2:)
m
log 2/0 2m
< Exz || (X) Ex|2 fabx (X)1|]² + 2Dss, (H1) - 12
with probability 1 - 8. From the optimality Ex1z (abox (X),, we have
112
bbax (3ra) Æxiz [obo (z:)
||
1
m | (3re) Exiz [bbox (X) (z:)]²²
m
m
Hence, we have
Exz || (X) -Ex1 1xbox (X)] [
< Exz Illbex (X) Exiz I (X]]1 498, (Hl)) 2
log 2/8 2m
with probability 1 - 26. Now we have
Exz bbxx (X) -Exiz [xbox (301][
Exz [1 (x) Exiz (bbx (x)11²) Ez
E Ex12 (bba [X)) Exiz [ubox (X)) (x)1
and thus
Ez |X (sba (X)) Exiz (bb (x21|"] 198s, (Hl) + 241
log 2/5 2m
Therefore, by taking the square root of both sides, we can see
|xxz [f(x) Ex1z [f(X]] P(Z)
log 2/5 2m
s MV 19s, (Hi) + 2
The generalization error for stage is given in the following lemma.
15

---

Published as a conference paper at ICLR 2021
Lemma 2. Under Assumption 2, and given stage 1 data S {x), 21) stage data S2 {(), d=);: and the estimated structural function fatruct (z) (ū(m)) v0x (x), then for any 0, with at least probability 1 1-4 we have
Eriz [ -xxz [ssrucet (X) P(Z)
log 2/6
log 2/0
< 194ss (H2) + 24M2
+ M (HH) + 24V
2n
2mn
where Hi is desined (13) and H2 is defined as
H2 = (1,,2)) E y Z (y g()))) E g R R,, //2||| S M..
(14)
Proof. Since Exiz [vox] EG, by applying Proposition to hypothesis space H2, we have
Eyz (Y - Exxz [fattcc (x))
U. - -xizz (farue)())) (a))²+ + 28s, (H2) + 12M²
log 2/8
2n
with probability of 1 - 8. Note that all functions h E H2 are bounded in ||||| S 4M² since Y S M, 1o (x)Il s 1, || I M from Assumption 2. Similarly, since fstruct F from
Assumption 2, we have
Vi Ex12 [Struet (X)
71
log 2/0 2n
Eyz (r -x
Exiz [faucet [Xar)) (x)1)² + 29s, (H2)) + 12M2
with probability 1 - 8. From the minimality of farruct, we have
jr Exiz (X) (2i) < Σ (ū. E Exiz [fstruct [ (2:))²
(2)))
Hence, we have
Eyz -x-1Xx1 Æx1z ((armaa())) (x)
< Eyz [or (Y--BxI Exiz [strrnct (x)))) 19s,, (H2) 21M2 V
log 2/8 2n
with probability 1- 28. Now we also have
Eyz (x 1Bx1xx2 [farruss) 1) = Eyz [[Y - Eyiz [x + Ez [(x12 [fauruca] - -Eriz [T1)
Eyz ( -x-Ex [fstruet)) = Erz - [+]) + Ez [Bx]z [fstruct] Eriz [M1)2]
Therefore, with probability 20,
Ez (Zyiz [Y] Ex z [[armc(())]
/log2/0
2/6
< 49ss, (H2) + 24
Ez [Éx z (fatruct(X)) Eriz [Y])2]
2n
From Lemma and (12), with probability at least 1 - -25,
log 2/8 2m
Ez [Bx [fstruct - Eriz [M]] < RM (H2) + 24M2
16

---

Published as a conference paper at ICLR 2021
By combining them, we see that with probability - 46,
Eriz [Y] Ex|z [farrcc (X)) P(Z)
log 2/5
log 1/g2/0 2/0 2m
V (H2) + +24M²
+ 4M2RR, (H.) + 24M²
2n
log 2/5
1/g
log 2/8
V 19s, (H2) + 24M22
+ MV 49s, (Hl)+ 24V
2n
2m
B.3 CONSISTENCY PROOF OF DFIV
The goal is to bound the deviation between fatrect and fatruct. This discrepancy is measured by Lg-norm with respect to P(X), defined as
[M(X))p(()) = -f (IIMX)PPP(())
However, we used the norm || (l((()) in Lemmas 1 and 2. To bridge this gap, we state the necessary condition for identification introduced (Newey and Powell, 2003). Proposition 2. (Proposition 2.1 Newey and Powell, 2003) For all 8(x) with finite expecta- lion, E (X) = 0 a.e. on Z implies 8(x)) 0.
Proposition 2 is the minimum requirement for identification and Assumption a sufficient condition for it. Given Proposition 2, we consider constant < defined as
f (X) f(X)l
sup AFF.Iffanue
(15)
|E [fstruct (X) - f(X)
Note that チ 2 1 from definition and r This measures the ill-posedness of IV problem. A similar quantity is introduced in (Blundell et al., 2007). Given this quantity, we derive the convergence rate of DFIV as follows.
and only if X is measurable with respect to Z.
Theorem Let Assumplion hold. Given stage data S = {( 21) })); slage data S2 = ((i, km and t defined in (15), for any > O, with at least probabilty of 1 68, we have
fl (X) -faruc((X))l (X))lp(x)
2/8
log 2/8 2n
$ MT RRs (H1) + 2
+ TV 4ns (H2) + 24M²
2m
where Hi and H2 are defined in (13) and (14), respeclively.
Proof. Using in (15), we have [arrrc (X) Satraaa (X) (X)l)(()) 5
E Sarmor(X) Sarmect (X)
p(())
(aaruo - Exiz [Farrcc P(Z)
< EE
+ E [fstruct (oaeme(xx115 (x)12) - Ex1z [Farmau(x)] P(Z)
= r E f fatruce (X) - Exiz [Farrce(x)) p(Z))
E ExIZ [(atrccc p(zz
where the last inequality holds from (12). Using Lemmas and the result thus follows. n
17

---

Published as a conference paper at ICLR 2021
Figure 7: Causal graph with observable confounder
From this result, we obtain directly the following corollary.
Corollary 1. Let Assumption 2 hold. If Rs, (H1) - and Rs (H2) 0 probability as datasize increases, fatruct converyes to fetruct in probability.
The proof of vanishing Rademacher complexities for particular Stage 1 and Stage 2 function classes is a topic for future work.
C 2SLS ALGORITHM WITH OBSERVABLE CONFOUNDERS
In this appendix, we formulate the DFIV method when observable confounders are available. Here, we consider the causal graph given in Figure 7. In addition to treatment X E X, outcome E V, and instrument Z E 2, we have obscrvable confounder O E O. The structural function fatruct we aim to learn is now X' x O + V/ and the structural causal model is represented
Y= fstruct (X,0) E [
[ * 0.
For hidden confounders, we rely on Assumption For observable confounders, we int.ooduce similar assumption.
A Assumption 3. The conditional distribution P(XIZ,O) is not constant in Z and [= =.
Following similar reasoning in Section 2, we estimate the structural function sstruct by minimizing the following loss:
Patruct arg min Σ(f), ī(f) zo [(Y Ex (xX, + +(f).
fEF
One universal way to deal with the observable confounder is to augment both the treatment and instrumental variables. Let us introduce the new treatment X = (X, O) and instrument S = (2,0), then the loss E becomes
Ł(f) = = [Y-EE, Ex Ex2 [(( + +((),
which is equivalent, to the original loss L. This approach is adopted in KTV (Singh el al., 2019), and used it here for DeepGMM method (Bennett al., 2019) in the demand design experiment. However, this ignores the fact that we only have to consider the conditional expectation of X given Ž = O). Hence, we introduce another approach which is to model f(X, O) u (x) & {(O)), where (x( and {(O) are feature maps and & denotes the tensor product defined as a @ b vec(ab It follows that Exiz, [f ([X,O)] = u (Exiz,, (x)] & {(O)), which yields the following two-stage regression procedure.
In stage 1, we learn the matrix V. that
V arg min (V( (V(V) Ex.00 [1[x(x) - Vo(Z,)) 1 Vegdix4; xd2
which estimates the conditional expectation Ex 2,0 (x)) Then, in stage 2, we learn á using
i = arg min L2(w) Le(w) = Ey,z, [|r - (uT(V((2 0 {(0)))] - 2211za|?? "
18

---

Published as a conference paper at ICLR 2021
Again, both stages can be formulated as ridge regressions, and thus enjoy closed-form solutions.
We can further extend this to learn deep feature maps. Let Φø (2, 48 (X), Sa (O) be the feature maps parameterized by 0z xx 00 respectively. Using notation similar to Section 3, the corresponding DFIV algorithm with observable confounders shown in Algorithm 2. Note that in this algorithm, steps 3, 5, 6 are until convergence, unlike for Algorithm 1.
Algorithm 2 Deep Feature Instrumental Variable with Observable Confounder
Input: Stage data (Iy 24, o,); Stage 2 data (gt, 2,, Initial values d 0x 0z. Regulariz- ing parameters (A1,
Output: Estimated structural function fstruct(a))
1: while (.")) has not converged do
2: while E/T) has not converged do
3: Update Oz by 6z t 82 - aV ((m)) ( (m)) (00 20 0z 1\ Stage
4:
end while
5: Update Ox by 0x 0x a a ((m) (ū Ox
0z) 0x =xx
11 Stage 2 Stage 2
6: Update to by do 8 Voo 22 (2) 0x
7: end while
8: Compute a(n) from (9)
9: return (str = (ū(m)) 2/ (x)
D APPLICATION OF 2SLS AND DFIV TO OFF-POLICY POLICY EVALUATION
Here we first show that the optimal solution of (2) in the OPE problem is equivalent to that of mean squared Bellman error, and then describe how apply 2SL. algorithm.
We interpret the Bellman equation in (10) as IV regression problem, where X (s, s dd), , Z (s, a) and Y = and
fstrocr a,,,,,,,,, 6.,8', (= a) (Q"
Let R(s,a) a) be the conditional expectation of reward given 8,a defined as
R(s, a) = E, [r] - f r((''ls, (s a) R(rls, s')ds'd dr.
Then, we can prove that the solutions of (11) and MSBE are equivalent. Indeed, we have (11)
arg min Es [(r Q" (s) + |a 12 (s', d')))
Q
= arg min EL, Eg, (r- R(s a) + R(s, a) - Q" (s, + + la,s (2"
Q
arg min Eg,a
Var(r|s,a) 1/ const wrt Q"
Q"
+ +By/e [(r R(s a))] (M(s, (* y.. la.s ( /1 =0
(H(ss, a) (Q, a) E. a' (s' ='))²
aarg min Es,a
(R(ss,a) (s, - F.. a. 10 (8', (s(,
(16)
Q"
In this context, we model (=) = u (0(8 a) so that.
(s,a,s', a') f
u (*b(s, a) - 7/b(s', a')).
(17)
It follows that
Ex [f(X)] = ( a) " (bs''a')))
19

---

Published as a conference paper at ICLR 2021
We will model By (8i, Oz, 84), stage regression becomes
(s/(s', d') a))ssa). In this case, given stage 1 data
(m) arg min cim, VERlixd
c(m)
(bss V A
where we sample of ~ (-|s()). However, given the specific form (17) of the structural function, stage is slightly modified and requires minimizing the loss
à (n) arg min C
72 E F. I
u (1b(8,,
+ A2||u|l?
uER4
given stage data (&,, R.). We can further learn deep feature maps by parameterized feature maps up and o as described in Section 3 to obtain the DFIV algorithm for OPE.
E EXPERIMENT DETAILS AND ADDITIONAL RESULTS
E.1 DETAILS OF DEMAND DESIGN EXPERIMENTS
Here, we introduce the details of demand design experiments. We follow the procedure in Singh et al. (2019). The observations are generated from the IV model,
fstruct S.) E, T,
where Y is sales, P is the treatment variable (price) instrumented by supply cost-shifter C. T.S S are the observable confounder, interpretable as time of year and customer sentiment. The true structural function is
fstruct (P,T,S) 100 + (10 P)Sh(T) - 2P,
(t (t-5)4 exp((4((1- 5)3) + 10
h((1 2
-
600
Data sampled as
S Unif(1,.....
T ~ Unif[o, 10]
C 1((,
V ~ 1(0,1)
8 N(VV p
P 25 25 (C33))(T) +V
From observations of (Y,P,T,S,C) C); we estimate Sareet by several methods. For each estimated fstruct: measure out-of-sample crror as the square error of f versus true fstruct applied to 2800 values of (p. t, s). Specifically, we consider 20 evenly spaced values of p E [10, 25), ; 20 evenly spaced values of / (0, 10), and all 7 values E {I,...,7].
E.2 EFFECT ESTIMATION IN DEMAND DESIGN EXPERIMENTS
In this section, we report the result of causal effect estimation based on demand design experiments. Specifically, we consider two effects: one is average treatment effect (ATE) and another is conditional average treatment effect (CATE).
ATE Estimation
ATE is central target quantity in causal inference defined as ATE E [Y]do(X = 1)] EE [Y]do 0)]
for binary treatment X E {0. 13. ATE thus requires estimating the counterfactual mean out- comes E [Y]do(X= 1)), E do 0)]. Here, we generalize this to continuous treatment
20

---

Published as a conference paper at ICLR 2021
Agortthm
Aigorthem Ground Truth DFV
-325
DFN
RV
-100-
DeupiV
CATE
DeepGMM
-200
Desp
-200
-250
15
10
12
22
24
Time of the Year!
Price p
Figure 9: Estimation of CATE condi-
Figure 8: Estimation ATE.
tioned on T given 2
and consider estimating E [Y]do((X x)). This is also known continnous treatment effect or dose-response curve.
In the demand design problem, the ground truth is given by
E [Y]o(( p)] Er.s [fsuruet (p, TT
= (G - 2)p 1+p+ + 100 + 10G.
where Es ET (T)) The important point here that this quantity must be monotonically decreasing with respect to price P, since we should observe drop of demand as we increase the ticket price.
We estimate ATE by averaging the estimated structural function faruet over S and The estimated ATE given by DeepGMM, KTV, DeepIV, DFTV are shown in Figure 8. From Figure 8, we can see that KIV and DeepGMM fail to recover the monotonic structure in ATE. DeeplV and DFIV are able to capture this structure, but DFIV estimation is consistently better than DeepIV.
CATE Estimation Although ATE captures the treatment effect for the entire population, treatment effects may be heterogeneous for different sub-populations, which we might be interested in. In such case, we can consider CATE defined
CATE(O) E [r do(X 1),ō= -E Y do 1o(X=0),0-2
0)
for binary treatment X E {0, 1}. Here, is sub-vector of observable confounder O. Again, we generalize this idea and consider [Y do(X = 2), Ô - ō] which allows treatment X to be continuous. This quantity also known the heterogeneous treatment effect.
In demand design experiment, we can consider the CATE conditioned on the time of the year T. The true CATE defined as
E Y do(P p),T = = [stuue (p,t, S)] = =100 + (10 + p) [S] h(t) - 2p,
which can be obtained by averaging the estimated structural function fatrect over S. Figure shows the prediction of CATE with respect to T where treatment P is fixed to P = 25. Again, we can observe that KIV and DecpGMM fail to capture the shape of the ground truth curve. DeeplV performs significantly better but is less accurate than DFIV.
E.3 DATA GENERATION PROCESS IN DSPRITES EXPERIMENTS
Here, we describe the data generation process for the dSprites dataset experiment. This is image datasct described via five latent parameters (shape, scale, rotation, posX and posY). The images are 64 x 64 = 4096-dimensionall. In this experiment, we fixed shape parameter to heart, i.c. we only used the heart-shaped images. The other latent paramcters take values of scale E [0.5, 1], rotation E [0, 2T), posX E (0, 1], posY E [0, 1].
From this dataset, generate the treatment variable X and outcome Y follows.
21

---

Published as a conference paper at ICLR 2021
300
DeePGMM
Tut-of-Sample MSE
DFIV
200
100-
20
40
60
80
100
Epoch
Figure 10: Out-of-Sample MSE in dSprite experiment during the training
1. Uniformly samples latent parameters (scale, rotation, posX, posY).
2. Generate treatment variable X
X Fig(scale, rotation, poSX, posY) - 77.
3. Generate outcome variable Y as
Y AX 1 - 5000
+ 32(posy - 0.5) + E.
1000
Here, function Fig returns the corresponding image to the latent parameters, and m, e noise variables generated from 7, ~ (0.0, 0.1/) and e (0.0,, 0.5). Each element the matrix A E R 10x 4096 is generated from Un.f 1.0) and fixed throughout the experiment. From the data generation process, we can see that X and are confounded by posY. We the instrumental variable Z (scale, rotation, post) IR,, and figures with random noise as treatment variable X. The variable posY is not revealed to the model, and there is no observable confounder. The structural function for this setting is
|AA 5000
fstruct
1000
We use 588 test points for measuring out-of-sample error, which is generated from the grid points of latent variables. The grids consist of 7 evenly spaced values for posX, posY, 3 evenly spaced values for scale, and 4 evenly spaced values for orientation.
E.4 INSTABILITY OF DEEPGMM IN DSPRITES EXPERIMENTS
In the dSprite experiments, we observed that the training procedure of DeepGMM can be unstable. Figure 10 displays the MSE for the models learned in the first 100 epochs. Here, we can see that DeepGMM performs poorly in the early stage of the learning. Furthermore, even after appears to have converged, we observe a sudden increase of MSE around 80th epoch, which makes difficult to determine when to stop. We conjecture that this is duc to the instability of the smooth zero-sum game solved by DeepGMM. By contrast, DFIV converges quickly and performs consistently better than DeepGMM this task.
E.5 MNIST EXPERIMENTS
Here, we report the result of MNIST experiments proposed by Bennett et al. (2019), who consider the following datasets:
~~ [nif([-3,3]
e NN(0,)) 1), 7.8 NN(.0.11)
= Zi +
y |X +
22

---

Published as a conference paper at ICLR 2021
M
MNIST,
MNIST 82
De
.15 + .02
.07 + .02
.14 = .02
DFIV
.18 .01
.07 .001
.10 - .003
Table 1: Outple-Sample MSE in MNIST experiment of Bennett et al. (2019)
Catch Mountain Car
Cartpole
50
A
3
3
Table 2: Dimensions of BSuite tasks
Here, the structural function aim learn is fstruct (X) [X]. Additionally, map Z, X, or both X and Z to MNIST images to see whether the model can handle the high- dimensional trealment and instrumental variables. Let the output of original IV problem (mammmmmimxxxxmm..e
above to be Xow; Zoo and Txx) function that maps imputs to an integer between 0 and 9, and let Randomlmage(dd be function that selects a random MNIST image from the digit class d. The images are 28 x 28 784-dimensional. We consider the three following scenarios.
+ 5, 0),9) be a transformation
MNIST X f- Xlow; Z +- Randomlmagc((T Zlow))
MNIST.:: X <- (andomlmago((((X.))) Jow)), )), Z < Zlow
MNIST m:: X +- Randomlnagee( )), +- Randomlmage( x (Zow)) ))
We refer the reader to Bennelt et al. (2019) for detailed description. We applied DFIV using the same architecture as DeepGMM. In Table 1, we present the mean and standard error of out-of-sample MSE for DFIV and report the results obtained for DeepGMM in Bennett et al. (2019). From Table we can see that DFIV performs essentially like DeePGMM.
E.6
OPE EXPERIMENT DETAILS
E.6.1 BSUITE TASKS
We provide brief description below of the three behavior suite (BSuite) reinforcement learning tasks in Osband et al. (2019):
1. Catch: 10x5 Tetris-grid with single block falling per column. The agent can move left//rght in the bottom row to 'catch' the block. Illustrated in Figure 1la.
2. Mountain Car: The agent drives an underpowered car up a hill (Moore, 1990). Illustrated in Figure 11b.
3. Cartpole: The agent can move a cart left/right on plane keep a balanced pole upright (Barto et al., 1983). Illustrated in Figure 11c.
(a) Catch Figure 11: Three BSuite tasks. Figures are from Osband et al. (2019).
(b) Mountain Car
(c) Cartpole
All tasks have a real-valued state space C RD and a discrete action space AA (0. 1,2,....,
.,...,, A --1 1}. The state dimension and number of actions provided in Table 2.
23

---

Published as a conference paper at ICLR 2021
E.6.2 FITTED Q EVALUATION
Fitted Q Evaluation (FQE) (Le al., 2019) is simple variant of the Fitted Iteration (FQI) algorithm (Ernst et al., 2005). Instead of learning the Q function of an optimal policy as FQI, FQE estimates the Q function of fixed policy æ. It is an iterative algorithm with randomly initialized the Q function parameters By- At iteration k 2 with the current estimate of Q function Q" (s, al0k-1), it updates parameters 0g by solving the following regression problem using a least squares approach:
('' -0 (s,
(18)
+7 (Q (6 ,'100=))) Ptsa a''=
+r R(,,R.) [r].
where the regression function estimate is (s, (k), the observed outcome is the term on the LHS of (18) and the residual is the sum of terms in the second and third lines of (18). Comparing the equation above with (10), FQE reformulates the regression problem and moves the confounded part in the treatment of (10), that is 7Q" a'), to the oulcome. The regression problem al each ileration is therefore not confounded any more. FQE converges, it finds solution of the following equation
Q" (s, 0) = P(Erjss [r] + 7Bsra'ls IQ (s', a'')))
(19)
where P.-) is the T22 projection operator that maps function of (s, a) onto the (parameterized) Q function space.
F FAILURE OF JOINT OPTIMIZATION
One might want to jointly minimize the loss (lm), which is the empirical approximation of C. However, this fails to learn the true structural function fatmet, as shown in this appendix.
Figure 12 shows the learning curve obtained when we jointly minimize Ox and Oz with respect to Z("), where Ltest is the empirical approximation of
Lesst = Ex [llfatrnca (X) - (z)(")) T (x))P]
(20)
We observe in Figure 12 that the decrease of stage loss does not improve the performance of the learned structural function. This is because the model focuses on learning the relationship between instrument Z and outcome Y, while ignoring treatment X. We can see this from the fact that stage 1 loss becomes large and unstable. This is against the goal of TV regression, which is to learn a causal relationship between and Y.
On the other hand, Figure 13 shows the learning curve obtained using DFIV. Now, can see that stage 2 loss matches Lesst- Also, we can confirm that stage loss stays small and stable.
24

---

Published as a conference paper at ICLR 2021
Stage1 1 1015
Stape 2 003 Clad
Stage
Stage soa
-1-
-2
200
630
620
1000
Coochs
0
200
650
620
a20
1000
Spechs
Figure 12: Learning curve of joint minimiza- tion
Figure 13: Learning curve of DFIV
Table Network structures of DeepIV for demand design datasct. For the input layer, we provide the input variable. For the fully-connected layers (FC), we provide the imput and output dimensions. For mixture Gaussian output, we report the number of components. Dropout rate is given in the main text.
Instrument Net
Treatment Net
Layer
Configuration
Layer
Configuration
1
Input [C, S)
1
Input (P,T,
2
FC(3, 128), ReLU
2
FC(3, 128), ReLU
Dropout
3
Dropout
4
FC((28, 64), ReLU
4
FC(128, 64), ReLU
Dropout
5
Dropout FC(64, 32), ReLU
6
FC(64, 32), ReLU
6
7
Dropout
7
Dropout
MixtureGaausian(11)
8
FC(32, 1)
G NETWORK STRUCTURES AND
Here, we describe the network architecture and hyper param--areeeters of all experiments. Unless otherwise specified, all neural network-based algorithms are optimized using Adam with learning rate - 0.001, $1 = 0.9, B2 0.999 and E = 10-8
Demand Design For DeepIV, we used the original structure proposed in Hartford et al. (2017), which described in Table 3. We follow the default dropout rate in Hartford el al. (2017), which depends on the data size. For DFIV, used the structure described in Table 4. The regularizer d1, A2 are both set to 0.1 as a result the tuning procedure described in Appendix A. For KTV, we used the Gaussian kernel where the bandwidth is determined by the median trick described by Singh et al. (2019). We used random Fourier feature trick (Rahimi and Recht, 2008) with 100-dimensions. For DeePGMM, we used the same structure as DeepIV but no dropout is applied and the last layer of the Instrument Net is changed to cully-conneeted layer which maps 32 dimensions to dimension.
Demand Design with MNIST The feature extractor for MNIST image data given in Table 5, which is used for both stage 1 and 2. For DecpIV, we used the original structure proposed in Hartford et al. (2017), which is described in Table 6. We follow the default dropout rate in Hartford et al. (2017), which depends the structure described in Table 7. The regularizer A1. are both set to 0.1 as a result of the tuning procedure described in Appendix A. For KIV, we used the Gaussian kernel where the bandwidth is determined by the median trick and also used random Fourier features with 100-dimensions. For DeepGMM, we used the same structure DeepIV but no dropout
the data size. For DFIV, we used
25

---

Published as a conference paper at ICLR 2021
Table 4: Network structures of DFIV for demand design datasets. For the input layer, we provide the input variable. For the fully-connected layers (FC), we provide the input and output dimensions.
Instrument Feature Φo
Treatment Feature 4box
Layer
Configuration
Layer
Configuration
Input(C,T,s)) FC(3, 128), ReLU
1
Input(P)
2
FC(1, 16), ReLU
3
FC((28, 64), ReLU
3
FC(16, 1)
FC(64, 32), ReLU
Observable Feature too
Layer
Configuration
Imput(T, S) FC(2, 128), ReLU
4 FC(128, 64), ReLU
FC(64, 32), BN, ReLU
Table 5: Network structures of feature extractor used in demand design experiment with MNIST. For each convolution layer, we list the input dimension, output dimension, kernel size, stride, and padding. For the input layer, we provide the imput variable. For the fully-connected layers (FC), we provide the imput and output dimensions. For max-pool, we list the size of the kernel. Dropout rate here is set to 0.1. SN denotes Spectral Normalization (Miyato et al., 2018).
ImageFeature
Layer
Configuration
Input(S)
2
Conv2D (1, 64, 3, 1, 1), ReLU, SN
3
Conv2D (64, 64, 3, 1, 1), ReLU, SN
4
MaxPool(2, 2)
5
Dropout
6
FC(9216, 64), ReLU
7
Dropout
8
FC(64, 32), ReLU
is applied and the last layer of instrument net is changed to fully connected layer which maps 32 dimensions to dimension.
dSprite experiment For DeepGMM, we used the structure described in Table 8. For DFIV, we used the structure described in Table 9. The regularizer X1, A2 are both set to 0.01 as result of the tuning procedure described in Appendix A. For KIV, we used the Gaussian kernel where the bandwidth is determined by the median trick. We used random Fourier feature trick (Rahimi and Recht, 2008) with 100 dimensions.
OPE experiment For DeepIV, we used the structure described in Table 12. For DFIV, ; we used the structure described in Table 10. The regularizer Xı A2 are both set to 10-3 result of the tuning procedure described in Appendix A. For KTV, we used the Gaussian kernel where the bandwidth is determined by the median trick. We used random Fourier features (Rahimi and Recht, 2008) with 100 dimensions. For DeePGMM, we use the structure described in Table 11.
26

---

Published as a conference paper at ICLR 2021
Table 6: Network structures of DoepIV in demand design with MNIST data. For the input layer, we provide the imput variable. For the fully-connected layers (FC), we provide the imput and output dimensions. For mixure Gaussian oulput, we report the number of components. ImageFealure denotes the module given in Table 5. Dropout rale is described in the main text.
Instrument Net
Treatment Net
Layer
Configuration
Layer
Configuration
1
Input(C,T, ImageFeature(S))
Input(P,, (ma)
2
FC 32), ReLU
FC(66, 32), ReLU
3
Dropout MixtureGaussian(())
3
Dropout FC(32, 1), ReLU
4
Table 7: Network structures of DFIV in demand design with MNIST. For the input layer, we provide the imput variable. For the fully-connected layers (FC), we provide the input and output dimensions. ImageFeature denotes the module given in Table 5.
Treatment Feature tbax
Instrumental Feature Φ6z
Layer
Configuration Inpul(P)
Layer
Configuration [C, ImageFeature(S))
1
1
2
FC(1, 16), ReLU
2
FC(66, 32), BN, ReLU
3
FC(16, 1)
Obsevable Feature Net Eoo
Layer
Configuration Input([,, (
1
2
FC(65, 32), BN, ReLU
Table 8: Network structures of DeepGMM in dSprite experiment. For the input layer, we provide the input variable. For the fully-connected layers (FC), we provide the input and output dimensions. SN denotes Spectral Normalization (Miyato et al., 2018).
Instrument Net
Treatment Net
Layer
Configuration
Layer
Configuration
Input(2)
1
Input(X)
FC(3, 256), SN, ReLU
2
FC(4096, 1024), SN, ReLU
3
FC(256, 128), SN, ReLU, BN
3
FC(1024, 512), SN, ReLU, BN
4
FC(128, 128), SN, ReLU, BN
4
FC(512, 128), SN, ReLU
FC(128, 32), SN, BN, ReLU FC(32, 1)
5
FC(128, 32), SN, BN, Tanh FC(32,1)
6
6
Table 9: Network structures of DFIV in dSprite experiment. For the input layer, we provide the input variable. For the fully-connected layers (FC), we provide the input and output dimensions. SN denotes Spectral Normalization (Miyato et al., 2018).
Instrument Feature Φo
Treatment Feature vhex
Layer
Configuration
Layer
Configuration Ixp
Input(Z) FC(3, 256), SN, ReLU 3 FC(256, 128), SN, ReLU, BN
1
2
2
FC(4096, 1024), SN, ReLU
3
FC(1024, 512), SN, ReLU, BN
FC(128, 128), SN, RcLU, BN
4
FC(512, 128), SN, ReLU
5
FC(128 28, 32), SN, BN, ReLU
5
FC(128, 32), SN, BN, Tanh
27

---

Published as a conference paper at ICLR 2021
Table 10: Network structures of DFIV in OPE experiment. For the imput layer, we provide the input variable. For the fully-connected layers (FC), we provide the input and outpul dimensions.
Instrument Feature Φ9
Treatment Feature 40ox
Layer
Configuration
Layer
Configuration
1
Input(s, one-hot(a)) FC(*, 150), ReLU
1
Input(s, one-hot(a))
2
FC(*, 50), ReLU
3
FC(150, 100), ReLU
3
FC(50, 50), ReLU
4
FC(000, 50), ReLU
Table 11: Network structures of DeepGMM in OPE experiment. For the input layer, we provide the imput variable. For the fully-connected layers (FC), we provide the input and output dimensions.
Instrument Net
Treatment Net
Layer
Configuration
Layer
Configuration
Input(s, on)-hot(a))
Input(s, one-hot(a))
2
FC(*, 150), ReLU FC(150, 100), ReLU
2 FC(*, 50), ReLU
3
3
FC(50, 50), ReLU
4
FC(100, 50), ReLU
FC(50, 1), ReLU
5
FC(50, 1)
Table 12: Network structures of DeepIV in OPE experiment. For the input layer, we provide the input variable. For the fully-connected layers (FC), we provide the input and output dimensions. The MixutureGannsiann layer maps the input linearly to required parameter dimensions for mixture of Gaussian distribution with diagonal covariance matrices. The Bernoulli layer maps the imput linearly to single dimension to represent the logit of a Bernoulli distribution to predict if the next state terminating state.
Instrument Net
Treatment Net
Layer
Configuration Imput(s, on))
Layer
Configuration
1
1
Input(s, one-hot(a))
2
FC(*, 50), ReLU
FC(*, 50), ReLU
3
FC(50, 50), ReLU
3
FC(50, 50), ReLU
4
MixtureGGaussian(3 and Bernouli
4
FC(50, 1), ReLU
28


Reviews:
[{"review_id": "k5K3ltbKc63", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main idea in this paper is very nice and is as follows:\n\nSuppose we have the standard IV setup which is\nX = m(U) + f(Z) \nY = q(U) + b(X)\n\nWhere U is some unobserved confounder.\n\nTypically we are interested in the linearized version of this model. If we regress Y on X, we get a biased estimate of B(X) because U is a common unobserved cause. We can deal with this by first regressing X on Z, then regressing Y on the predicted values of X given Z. It is well known that using any other method than linear regression in this context leads to \"forbidden regression\" problems. So, typically people just estimate the average effect.\n\nHowever, with bigger data we might be interested in trying to get more precise estimates of the function b(X) and also f(Z). Typically this is done by hand-crafting a basis (interacting instruments with things, usually). The authors propose using deep networks to learn linear bases and then essentially do 2SLS on top of them (technically this is all trained together, but I am simplifying for the review).\n\nOverall, while I think there are many unanswered questions in this paper, I think it should be accepted for the community to build on it. IV is a tricky area and there is no way to show everything we need to know about a method in one paper.\n\nBelow I list several questions I had about the paper which I think the authors could incorporate in various formats (or educate me on how they have already answered them and I missed it):\n\n- Assumptions required\nThe authors state that the only assumption that is required is the standard exclusion restriction. I feel like that is not necessarily correct. For example, in 2SLS if we assume underlying heterogeneity (which, we must assume in this case otherwise why would we be learning a basis for this heterogeneity?) then if it the heterogeneity is not proxied by observed variables correctly we need some additional assumptions like monotonicity to get an interpretation of the IV estimate as some kind of causal effect (e.g. a LATE). Similarly, even with no covariates if we're trying to estimate the full function b(X), don't we need some assumption about how the IV affects X in the whole support in order to guarantee that we can use the IV to learn b(X)?\n\n- Regularization parameters \nAs I understand it, and the authors can correct me if I'm wrong, the hyperparameters for the regularizers are set by holding out some (x,z) and some (y, x) and then evaluating standard out of sample predictive loss? This is also what is done in DeepIV. This has been shown to not be an optimal way of choosing hyperparameters in Peysakhovich and Eckles (2018) which talks about \"Causal Cross-validation\" for adapting split-sample IV ideas from (Angrist & Krueger, 1995; Imbens et al., 1999; Hansen & Kozbur, 2014). The setting in that paper is specifically discrete instruments rather than continuous ones, so that procedure can't exactly be adapted in all the cases here, but it still may be worthwhile to mention as a future direction. Overall, hyperparameter tuning for these problems is really important and even more so when you have these deep feature generators.\n\n- Comparison to control functions / 2 stage residual inclusion\nCurrently the only comparison made are to other 2sls type methods where endogenous variables are changed to their predicted values. There is another way to estimate IV which is the control function/2 stage residual inclusion approach which seems to lend itself naturally to neural networks/function approximators since all you need to do is include the residual from the first stage into the second stage to get a correct estimate of b(X). It seems like it would be pretty easy to add this as a baseline to at least the high-D demand experiment.\n\nTypo:\nThe abstract states: \"In this case, deep neural nets are trained to define informative nonlinear features on the instruments and treatments.\" I believe this should say \"linear features\" in the sense that you are learning the correct basis for 2SLS?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "bofvPzZiDla", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I liked reading this paper. It is reasonably well-written and tackles an important problem in an interesting way. I list the issues I had with the paper below. I believe one advantage of DFIV is that it does not rely on some relaxed optimization problem like DeepIV. The main idea in the paper is learning a set of basis functions such that the structural function is a linear combination on them; the learning itself relies on predicting the basic functions from the IV which ensures that confounding information is projected out.\n\nFirst, while I think the method is well motivated, I am confused by the discussion around 'forbidden regression'. I believe the statement made in the paper about 'high variance' may be misleading. 'Forbidden regression' is a statement about the mis-specification of the conditional density model which may lead to an identification failure. Can the authors point out the exact part of Angrist where high variance is called the 'forbidden regression' problem?\n\nI think the proposed method does not face the forbidden regression problem because of the linear relationship between the IV and the outcome implied by the proposed model. \n\nSecond, I believe it would be in support of the method to expand the experiments to include the high-dimensional IV and treatment experiments from DeepGMM. \n\nI would like the authors to clarify the following:\n\n1. In 4.2, if the image is given as treatment to the model, isn't the confounder posY be specified as a function of the treatment?\n2. In the OPE experiment, the DFIV (and DeepGMM to a lesser extent) does not  exhibit monotonic behavior with noise increase. Could the authors explain if this is randomness? if it is not, could the authors provides results over a larger number of seeds?\n\nFinally, could the authors expand on any convergence issues during training due to the coupled optimization and how they fixed them? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "ESIsRvwRYA8", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes the deep feature instrumental variable algorithm that has two stages of regressions and uses deep features. This paper demonstrates that this algorithm has a good performance on several applications.\n\nThis paper is generally well written. Moreover, the experimental results look promising. However, this paper does not theoretically justify its solution that is in general relevant in the IV literature. Instead, this paper emphasizes its algorithm's prediction performance.  It needs some clarification and justification on why the results in this paper can help people better identify and correctly estimate causal effects, which is the primary goal of IV (inference is important in IV). Also, there seem to have some issues with the assumption. \n\nMajor comments:\n\n1. Identification: The purpose of IV is to identify and consistently estimate the causal effect in the presence of the unobserved confounders (Angrist et al., 1996). How can we uniquely identify the treatment effect using the DFIV algorithm? And why is the causal effect estimated from the DFIV algorithm consistent? As a reference, causal effects estimated from 2SLS are consistent and asymptotically normal. Moreover, Theorem 2 in Bennett et al (2019) shows the estimated parameters (and treatment effect) converge in probability to the true value. Since IV is not generally used for prediction problems, a careful discussion on identification (and consistency) will be helpful.\n2. Assumption 1: In 2SLS, Assumption 1 ($\\mathbb{E}[\\varepsilon|Z]=0$) is sufficient to prove the estimated treatment effect is consistent and asymptotic normal because the model is linear. However, this assumption is not sufficient for the DFIV algorithm because $\\phi(Z)$ is nonlinear. It seems the correct assumption is if $\\phi(\\cdot)$ is unknown, then $\\varepsilon$ needs to be independent of $Z$; if $\\phi(\\cdot)$ is known, then $\\mathbb{E}[\\varepsilon|\\phi(Z)]=0$.\n3. Off-policy policy evaluation experiment:  First, what is the treatment variable in the problem? Second, why does this application use absolute error while the demand and dsprites application use out-of-sample MSE? Third and most importantly, the IVs used in this paper are $Z=(s,a)$.  Why is $\\mathbb{E}[\\varepsilon|Z]=0$ or $\\varepsilon$ independent of $Z$ given $\\varepsilon$ is also a function of $s$ and $a$ because in the two expectations in $\\varepsilon$, $s^\\prime  \\sim P(\\cdot|s, a)$ and $r\\sim R(\\cdot|s,a)$? \n4. dSprites experiments: The outcome variable $Y$ is quite important when people use IV, but it is not stated in the main text that seems to be confusing. What is the interpretation of $Y$ based on the DGP provided in Appendix D.2? Why should we care about this $Y$?\n\nMinor comments:\n\n1. Typo: P2. six to the last line: \". an unobserved confounder\" -> \". $Z$ is an unobserved confounder\". \nP2. last line: \"which we assume to be a continuous\" -> \"which we assume to be continuous.\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "42YWhgPkJTm", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors consider the problem of learning the structural equation governing the relationship between a treatment and outcome in a causal model. They assume an instrumental variable setting and use neural networks to fit non-linear models as part of the conventional 2SLS approach to parameter estimation in IV models.\n\nIn general I found the paper to be pretty clearly written and I think it shows promise. That said, there are some points of confusion/clarification discussed below. In light of these, I've opted to conservatively score this paper as a weak reject for the pre-rebuttal/pre-discussion phase and will reevaluate and consider raising my score later in the review process.\n\nDetailed comments (major concerns denoted by **):\n\nIn the intro (paragraph 3) the authors describe the characteristics necessary for something to be a valid instrument. I recommend also adding that the instrument Z cannot have an incoming edge from the latent confounder that is also a parent of the outcome.\n\nIn the intro the authors state \"Although these emthods enjoy desireable theoretical properties, the flexibility of the model is limited, since existing work uses prespecified features\". Can the authors clarify what's meant by \"prespecified\"? It's not obvious how exactly this is a limitation.\n\nIn a similar vein, later in the intro, the authors refer to DeepGMM as having an \"unstable\" learning procedure since it involves solving a smooth zero-sum game. The authors should clarify why this is the case, since many readers, especially in the more traditionalist IV community, may not be familiar with the subtleties of recent competitor methods like DeepGMM.\n\n**How does the assumption of additive noise constrain the problem? What happens if this assumption is dropped? Is the structural function then not identifiable? The authors don't seem to use this assumption later in their argumentation for why their method works and so I'm curious whether either i) the assumption isn't necessary or ii) the assumption is trivializing the problem in some way. In general additive noise is a pretty strong assumption and so this could drastically limit the efficacy of the proposed method.\n\nIn line with above comments about clarifying background details about competitor models, can the authors provide more background on Hermite polynomials?\n\n**I'm a bit concerned about the use of regularization in the fitting procedure (Equations 6 and 8). Both of these models are _nuisance_ models. While the authors frame the problem as trying to learn the structural function that governs the X->Y relationship, ultimately what a practitioner cares about is evaluating effects like E[Y(x) - Y(x')]. Traditionally, one has to be very careful about employing regularization in causal modeling since you might regularize an effect to zero in the service of trying to improve prediction performance for the nuisance model. As a toy example, consider a simple linear causal model where Y = Beta X. If the true effect for some element of X, say X_2, is small but nonzero (i.e. epsilon > Beta_2 > 0), then regularizing could make us learn a \\hat{Beta} such that \\hat{Beta}_2 == 0 which is a biased effect estimate. Perhaps the dynamics are different here because of non-linearities and the regularization does help empirically with the problem the authors consider. But in general I'm not sure that's the case -- the authors should clarify this point in their argumentation and think carefully about this issue in the context of their experiments.\n\n**Some questions about sample efficiency. The authors give some commentary on the computational complexity. This is certainly important, but typically of equal or greater concern is whether the models can actually converge to the true parameter. As far as I know, there are few (if any) results on the sample efficiency of NNs in general and so it's hard (impossible) to say whether or not they are consistent (converge to an unbiased estimator of the true parameter). Is there a sense in which this can be known for the class of NNs the authors consider? What is the rate of convergence of the fitting procedure?\n\nIn the airplane experiment, why does time range from 0 to 10 rather than, say 0 to 11 or 1 to 12?\n\nIs there a substantive interpretation for rho in the airplane example? Is there a reasonable latent confounder that this might correspond to that does not act as a causal parent of Z? I realize that it exists as an artifact of the simulation DGP from the Hartford paper but it makes the example somewhat weaker that it might not be something that can be mapped to a fully realistic modeling problem.\n\n\"This may be due to the current DeepGMM approach to handling observable confounders, which may not be optimal\" -- what is the 'current approach' and why is it suboptimal?\n\n**In the DSprites experiment, what is the outcome Y?\n\n**The authors should strongly consider adding an experiment in which they actually intervene on X. The task of learning f(X) is very close to estimating an effect, but I'm curious to know i) how well their algorithm actually does for effect estimation and ii) whether their model's outperformance relative to competitors is as drastic in that situation. In that case, because they are using a synthetic DGP, it would be possible to know the ground truth effect of the intervention and thus they'd have a principled way to evaluate the performance on the effect estimation task.\n\nTo the other reviewers/ACs, I'm not really qualified to evaluate the experiment and claims discussed in section 4.3 regarding off-policy evaluation. To the authors, however, in line with above comments about defining terms from the literature, many readers will not have a clue what things like 'catch and mountain car' are or what the details of the FQE model are and so you should strongly consider clarifying the task and details of the competitor model if space allows.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated the paper's applied neural net approach to the problem of designing features for 2SLS regression for IV analysis as an alternative to sieve approaches. The paper would make a good contribution to ICLR. While the paper does not focus on theory -- learning data-driven features appears to be mostly heuristic -- it should still be grounded in a sound approach to the IV problem, and the reviewers recommend various important technical clarifications regarding the foundations of IV models; the authors should implement these suggestions very carefully and correctly in future versions. For example, even if the structural models are well-specified in that Eq. (5) holds for some parameters, since the dependence is non-linear on parameters, it is not clear when we should expect this to be identifying of (theta_X,theta_Z) (these are in fact not identifiable in general) and when we should expect the proposed method to be consistent.
Decision: Accept (Poster)