Deep Multi-Modal Structural Equations For Causal Effect Estimation With Unstructured Proxies
Authors: Shachi Deshpande
Kaiwen Wang
Dhruv Sreenivas
Zheng Li
Volodymyr Kuleshov
Venue: NeurIPS 2022 Accept (2022)

Abstract: Estimating the effect of intervention from observational data while accounting for confounding variables is a key task in causal inference. Oftentimes, the confounders are unobserved, but we have access to large amounts of additional unstructured data (images, text) that contain valuable proxy signal about the missing confounders. This paper argues that leveraging this unstructured data can greatly improve the accuracy of causal effect estimation. Specifically, we introduce deep multi-modal structural equations, a generative model for causal effect estimation in which confounders are latent variables and unstructured data are proxy variables. This model supports multiple multimodal proxies (images, text) as well as missing data. We empirically demonstrate that our approach outperforms existing methods based on propensity scores and corrects for confounding using unstructured inputs on tasks in genomics and healthcare. Our methods can potentially support the use of large amounts of data that were previously not used in causal inference

Deep Multi-Modal Structural Equations For Causal Effect Estimation With Unstructured Proxies
Shachi Deshpande Kaiwen Wang :2 Dhruv Sreenivas", Zheng Li Volodymyr Kuleshoy Department of Computer Science, Comell Tech and Cornell University {ssd86, kw437, ds844, z1634, kuleehhor@@comeeel.ccm
Abstract
Estimating the effect of an intervention from observational data while account- ing for confounding variables is a key task in causal inference. Oftentimes, the confounders are unobserved, but we have access to large amounts additional un- structured data (images, text) that contain valuable proxy signal about the missing confounders. This paper argues that leveraging this unstructured data can greatly improve the accuracy of causal effect estimation. Specifically, we introduce deep multi-modal structural equations, generative model for causal effect estimation which confounders
latent variables and unstructured data proxy variables.
This model supports multiple multi-modal proxies (images, text) as well as missing data. We empirically demonstrate that our approach outperforms existing methods based on propensity scores and corrects for confounding using unstructured inputs tasks in genomics and healthcare. Our methods can potentially support the of large amounts of data that were previously not used in causal inference.
Introduction
An important goal of causal inference is to understand from observational data the causal effect of performing an intervention-44 .. the effect of behavioral choice on an individual's 's health [36] As initial motivating example for this work, consider the problem of determining the effect of smoking on an individual's risk of heart disease.
This problem is complicated by the presence of confounders: e.g., it is possible that individuals who smoke have higher likelihood to be sedentary, which is lifestyle choice that also negatively impacts their heart disease risk. If the individual's 's lifestyle is available we may adjust for this factor while computing treatment effects [36]. However, confounders are often not observed and not available in the form of features, making accurate causal inference challenging.
well-defined feature,
Oftentimes, datasets in domains such as medicine or genomics come with large amounts of wn- structured data-e.g. medical images, clinical notes, wearable sensor measurements 7 This data contains strong proxy signal about unobserved confounding factors-c.g, wearable sensor measurements help reveal individuals who are sedentary. However, existing causal inference methods are often not able to leverage this "dark data" for causal effect estimation [8]
The goal of this paper is to develop methods in causal inference that improve the estimation of causal effects in the presence unobserved confounders by leveraging additional sources of unstructured multi-modal data, such as images and text. For example, given time series from patients' wearables, our methods may disentangle the effects of being sedentary from the effects of smoking by using clusters in the sensor measurements (which would correspond to groups of active and sedentary individuals) as a proxy for the patients' lifestyles and without requiring explicit lifestyle features.
Concretely, our paper formalizes the task of estimating causal effects using rich, unstructured, multi- modal proxy variables (e.g. images, text, time series) and introduces deep multi-modal structural
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

equations, generative model in which confounders are latent variables. This model can perform causal effect estimation with missing data by leveraging approximate variational inference and learning algorithms [6] developed for this task. Previous methods relied on propensity scoring with neural approximators [41
50 5I which may output volatile probabilistic outputs that lead to unreliable effect estimates [8]. Our methods are generative, and thus naturally sidestep some of the aforementioned instabilities, support high-dimensional treatment variables and incomplete data, and as result are applicable to a broader class of causal inference problems.
We evaluate our methods on an important real- weal-world causal inference task- eask-estiating the effects of genetic mutations in genome-wide association studies (GWASs) as well as on benchmarks derived from popular causal inference datasets. The intervention variables in a GWAS are high-dimensional genomic sequences, hence existing methods based on propensity scoring are not easily applicable. In contrast, we demonstrate that our algorithms naturally leverage high-dimensional genetic and environmental data (e.g., historical weather time series) and can discover causal genetic factors in plants and humans more accurately than existing GWAS analysis methods.
Contributions In summary, this paper makes three contributions: (1) we define the task of esti- mating causal effects using rich, unstructured multi-modal proxy variables; (2) introduce deep multi-modal structural equations, a generative model tailored to this problem, and we describe associ- ated variational learning and inference algorithms; (3) we demonstrate on an important real-wor worldworld problem (GWAS) that unstructured data can improve causal effect estimation, enabling the use of large amounts of "dark data" that were previously not used in causal inference.
2 Background
Formally, we are given an observational dataset DD = {(x11))))))))))))) (0 (000,,0)))) ) consisting of individuals, each characterized by features z(() E X C Rd, a binary treatment ( E [0, and scalar outcome (( E R. We initially assume binary treatments and scalar outcomes, and later discuss how our approach naturally extends beyond this setting. We also use 2 E R2 to model latent confounding factors that influence both the treatment and the outcome
Notation
We are interested in recovering the true effect of T = t in terms of its conditional average treatment effect (CATE), also known as the individual treatment effect (ITE) and average treatment effect (ATE).
28
(1) h Many methods for this task rely on propensity scoring 41 50 51], which uses model p(4]ar) to assign weights to individual datapoints; however, when
Y|2, t = Y = x, do = =t)]
ITE(()) = Y|x, 1 Y(z,0) ATE = (ITT(())),
where do(-) denotes an intervention
x is high-dimensional and unstructured, neural approximator for p(t[x) may output volatile and miscalibrated probabilities close to {0, 1} that lead to unreliable effect estimates [[]]
Structural Equations
An alternative approach are structural equation models of the form == f2(2,52)
(2) where Z ~ p(Z) is drawn from a prior and the E, are noise variables drawn independently from their distributions [10]. Structural equations define a generative model p(æ, y, z, t) of the data. When this model encodes the true dependency structure of the data distribution, we can estimate the true effect of an intervention by clamping to its desired value and drawing samples.
== (2,
= ff(2zt,,3)
Deep Structural Equations Equations 2 can be parameterized with deep neural networks, which 28 Expressive neural networks may learn a more
yields deep structural equation models accurate model of the true data distribution on large datasets, which improves causal effect estimation. Such models have been used for GWAS analysis [49 and to correct for proxy variables 28).
Causal Effect Estimation With Unstructured Proxy Variables
Oftentimes, datasets in domains such as medicine or genomics come with large amounts of unstruc- tured data (medical images, clinical notes), which contains strong proxy signal about unobserved confounding factors. Our paper seeks to develop methods that leverage unstructured data within causal inference. We start by formalizing this task as causal effect estimation with unstructured proxy variables; these proxies may come from multiple diverse modalities (images, text).

---

3.1 Task Definition
Formally, consider causal inference dataset D (0)
[(x(), 1le00y00 y00, [t)) in which x0)) (1)
(x) 42 ................................ Fm is a vector of m distinct input modalities zjj E X, (e.g.. images, text, time series, etc.). In other words, X = =X, x... x Xm. where each X corresponds to a space of images, time series, or other unstructured modalities, Here, g(i) E 'J (binary or continuous) is the treatment and (+)) E y is the output. Some modalities may also be missing at training or inference time.
We are interested in recovering the true effect of in terms of the individual and average treatment effects. We are specifically interested in estimating the individual treatment effect (ITE) from arbitrary subsets of modalities M {1, 2, .... m. indicating that certain inputs may be missing at test time. ((z, 1, M] = E[Y]do((? 1), X3 Iz for j in M] ITE( M) Y(e, /
1, M Y ( 0, M
To help make this setup more concrete, we define two motivating applications.
Consider the task of determining the effect of smoking on heart disease from an observational dataset of patients. The observational study may contain additional unstructured data about individuals, e.g., clinician notes, medical images, wearable sensor data, etc. This data may hold information about hidden confounders: for example, raw wearable sensor data can be clustered to uncover sedentary and active indivudals, revealing a latent confounding factor, sedentary lifestyle.
Healthcare
Genomics Consider the problem of estimating the effects of genetic variants via a genome-wi wide association study (GWAS). Modern GWAS datasets in plants or humans feature large amounts of unstructured inputs 7 clinical notes, medical records, meteorological time series. For example, historical weather data (e.g., precipitation, wind strength, etc.) can reveal distinct climatic regions that affect plant phenotypes and whose confounding effects should be corrected for in a GWAS (55]
4 Deep Structural Equations for Causal Effect Estimation
Next, we derive models and inference algorithms for the task of causal effect estimation with unstructured proxy variables. Our approach uses deep structural equations to extract confounding signal from the multi-modal proxies (1) We use neural networks because they naturally handle unstructured modalities via specialized architectures (e.g., convolutions for images) that can learn high-level representations over raw unstructured inputs (e.g., pixels).
Parameterizing structural equations with neural networks also presents challenges: they induce complex latent variable models that require the development efficient approximate inference algorithms [C]
We present an instantiation of Equations 2 that admits such efficient algorithms.
4.1 Deep Multi-Modal Structural Equations
We start by introducing deep multi-modal structural equations (DMSEs), a generative model for estimating causal effects in which confounders are latent variables and unstructured data are proxy variables. We define a DMSE model follows:
(4) where Pay,Py Py are probability distributions with tractable density over Ty and y. respectively, and the Oz,, O2 are the parameters of Pæ, Py. The Oz 0, are themselves functions of 2, parameterized by neural networks-e..
2 N~,:00:4pp)
2 ~ ~ Ps, (- (z) Vj
t~ te)(TT((
u>pp(-;0,(z,))
when Ty is Gaussian, the (2) are a mean and a covariance matrix (zy(2) Ex,(2) (z) that are parameterized by a neural network as function of (see [21 28 ). Note that other modeling choices (e.g., Bernoulli distributions for discrete variables) are also possible.
Note that the models for (z) can benefit from domain-specific neural architecture--e.g convolutional parameterization for Pz,, (z). oz 2, (2) as function of 2 is more appropriate when the zy are images. See Appendix B for details on recommended architectures.
While Section 3.1 defines y, / as scalars following existing literature 28
DMSEs can also 54 50
define a model with high-dimensional y, l-we simply choose the distributions over y, to be multi- variate. Our inference and learning algorithms will remain unchanged, except for the parameterization
3

---

of specific approximate posteriors (e.g., g(2g.. t)). In fact, we apply DMSEs to high- dimensional in our GWAS experiments in Section Note that this is a setting where existing propensity scoring methods (which learn a model of p(4))) are not directly applicable [4 550 our approach, on the other hand, can easily be used with high-dimensional on tasks like GWAS analysis.
Dependency Structure Equations define density p(=)p(t|z)p( 2, [T==1 p(zzíz). Note that each proxy z is independent of the others conditioned on z. Figure shows these dependencies as solid lines. In our setting, the z, represent image pixels, waveform measurements, etc: thus, they should not directly influence each other or f. For example, it would not make sense for the pixels of an image causally influence the samples of waveform measurement- influence each other only through a latent confounder (e.g., patient health status), and the x are therefore assumed to be conditionally independent given 2. These independence assumptions will also enable us to derive efficient stochastic variational inference algorithms, as we show below.
In certain settings, we may also want model observed confounders as well as additional strucfured proxies 2ij that have direct causal ef- fects on each other and on f. Figure D shows these additional dependencies as dashed lines. Our model admits simple generalizations that support these modeling assumptions. In brief, our variational inference algorithms can be con- ditioned on the observed variables, and mutu- ally dependent sets of proxies can be treated as a group represented by one high-dimensional variable. See Appendix for details.
X1
X2
X
T
Z
Figure 1: Causal graph for the DMSE model. Solid lines depict dependencies between variables. Ap- pendix H contains simple extensions of DMSEs that support extra dependencies (dashed lines).
4.2 Approximate Inference and Learning Algorithms
In the full multi-modal setting, the Iy are conditionally independent given z (Figure D. which enables us to apply efficient algorithms inspired by Wu & Goodman [56]. These algorithms offer the following improvements: (1) we may perform learning, inference, and causal estimation with missing modalities zy; (2) the process for performing causal inference does not require training auxiliary inference networks as previous work (8
The DMSE model induces a tractable joint density p(r, y. f,, z), which allows us to fit its parameters using stochastic variational inference by optimizing the evidence lower bound (ELBO):
m
ELBOx
Σ
log pz "9) 10, - logg(e)xx() y (0))
(5)
where ()) 20
2) p) (()), tional posterior. We assume a total of m modalities.
12)p(() and ((212(0),))))))))))))))))))))))) ,100) (0)) is the approximate varia-
Structured Multi-Modal Variational Inference We may use the independence structure of (Figure D to derive an efficient structured form for q. First, observe that because the true posterior factorizes as p(z|2, v) x (p(z)4, v) I1 p(=l=;))/ (eel=)))[[[==> p(=), the optimal approximate posterior q must also factorize as q[=|ar, (, xx (g(=)4,)) v) I[ ((=lx,)) I1;-- p(z). This decomposition implies that we can maintain the optimal structure of by training mpdality-peecific inference
networks ğ(z y) and {(z/2,) such that q(=(r))) ((2lx;))(() and g(2(t, v) = 0[2 u))(2) and
by defining a joint posterior as
v2l1,, x ((2) t) II ((
(6)
This network can be seen as a product of experts (PoE) [56]
Computing the density q is in general not possible. However, because p(z), a(2((,,) v) and q(=(1;)) are Gaussians, we may use the fact that a product of Gaussians with means μ4 and covariances V;
is (E,TT)/ T,) and V (E, T where T; 1/V.. Thus, computing q(2 xr, t, y) for

---

any subset of modalities is possible without having to train an inference network for each subset of modalities separately.
A Multi-Modal ELBO Training this architecture with ELBOx will not necessarily yield good eaggllty inference networks (z)) while training with each modality separately will prevent the network from learning how the modalities are related to each other. Hence, we train our model with a sub-sampled ELBO objective (56 that is computed on the full set modalities, each individual modality, and few subsets of modalities together. For this, we randomly pick at each gradient step s non-empty subsets {M& k==1 of the set of modalities M2 S 21,2...., m]. The final objective
ELBOx اド= ELBO (3) + k= ELBOM,, where
ELBOM = Σ Big
(ogp(a)) 2²) + log (,000, 1og(p()))(=11)) {0)) II
((=laf)))
JEM
JEM
Our model doesn't 't train extra auxiliary inference networks, unlike that of Louizos et al. 128
4.3
Deep Gaussian Structural Equations
The DMSE model can be simplified in settings in which there is only one type of proxy I (i.e., m 1). This simplified model, which call deep Gaussian structural equations (DGSEs), has tractable joint density p(r, 4, 2) = p( ())p:r))))((()))ppppl....))))))/ (), where the latent z is Gaussian.
The DGSE model can also be fit using stochastic variational inference by optimizing the ELBO ob- jective Σ;=1 Eg [log ((z(1), ( t(()) 2) log (=2|x(1), , { where (<222, y, t) is an approx- imate variational posterior. We optimize the above objective using gradient descent, applying the
reparameterization trick to estimate the gradient. We compute the counterfactual Y Y|2, (] using
auxiliary inference networks as in earlier work 28
See Appendix for the full derivation.
4.4 Properties of Deep Multi-Modal Structural Equations
Recovering Causal Effects The DMSE and DGSE models determine the true causal effect when their causal graph correct and they recover the true data distribution. The following argument is analogous to that made in most previous works on causal deep generative models [28, 57 351
Theorem 1. The DMSE and DGSE models recover the true ITE(x, M) for any subset M C (1,2, ..., m of observed modalities whenever they represent the true data distribution p(a, y, 2).
Proof: We establish the theorem for DMSEs; the proof for DGSEs is analogous with m = 1. Let IM {ay j E M) be the data from the observed subset of modalities. We need show that p(yleM, do = t²)) is identifiable for any t'. Observe that
p(ylTM, do = )²)) = plul2, TMM, do = t'))p((( (iftt=(]]dd - ((vll p0 XM,, e' ())(((
where the second equality follows from the rule of do-caleulus (applying backdoor adjustment). Since our proof holds for any t' and all elements on the right-hand side are identifiable, the claim follows.
Note that in practice and may fail), but there is evidence of both failure modes [4 as well as successful settings in which
assumption may not hold (e.g., neural network optimization is non-convex
deep latent variable models provide useful causal estimates 571 section for additional details.
See our Discussion 30
48
Identifiability in Linear Models Structural equations parameterized by non-convex neural net- works are less amenable to analysis than simpler model classes. However, we may provide theoretical guarantees in the special case where DMSE model (Equations 4 is linear, i.c., each equation with input variables u E dgd; has the form A u + 6 for some A E dd Xd1, E IR.. Specifically, we establish in Appendix the following result.
5

---

Theorem 2. Given a binary treatment t, univariate outcome y, confounder & and proxy variables u, U, w, the causal effect P(y)do(t) is identifiable if
1. The structural equations follow a DMSE model (Figure solid edges) and are linear.
2. Three independent views of are available the form of proxies u, v, w such that u I v 1 w|z and the equations between and u, U, w are parameterized by matrices of rank dim(z)
Our proof extends techniques developed by Kuroki and Pearl PP] to high-dimensional proxy variables. Interestingly, our result crucially relies on independence structure specified by Figure ( (specificalll, the existence of three independent proxy variables), which lends additional support for this modeling assumption and for the development of variational techniques specialized to this model family.
Computing Causal Effects Given subset of modalities M, we can compute the ATE & ITE as E(ITE(z, M)), where ITE(2, M) (xx t 1.M] Y [x, 0, 0, and
plyl, do(t = t')) = x';t==tt))(())))d= / (piult = (p(u)t=t(,=)dd =))))(())))))))))))))))))))()))) II
m
(clo;))) (7) j=1
where variational posterior formulation from Equation o approximate the true posterior p(z[z).
5 Experimental Results
5.1 Synthetic Demonstration Dataset
We start with demonstration that provides intuition for why proxy variables are important, and how unstructured proxies can serve in place of featurized (structured) ones. The following small-scale synthetic setup (Louizos et al. (2 involves data distribution over binary variables y, t, 2,
(P 1) P(= 0) = 0.5
P(r 1/= 1) = P=1 0.3
(a = 1| 0) P 0.1
y = 1 ④
IP(L 1|: = Pe1 0.4 IP(1 = 1| 0) Pro 0.2
where 0 < Pa1, PaO. Pri, Pro < are parameters. We also introduce an unstructured proxy variable X that represents an "image version" " of æ. The variable X will be a random MNIST image zero one, depending on whether 0 or 1. Formally, X is distributed as follows:
(PX 1) is unif. over MNIST images of '1' P(X 0) is unif. over MNIST images of '0'
First, this setup that requires us to model proxies : treating X as a con- founder as using model of P(y X, t) recovers the true ATE only when Pt1 = Pta and Px1 = 1 Pzo (i.e., when X is perfectly informative of 2), otherwise it fails (see also Appendix
Table 1: Treatment effect estimation on the synthetic demonstration dataset.
Setting
SATE (Train)
EATE (Test)
Deep Str Eqns
Binary
0.062 (0.012)
0.069 (0.015)
Image
0.068 (0.018)
0.096 (0.018)
We also show that struetural equations solve this task. We sample 3000 data points from and fit DGSE models to 80% of the data points {z, y, t} (the B1- NARY setting) as well as on (X, y, t] (the IMAGE setting). We note the Aver- age Treatment Effect (ATE) on the train- ing and test sets, and we report results in Table We compare DGSE with the Inverse Probability of Treatment Weighted estimator (IPTW) 29 and doubly robust Augmented-IPTT
IPTW
Binary
0.090 (0.005)
0.127 (0.016)
Image
5.050 (0.607)
4.067 (0.53))
Augmented IPTW 45
Binary Image
0.442 (0.040)
0.487 (0.037)
4.717 (0.670)
6.426 (1.603)
Non-Causal
Binary
0.197 (0.003)
0.206 (0.004)
Baseline
Image
0.214 ( (.026)
0.228 (0.025)
45] each case the propensity score model is an MIP trained to predict / from either or X. We found that replacing with an image X causes the model to output highly miscalibrated probabilities close to 0, (while maintaining good accuracy), which results in large and volatile inverse propensity weights and in poor ATE estimates.

---

5.2 Benchmark Datasets for Causal Effect Estimation
IHDP The Infant Health and Development Project (IHDP) is a popular benchmark for causal inference algorithms [33] that contains the outcomes of comprehensive early interventions for premature, low birth weight infants. We create a benchmark for multi-modal causal inference based on IHDP in which we can replace certain features with their "unstructured version". We choose 9 of the 25 features available in IHDP in order to magnify their relative importance and accurately measure the effects of their removal. Please refer Appendix D and E for detailed setup.
The Student-Teacher Achievement Ratio (STAR) experiment ① studied the effect of class size on the performance of students. We consider small class size as treatment; the outcome is the sum of the reading and math scores of a student. We "derandomize" this dataset by removing 80% of the data corresponding to white students in the treated population. Similarly to IHDP, we select 8 attributes for the multi-modal experiment. Further details can be found in the Appendix D and
STAR
Table 2: Multimodal Experiments on IHDP Dataset: With deep structural equations, replac- ing baby's s gender with corresponding image embedding (8 attrs + image) shows some in- crease in ATE error compared to IHDP-Mini setting (9 attrs) but is better than dropping this modality altogether (8 attrs).
Model
SATE (Train+ Val) ATE error(Test)
Deep Str Equs
Adding Unstructured Modalities We create a benchmark for multi-modal causal inference de- rived from IHDP and STAR in which we replace features with unstructured inputs that contain the same information as their featurized versions. On IHDP, we replace the attribute 'baby's s gender' with the CLIP embedding 244 of an image of child between ages 3 to years, drawn from the UTK dataset [11. On STAR, we replace the attributes corresponding to the student's ethnicity and gen- der by selecting image of child with the same ethnicity and gender from the UTK dataset.
9 attrs
0,259 (0.037)
0.487 (0.078)
8 attrs
0.392 (0.141)
0.620 (0.158)
8 attrs + image
0.372 (0.107)
0.575 (0.130)
CFRNet
9 attrs
0.433 (0.063)
0.549 (0.090)
8 attrs
0.412 (0.062)
0.608 (0.107)
8 attrs image
0.501 (0.076)
0.617 (0.114)
OLS
9 attrs
0,424 (0.061)
0.584 (0.
8 attrs
0.429 (0.066)
0.593 (0.103)
8 attrs + image
0.428 (0.064)
0.590 (0.101)
We train and evaluate models on datasets where the image 'replaces' the attribute. (e.g., 8 ATTRS + IMAGE). We also consider two other settings for comparison: a) the original attribute is included (e.g.. 9 ATTRS) and b) the attribute is dropped from the reduced set of input features (8 ATTRS).
Results As seen in Table 2 the degradation in ATE error from replacing the baby's 's gender by a photograph is lower as compared to removing the attribute entirely. This shows our models leverage signal found the unstructured image modality with the help of deep neural networks.
We compare these results with simple Ordinary Least Squares model (OLS) baseline as described by Shalit et al. 47 to predict treatment effect. OLS shows a similar behavior when replacing baby's gender with corresponding image, however ATE eтrоrs are generally worse as compared to ATE eтror produced by DGSE. We also compare this with Counterfactual Regression Network (CFRNet) [16] baseline. However CFRNet did not show benefits of using image modality unlike our approach.
In Table replacing gender and ethnicity attributes on STAR with the corresponding image improves ATE errors as compared to dropping these two attributes entirely. This shows that we can use an image to extract multiple attributes while doing causal inference. The CFRNet baseline shows similar behavior, but the difference between average ATE errors across different setups is small.
5.3 Genome- Sendee-Wiie Association Studies
We evaluate our methods on an important real-world causal inference problem genome-wide association study analysis (GWASs). GWAS is a large observational study that seeks to determine the causal effects of genetic markers (or genotypes) on specific traits (known as phenotypes). In this setting, treatment variables are high-dimensional genomic sequences, and existing propensity scoring methods (which learn model of p(t[z) for a binary t) are not casily applicable 4] 50
they may
7

---

require training an impractical number of models. Our approach, on the other hand, can easily be used with tigh-dimensional. t.
Background and Notation As motivation, consider the problem of linking a plant's genetic variants t E {0, 1 with nutritional yield, which we model via a variable y E R. Our goal is to determine if each variant t, is causal for yield, meaning that it influences biological mechanisms which affect this phenotype [8] We also want to leverage large amounts of unstructured data (e.g., health records, physiological data) that are often available in modern datasets 7 25
key challenge in finding causal variants is ancestry-based con- founding 52. Suppose that we are doing a GWAS of plants from Countries and B; plants in Country A get more rainfall, and thus grow faster and are more nu- tritious. A simple linear model of y and will find that any variant that is characteristic of plants in Country A (e.g., bigger leaves to capture rain) is causal for nutri- tional yield.
Table 3: Comparison of standard DSE methods with linear base- lines. The & column refers to ||5 1 where 7y is the vector of estimated causal effects, and 7 is the vector of ground truth causal effects. Precision and recall are defined in Appendix G
Standard error of the Mean (sem) is computed over 10 seeds.
Model
(1)
Precision (t)
Recall (t)
Mean (sem)
Mean (sem)
Mean (sem)
Optimal
0.22 (0.04)
0.97 (0.03)
1.0 (0.00)
DSE (2 modalities)
0.30 (0.06)
0.93 (0.04)
1.0 (0.00)
LMM
0.44 (0.06)
0.85 (0.08)
1.0 (0.00)
DSE (1 modality)
0.60 (0.09)
0.78 (0.08)
1.0 (0.00)
PCA (1 component)
0.93 (0.17)
0,58 (0.09)
1.0 (0.00)
FA (1 component)
1.08 (0.17)
0.62 (0.08)
1,0 (0.00)
PCA components)
1.38 (0.24)
0.44 (0.09)
0.9 (0.07)
Methods and Baselines Most
FA (2 components)
1.44 (0.30)
0.55 (0.09)
1.0 (0.00)
existing GWAS analysis methods for estimating the effect of vari- ant Lj rely on latent variable mod-
PCA (3 components)
1,66 (0.23)
0.37 (0.08)
0.8 (0.08)
FA (3 components)
1.89 (0.45)
0.44 (0.08)
0.9 (0.07)
els: (1) they treat all remaining variants æ as proxies and obtain & via a linear projection (e.g., PCA
38 39 or LMM [59 26 of into a lower dimensional space where genomes from Countries A
and B tend to form distinct clusters (because plants from the same country breed and are similar); (2) we assume a linear model B t of y and add into it, which effectively adds the country as a feature (z reveals the cluster for each country); this allows the model regress out the effects of ancestry and assign the correct effect to variants t (one at a time).
Baselines. Our main baselines are Principal Component Analysis (PCA) and Linear Mixed Model (LMM), as described above and implemented via the popular LIMIX library [27]. We also compare against Factor Analysis (FA). standard linear technique for deriving latent variables, Uniform Manifold Approximation and Projection (UMAP), a manifold learning technique for dimensionality reduction BI and a linear model with no correction for confounding.
DSE Models. We compare against deep structural equation models that leverage
Figure 2: Latent z extracted by DSEs. Plants from dif- ferent countries form clusters (in color).
more sets of proxy variables coming from the following unstructured modalities: genomic sequences, weather time series, simulated physiological time series. We fit DSE models via a stagewise strategy analogous to how classical GWAS models are fit: (1) fit the component p(=g(=r))) for each proxy zggi (2) we fit linear model of y given and the z to estimate causal effects. Thus, our p( a 22) components are deep, while p(y|æ, 2) is shallow (following standard assumptions on epistasis in GWAS).
5.3.1 Simulated Human GWAS
We used the 1000 Human Genomes 3 dataset to generate simulated multi-modal GWAS dataset, following the aagloiicalll-inspiree "Spatial" strategy studied in 49]. In addition to genotypes, we generated random physiological time series by sampling Fourier series conditioned on the confounders. Please see Appendix G for details.
Table 3 shows our results. Amongst the non-oracle baselines, multi-modal DSE has the smallest error in estimating causal effect, as well as the highest precision and recall identifying causal SNPs. The uni-modal DSE, while worse than LMM, still beats PCA and FA. Note that the LMM
8

---

model is not compatible with multiple unstructured proxies. In general, we see that precision starts to deteriorate faster than recall, suggesting that false positives are more likely from the weaker linear deconfounding methods such as PCA/FA. Our results here further support that additional sources of unstructured multi-modal data can improve GWAS.
5.3.2 Real-Worle Genomic Prediction and GWAS in Plants
We also tested our methods on real- world plant GWAS dataset from the 1001 Genomes Project for Arabidopsis Thaliana plants [19 [55] The challenge here is that we don't know true causal effects; therefore, we define pheno- type y for which the true causal effect is zero specifically, we set y to the GDP of the country where each plant grows. Because the genomes of plants from the same country are similar, there exist spurious correlations caused by la- tent subpopulation groups 2. Our goal detect and correct for these con- founding effects. In addition to genomic data, we gather historical weather time series from the location of each plant (see Appendix [F) and use both modalities to correct for confounding.
Table 4: Multimodal Experiments on IHDP and STAR Datasets comparing DMSE and DGSE methods. When the image modality is missing, DMSE still produces reason- able ATE errors as it can compute the latent representation subset of modalities better compared to DMSE
Dataset
Model
еaти (train+val)
EAT z error(test)
IHDP
DMSE
0.433 (0,057)
0.627 (0,094)
DGSE
0.794 (0.308)
1.080 (0.337)
STAR
DMSE
32.575 (1.634)
33.743 (1 (1.890) 60.152 (4.788)
DGSE
59.102 (4.734)
Table 5: Correcting for confounding on a plant GWAS dataset. DSEs can discover complex, non-linear clusters over genomic and weather data to identify the latent con- founding variable better.
We evaluate whether each model learned to correctly account for latent confound- ing effects by measuring the predictive power of 3r for y, where / are the causal effects and is the vector of vari- ants. Specifically, we compare the Ria correlation between 8 of and y-here, lower better, since a model that has learned the causal effects should not be predictive of the phenotype. In Table we see the effect of extracting confound- ing variables using DSEs as opposed to using the standard PCA technique. We can see that the R² values produced us- ing DSES are closer to 0 as compared to using PCA. This experiment shows that neural network architectures are ef- fective in dealing with unstructured ge- nomic and weather data while correcting
Model
Input
R2)
Deep Sir Egns (Ours)
Weather+SNPs
0.049 (0.022)
PCA
Weather+SNPs
0.097 (0.019)
UMAP
Weather+SNPs
0.412 (0.026)
Deep Str Egns (Ours)
Weather
0.555 (0.027)
PCA
Weather
0,545 (0.017)
UMAP
Weather
0.653 (0.013)
Deep Str Eqns (Ours)
SNPs
0.068 (0.029)
PCA
SNPs
0.130 (0.020)
UMAP
SNPs
0.406 (0.020)
LMM
SNPs
0.804 (0.009)
Linear Model
0.665 (0.012)
for confounding. Please refer to Appendix for details.
5.4 Multimodal Experiments With Missing Modalities
We demonstrate the ability of the DMSE model to handle missing data. Our IHDP and STAR benchmarks involve two modalities: images (e.g.. baby's 'ss gender in IHDP) and tabular data (c.g.. the remaining features). We compare DMSE and DGSE models on these datasets when some of the modalities may be missing.
For DMSE, we define two different modalities X1 and X₂ for the tabular and image modalities respectively. For DGSE, we concatenate the image embedding to the tabular modality while training the network. We evaluate ATE while randomly dropping 50% of the images. DMSE utilizes its product-of-experts inference network to approximate the posterior distribution when modalities are missing. DGSE cannot do this, and we resort to feeding a vector of zeros when an image missing. Table 4 shows that DMSE produces improved ATE estimates as compared to DGSE.

---

Related Work
While previous work analyzed unstructured interventions t con- (e.g., determining the effect of polite a rude response) as well as unstructured [ 35 (e.g.. MRI images), our work proposes methods to handle unstructured
Multi-Modal Causal Inference
sisting of natural language 43
x. Veitch et al. [50] developed models that correct for confounders from a single unstructured proxy x derived from text 36 5T] or graph [50]. These approaches rely on propensity scoring tramework-they train a discriminative model of (((((z)--ence do not support proxy variables or missing data, and require pre-trained text embeddings. Additionally, propensity scoring methods rely on neural approximators for p(t]z), which may output volatile probabilistic outputs that lead to unreliable effect estimates [8. Our method (i) works across all modalities (beyond text or graphs), (ii) supports arbitrary numbers of proxies, (iii) supports missing data by virtue of being generative.
Deep Latent Variable Models Representation learning in causal inference has been studied by Johansson et al. T6
15 17 and Schölkopf et al.
46 Deep latent variable models find applications
throughout causal inference [28
30
53 20
Pawlowski al. [35 study unstructured
outcomes y (MRI scans), but do not support proxies. Louizos et al. 22 use variational auto-encoders
to estimate confounders from proxies; we introduce more structured model that handles multiple proxies that can be missing, and obviates the need for auxiliary modules. Tran & Blei 49 propose implicit deep structural equations for GWAS; ours are explicit and thus easier to train.
7 Discussion
Identifiability
Approaches to causal effect estimation that rely on deep learning 2 48 can never guarantee the recovery of causal effects-nnural network optimization is itself non-convex and has no guarantees. Other failure modes of deep latent variable models (DLVMs) include potentially not having a sufficiently expressive model, not having enough data to learn the model, as well as shortcomings of approximate inference algorithms. That said, there is
35
30 | 60, 53
50). The DLVM approach is appealing over existing propensity scoring methods AI 50 because: (i) it naturally handles
ample evidence of both failures [44] and successes of DLVMs 57
35
unstructured proxies that may be missing at random; (ii) it supports high-dimensional treatment variables in settings like GWAS, where propensity scoring algorithms are not easily applicable. Rissanen & Marttinen < empirically identify multiple failure modes of DLVMs; our work and that of others identifies success cases (particularly in GWAS 49 (4 and ultimately the validation of
DLVM methods needs to be empirical 48 58
Missing Data We make the common assumption that data is missing at random (MAR). This poses challenges if, for example, patients missing outcomes ones that are more likely to be sick. When two modalities and their missingness are correlated, their Zj, I& nodes in Figure I could be merged, somewhat addessing the issue. We leave the full exploration of non-MAR models to future work.
Conclusion
In conclusion, we proposed an approach based on deep structural equations that can leverage useful signal present in unstructured data to improve the accuracy of causal effect estimation. Unlike previous methods that relied on propensity scores [1 500, ours does not suffer from instabilities caused by volatile predictive probabilities coming out of neural networks, naturally handle missing data, and are applicable in settings in which the treatment variable is high-dimensional (such as in genome-wide association studies). Our work highlights the benefits of using large amounts of "dark" data that were previously left unused by existing methods to improve the accuracy of causal effect estimation.
Acknowledgements
This work was supported by Tata Consulting Services, the Cornell Initiative for Digital Agriculture, and an NSF CAREER grant (#2145577).
10

---

References
[1] Achilles, C., Bain, H. P., Bellott, F., Boyd-Zaharias, J., Finn, J., Folger, J., Johnston, J., and Word, E. Tennessee's Student Teacher Achievement Ratio (STAR) project, 2008. URL https: :/tp::://ddi... ooddi.ooo//...ooo /wt../orr//...////////SSTTT 7910/DVN//SWHHFF
[2] Astle, W. and Balding, J. J. Population structure and cryptic relatedness in genetic association studies. Statistical Science, 24(4):451-471, 2009.
[3] Auton, A., Abecasis, G. R., Altshuler, D. M., Durbin, R. M., Bentley, D. R., Chakravarti, A., Clark, A. G., Donnelly, P., Eichler, E. E., Flicek, P., Gabriel, B., Gibbs, A., Green, E. D., Hurles, M. E., Knoppers, B. M., Korbel, J. O., Lander, E. S., Lee, C., Lehrach, H., ...
University, M., and National Eye Institute, N. H. A global reference for human genetic variation. Nature, (26(757)):687777 Oct 2015. ISSN 1476-4687. doi: 10. 10.1038/nature15535
URL https: org 10. http:////......///aaaaeee...100. 1038/natuure13993
[4] Balding, D. J. and Nichols, R. A. A method for quantifying differentiation between populations multi-allelic loci and its implications for investigating identity and paternity. Genetica, 96(1); 3-12, 1995.
[5] Batmanghelich, N. K., Dalca, A., Quon, G., Sabuncu, M., and Golland, P. Probabilistic modeling of imaging, genetics and diagnosis. IEEE transactions on medical imaging. ((7)::7765-179) 2016.
[6] Blei, D. M., Kucukelbir, A., and McAuliffe, D. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):59-887, 2017.
[7] Bycroft, C., Freeman, C., Petkova, D., Band, G., Elliott, L. T., Sharp. K., Motyer, A., Vukcevic, D., Delaneau, O., O'Connell, J., et al. The uk biobank resource with deep phenotyping and 2018.
genomic data. Nature, 562(7726):203-222
[8] Crouch, D. J. and Bodmer, W. F. Polygenic inheritance, gwas, polygenic risk scores, and the search for functional variants. Proceedings of the National Academy of Sciences, 117(32): 18924-18933, 2020.
[9] Dorie, V. Non- non-parametrics for causal inference, 2021. URL http:::/ggtttubb.cm///// npci
[10] Duncan, O. Introduction structural equation models. Elsevier, 2014.
[11] Hartford, J., Lewis, G., Leyton-Brown, K., and Taddy, M. Counterfactuall prediction with deep instrumental variables networks, 2016. URL htll:::t/////xi.orrrgg
09596
[12] Higgins, I., Matthey, L., Pal, A., Burgess, C. P., Glorot, X., Botvinick, M. M., Mohamed, S., and Lerchner, A. beta-vae: Learning basic visual concepts with constrained variational framework. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
/stps://openrevlte http:ppawwe.re.ooooooooara//aaa wttp://orruiidddy2f2eleee.
[13] Hill, L. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217-240, 201
[14] Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 9(8): 1735-1788 1780, 1997.
[15] Johansson, F. D., Kallus, N., Shalit, U., and Sontag, D. Learning weighted representations for generalization across designs, 2018.
[16] Johansson, F. D., Shalit, U., and Sontag, D. Learning representations for counterfactual inference, 2018.
[17] Johansson, D., Sontag, D., and Ranganath, R. Support and invertibility in representations, 2019.

---

[18] Kallus, N. Deepmatch: Balancing deep covariate representations for causal inference using adversarial training. In International Conference on Machine Learning. pp. 5067-5077. PMLR, 2020.
[19] Kern, A. Polygenic score prediction of gross domestic product using arabidopsis thaliana gwas. http:://hw.a..cccccaaccacr/e///a aarpeearrrraaaaaaaa..a iippy ipynb,
[20] Kim, H., Shin, S., Jang. J., Song, K., Joo, W., Kang, W., and Moon, I.-C. Counterfactual faimess with disentangled causal effect variational autoencoder, 2020. URL https://arxiv org/
abs/2011. 11878
[21] Kingma, D. P. and Welling, M. Auto-encoding variational bayes, 2013. URL https://axxiv
org/abs/1312.66 6114
[22] Kuroki, M. and Pearl, J. Measurement bias and effect restoration in causal inference. Biometrika, 101(2):423-437, 03 2014. ISSN 0006-3444,
doi: 10. n0.1093//iomee/as0666. URI. https
1093/ //1////iie bioomm/astt0
[23] LeCun, Y., Bengio, Y., al. Convolutional networks for images, speech, and time series. The handbook of bruin theory and neural networks, 3361(10):1995, 1995.
[24] Li, M., Xu, R., Wang, S., Zhou, L., Lin, X., Zhu, C., Zeng, M. Ji, H., and Chang, S.-F. Clip- event: Connecting text and images with event structures. arXiv preprint arXiv:2201.050% 2022.
[25] Li. R., Chen, Y., Ritchie, M. D., and Moore, J. H. Electronic health records and polygenic risk scores for predicting disease risk. Nature Reviews Genetics, 21(8):493-500 -0233,5022220 2020.
[26] Lippert, C., Listgarten, J., Liu, Y., Kadie, M., Davidson, R. I., and Heckerman, D. Fast linear mixed models for gedome-wide association studies. Nature methods, 8( 8(10):833-835, 2011.
[27] Lippert, C., Casale, F. P., Rakitsch, B., and Stegle, O. Limix: genetic analysis of multiple traits. BioRxiv, 2014.
[28] Louizos, C., Shalit, U., Mooij, J., Sontag, D., Zemel, R., and Welling, M. Causal effect inference with deep varial-v models. arXiv preprint arXivv1705.0882) 2017.
[29] Lunceford, J. K. and Davidian, M. Stratification and weighting via the propensity score in estimation causal treatment effects: a comparative study. Stat. Med., 23(19))2937-2960, October 2004.
[30] Mayer, I., Josse, J., Raimundo, F., and Vert, J.-P. Missdeepcausal: Causal inference from incomplete data using deep latent variable models, 2020. URL 2002. 10837
[31] McInnes, L., Healy, J., and Melville, Umap: Uniform manifold approximation and projection for dimension reduction, 2018.
org/ abs 180208822000 03426
[32] Menne, M., Durre, I., Vose, R., Gleason, B., and Houston, T. An overview of the global historical climatology network-daily database. Journal of Atmospheric and Oceanic Technology, 29:897-910, 07 2012. doi: 10. 10.1155//ECH-D- 1-ITECCH----1-00000- 1000103..
[33] Naoki, E., Christian, J. F., Justin, G., Margaret, E. R., and Brandon, M. S. How to make causal inferences using texts, 2018.
[34] Nussbaumer, J. Fast fourier transform and convolution algorithms, 1981
[35] Pawlowski, N., Coelho de Castro, D., and Glocker, B. Deep structural causal models for tractable counterfactual inference. Advances in Neural Information Processing Systems, 33: 857-869, 2020.
[36] Pearl, Causality. Cambridge university press, 2009.
[37] Pearl, et al. Models, reasoning and inference. Cambridge, UK: CumbridggUUniversiyprress,
19, 2000.
12

---

[38] Price, A. L., Patterson, N. J., Plenge, R. M., Weinblatt, M. E., Shadick, N. A., and Reich, D. Principal components analysis corrects for stratification in genome-wide association studies. Nature genetics, 38(8):004 909, 2006.
[39] Price, A. L.. Zaitlen, N. A., Reich, D., and Patterson, N. New approaches to population stratification in genome-wide association studies. Nature reviews genetics, 11(7):459-465 2010.
[40] Pritchard, J. K., Stephens, M., and Donnelly, P. Inference of population structure using multilocus genotype data. Genetics, 155(2):945-959, 2000.
[41] Pryzant, R., Chung, Y., and Jurafsky, D. Predicting sales from the language of product descriptions. In eCOM@ SIGIR, 2017.
[42] Pryzant, R., Basu, S., and Sone, K. Interpretable neural architectures for attributing an ad's performance to its writing style. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 125-135, 2018.
[43] Pryzant, R., Card, D., Jurafsky, D., Veitch, V., and Sridhar, D. Causal effects of linguistic properties. arXiv preprint arXiv:2010.1291 2020.
[44] Rissanen, S. and Marttinen, P. A critical look at the identifiability of causal effects with deep latent variable models, 02 2021.
[45] Robins, J. M., Rotmitzky, A., and van der Laan, M. On profile likelihood: Comment. Journal of the American Statisticul Association, 95(450))477-482, 2000. ISSN 01621459. URL http: // wwwp:/wwwwww. jstor. htt.://aablee266933...
[46] Schölkopf. B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., and Bengio,
Y. Towards causal representation learning. CoRR, abs/2102.11107, 2021. URL https
//arxiv. saaxiv.org/aee www.oorgg/////eeeeoeoooeee 2102 11107
[47] Shalit, U., Johansson, F. D., and Sontag, D. Estimating individual treatment effect: generaliza- tion bounds and algorithms, 2017.
[48] Shi, C., Blei, D. M., and Veitch, V. Adapting neural networks for the estimation of treatment effects, 2019. URL http:::tt///. /ttp:://aaiivevia org/ abs
[49] Tran, D. and Blei, M. Implicit causal models for genome-wide wide association studies. arXiv preprint
2017.
[50] Veitch, V., Wang. Y., and Blei, D. M. Using embeddings to correct for unobserved confounding in networks, 2019.
[51] Veitch, V., Sridhar, D., and Blei, D. M. Adapting text embeddings for causal inference, 2020.
[52] Vilhjälmsson, B. J. and Nordborg, M. The nature of confounding in genome-wiet association studies. Nature Reviews Genetics, 14(1):1-2, 2013.
[53] Vowels, M. J., Camgoz, N. C., and Bowden, R. Targeted vae: Variational and targeted learning for causal inference, 2020. URL httpss//////xi werg/aba/2000 13472
[54] Wang, Y. and Blei, D. M. The blessings of multiple causes. Journal of the American Statistical Association, 1174529) 2019.
[55] Weigel, D. and Mott, R. The 1001 genomes project for arabidopsis thaliana. Genome biology, 10(5):155, 2009. 2009.
[56] Wu, M. and Goodman, N. Multimodal generative models for scalable weakly-supervised learning, arXiv preprint arXiv:1802.05333. 2018.
[57] Wu, P. and Fukumizu, K. Towards principled causal effect estimation by deep identifiable models, 2021. URL
13

---

[58] Yoon, J., Jordon. J., and van der Schaar, M. Ganite: Estimation of individualized treatment effects using generative adversarial nets, In ICLR, 2018.
[59] Yu, J., Pressoir, G., Briggs, W. H., Bi, I. V., Yamasaki, M., Doebley, F., McMullen, M. D., Gaut, B. S., Nielsen, D. M., Holland, J. B., et al. A unified mixed-model method for association mapping that accounts for multiple levels of relatedness. Nature genetics, 38(2):203-22)) 2088,,2000--0 2006.
[60] Zhang. W., Liu, L., and Li, J. Treatment effect estimation with disentangled latent factors, 2020.
URL https: : //arxiv org/ http:/abb2000.00000 10652
[61] Zhang, Z., Song, Y., and Qi, H. Age progression/eegrssion by conditional adversarial autoen- coder. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5810-5818, 2017.
14


Reviews:
[{"review_id": "SLcdPwRjJaW", "reviewer": "Reviewer_zkCX", "summary": "This work addresses the problem of estimating the unbiased causal effect of an intervention on the outcome using unstructured proxy variables. The authors first formulate the underlying causal structure of the problem. The authors then develop a generative approach, i.e., estimating the parameters of the deep multi-modal structural equations (DMSEs) or Deep Gaussian Structural Equations (DGSEs) via optimizing the derived multi-model evidence lower bound. The causal effects, therefore, can be computed from the learned models. The experiments used real-world datasets (IHDP, STAR, and GWAS datasets) to demonstrate that the developed models can estimate ATE with fewer errors than competing methods.", "questions": "\nThis work heavily relies on the assumption that the causal structured is the form $$ X = f_1(Z, \\epsilon_1), T = f_2(Z, \\epsilon_2), Y = f_3(Z, Y, \\epsilon_3). $$  The causal effect estimation from the DMSE or DGSE is unbiased when the causal structure is indeed true. However, many other dependencies between these variables make sense in the real-world setting (such as the existence of the interdependence between proxy variables or the proxy variables also control the distribution of the intervention variable and the outcome variable), which the author did not discuss.  \n\nThe following questions should be discussed:\n\nWhy is only the causal structure shown in Figure 1 discussed?\nWhat are the relevant issues, such as whether it is possible to model it using a generative model?\nIs it able to estimate the ATE from it? \nWhat might go wrong if the underlying model is inconsistent? \n\nAlso, the following paper might be helpful for the discussion:\n\nMiao, Wang, Zhi Geng, and Eric J. Tchetgen Tchetgen. \"Identifying causal effects with proxy variables of an unmeasured confounder.\" Biometrika 105, no. 4 (2018): 987-993.", "limitations": "Similar to the above discussion, it is beneficial that the authors could discuss the assumption made on the causal model and the possible limitations or societal impact on the real-world applications.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n1) The authors provide motivating real-world applications (Healthcare and Genomics) to show the context where a large amount of unstructured data is available and the missing confounders' issues in these respective applications.\n2) The notations, problem formulation, the underlying causal model, the derivation of the objective, and the estimation of causal effects are presented.\n3) Details of the experimental setup are well described in the manuscript and the appendix.\n\nWeaknesses:\n1) The central issue of this work heavily relies on the causal structure they are considering (shown in Figure 1), which I will discuss in more detail in the next section). \n2) In the experimental section if the STAR dataset were used for the experimental validation, its results (i.e., Table 6 in the appendix) should also be presented in the manuscript to help readers read smoothly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "M2obpj9-UwF", "reviewer": "Reviewer_GQXK", "summary": "This paper addresses causal effect estimation when there are unobserved confounders but there exist (often unused) unconstrained data that could be used as proxies for the unobserved confounders. A model named Deep Multi-modal Structural Equations (DMSE) is proposed to leverage the multi-modal data. The experimental results demonstrate the effectiveness of the proposed method on two real-world tasks.\n", "questions": "- Lines 103-104 state that “our approach uses deep structural equations to extract confounding signal from the multi-modal proxies”. The “deep” part does this not the “structural equations” part, correct? Because SEs cannot do representation learning.\n- How did you arrive at Eq. (5)? ELBO should have a reconstruction error part and a KL divergence part. Please provide detail on how you derived this ELBO.\n- Lines 137-139 state that the posterior factorizes in that way. Could you please elaborate on why this is the case?\n- Line 155: please elaborate why auxiliary inference networks are not necessary in your application?\n", "limitations": "N/A\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- The paper is written clearly and it’s easy to read.\n- The empirical experiments are thorough.\n- The contribution is significant in that the proposed model can leverage often unused and sometimes missing multi-modal data to enhance causal effect estimation performance. \n\nCons:\n- The originality of the paper seems a bit limited as the proposed method is composed of already available components (i.e., VAE, SE).\n- There is little explanation for the main equations stated in the paper. I will comment on these in the “Questions” section.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JNHchPwGhH", "reviewer": "Reviewer_5MY4", "summary": "This paper presents novel deep multi-modal structural equations for causal effect estimation from unstructured data with unobserved confounders. The correctness of the developed method relies on the set of rich unstructured proxy variables and perfect modelling. The experiments on synthetic and semi-synthetic datasets show the performance of the developed method.", "questions": "Q1. Why not compare the developed methods with CEVAE and other VAE-based estimators in the experiments?\n\nQ2. What are the missing modalities $X_j$ in this work?", "limitations": "Yes, I have not found a negative societal impact. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper considers a very important problem in causal inference, i.e. estimating the causal effect of intervention from unstructured data with latent confounders. The paper seems to take the main step of CEVAE over some specialized architectures, such as convolutions for images. The idea of the paper is good and interesting. However, to my understanding, the assumption of all covariates are the unstructured proxy variables, seems too strong to be satisfied in many real-world applications. For example, what is the latent confounder of the proxy variable \"sex of baby\" in terms of IHDP? This may not be very practical. Moreover, there are many issues with the presentations that make them not easy to follow. Overall, the paper is not good enough.\n \nSome minors:\n\n*multiple multi-model, it is better to remove `multiple'.\n\n*Confounder does not define in this paper.\n\n*Are \"unstructured proxy variables\", \"unstructured multi-modal proxy variables\" and \"unstructured data\" the same in this work? It is better to keep consistent.\n\n*Line 143 to 147, It would be better to provide a more detailed explanation. The current evidence and conclusion given are somewhat sloppy.\n\n*After Line 198, \"1\", \"0\" -> \\``1\", \\``0\"\n\n*The conclusion in Section 5.1 is not clear from the current descriptions.  \n \n=== Strengths ===\n\n1. A novel deep multi-modal  structure equation is developed for unstructured data \n\n2. Experiments conducted on a number of datasets show the performance of the developed algorithm. \n  \n=== Weaknesses ===\n\nThere is not a theoretical analysis of the developed deep multi-modal structure equations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept