Answering Complex Causal Queries With the Maximum Causal Set Effect
Authors: Zachary Markovich
Venue: NeurIPS 2021 Poster (2021)
Keywords: Causal Inference
frequentist inference
hypothesis testing

Abstract: The standard tools of causal inference have been developed to answer simple causal queries which can be easily formalized as a small number of statistical estimands in the context of a particular structural causal model (SCM); however, scientific theories often make diffuse predictions about a large number of causal variables. This article proposes a framework for parameterizing such complex causal queries as the maximum difference in causal effects associated with two sets of causal variables that have a researcher specified probability of occurring. We term this estimand the Maximum Causal Set Effect (MCSE) and develop an estimator for it that is asymptotically consistent and conservative in finite samples under assumptions that are standard in the causal inference literature. This estimator is also asymptotically normal and amenable to the non-parametric bootstrap, facilitating classical statistical inference about this novel estimand. We compare this estimator to more common latent variable approaches and find that it can uncover larger causal effects in both real world and simulated data. 

Answering Complex Causal Queries With the Maximum Causal Set Effect (MCSE)
Zachary Markovich"
Massachussetts Institute of Technology Cambridge, MA, 02139 zmarko@mit.ed
Abstract
The standard tools causal inference have been developed to answer simple causal queries which can be easily formalized as small number of statistical estimands in the context of a particular structural causal model (SCM); however, scientific theories often make diffuse predictions about large number of causal variables. This article proposes a framework for parameterizing such complex causal queries as the maximum difference in causal effects associated with two sets of causal variables that have a researcher specified probability of occurring. We term this estimand the Maximum Causal Set Effect (MCSE) and develop finite
an estimator for it that is asymptotically consistent and conservative samples under assumptions that are standard in the causal inference literature. This estimator is also asymptotically normal and amenable to the non- parametric bootstrap. facilitating classical statistical inference about this novel estimand. We compare this estimator to more common latent variable approaches and find that it can uncover larger causal effects in both real world and simulated data.
Introduction
Recent advances in machine learning technology have made possible non- con-parametrically estimate many parameters present in complex structural causal models (SCMs). Specifically, such estimating technology has rapidly advanced for three major causal inference settings: the many causes setting, the many moderators setting, and the many mediators setting. All three settings represent a situation in which a particular causal query can be stated in terms of large number of combinations of different variables. Specifically, researcher could estimate a different treatment effect associated with each of the many different possible combinations of causes [Imbens] 2000 Wang and Blei 2019 et al. 2019. Wang et al., 2018 Zheng et al. Forthcoming). a different conditional treatment effect for each of the many different combinations of moderators |reen and Kern 2012 (Athey and Imbens, 2016 Grimmer et al.) 2017 Wager and Athey 2018 Künzel et 20.1| and different mediated effect for each of the many different combinations mediators [Zhou and Yamamoto 2020 Daniel et al. 2015). Such causal queries are complex in the sense that they require summarizing the combined influence of a large number causal variables.
The main challenge for applied researchers in such settings is that standard causal inference algorithms are designed to provide a different estimate associated with each of the many causal variables rather than a single number summarizing the combined influence of all the causal variables together. Consider, for example, the setting of inferring the causal effect of actors on film's box office performance. Wang and Blei [2099] provide framework for estimating the average treatment effect associated with every actor on a film's 's performance. While certainly useful for making predictions
*PHD Candidate, Interdisciplinary hutps//zmarkovichhggluubbii
Doctoral
Program in Statistics and Political Science,
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

about which actors a director should cast, an economist studying the film industry might prefer single number which summarizes the general importance of actors in general for film 's box office success. As discussed in the next section, such settings are common in scientific research, suggesting the need for novel causal estimands to parameterize the predictions of such theories in the context of a particular SCM.
Contribution The contribution of this paper is threefold. First, introduces the notion of a complex causal query and argues that existing causal estimands are of limited utility to applied researchers in the face of such queries. Second, it defines a novel estimand - the Maximum Causal Set Effect (MCSE) which can be used to provide an interpretable answer to such complex queries. Finally, the paper introduces an estimator for this estimand. The estimator is based on techniques proposed in the double Q-learning literature [Hasselt 2010] and is asymptotically consistent and conservative in finite samples under assumptions that are standard in the causal inference literature. It is also asymptotically normal and amenable to non-parametric bootstrap techniques, facilitating classical statistical inference about the MCSE.
2 Setting and Previous Work
2.1 Problem Overview
Standard approaches to causal inference [Pearl 2009 typically begin with the researcher specifying an SCM and then defining causal query which can be answered based on the assumed SCM. Under certain assumptions about the SCM, it may be possible to estimate the answer to that causal query using the conventional tools of statistical inference. The standard tools of causal inference are designed with settings in mind where the predictions of scientific theory take the form of simple causal query. Such queries are stated in terms of some low dimensional causal variable and some outcome Y. For example, question like how much does a medical procedure reduce the risk of disease, represents simple causal query because it is defined terms of a single unidimensional treatment. Such queries can be easily quantified using conventional statistical estimands because they are directly formulated in terms of small number of theoretically motivated variables.
This paper instead focuses on situations where a scientific theory makes diffuse predictions about the importance of large number of causal variables, defying the stylization of simple causal queries. Such queries are common in scientific research. For example:
Genome Wide Association Studies (GWAS) GWAS attempt to quantify the causal effect of huge number of individual genotypes on the likelihood that some trait is expressed [Stephens and Balding 2009 Visscher et al. 2017].
Personality- - psychologists are often interested in the effect certain personality traits (such as extraversion or neuroticism) might have on life outcomes [Pervin] 2003). but such traits
are only observed by the researcher as responses
large number of survey questions.
Text language is complex and multi-faceter and the causal effect of the wording of document on a user's response requires an assessment of the contribution of many different topics or words together [Fong and Grimmer 2016 Egami et al. 2018 Fong and Grimmer Forthcoming).
Complex medical treatments many medical treatments cannot be reduced to a single low dimensional representation. For example, radiation exposure is observed as high dimensional vector [Nabi et al., [017] and medical researchers might also wish to understand the combined importance of many procedures using electronic medical records [Gottesman] et al. 2013].
Such causal queries are complex because they require estimating the joint influence of many causal variables.
The SCM undergirding such complex queries can take many forms. Three major examples are: (a) the many causes setting where the researcher wishes to understand the joint influence of many treatments (b) many moderators setting where the researcher wishes to understand how effect of binary treatment varies based on many variables (c) the many mediators setting where the researcher wishes to model how causal effect can be decomposed into many different pieces. These SCM's 's are

---

Figure 1: Visualization of Causal Graphs With Complex Queries
(N)
Figure visualizes SCMs corresponding to complex causal queries in the case (a) many causes, (b) many moderators, and (c) many mediators. (a) visualizes the case where treatment types to... t, each influence y described in Wang and Blei [2019. (b) visualizes the case where the causal effect of to on is directly modified by t ... in as described in VanderWeele and Robins [2007). (C) visualizes case where the effect of to on y is mediated by .1...ly ln as described by
Note:
Zhou and Yamamoto 2020).
visualized in Figure I in the form of directed acylical graphs (DAGs). The unifying trait of a complex causal query that it asks about the importance of many arrows present in each DAG.
Techniques developed in the context of simple causal queries cannot readily used to answer complex ones. While the standard tools of causal inference be used estimate causal effects corresponding to every combination of causal variables in SCM's like those visualized in Figure they do provide applied researchers with single unambiguous estimate with which summarize the joint causal effect of many such variables.
2.2 Previous Work
The only existent proposal for addressing the challenge presented by complex causal queries in the machine learning literature is to dimension reduce the relevant causal variables and then focus on simple causal query defined in terms of that latent trait (Fong and Grimmer] 2016 forthcoming Nabi et al. (017). This strategy has only been proposed in the many causes setting, but could also be extended to the many moderators many mediators reductive and risks understating the magnitude of causal effects because it disregards all variation in the treatment types that is not accounted for in the latent trait. Additionally such latent traits are often scale invariant and so may lack a scientifically meaningful interpretation.
well. Such strategy inherently
2.3 Assumptions and Notation
We assume that the researcher observes a N independent (t,, Y,, x:) triplets where Y; is the outcome, and t; is length K vector indicating the treatment type received by unit i, and x; is length J vector representing a of background covariates that causal effects should be adjusted for. Additionally, let T denote the support of the distribution of t;.
We also assume that the researcher has knowledge of the population distribution of t;: gtt). In many settings, the empirical distribution of t; will be the most logical choice, but other choices may be reasonable well if the population distribution is known the researcher, might be the case when conducting survey research or if the treatment types were experimentally randomized.
Finally, we assume that the researcher has specified some SCM and has specified a simple causal query, T(T', T"), which is defined in terms two subsets: T', TT T. In the many causes case, (TT.T") might take the form:
T(T. E (E (Y/)do(t)) TT) - E EE (Y/)do(t)) [t T")
where do(-) represents some causal intervention Pearl 2009 This estimand represents the average
effect of receiving set of treatments contained in T rather than T

---

In the many moderators case on the other hand let the zeroth element of the treatment types vector recieved by unit ti,0 E {0, 1), denote the level of some binary treatment received by unit i. Similarly, let the remaining elements of f, be denoted = and indicate the level received by unit i on the many moderators. Then possible choice for rTT', ,'7",T" T might be:
(T'' T") E (I YY, do 1)) E (Y, do(t.,o 0)) 1 T')
E E (Y do((s,o = 1)) E (Y:]do(tz,0 0)) t1 ET)
which represents the difference in average treatment effects between units with moderators contained in T' rather than T" Defining such estimands in the many mediators case requires more cumbersome notation, but can be accomplished in terms of an average of different combinations of path specific effects [Daniel et al. 2015 Zhou and Yamamoto, 2020].
The Maximum Causal Set Effect
The challenge for applied researchers in the presence of such complex causal queries is that different value of r(T'. T") can be defined for every distinct pair of sets T T C T, leaving the analyst without single unambiguous causal estimand to summarize their findings. In this section, we define a causal quantity of interest which overcomes this challenge by focusing on the contrast between two MMxax and MiMin which maximize correspond to unrepresentative edge cases, we require that the sets be of a researcher specified size: Formally, let the set of subsets of T such that the probability that t; is in T least g be defined as: T = (T' S T P(t E T') 2 /} where Ptt,- E T') gtg(t) E Jdt We then define MCSE, as:
T"). To avoid choosing sets T.Max and T.Min which
MCSE
(TT.T") (T.Max, T-Miin) TT,
max
We refer to TMxax as the maximum causal set and T,Min as the minimum causal set. For many applications, the MCSE will have an intuitive and scientifically meaningful interpretation. In the actors example, it might be used to answer a question like what is the expected difference in box office performance between film cast with one of the 10% best performing casts rather than one of the bottom 10% worst performing casts? Similarly, in the genetics example, it might answer the question, what the difference the efficacy of some drug for patients with one of the top 10% most treatment enhancing sets of genes rather than one of the bottom 10% most treatment diminishing sets of genes?
Estimation
This section outlines an algorithm for estimating MCSE,, Sample splitting is major part of this algorithm and this section develops the procedure in the context of a single data split. The efficiency of this estimator and then averaging the results, a procedure known as crossfitting (Chrnozzukoo et al. (017), which discuss in Appendix
also easily be improved by rotating the roles that each subset of the data plays
4.1 Algorithm Overview
basic result in the Q-learning literature is that single sample estimator for the maximum expected value will have an upward bias [Hasseli, 2010]. Since conservative estimators are easier to interpret and necessary for valid hypothesis testing, we follow the lead of Hasselt [2010) in using a split sample estimator for this estimation task. This approach is also useful in demonstrating the asymptotic normality of the resulting estimator as well.
Specifically, we begin by assuming that the analyst has randomly split the observations into two equally sized sets, Fest and SPob We further assume that the analyst has specified two models. The first uses the elements of the splitting set to make predictions about the probability that any T', T Ty are the true maximum and minimum causal sets and we denote its predictions: P(T'

---

T Ma = - M.n.). The second model makes prediction about T T)" for any two T', T ST. and we denote its predictions ;(T', Note P(T' = Tyuas n nT" = Tg should make use only of outcomes that are included in SProb while that, VT', ,T E Tq. P(T MT n T" M 1 (TT,T") conditional on observing the sample values of t; and z; for all units. After specifying models for P(T = -Tuax n nT" = T and ("), estimation proceeds as a weighted average of the estimates for ((TTT") for every T,, T,T" TTi
should only use the outeomes in S so
MCSE
P(T' T',T"ET,
E
TaMaxx nT" TyMinn) (aTT)
4.2 Point Estimation Properties
major requirement for the good behavior of this estimator is that P(T' Max nT" T,Minn
obey the basic probability axioms and that P(T' T.Max n T.Min)) assign zero probability to subsets of T not in Tq. These requirements are entirely verifiable by the analyst through the careful construction of P(T' = TyMax nT" = Min and are formalized in the following assumption:
Assumption 1. P(T' = -Taax T T satisfies the following conditions:
ET.T.ET, P(T T.Max n T T.Min) =
vT'T" E Tgp 5 P(T' = Tamax T = =Toling S
P(T T.Mar n T Miii = 0 for all T o To
Under Assumption MCSE, can be interpreted effect of being treated with treatment type in one set rather than another. Because MCSE, is defined as the maximum such causal effects for any two subsets greater than the expectation of this average, leading to the following proposition: and the conditions of Assumption
weighted average of estimators for the causal
of the required size, will always be
Proposition 1. IT T E Tg. (f(T', T)) S hold, then:
MCSE,))
MCSE
Proof in appendix C.1
The conditions for finite sample conservatism are relatively mild (for example, PPTT' = T-Max nT" TyMin) could be misspecified or inconsistent); however, as formalized in the next proposition, the conditions for the consistency of MCSE, are a bit stronger and require that P(T' = -T,Max n nT" Min) converge to binary indicator identifying T,Min and Max: Proposition 2. IfVT', T E T
P(T' = T.Max nT" = T.Mimi (trob) n-000
P, 11T' = T.Max 11(T' T.Mary
and
p T(T', T") 7(T', (T")
then
MCSE, R-
MCSE,
This result will also hold if convergence in probubility is replaced with almost sure convergence.
Proof in Appendix C.2
Some machine learning techniques (e.g. support vector machines, regression trees, etc.) will not readily produce probabilistic estimates for P(T M.Max n T T-Min), instead generating only
5

---

binary prediction for the two sets Mi and T.Min 2 The following proposition shows that such binary estimators will perform best as well probabilistic estimators as long as the two estimators have the same expectation:
Proposition Let, d(T'TT") {0, 13 and u(T', E [0, 1] represent two choices for P(T' TMaxx n NT" ) Let MCSE, d and MCSE, represent the corresponding estimators for MCSE.. Then VT T E Ty. E (d")", = (Wo((T
(MCSE - MCSE, ")) S ((MCSE,, -MCSE, (MCSS, 2 )
Proof in Appendix C.3
direct implication this result that bootstrap aggregation can be used to improve the performance of any binary predictor for P(T' = T n nT" = Txxlnn) to create a probabilistic estimator without changing the expected value of the predictions.
4.3 Interval Estimation
While the previous section establishes the properties of the point estimator for MCSE,, such results will be of little utility for applied researchers without a corresponding framework for measuring uncertainty those estimates. In this section, we begin the process of providing such results by introducing the assumption that "(T', TT") can be represented as a linear combination of the estimation set outcomes:
Assumption 2. Let Z {t,, x4 i Sta). For any T', TT Ty there exists set of transformations
(f, T") : 5 such that:
+(T,T")
Σ M(Z,TTT")) iESE
Many common estimators for causal effects (e.g. matching, weighting, regression techniques, etc) fit this form, so such an assumption will not be unduly restrictive many settings.
This assumption eases the derivation of asymptotic normality because it shows that "('', T") can be represented the sum of independent random variables. The following proposition uses the central limit theorem derived by Neumann [203) to show that multiplication by P(T Max Mri)
will impact this convergence so that asymptotic normality of MCSE, can be preserved under some mild regularity conditions:
Proposition 4. P(T'
T Max NT" T.Min) satisfies assumption 1
Vi, E (Y??) co: and Ve 0.
Σ SE
{))
0
|50
Then, conditional observing the estimation set values of t, and x,,
MCSFE, -E (MCSE.)) D, 10,
Var(MCSEE)
Proof in Appendix C.4
The final result necessary for conducting classical statistical inference a corresponding variance estimator. This can be most easily accomplished via the non-parametric bootstrap. Specifically,
NNote, many of these algorithuss can be tweaked provide such probabilistic estimates. For example, Bayesian regression trees [Chipman et al. 2010 a tree based method that can easily make these sorts of probabilistic predictions. Proposition suggest that such approaches would also be preferable to a coarse binary prediction.

---

Mammen 1992] shows that the non parametric bootstrap is consistent for an asymptotically normal estimator that can represented linear transformation of some set independent observations. The following lemma uses assumption 2 to provide just such a result:
Lemma 1.
MCSE =
Σ
Y,w;
where w4 ETITTET, P(T' = Ty n = = f.
TT) for Proof. The proof follows trivially by using assumption 2 to substitute Eiego fi(Z,', ,T, "(T'TT") in the definition of MCSE, and then changing the order summation.
So the variance and confidence intervals of MCSE, can be consistently estimated by bootstrap resampling from the set (Yww; : E Slis)
5 Experiments
5.1 Benchmarks on Synthetic Data
We first consider the performance of this estimation procedure using synthetic data. Specifically, to asses the performance of this estimator, we implemented it on synthetic version of the many causes setting. First, we generated a of N length K vectors of causes for each unit where Σ is some matrix with ones on the diagonal elements and some value diagonal elements. We then generated the outcome as μı; tī where / is a length K vector composed of i.i.d draws from the standard normal distribution. Finally, we normalized μ4, so that the corresponding value of MCSE, was always and generated the outcome variables as Y, = bt + e; where E; ~ (0, 1).
t, NNO 2) [0, 1] in the off
We implemented
estimators on this dataset. The first is the split sample MCSE, estimator described in this paper Note that under this simulation set up, all the assumptions needed for the theoretical results presented in Section 4 to hold are known to be true, so MCSE, should be unbiased and consistent. We compared the performance of MCSE, with estimate for MCSE, generated using a linear regression of Y; on the first principal component of t; This estimator corresponds to the current state of the art for drawing causal inferences in the face of a complex causal query, which involves using dimension reduction techniques to simplify the complex causal query into simple one. We repeated this procedure 100 times for each combination of = 2, 10, and 50; p = 0, and, 99; and values of N between 100 and 1,000.
Figure 2 visualizes the results of this analysis. Each point in the figure represents the average of all 300 iterations of the simulation procedure with the same values of and K or n and Because the bias of both estimators is large relative to their variance in this setting, Figure 2 focuses on the
bias of the estimators These estimates show that MCSE, is large improvement over the latent trait model, generating significantly less biased estimates even when p large and the principal components analysis (PCA) should perform well. Importantly, the bias of MCSE, appears to vanish asymptotically while the PCA estimator shows little convergence as the sample size increases &
Note, clustered standard errors can also easily generated using the hlock bootstrap. "Specifically, one using monte carlo sampling from the asymptotic distribution of linear regression of Y; on t; as P(T TyMin n T TiMin and a linear regression for f(T', T". See Appendix B.1 for more details on the implementation of MCSE,
"See appendix B.2 details the implementation of this estimator. 'Note, the monte carlo error in these estimates is quite low. The standard erorr associated with these average is never higher than .019 for any of the points.
'Appendix (ppenddxx/ppeeeentt presents estimates for root mean squared error, which show similar pattern
"Note additional simulation results are also presented in Sections B.3 and B.4 of the supplementary informa-
tion.
7

---

Figure 2: Simulation Results
ke2
k-10
k=50
0.0
-0.2
Model
Blas
MCSE
PCA
-0.6
200
400
600
800
1000200
400
600
800
1000200
400
600
800
1000
rho=0
rha=0.5
rho=0.9
-0.2
Model
Blas
-0.4
MCSE
PCA
-0.6-
200
400
600
800
1000200
400
600
800
1000200
400
600
800
1000
Note: Red dots identify the bias of the method for quantifying the combined effect of many causes proposed in this paper while blue dots show the bias dimension reduction techniques that represent the current state of the art for this same task.
5.2 An Application to Real World Data
Our second application focuses on the role of democratic political institutions in reducing the likelihood of civil war onset. Democracy fundamental concept when modeling the quality of governance, but drawing inferences about it's effect represents straightforward example of the multiple causes setting. In particular, democracy cannot be measured as a single unambiguous feature instead it is confluence of many conceptually related by empirically distinct features describing different aspects of a system of governance. The causal effect of democracy on outcomes like conflict initiation is typically measured using dimension reduction of the features representing the individual institutions [Treier and Jackman] 2008]; however, such strategies have led to conflicting results about the importance of democracy for political stability [Vreeland] 2008 Fearon and Laitin 20033 Consequently, the role of democratic political institutions in reducing civil war onset represents useful case for comparing latent trait models with with the MCSE.
Specifically, we used a linear model with fixed effects for the country and year for both P(T Max nT = Min) and +TT', (") and measured democratic political institutions using 111 features describing the system of governance present in a country in the Varieties of Democracy Dataset (V-Dem) 9 quantifying the effect of these political institutions on civil war onset for many different values of q. In particular, they suggest that countries with one the 10% most conflict reducing sets of institutions have roughly 1.2% lower risk of civil war than countries with some of the 10% most conflict inducing institutions. The dots on the right instead represent estimates for the MCSE, using just the V-Dem polyarchy indicator, which is standard measure of Democracy in the political science literature. The estimates for MCSE are larger than those achieved using the univariate model,
The dots and confidence intervals on the left Figure show the estimates for MCSE,
See appendix B.1 for more details these models.

---

Figure 3: The Causal Effect of Democratic Political Institutions on the Probability of Civil War Onset
0.02
Model
est
0.01
All Democratic Institutions Democracy Index
0.00
-0.01
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
The red dots identify estimates for the MCSEg made using the methodology outlined this paper and represent the combined influence of many different democratic institutions together. The blue dots instead represent the influence of just univariate latent trait produced by the maintainers of the V-Dem Dataset that is frequently used model democracy. Note 2: Confidence intervals adjusted for clustering by country.
Note:
suggesting that the the MCSE can successfully recover causal effects that standard latent variable approaches cannot.
Conclusion
Non-parametric estimation techniques and high dimensional datasets increasingly confront re- searchers with estimates for huge number of distinct causal estimands. While the capacity to fit such models represents tremendous progress for the estimation and computational techniques that support them, scientific theories rarely make predictions about such large number of distinct parameters. this article, we propose framework for making sense of such model outputs by focusing on the maximum causal contrast between two sets of researcher specified size We also develop an estimator for this estimand that consistent, conservative in finite samples, and asymptotically normal. While the estimator is developed with the many causes and treatment effect heterogeneity settings in mind, the framework extremely flexible and could be extended myriad of other causal qauntities of interest, speaking to its wide applicability and utility for applied researchers. While single causal estimand will never replace the kind of careful synthetic and analysis of individual causal variables should accompany the study of any complex phenomenon, we believe the MCSE will be a useful tool a wide array scientific disciplines.
References
Susan Athey and Guido Imbens. Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences, 113(27):3533-766) 2016.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, and Whitney Newey. Doubler/dehiaseddneymmn machine learning of treatment effects. American Economic Review, 107(())261-66))) 2017.

---

Hugh A Chipman, Edward I George. and Robert E McCulloch. Bart: Bayesian additive regression trees. The Annals of Applied Statisties, 4(1):266-299) 2988 2010.
Rhian M Daniel, Bianca L De Stavola, SN Cousens, and Stijn Vansteelandt. Causal mediation analysis with multiple mediators. Biometrics, 71(1):1-14. 2015.
Naoki Egami, Christian J Fong, Justin Grimmer, Margaret E Roberts, and Brandon M Stewart. How to make causal inferences using texts. arXiv preprini arXiv: 1802.02260 2018.
James D Fearon and David D Laitin. Ethnicity, insurgency, and civil war. American political science review, pages 75-90, 2003.
Christian Fong and Justin Grimmer. Discovery of treatments from text corpora. Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1600-1609, 2016.
Christian Fong and Justin Grimmer. Causal inference with latent treatments. American Journal of Political Science, forthcoming. URL http:://ww: /tRL:tttts////wdddoppoooo drw.wox.oooo.ooooo cow.s///cccaaaoooaooaaoo.coo. htmm/////xxsStpjurrr..//
Sexp_rev.ee sesp_rev..dffdd..
Omri Gottesman, Helena Kuivaniemi, Gerard Tromp, Andrew Faucett, Rongling Li, Teri
Manolio, Saskia C Sanderson, Joseph Kannry, Randi Zinberg, Melissa A Basford, et al. The electronic medical records and genomies (emerge) network: past, present, and future. Genetics in Medicine, 15(10):661771, 2013.
Donald Green and Holger Kern. Modeling heterogeneous treatment effects in survey experiments with bayesian additive regression trees. Public opinion quarterly, 76(3):491-511, 2012.
Justin Grimmer, Solomon Messing, and Sean Westwood. Estimating heterogeneous treatment effects and the effects of heterogeneous treatments with ensemble methods. Political Analysis, 25 (4):413 434, 2017.
Hado Hasselt. Double q-learning. Advances in neural information processing systems, 23:2613-2621, 2010.
Guido W Imbens. The role of the propensity score in estimating dose-response functions. Biometrika, 8763)700--70, 2000.
Sören R Künzel, Jasjeet Sekhon, Peter Bickel, and Bin Yu. Metaleamerss for estimating heteroge- neous treatment effects using machine learning. Proceedings of the national academy of sciences, 116(10)4416-4166 2019.
Fan Li et al. Propensity weighting for causal inference with multiple treatments. The Annals of Applied Statistics, (3(4):2389-2411) 2019.
Enno Mammen. Bootstrap, wild bootstrap, and asymptotic normality, Probability Theory and Related Fields, 93(4):439-455, 1992.
Razieh Nabi, Todd McNutt, and Ilya Shpitser. Semiparametric causal sufficient dimension reduction of high dimensional treatments. arXiv preprint 1rXiv:1710.06727. 2017.
Michael H Neumann. A central limit theorem for triangular arrays of weakly dependent random variables, with applications in statistics. ESAIM: Probability and Statistics, 17:120-134, 2013.
Judea Pearl. Causality. Cambridge university press, 2009.
Lawrence A Pervin. The science of personality. Oxford university press, 2003.
Matthew Stephens and David Balding. Bayesian statistical methods for genetic association studics. Nature Reviews Genetics, 10(10):681-690 2009.
Shawn Treier and Simon Jackman. Democracy as latent variable. American Journal of Political Science, 22(1)2201-217, 2008
10

---

Tyler VanderWeele and James M Robins. Four types of effect modification: a classification based on directed acyclic graphs. Epidemiology, 1865)5611556, 2007.
Peter M Visscher, Naomi Wray, Qian Zhang, Pamela Sklar, Mark McCarthy, Matthew Brown, and Jian Yang. 10 years of gwas discovery: biology, function, and translation. The American Journal of Human Genetics, 101(1):3 22, 2017.
James Raymond Vreeland. The effect of political regime on civil war: Unpacking anocracy. Journal of conflict Resolution, 52(3):401-425, 2008.
Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):1228-12422 2018.
Yixin Wang and David M Blei. The blessings of multiple causes. Journal of the American Statistical Association, pages 1-71, 2019.
Yixin Wang, Dawen Liang, Laurent Charlin, and David M Blei. The deconfounded recommender. causal inference approach to recommendation. arXiv preprint arXiv:/808.066: 1800006811. 2018.
Jeffrey M Wooldridge. Econometric analysis of cross section and panel data. MIT press, 2010.
Jiajing Zheng, Alexander D) Amour, and Alexander Franks. Copula-based sensitivity analysis for multi-treatment causal inference with unobserved confounding. Journal of the American Statistical Association, Forthcoming.
Xiang Zhou and Teppei Yamamoto. Tracing causal paths from experimental and observational data. SocArXiv: January, 11, 2020.


Reviews:
[{"review_id": "m4pLpU-xQui", "reviewer": "Reviewer_jFfp", "summary": "This paper describes and procedure for estimating causal effects in the setting where there are many treatments of interest and the analyst wishes to summarize the relationship between those many treatments and the outcome. The authors define an estimand, the maximum causal set effect, which describes the treatment effect implied by intervening on the sets of treatment variables with different values. The authors provide theoretical results demonstrating the convergence/asymptotic normality of their proposed estimator for this effect using sample splitting. They then apply their estimator to synthetic and real-world data.", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the proposed methodology in this paper to be modestly interesting and, once you get past some notational opaqueness, the backing theory appears to be sound (i.e. I can tell what the authors are getting at). There are some rather fundamental issues with the exposition and motivation that make it unclear why exactly this approach is \"necessary\". Perhaps some of these issues can be cleared up through the review process but for now I think the paper warrants major revisions. \n\nI'll also add that a full paragraph (Broader Impacts section, about 15 lines) spills over onto a 10th page.\n\nDetailed comments follow:\n\n**Motivation** -- Throughout the paper I struggled to understand what is fundamentally wrong with what the authors cast as the woeful status quo. The authors point to applied researchers as being unable to properly characterize the effects of many variables on one outcome. I think this is...sorta the point of applied science? The world is complex and often _shouldn't_ be characterized in terms of univariate summaries (take IQ as an exceedingly good example of something that does an exceedingly bad job of capturing something multifaceted, intelligence). This has parallels with uncertainty quantification or set identifiability. People might find it upsetting or dissatisfying that there isn't _one_ right answer. The reality, though, is that the complexity in the modeling/statistics exists because the problems themselves are complex.\n\nFollowing that point, the authors don't really make clear why it's infeasible (or not desirable) to intervene on all the treatments of interest with separate treatment settings for the sake of comparison, or, alternatively, intervene on each treatment successively to get a characterization of each treatment's causal relationship with the outcome.\n\n\"Techniques developed in the context of simple causal queries cannot be used to answer complex ones\" -- why not? Is there a citation? What is the \"challenge\" to which the authors refer? Statistical? Interpretation?\n\n**Multiple cause vs. multiple moderators** -- It's not clear how the two graphs in Fig. 1 are different. In a non-parametric causal model, it's standard for there to be interaction effects so I don't see the point of the distinction.\n\n**Notation/Assumptions** -- What is meant by \"treatment type\" (line 103)? Is it the value of the treatments? Are they all binary, continuous, other?\n\nUp to section 3, caligraphy{T} isn't clearly defined. It appears to be a set of treatment variables which are (implicitly?) intervened upon. Is that understanding correct? This should be stated explicitly before it's used in downstream definitions\n\nThe definitions of Tmax and Tmin aren't clear. It appears the authors mean \"Tmax and Tmin are the settings of the treatment variables T such that the difference in average outcome Y under Tmax and under Tmin is maximal\". Even the definition on line 135 doesn't quite make this explicit.\n\nThe interpretations of Tmax and Tmin are further confused by the authors' example of picking the \"10% best-performing casts\" vs. \"10% worst-performing casts\". It seems the idea is to \"peak\" at Y to determine the relative effects of various treatment variables. To determine the top 10%, I don't see how the analyst could avoid an exhaustive \"intervene on everything successively\" approach like the one I alluded to above.\n\n**\"A basic result in the Q-learning literature** -- citation? Is it really basic?\n\n**ML techniques for estimating the probability chunk** -- The authors claim ML techniques are incapable of providing probabilistic estimates for Phat(Tmax = T' \\cap Tmin = T''). I'm not sure I agree with this. There are several out-of-the-box multi-class classification models that can provide probability estimates. For instance, RFs inherently are multi-class and sklearn's implementation provides out-of-box probability predictions.\n\n**Experiments** -- what exactly is the \"intervention\" here? Are you picking a specific q and then finding sets to maximize Y according to that q?\n\n\"In particular, they suggest that countries with one of the 10% most conflict reducing institutions have...\" -- similar to above comment, this seems a bit like the tail wagging the dog. The authors are explicitly looking at the outcome to find effects of interest and then casting it as surprising that high \"performing\" interventions do \"better\" than low \"performing\" interventions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above comments about the efficacy of summarizing complex research. I don't think the authors make a strong case. In fact, they explicitly allude to negative examples (in their over-the-limit Broader Impacts section) like IQ. It's well-understood that IQ is an inappropriate tool that has been used in the perpetuation of racist and sexist practices (see, e.g. Weiten 2016). Intelligence is a multi-faceted characteristic of a person that CANNOT be boiled down to one number and therefore SHOULD not. I am concerned the authors are falling into the standard NeurIPS trap of saying \"this seems like an interesting technical idea\" and casting it as ground breaking for applied science. It has the potential for damage and the potential for benefit is, as I've stated, limited and unclear.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {"ethical_concerns": "I don't think my above point rises to the status of \"needs ethical review\". It's unlikely this paper will immediately see wide adoption. The authors, rather, should carefully ponder what exactly their broader contribution is as they revise."}}, {"review_id": "HHKGA2yq8ec", "reviewer": "Reviewer_XkeP", "summary": "### 1. Summary\n\nThe paper introduces a novel estimator for the multiple causes / moderators/ mediatiors scenario within causal inference processes.\n\nThe paper claims a threefold contribution, namely\n\n(i) to introduce the notion of \"complex causal queries\"\n\n(ii) to introduce a novel estimand (MCSE)\n\n(iii) to introduce a related estimator", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### 2. Rationale for the score\n\nOverall the paper follows a good structure and shows a adequate style of writing. A good structure with highlighted Lemmata and definitions is used. The choice of the problem is interesting and up-to-date. The lack of previous work on the topic convinces me that the paper should be accepted. \n\n### 3. Positive aspects\n\n- the problem is nicely introduced\n\n- the accompanying expirements with synthetic and real data are fitting to the problem description\n\n- using the algorithm on socio-political data (here civil war onset), is a very interesting application and fits the outlook regarding broader impacts\n\n-the overall structure is well formulated and the seperate handling of point estimation and interval estimation properties seems advisable\n\n### 4. Negative aspects\n\n- while examples are provided within the theoretical parts of the paper, I feel these could be better integrated to serve as illustration of the concepts explained.\n\n-I would suggest to highlight the claimed contributions of the paper more clearly and also refer to these in the conclusion. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact are adressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "5bvpIq601d", "reviewer": "Reviewer_CbdZ", "summary": "This paper proposes a single-number measure called the maximum causal set effect to provide a simple and informative answer to complex causal queries that feature many causal variables. The measure is roughly the biggest difference alternative treatments can make, among treatments that appear sufficiently often in the population. An estimator for this quantity is developed and some useful statistical properties of the estimator are established. Empirical results are also presented, including an interesting application to real data from political science.   ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a solid paper, with original and useful results that are relevant to addressing an important question. One slightly misleading aspect is that the paper sometimes (e.g., in the abstract) describes the maximum causal set effect to be \"the maximum difference in causal effects associated with two sets of causal variables of a researcher specified size.\" However, as I understand it, the two sets are not sets of variables, but rather sets of values of variables or sets of states, and the researcher specified size does not refer to the cardinality of a set but its probability. If my understanding is correct, I suggest such remarks should be reformulated. It seems to me that the need for such a measure as MCSE arises when there are a large number of alternative treatments. This will be the case if there are a large number of variables one can simultaneously control, but it may also be the case if some causal variable has many possible values. \n\nAnother puzzling point is on p. 4, lines 9-10, where a formula is introduced to quantify the effect by moderators or mediators. For moderators the formula makes sense to me, but I don't see why it is a sensible choice for mediators. Since mediators are affected by t_0, it is unclear to me what kind of effect the formula is quantifying. \n\nThe experiments take the PCA estimator as a baseline, but it does not seem fair to evaluate the PCA estimator against the MCSE, as the former is intended to capture a different aspect of a complex causal mechanism. I wonder if a single sample estimator of the maximum, despite its upward bias, will make a better comparison. \n\nMinor comment: on p. 6, the top line has a typo: the last term should be about the Min set.  \n  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers a relevant problem, but reviewers have identified some sloppiness regarding causal claims: 

The discussion of the mediators case on p. 4 struck several reviewers as confusing (and possibly confused.) The formula given between lines 119 and 120 is sensible with respect to the moderators case, but hardly makes sense with respect to the mediators case. The authors should either explain how to read or understand the formula in the mediators case so that it evidently expresses a meaningful effect differential, or use a different, clearly applicable formula for the mediators case, or at least restraint from suggesting that the said formula is also applicable to the mediators case.

Unfortunately, the authors replied to this issue in a very non-explicit way although it's a crucial point from the causal perspective.
We decided to accept this paper despite this problem. It should be emphasised, however, that we considered it serious enough to discuss rejection since a paper on causality should be careful about causal claims. We therefore ask the authors to consider the issue carefully in the revised version.  

Decision: Accept (Poster)